Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Using gradients to detect backdoors in neural networks

A backdoor and gradient technology, applied in biological neural network models, neural architectures, neural learning methods, etc.

Pending Publication Date: 2020-11-20
IBM CORP
View PDF1 Cites 5 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0011] For example, in the context of autonomous driving implementations, intruders or attackers may wish to provide backdoor street sign detection , which has good accuracy for classifying street signs in most cases, but it classifies stop signs with specific graphical features (e.g., stickers) as speed limit signs, potentially causing the ego vehicle to continue through the intersection without non-stop

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Using gradients to detect backdoors in neural networks
  • Using gradients to detect backdoors in neural networks
  • Using gradients to detect backdoors in neural networks

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0028] figure 1 An example scenario is shown of input provided to generate a backdoor model for classifying stop signs. For the purpose of this example, it is assumed that the machine learning or cognitive model being trained is specifically trained to recognize street signs in the image and is based on predefined output categories (e.g., stop signs, speed limit signs, Concession signs, street name signs, road works signs, etc.) classify them.

[0029] Although Gu et al. publicly proposed a method of creating a network with backdoors, they did not provide a systematic method for identifying backdoors that exist in machine learning or cognitive models. The publication of Gu et al. also shows that the pattern backdoor is less visible in the convolutional layer filter, making it difficult to identify the backdoor by examining the convolutional layer filter. In addition, the backdoor can be encoded in layers other than the convolutional layer. Gu et al.'s reference to the "third ne...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

Mechanisms are provided for evaluating a trained machine learning model to determine whether the machine learning model has a backdoor trigger. The mechanisms process a test dataset to generate outputclassifications for the test dataset, and generate, for the test dataset, gradient data indicating a degree of change of elements within the test dataset based on the output generated by processing the test dataset. The mechanisms analyze the gradient data to identify a pattern of elements within the test dataset indicative of a backdoor trigger. The mechanisms generate, in response to the analysis identifying the pattern of elements indicative of a backdoor trigger, an output indicating the existence of the backdoor trigger in the trained machine learning model.

Description

Technical field [0001] This application generally relates to improved data processing devices and methods, and more specifically to the mechanism of using gradients in neural networks to detect backdoors. Background technique [0002] Deep learning is part of a broader family of machine learning methods based on learning data representation as opposed to task-specific algorithms. Some representations are loosely based on the interpretation of information processing and communication patterns in the biological nervous system, such as neural codes that attempt to define the relationship between various stimuli in the brain and related neuronal responses. Research attempts to create effective systems to learn these representations from large-scale, unlabeled data sets. [0003] Deep learning architectures such as deep neural networks, deep belief networks, and recurrent neural networks have been applied to fields including computer vision, speech recognition, natural language process...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06N3/08
CPCG06F21/577G06N3/08G06N3/045G06F2221/033G06N20/00
Inventor 李泰星I·M·莫洛伊W·喀瓦罗B·J·爱德华兹张佳龙B·陈
Owner IBM CORP
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com