Image classification using shape biases with deep convolutional networks

Abstract

A system for analyzing images includes: a processing device comprising a receiving module configured to receive images; and an analysis module configured to apply the received images to a machine learning network and classify one or more features in the received images. The machine learning network is configured to propagate image data through multiple convolutional layers, each of which includes multiple filter channels. The machine learning network includes a bottleneck layer configured to identify image features based on the shape of image components. The system also includes an output module configured to output classification representation data including one or more features.

Description

Technical Field

[0001] This subject matter relates to the field of image analysis, and more specifically, to systems and methods for image recognition that train classification models and represent images in different domains. Background Technology

[0002] Modern vehicles are increasingly equipped with cameras and / or other imaging devices and sensors to facilitate vehicle operation and improve safety. Cameras can be included in vehicles for a variety of purposes, such as increasing visibility and driver awareness, assisting the driver, and performing vehicle control functions. Autonomous vehicle control is becoming increasingly common, with autonomous control systems equipped with the ability to identify environmental objects and features using cameras and other sensors such as radar sensors. Summary of the Invention

[0003] In one exemplary embodiment, a system for analyzing an image includes: a processing device including a receiving module configured to receive an image; and an analysis module configured to apply the received image to a machine learning network and classify one or more features in the received image. The machine learning network is configured to propagate image data through multiple convolutional layers, each of which includes multiple filter channels. The machine learning network includes a bottleneck layer configured to identify image features based on the shape of image components. The system also includes an output module configured to output classification representation data including one or more features.

[0004] In addition to one or more features described in this article, the machine learning network is a deep convolutional neural network.

[0005] In addition to one or more features described in this paper, the bottleneck layer is a convolutional layer, and the machine learning network includes at least one pooling layer and a fully connected layer configured to apply one or more classifications to the received image based on the outputs from the bottleneck layer and the multiple convolutional layers.

[0006] In addition to one or more features described in this paper, the bottleneck layer is configured to apply a single filter to the image data.

[0007] In addition to one or more features described herein, the received image comprises multiple image channels, and the bottleneck layer is configured to apply a single filter to the multiple image channels.

[0008] In addition to one or more features described in this paper, the analysis module is configured to train the machine learning network by propagating multiple training images via the machine learning network.

[0009] In addition to one or more features described in this paper, the analysis module is configured to apply sparse regularization to the bottleneck layer to enforce the representation of shape.

[0010] In addition to one or more features described in this paper, sparse regularization is selected from at least one of L1 regularization and the Sigmoid function.

[0011] In one exemplary embodiment, a method for analyzing an image includes: receiving an image by a receiving module; and applying the received image to a machine learning network by the analysis module, and classifying one or more features in the received image. The machine learning network is configured to propagate image data through multiple convolutional layers, each of which includes multiple filter channels. The machine learning network includes a bottleneck layer configured to identify image features based on the shape of image components. The method also includes outputting classification representation data comprising one or more features.

[0012] In addition to one or more features described in this article, the machine learning network is a deep convolutional neural network.

[0013] In addition to one or more features described in this paper, the bottleneck layer is a convolutional layer, and the machine learning network includes at least one pooling layer and a fully connected layer configured to apply one or more classifications to the received image based on the outputs from the bottleneck layer and the multiple convolutional layers.

[0014] In addition to one or more features described in this paper, the bottleneck layer is configured to apply a single filter to the image data.

[0015] In addition to one or more features described herein, the received image comprises multiple image channels, and the bottleneck layer is configured to apply a single filter to the multiple image channels.

[0016] In addition to one or more features described in this paper, the method also includes training a machine learning network by propagating multiple training images via the machine learning network.

[0017] In addition to one or more features described in this paper, training includes applying sparse regularization to the bottleneck layer to force the representation of shape.

[0018] In addition to one or more features described in this paper, sparse regularization is selected from at least one of L1 regularization and the Sigmoid function.

[0019] In one exemplary embodiment, a vehicle system includes a memory having computer-readable instructions and a processing device for executing the computer-readable instructions. The computer-readable instructions control the processing device to receive an image via a receiving module and to apply the received image to a machine learning network via an analysis module, classifying one or more features in the received image. The machine learning network is configured to propagate image data through multiple convolutional layers, each of which includes multiple filter channels. The machine learning network includes a bottleneck layer configured to recognize image features based on the shape of image components. The computer-readable instructions also control the processing device to output characterization data comprising the classification of one or more features.

[0020] In addition to one or more features described in this article, the machine learning network is a deep convolutional neural network.

[0021] In addition to one or more features described in this paper, the bottleneck layer is configured to apply a single filter to the image data.

[0022] In addition to one or more features described herein, the received image comprises multiple image channels, and the bottleneck layer is configured to apply a single filter to the multiple image channels.

[0023] The above-described features and advantages, as well as other features and advantages, of this disclosure will become apparent when taken in conjunction with the accompanying drawings and the following detailed description. Attached Figure Description

[0024] Other features, advantages, and details appear by way of example only in the following detailed description, which refers to the accompanying drawings, wherein:

[0025] Figure 1 This is a top view of a motor vehicle including aspects of an image analysis system according to an exemplary embodiment;

[0026] Figure 2 A computer system configured to perform aspects of image analysis according to an exemplary embodiment is described;

[0027] Figure 3 A deep convolutional neural network according to an exemplary embodiment is depicted;

[0028] Figure 4 This describes the use according to exemplary embodiments. Figure 3 Flowcharts of various aspects of convolutional neural network methods for image analysis;

[0029] Figure 5 An example of a deep convolutional neural network according to an exemplary embodiment is depicted; and

[0030] Figure 6 It is used for depiction Figure 4A graph showing an example of the analysis results from this method compared to the analysis results using a conventional convolutional neural network. Detailed Implementation

[0031] The following description is merely exemplary in nature and is not intended to limit this disclosure, its application, or use. It should be understood that in all the drawings, corresponding reference numerals denote similar or corresponding parts and features.

[0032] According to one or more exemplary embodiments, methods and systems for image analysis are described herein. Embodiments of the image analysis and / or detection system are configured to receive image data from an imaging device (e.g., a camera) and apply the image data to a machine learning network using a machine learning algorithm. In an embodiment, the machine learning network is a deep convolutional neural network (CNN). The image data may include an image having one image channel (e.g., a grayscale or black-and-white image) or an image having multiple image channels (e.g., color component channels).

[0033] Implementations of image analysis systems include machine learning architectures containing multiple convolutional layers, such as convolutional neural network architectures. Each convolutional layer includes multiple filter channels for detecting image features. The architecture also includes one or more "bottleneck" layers. A bottleneck layer is a convolutional layer with a finite number of channels, thus forcing the algorithm to detect only features that can be detected using a single filter (for a given image channel). In one embodiment, the bottleneck layer has only a single filter channel, or a single filter channel for each image channel.

[0034] The embodiments also include methods for training machine learning networks. The provided training aspects increase the network's shape bias, thereby improving the network's ability to recognize shapes and reduce texture bias. For example, sparse regularization may be included at bottleneck channels. In another example, an auxiliary pre-text task is performed during training to enable the network to learn the global object shape. The "global object shape" of an object refers to the shape of the object in various rotational orientations or angular positions.

[0035] One or more machine learning algorithms can be used to generate and / or update networks. Examples of such algorithms include deep learning, supervised learning, unsupervised learning, semi-supervised learning, multi-task learning, etc.

[0036] The embodiments described herein present numerous advantages. Compared to conventional analysis techniques, including conventional deep convolutional networks, this system provides an efficient method for analyzing images with improved shape recognition. These embodiments offer improved shape bias compared to conventional networks. "Shape bias" refers to the network's ability to classify images based on their shape rather than their texture, according to human vision. Another advantage is improved interpretability of network decisions. Improved shape bias has the potential to improve network test time in three ways: better generalization in scenes with reduced visibility, generalization to new tasks, and robustness against adversarial attacks.

[0037] Figure 1 An embodiment of a motor vehicle 10 is shown, which includes a body 12 that at least partially defines a passenger compartment 14. The body 12 also supports various vehicle subsystems, including the engine system 16 and other subsystems that support the functions of the engine system 16 and other vehicle components, such as the braking subsystem, steering subsystem, fuel injection subsystem, exhaust subsystem, etc.

[0038] One or more aspects of the image acquisition and analysis system 18 may be integrated into or connected to the vehicle 10. In this embodiment, the image analysis system 18 includes one or more optical cameras 20 configured to capture images, such as color (RGB) images. The images may be still images or video images. Additional devices or sensors may be included in the image analysis system 18. For example, the vehicle 10 may include one or more radar components 22.

[0039] Camera 20 and / or radar component 22 communicate with one or more processing devices, such as onboard processing device 24 and / or remote processor 26, such as a processor in a mapping or imaging system. Vehicle 10 may also include a user interface system 28 for allowing users (e.g., a driver or passenger) to input data, view images, view features identified by image analysis as discussed herein, and otherwise interact with the processing devices and / or image analysis system 18.

[0040] Figure 2Various aspects of an embodiment of a computer system 30 are illustrated. This computer system 30 communicates with or is part of an image analysis system 18 and can perform various aspects of the embodiments described herein. The computer system 30 includes at least one processing device 32, which typically includes one or more processors for performing various aspects of the image acquisition and analysis methods described herein. The processing device 32 may be integrated into the vehicle 10, for example as an onboard processor 24, or it may be a processing device separate from the vehicle 10, such as a server, personal computer, or mobile device (e.g., a smartphone or tablet). For example, the processing device 32 may be part of or communicate with one or more engine control units (ECUs), one or more vehicle control modules, cloud computing devices, vehicle satellite communication systems, and / or others. The processing device 32 may be configured to perform the image detection and analysis methods described herein and may also perform functions related to the control of various vehicle subsystems.

[0041] The components of computer system 30 include processing device 32 (such as one or more processors or processing units), system memory 34, and bus 36 coupling various system components, including system memory 34, to processing device 32. System memory 34 may include various computer system readable media. Such media may be any available media accessible by processing device 32, and include volatile and non-volatile media, removable and non-removable media.

[0042] For example, system memory 34 includes non-volatile memory 38, such as a hard disk drive, and may also include volatile memory 40, such as random access memory (RAM) and / or cache memory. Computer system 30 may also include other removable / non-removable, volatile / non-volatile computer system storage media.

[0043] System memory 34 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments described herein. For example, system memory 34 stores various program modules that generally perform the functions and / or methods of the embodiments described herein. It may include a receiving module 42 to perform functions related to acquiring and processing images and information received from sensors, and may include an image analysis or processing module 44 to perform functions related to image analysis, such as executing machine learning algorithms, generating models, and labeling images. System memory 34 may also store various data structures 46, such as data files or other structures storing data related to image detection and analysis. Examples of such data structures include camera images, radar images, models, and labeled images. As used herein, the term "module" refers to processing circuitry, which may include application-specific integrated circuits (ASICs), electronic circuitry, processors (shared, dedicated, or grouped), and memory executing one or more software or firmware programs, combinational logic circuitry, and / or other suitable components providing the described functions.

[0044] The processing device 32 can also communicate with one or more external devices 48, such as a keyboard, a pointing device, and / or any device that enables the processing device 32 to communicate with one or more other computing devices (e.g., a network interface card, a modem, etc.). Furthermore, the processing device 32 can communicate with one or more devices, such as a camera 20 and a radar assembly 22 for image analysis. The processing device 32 can also communicate with other devices that can be used in conjunction with image analysis, such as a Global Positioning System (GPS) device 50 and a vehicle control device or system 52 (e.g., for driver assistance and / or autonomous vehicle control). Communication with various devices can be made via an input / output (I / O) interface 54.

[0045] Processing device 32 can also communicate with one or more networks 56 via network adapter 58, such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet). It should be understood that, although not shown, other hardware and / or software components may be used in conjunction with computer system 30. Examples include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, and data archiving storage systems.

[0046] Figure 3This is a block diagram depicting various aspects of an embodiment of the architecture of a deep convolutional neural network 70. The network 70 is configured to receive images (e.g., camera images from vehicle 10) and pass image data through one or more convolutional layers. Each convolutional layer includes a set of learnable filters (value matrices) that slide across the image to detect corresponding features. Each filter is applied to the image data and mapped onto an output matrix called an activation map or feature map. The feature map includes the values ​​of the filter at each spatial location the filter passes through, indicating the presence or absence of the feature the filter is trying to detect.

[0047] Network 70 can have any number of convolutional layers, and each convolutional layer can include any number of filters. Filters in a convolutional layer can also be referred to as channels. As discussed further in this paper, the network can include additional layers, such as pooling layers.

[0048] For example, an input image 72, such as a camera image, is fed into a first convolutional layer 74, which includes multiple filters, each configured to detect a corresponding feature (e.g., line curves, colors, etc.). These features may be low-level features, which are combined in subsequent layers to identify more complex features. The first convolutional layer 74 outputs a set of feature maps, which are fed into a second convolutional layer 76 that has its own set of filters and outputs a corresponding set of feature maps. The network 70 may include any number of additional convolutional layers, such as a third convolutional layer 78. The outputs of the previous layers are then fed into a fully connected (FC) layer 80, which acts as a classifier and outputs classification labels based on the layer outputs. Note that the convolutional network architecture is not limited to any particular type or number of filters and / or layers.

[0049] Embodiments of network 70 also include convolutional layers 82 having a single convolutional channel, or multiple convolutional channels less than a threshold number. The threshold number is chosen to force the convolutional network algorithm to use shape-related filters and prevent convolutional layers 82 from encoding texture or other features that typically require multiple filters or channels to encode. For example, a typical convolutional neural network includes multiple filters in convolutional layers to identify texture; by reducing the number of available channels, the algorithm cannot encode texture at convolutional layer 82 and is instead “forced” to encode shape. Thus, convolutional network 70 has increased shape bias compared to a conventional network because the number of times shape is identified instead of texture increases. For example, the architecture of network 70 can increase shape bias to approximately 90% or higher, reaching a level of shape bias comparable to human vision. Convolutional layer 82 is also referred to as a “bottleneck layer” 82.

[0050] Example of shape output activation in Figure 3 The image shows a bird-shaped filter, configured based on previously applied training data. Note that this shape is provided for illustrative purposes, not for limitation.

[0051] In this embodiment, the bottleneck layer 82 has a single filter (referred to herein as a "channel" or "filter channel"). Image data input to the bottleneck layer (e.g., an image received from a camera or feature map) may have more than one image channel. For example, if the image is a grayscale image, the image input to network 70 has a single image channel, or if the image is an RGB (red-green-blue) image, it has three image channels. In either case, the bottleneck layer has a single filter channel.

[0052] For example, when performing image analysis and / or network training, the first convolutional layer 74 slides each filter in layer 74 onto all spatial locations in the received image 72 and outputs an activation map (feature map) for each filter. The first convolutional layer 72 outputs the feature map, which is then applied as input to the next convolutional layer 76. The feature map from the convolutional layer 76 is applied to a bottleneck layer 82, which applies a single filter to each applied feature map and outputs the feature map to the convolutional layer 78. The outputs from all convolutional layers are then applied to a fully connected layer 80 for classification. As discussed further herein, a pooling layer can be placed between two convolutional layers to simplify the feature maps and reduce their size.

[0053] Figure 4 An embodiment of a method 90 for analyzing image data is depicted. An image analysis system 18 and / or an image analysis module 44, or other processing devices or systems, may be used to perform aspects of method 90. Method 90 is discussed in conjunction with boxes 91-94. Method 90 is not limited to the number or order of its steps, as some steps represented by boxes 91-94 may be performed in a different order than described below, or fewer than all steps may be performed.

[0054] In box 91, a deep convolutional neural network 70 is trained using multiple training images to configure various filters that can later be used in image recognition and classification. Typically, training involves applying multiple images (usually thousands or more) to the network 70. The training images may be accompanied by representational data, such as object or feature labels.

[0055] In box 92, network 70 is trained by propagating the training images through multiple convolutional layers, including bottleneck layer 82. Note that the network can include any desired number of bottleneck layers 82 at any desired location within network 70. Training can be performed by any suitable training procedure, such as supervised or unsupervised training.

[0056] In one embodiment, during training, the bottleneck layer 82 includes a sparsity regularization function to enforce the representation of shape (as opposed to, for example, texture). For example, L1 regularization can be used to increase the sparsity of activations, L1 regularization of the spatial gradient operator can be used to increase the sparsity of gradients, and / or the sigmoid function can be applied to activations to increase the sparsity of activations.

[0057] In one embodiment, the training algorithm includes a self-supervised auxiliary training task to improve shape bias. The algorithm can perform rotation prediction of objects or features to learn the global object shape (e.g., the shape of an object in various orientations). For example, the auxiliary training task is trained on image features that are rotated about a selected image point relative to an initial orientation (zero degrees) to various orientations or angular positions (e.g., 90 degrees, 180 degrees).

[0058] In box 93, one or more images, such as camera images acquired by vehicle 10, are applied to the trained network 70. The acquired images are processed, and various classifications (e.g., shape, object type, such as vehicle, pedestrian, road features, etc.) are assigned to the identified features or objects.

[0059] In box 94, the categorized image is output to an appropriate location, such as storing the image and categorization data, and / or displaying the image to a user. For example, the categorized image may be stored in a database or other suitable location for various purposes, such as further training, vehicle diagnostics, mapping, etc. In one example, the categorized image is output to a vehicle control processor, such as a control unit for autonomous vehicle operation and / or driver assistance.

[0060] Figure 5 An embodiment of a deep convolutional neural network 100 (which may be an example of network 70) is depicted. Network 100 is configured to receive an image 102, which may be a training image used to train network 100 or an image input used for classification using previously learned filters. In this example, image 102 is an RGB image with a height dimension of 224 pixels, a width dimension of 224 pixels, and three image channels (i.e., an image size of 224×224×3).

[0061] The first set of convolutional layers 104 has a size of 224×224×64 (i.e., each of these layers 104 includes 64 filter channels with a height of 224 pixels and a width of 224 pixels). Feature maps (one for each filter) are output to a first pooling layer 106 to pool the feature maps, thereby reducing the size to 112×112. The reduced feature maps are then input to a second set of convolutional layers 108, which includes 128 filter channels.

[0062] The output from convolutional layer 108 is reduced again at a 56×56 pooling layer 110 and fed into a bottleneck layer 130, which has the same dimensions as pooling layer 110 but only one filter channel. Therefore, the size of bottleneck layer 130 is 56×56×1.

[0063] Note that network 100 may have more than one bottleneck layer, and may have one or more bottleneck layers, which are different locations in network 100.

[0064] The output from the bottleneck layer is then fed into a third set of convolutional layers 114, which has a size of 56×56 pixels and 256 channel filters. Network 100 also includes a pooling layer 116 that reduces the output feature maps to a 28×28 mapping, and a fourth set of 28×28 convolutional layers 118 (with 512 filters). Another pooling layer 120 and a set of convolutional layers 122 generate their respective output feature maps, which are then pooled again in pooling layer 124 and output to a fully connected (FC) layer 126 and activation function layers 128, such as Softmax, for classifying images 102 (e.g., vehicles, pedestrians, etc.).

[0065] Figure 6 The results of image analysis using the embodiments described herein are shown compared to conventional deep convolutional neural network analysis. Figure 6 A graph depicting shape bias as a function of image classification accuracy is plotted. Accuracy is expressed as the percentage of detections where image features are correctly classified, and shape bias is expressed as the percentage or proportion of instances that identify shape rather than texture. Point 150 shows the general analysis results for the first set of images, and point 160 shows the analysis results for the first set of images using networks 70 and / or 100.

[0066] Point 152 shows the results of the routine analysis of the second set of images, and point 162 shows the results of the analysis of the second set of images using network 70 and / or 100. Point 154 shows the results of the routine analysis of the third set of images, and point 164 shows the results of the analysis of the third set of images using network 70 and / or 100. Finally, point 156 shows the results of the routine analysis of the fourth set of images, and point 166 shows the results of the analysis of the fourth set of images using network 70 and / or 100.

[0067] As shown in the figure, by using a bottleneck layer, shape deviation increases significantly, in some cases approaching human-level shape deviation (points). Although accuracy is slightly reduced, the loss of accuracy in these examples is acceptable given the significant increase in shape deviation.

[0068] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that, when used in this specification, the term “comprising” and variations thereof specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof.

[0069] While the foregoing disclosure has been described with reference to exemplary embodiments, those skilled in the art will understand that various changes can be made without departing from its scope, and equivalents can replace its elements. Furthermore, many modifications can be made to adapt particular situations or materials to the teachings of this disclosure without departing from the basic scope of this disclosure. Therefore, it is intended that this disclosure be limited to the specific embodiments disclosed, but will include all embodiments falling within its scope.

Claims

1. A system for analyzing images, comprising: A processing device configured to receive images; An analysis module is configured to apply a received image to a machine learning network and classify one or more features in the received image. The machine learning network is configured to propagate image data through multiple convolutional layers, each of which includes multiple filter channels. The multiple filter channels are configured to apply multiple filters to the image data and output a feature map for each filter. The machine learning network includes a bottleneck layer, which is configured to identify image features based on the shape of the image components. The bottleneck layer is configured to prevent the encoding of textures and is configured to apply a single filter related to the shape of the image components to each feature map. as well as The output module is configured to output categorical representation data that includes one or more features.

2. The system according to claim 1, wherein, The machine learning network is a deep convolutional neural network.

3. The system according to claim 1, wherein, The bottleneck layer is a convolutional layer, and the machine learning network includes at least one pooling layer and a fully connected layer configured to apply one or more classifications to a received image based on the outputs from the bottleneck layer and the plurality of convolutional layers.

4. The system according to claim 1, wherein, The received image includes multiple image channels, and the bottleneck layer is configured to apply a single filter to multiple image channels.

5. The system according to claim 1, wherein, The analysis module is configured to train the machine learning network by propagating multiple training images via the machine learning network.

6. The system according to claim 5, wherein, The analysis module is configured to apply sparse regularization to the bottleneck layer to force a shape representation.

7. The system according to claim 6, wherein, The sparse regularization is selected from at least one of L1 regularization and the Sigmoid function.

8. A method for analyzing images, comprising: Receive image; The analysis module applies the received image to the machine learning network and classifies one or more features in the received image. The machine learning network is configured to propagate image data through multiple convolutional layers. Each of the multiple convolutional layers includes multiple filter channels. The multiple filter channels are configured to apply multiple filters to the image data and output a feature map for each filter. The machine learning network includes a bottleneck layer, which is configured to identify image features based on the shape of the image components. The bottleneck layer is configured to prevent the encoding of texture and is configured to apply a single filter related to the shape of the image components to each feature map. as well as The output includes categorical representation data containing one or more features.

9. The method according to claim 8, wherein, The machine learning network is a deep convolutional neural network, the received image includes one or more image channels, and the bottleneck layer is configured to apply a single filter to one or more image channels.

Images