Target recognition method and system, and readable storage medium

Abstract

The embodiments of the present application disclose a target recognition method and system, and a readable storage medium. A specific implementation of the method includes: calculating the depth information of the image to be recognized, and performing super-pixel segmentation on the image to be recognized to obtain a super-pixel image; generating three-dimensional image data based on the image data of the super-pixel image and the depth information; inputting the three-dimensional image data into a deep neural network for target recognition to obtain a recognition result. This implementation can perform accurate target recognition, especially accurately identify targets with similar appearances but different volumes, and can reduce the number of parameters of the network model and improve the recognition speed while ensuring the accuracy of target recognition.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a target recognition method and system, and a readable storage medium. Background Art

[0002] At present, the detection and recognition of targets in images (such as goods and other objects in surveillance images of unmanned supermarkets) are mostly based on two-dimensional images. First, the target objects are annotated in the two-dimensional images and used as training sets to train deep neural networks. Then, the trained deep neural networks can be used to recognize targets in the collected images. However, the above methods have problems such as low recognition accuracy, large number of parameters of the network model, and slow recognition speed. For example, for targets with similar appearances but different volumes (such as 2L and 500mL bottles of the same brand of beverages), since the image features of the two-dimensional images (such as RGB values, etc.) cannot directly reflect the volume of the targets, the above methods cannot finely identify the differences between these targets, which easily leads to recognition errors. Summary of the invention

[0003] The purpose of the present application is to provide a target recognition method and system, and a readable storage medium to solve at least one of the problems existing in the prior art.

[0004] In order to achieve the above objectives, this application adopts the following technical solutions:

[0005] The first aspect of the present application provides a target recognition method, comprising:

[0006] Calculating the depth information of the image to be identified, and performing super-pixel segmentation on the image to be identified to obtain a super-pixel image;

[0007] generating three-dimensional image data according to the image data of the superpixel image and the depth information;

[0008] The three-dimensional image data is input into a deep neural network for target recognition to obtain a recognition result.

[0009] The target recognition method provided in the first aspect of the present application uses depth information as a new channel and uses a deep neural network to perform image segmentation (i.e., target edge extraction, or pixel classification) on three-dimensional image data to achieve target recognition, which can improve recognition accuracy, especially accurately identify targets with similar appearance but different volumes. Moreover, due to the use of superpixel segmentation, the number of parameters of the network model can be reduced and the recognition speed can be improved while ensuring the target recognition accuracy.

[0010] Optionally, performing superpixel segmentation on the image to be identified includes: extracting texture features of pixel points of the image to be identified, and performing superpixel segmentation on the image to be identified according to the texture features of the pixel points.

[0011] Therefore, the image to be identified can be segmented into a series of small areas consisting of adjacent pixels with similar texture features through superpixel segmentation based on pixel texture features. These small areas retain effective information for further image segmentation without destroying the edge information of objects in the image.

[0012] Optionally, the target recognition includes: the deep neural network performs multiple downsampling on the input three-dimensional image data using multiple downsampling layers and then performs multiple upsampling using multiple upsampling layers; wherein the input of the first upsampling layer among the multiple upsampling layers is a feature map obtained by performing at least one convolution operation on the output of the last downsampling layer, and the input of other upsampling layers is a feature map formed by splicing the feature map corresponding to the corresponding downsampling layer and the feature map output by the previous upsampling layer.

[0013] Therefore, a deep neural network with a simple structure that can adapt to a very small training set can be used for target recognition. The input of the upsampling layer is a feature map corresponding to the corresponding downsampling layer and the feature map output by the previous upsampling layer. The feature map can combine deep and shallow features, refine the image, and obtain features of different dimensions of the image to be recognized, thereby improving the effects of image segmentation and target recognition.

[0014] Optionally, each of the multiple upsampling steps includes performing interpolation upsampling on a portion of channels randomly selected from a feature map of an input upsampling layer and performing transposed convolution upsampling on another portion of the channels.

[0015] This optional method uses an upsampling method that randomly combines interpolation and transposed convolution, which can further reduce the number of parameters of the network model and improve the recognition speed while ensuring the accuracy of image segmentation and target recognition.

[0016] Optionally, each of the multiple upsamplings includes performing interpolation upsampling on half of the channels randomly selected from the feature map of the input upsampling layer and performing transposed convolution upsampling on the other half of the channels.

[0017] In this way, it can be ensured that the number of parameters of the network model is reduced and the effectiveness of the recognition speed is improved.

[0018] Optionally, the downsampling layer includes at least one convolution layer and one pooling layer; the first upsampling layer among the multiple upsampling layers includes an interpolation module and a transposed convolution layer, and the other upsampling layers include an interpolation module, a transposed convolution layer and at least one convolution layer.

[0019] Optionally, the image to be identified is collected by a binocular camera, and calculating the depth information of the image to be identified includes: calculating the depth value of each coordinate point in the collected image to be identified according to the acquisition parameters of the binocular camera, normalizing the depth values ​​of all coordinate points, and obtaining the depth information of the image to be identified.

[0020] Therefore, the depth information of the image to be recognized can be calculated accurately and efficiently while using a relatively simple image acquisition device.

[0021] Optionally, the method further comprises:

[0022] Obtaining a training sample with annotations, wherein the training sample is a superpixel image with annotated target edges;

[0023] A training set is constructed according to the training samples, and the deep neural network is obtained by training with the training set.

[0024] The second aspect of the present invention provides a target recognition system, comprising: an image acquisition device, a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the image acquisition device is used to acquire an image to be recognized, and when the processor executes the program, the following are achieved: calculating the depth information of the image to be recognized, and performing super-pixel segmentation on the image to be recognized to obtain a super-pixel image; generating three-dimensional image data based on the image data of the super-pixel image and the depth information; and inputting the three-dimensional image data into a deep neural network for target recognition to obtain a recognition result.

[0025] Optionally, the image acquisition device is a binocular camera.

[0026] The third aspect of the present application provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method provided in the first aspect of the present application is implemented.

[0027] The beneficial effects of this application are as follows:

[0028] The present application can perform accurate target recognition, especially can accurately identify targets with similar appearance but different volumes, and can reduce the number of parameters of the network model and improve the recognition speed while ensuring the target recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The specific implementation methods of the present application are further described in detail below in conjunction with the accompanying drawings.

[0030] Figure 1 A flow chart of a target recognition method provided by an embodiment of the present application is shown.

[0031] Figure 2A schematic diagram showing the geometric parameters of a binocular camera.

[0032] Figure 3 A schematic diagram of the network structure of a deep neural network is shown.

[0033] Figure 4 Show Figure 3 Schematic diagram of the improved network structure of the deep neural network shown.

[0034] Figure 5 Shows a schematic diagram of the structure of the Inception model.

[0035] Figure 6 A schematic diagram of a target recognition system provided by another embodiment of the present application is shown.

[0036] Figure 7 A schematic diagram showing the structure of a computer system implementing a target recognition device. DETAILED DESCRIPTION

[0037] In order to more clearly illustrate the present application, the present application is further described below in conjunction with the embodiments and drawings. Similar components in the drawings are represented by the same reference numerals. It should be understood by those skilled in the art that the content specifically described below is illustrative rather than restrictive, and should not be used to limit the scope of protection of the present application.

[0038] like Figure 1 As shown, an embodiment of the present application provides a target recognition method, comprising the following steps:

[0039] Step S100: Calculate the depth information of the image to be identified, and perform super-pixel segmentation on the image to be identified to obtain a super-pixel image.

[0040] In a possible implementation, the image to be identified is collected by a binocular camera, and calculating the depth information of the image to be identified includes: calculating the depth value of each coordinate point in the collected image to be identified according to the acquisition parameters of the binocular camera, normalizing the depth values ​​of all coordinate points, and obtaining the depth information of the image to be identified.

[0041] Therefore, without the need for the assistance of other devices such as infrared sensors, a relatively simple binocular camera can be used to capture the image to be identified, and the depth information of the image to be identified can be accurately and efficiently calculated as the three-dimensional information for the subsequent generation of three-dimensional image data. The three-dimensional information is relatively simple and the amount of computational data is small.

[0042] In a specific example, the process of calculating the depth value of each coordinate point in the acquired image to be recognized according to the acquisition parameters of the binocular camera is as follows:

[0043] After the image to be identified is acquired by the binocular camera, the depth value of each coordinate point in the image to be identified is used as depth data according to the acquisition parameters of the binocular camera (or the camera internal parameters), and the depth data is matched with the coordinate points of the original image to be identified. The depth values ​​of all coordinate points are normalized and the depth value is used as a new channel.

[0044] Depth value calculation method is as follows:

[0045] Calculate the parallax d: d = u L -u R , where u L and u R Respectively Figure 2 The coordinates of point P shown in the left and right views;

[0046] Calculate the depth value z: z = (fb) / d, where, Figure 2 As shown, f is the focal length of the binocular camera, and b is the binocular baseline length of the binocular camera.

[0047] In a possible implementation, performing superpixel segmentation on the image to be identified includes: extracting texture features of pixel points of the image to be identified, and performing superpixel segmentation on the image to be identified according to the texture features of the pixel points.

[0048] Therefore, the image to be identified can be segmented into a series of small areas consisting of adjacent pixels with similar texture features through superpixel segmentation based on pixel texture features. These small areas retain effective information for further image segmentation without destroying the edge information of objects in the image.

[0049] Among them, texture is a visual feature that reflects homogeneous phenomena in an image. It reflects the slowly changing or periodically changing surface structure and organization properties of an object. Texture has three major characteristics: a certain local sequence is repeated continuously, non-random arrangement, and the texture area is roughly uniform. Texture is different from image features such as grayscale and color. It is expressed by the grayscale distribution of pixels and their surrounding spatial neighborhoods, that is, local texture information. The different degrees of repetitiveness of local texture information is global texture information.

[0050] Texture features can be divided into four types:

[0051] (1) Statistical texture features:

[0052] Based on the grayscale attributes of pixels and their neighborhoods, the statistical features in the texture area, or the first-order, second-order or higher-order statistical features of the grayscale in pixels and their neighborhoods, are studied. The main statistical texture feature is GLCM (gray-level co-occurrence matrix), which is a method based on estimating the second-order combined conditional probability density of the image. GLCM mainly describes the probability of a pair of pixels separated by d pixels in the theta direction having grayscale values ​​i and j respectively.

[0053] (2) Model-type texture features:

[0054] Assuming that texture is formed in a distribution model controlled by some parameters, the model parameters are estimated from the realization of texture image, and the image is segmented using the parameters as features or some strategy. Therefore, the estimation of model parameters is the core issue of this method. The model-based texture feature extraction method is mainly based on random field method and fractal method.

[0055] Random field model method: It attempts to describe the random process of texture with a probability model. It performs statistical operations on random data or random features to estimate the parameters of the texture model, and then clusters a series of model parameters to form model parameters consistent with the number of texture types. The estimated model parameters are used to estimate the maximum a posteriori probability of the grayscale image point by point to determine the probability of the most likely attribution of the pixel in the case of the pixel and its neighborhood. The random field model actually describes the statistical dependence of pixels in the image on neighboring pixels.

[0056] Fractal model method: As an important feature and measure of fractals, fractal dimension combines the spatial information and grayscale information of images in a simple and organic way, and therefore has attracted much attention in image processing. The application of fractal dimension in image processing is based on two points: (a) different types of morphological materials in nature generally have different fractal dimensions; (b) due to the researchers' assumptions, there is a certain correspondence between fractals in nature and the grayscale representation of images. Studies have shown that the human visual system's perception of roughness and concavity is closely related to the fractal dimension. Therefore, the fractal dimension of an image region can be used to describe the texture characteristics of an image region. The core issue of fractal dimension in describing texture is how to accurately estimate the fractal dimension.

[0057] (3) Signal processing texture features:

[0058] Based on time domain, frequency domain and multi-scale analysis, after a certain transformation is performed on a certain area in the texture image, a relatively stable eigenvalue is extracted, and this eigenvalue is used as a feature to represent the consistency within the area and the difference between areas. The texture features of signal processing mainly use some linear transformation, filter or filter group to convert the texture to the transformation domain, and then apply some energy criterion to extract the texture features. Therefore, the method based on signal processing is also called filtering method. Most signal processing methods are based on the assumption that the energy distribution in the frequency domain can identify the texture.

[0059] (4) Structural texture features:

[0060] Based on the analysis of texture features by "texture primitives", efforts are made to find texture primitives. It is believed that texture is composed of many texture primitives. Different types of texture primitives, different directions and numbers determine the expression of texture.

[0061] In addition, the superpixel segmentation in this embodiment can also be achieved based on features such as color and brightness of pixel points.

[0062] In a specific example, the process of performing super-pixel segmentation on the image to be identified to obtain the super-pixel image is as follows:

[0063] The original image to be identified (two-dimensional image, left view or right view) captured by the binocular camera is segmented into superpixels using a superpixel algorithm according to the texture features of the image to be identified. For example, the texture features of the image to be identified are extracted, and the image to be identified is segmented into superpixels according to the edges where the texture features change. Each unit obtained by superpixel segmentation is a superpixel. For each superpixel, the coordinate position of the superpixel is recorded (including the center coordinates of the superpixel and the coordinates of the original pixel point contained in the superpixel).

[0064] When performing superpixel segmentation, the edges obtained by superpixel segmentation can be annotated (along the edges of the target), and the superpixels within the target boundary can be marked as corresponding categories, that is, if a superpixel is within the target boundary in the original image, then this superpixel is marked as the corresponding target category, otherwise it is marked as background. In this way, the marking during superpixel segmentation can be used to assist subsequent target recognition.

[0065] It should be noted that in step S100, the execution order of "calculating the depth information of the image to be identified" and "performing superpixel segmentation on the image to be identified" is not restricted, that is, "calculating the depth information of the image to be identified" can be executed first and then "performing superpixel segmentation on the image to be identified", or "performing superpixel segmentation on the image to be identified" can be executed first and then "calculating the depth information of the image to be identified", or both can be executed in parallel.

[0066] After step S100, the image to be identified (e.g., an RGB image) can be represented as three-dimensional image data (e.g., a superpixel matrix diagram) containing depth information and two-dimensional image features (e.g., RGB values). Both the two-dimensional image features and the depth information are used as channels of superpixels. In this way, the image information of the original larger-sized image to be identified can be converted into high-dimensional and small-sized image information, and the size can be reduced without sacrificing the image features, thereby reducing the number of parameters of the network model of the deep neural network.

[0067] Step S200, as described above, generates three-dimensional image data based on the image data of the superpixel image and the depth information, thereby realizing the splicing of the two-dimensional information (such as RGB values, etc.) and the three-dimensional information (depth information) of the image.

[0068] Step S300: input the three-dimensional image data into a deep neural network for target recognition to obtain a recognition result.

[0069] In a possible implementation, the target recognition includes: the deep neural network performs multiple downsampling on the input three-dimensional image data using multiple downsampling layers and then performs multiple upsampling using multiple upsampling layers; wherein the input of the first upsampling layer among the multiple upsampling layers is a feature map obtained by performing at least one convolution operation on the output of the last downsampling layer, and the input of other upsampling layers is a feature map formed by splicing the feature map corresponding to the corresponding downsampling layer and the feature map output by the previous upsampling layer.

[0070] Therefore, a deep neural network with a simple structure that can adapt to a very small training set can be used for target recognition. The input of the upsampling layer is a feature map corresponding to the corresponding downsampling layer and the feature map output by the previous upsampling layer. The feature map can combine deep and shallow features, refine the image, and obtain features of different dimensions of the image to be recognized, thereby improving the effects of image segmentation and target recognition.

[0071] In a specific example, the deep neural network adopted by this optional method is a U-Net network, and the description of its specific network structure is as follows.

[0072] As the name suggests, the U-Net network has a U-shaped network structure. Figure 3As shown, its essence is an improved fully convolutional neural network. The U-Net network includes an encoder network 301 for downsampling and a decoder network 302 for upsampling. The input image is convoluted multiple times by the encoder network 301 to obtain multiple feature maps, and then deconvoluted multiple times by the decoder network 302. In this process, multiple feature maps corresponding to the encoder network 301 are spliced ​​to combine deep and shallow features and refine the image, so as to obtain features on different dimensions of the input image, thereby improving the effects of image segmentation and target recognition. Among them, if the two feature maps to be spliced ​​are of different sizes, after copying (Copy) the feature map corresponding to the encoder network 301, it is necessary to crop (Crop) and then transmit it to the decoder network 302 for splicing.

[0073] like Figure 3 As shown, in this example:

[0074] In order from shallow to deep, the encoder network 301 includes three downsampling layers, and the decoder network 302 includes three upsampling layers. It should be noted that the number of downsampling layers and the number of upsampling layers are not limited to three, but the number of downsampling layers and the number of upsampling layers should be the same.

[0075] Between the encoder network 301 and the decoder network 302, from shallow to deep ( Figure 3 Three feature propagation layers 3031, 3032, and 3033 are established in the order from top to bottom on the left side.

[0076] It should be noted that the U-Net network may also include a classification layer (not shown in the figure) after the decoder network 302 is set. The classifier can be set to calculate the probability that the pixels in the feature map (FeatureMap) corresponding to the three-dimensional image data belong to different categories, so as to achieve category prediction of the pixels in the feature map corresponding to the three-dimensional image data, thereby performing pixel classification, that is, image segmentation, to achieve target recognition.

[0077] For the encoder network 301 , feature extraction of three-dimensional image data can be performed through a number of three downsampling layers, and the feature map obtained by feature extraction is transmitted to the decoder network 302 through three feature propagation layers 3031 , 3032 , and 3033 .

[0078] Specifically, in this example, the downsampling layer includes two convolutional layers and one pooling layer, and the pooling layer is a maximum pooling layer. The three-dimensional image data is input to the shallowest downsampling layer ( Figure 3 The input 3D image data is convolved through the shallowest downsampling layer (such as Figure 3The figure shows two convolutions), and the feature map corresponding to the shallowest downsampling layer is obtained, and then input to the next shallowest downsampling layer after downsampling processing (maximum pooling downsampling processing).

[0079] From shallow to deep ( Figure 3 The other two downsampling layers in the encoder network 301 are traversed in the order from top to bottom on the left side in the figure to obtain feature maps corresponding to the other two downsampling layers traversed.

[0080] Between the encoder network 301 and the decoder network 302 , feature propagation of the feature graph is performed through feature propagation layers 3031 , 3032 , and 3033 , respectively.

[0081] The output of the deepest downsampling layer is convolved ( Figure 3 After that, the feature map is directly transmitted to the deepest upsampling layer ( Figure 3 ).

[0082] In the encoder network 301, the horizontal hollow arrows represent the convolution processing performed by the convolution layer, and the downward solid arrows represent the maximum pooling downsampling processing performed by the maximum pooling layer.

[0083] Among them, as the encoder network layer deepens, the feature extraction of three-dimensional image data is gradually abstracted from local description to global description, and then the three-dimensional image data is described more accurately, which is conducive to ensuring the accuracy of image segmentation.

[0084] For the decoder network 302, not only is feature extraction of the three-dimensional image data performed through several upsamplings, but feature fusion is also performed on the three-dimensional image data.

[0085] Specifically, in this example, the first upsampling layer (the deepest upsampling layer) includes a transposed convolution layer (transposed convolution is also called deconvolution, or up-convolution), and other upsampling layers include a transposed convolution layer and two convolution layers. The deepest upsampling layer performs a transposed convolution upsampling process on the feature map obtained after the output of the deepest downsampling layer is convoluted to obtain the first feature map to be fused.

[0086] The first feature map to be fused is input to the next deeper upsampling layer, and after being fused with the feature map corresponding to the deepest downsampling layer transmitted through the feature transmission layer 3033, a convolution process is performed (such as Figure 3 The figure shows two convolutions) to obtain the second feature map to be fused.

[0087] In order from deep to shallow ( Figure 3The other two upsampling layers are traversed to obtain the feature maps corresponding to the upsampling layers traversed.

[0088] After the traversal is completed, the feature map corresponding to the three-dimensional image data is obtained from the feature map corresponding to the shallowest upsampling layer.

[0089] In the decoder network 302, the horizontal hollow arrows represent the convolution processing performed by the convolution layer, and the upward solid arrows represent the transposed convolution upsampling processing performed by the transposed convolution layer.

[0090] Through the above process, through the combination of the encoder network and the decoder network, not only the computational complexity of image segmentation is effectively reduced, which is beneficial to improving the segmentation efficiency, but also the accuracy of image segmentation is fully guaranteed.

[0091] In a possible implementation, each of the multiple upsampling steps includes performing interpolation upsampling on a portion of channels randomly selected from a feature map of an input upsampling layer and performing transposed convolution upsampling on another portion of channels.

[0092] This implementation adopts an upsampling method of interpolation and transposed convolution random splicing, which can further reduce the number of parameters of the network model and improve the recognition speed while ensuring the accuracy of image segmentation and target recognition.

[0093] In a possible implementation, each of the multiple upsampling steps includes performing interpolation upsampling on half of the channels randomly selected from the feature map of the input upsampling layer and performing transposed convolution upsampling on the other half of the channels.

[0094] In this way, it can be ensured that the number of parameters of the network model is reduced and the effectiveness of the recognition speed is improved.

[0095] In one possible implementation, the downsampling layer includes at least one convolution layer and one pooling layer; the first upsampling layer among the multiple upsampling layers includes an interpolation module and a transposed convolution layer, and the other upsampling layers include an interpolation module, a transposed convolution layer and at least one convolution layer.

[0096] In a specific example, the deep neural network using the upsampling method of interpolation and transposed convolution random splicing can be regarded as an improvement on the U-Net network introduced above, which can be understood as replacing the transposed convolution upsampling process in the U-Net network with the upsampling process of interpolation and transposed convolution random splicing.

[0097] In a specific example, the network structure of the improved deep neural network is as follows: Figure 4As shown, it includes an encoder network 401 for downsampling and a decoder network 402 for upsampling. The input image is reduced by multiple convolutions through the encoder network 401 to obtain multiple feature maps, and then the decoder network 402 performs multiple expansions based on interpolation and transposed convolution random splicing. In this process, multiple feature maps corresponding to the encoder network 401 are also spliced ​​to combine deep and shallow features and refine the image, so as to obtain features on different dimensions of the input image, thereby improving the effects of image segmentation and target recognition. In addition, it can also effectively reduce the number of parameters of the network model and improve the recognition speed.

[0098] like Figure 4 As shown, in this example:

[0099] In order from shallow to deep, the encoder network 401 includes three downsampling layers, and the decoder network 402 includes three upsampling layers.

[0100] Between the encoder network 401 and the decoder network 402, three feature propagation layers 4031, 4032, and 4033 are established in order from shallow to deep.

[0101] It should be noted that the improved deep neural network may also include a classification layer (not shown in the figure) after the decoder network 402 is set. The classifier may be set to calculate the probability that the pixels in the feature map (FeatureMap) corresponding to the three-dimensional image data belong to different categories, thereby realizing the category prediction of the pixels in the feature map corresponding to the three-dimensional image data, thereby performing pixel classification, i.e., image segmentation, to achieve target recognition.

[0102] For the encoder network 401, feature extraction of three-dimensional image data can be performed through a number of three downsampling layers, and the feature map obtained by feature extraction is transmitted to the decoder network 402 through three feature propagation layers 4031, 4032, and 4033.

[0103] Specifically, in this example, the downsampling layer includes two convolutional layers and one pooling layer, and the pooling layer is a maximum pooling layer. The three-dimensional image data is input to the shallowest downsampling layer ( Figure 4 The input 3D image data is convolved through the shallowest downsampling layer (such as Figure 4 The figure shows two convolutions), and the feature map corresponding to the shallowest downsampling layer is obtained, and then input to the next shallowest downsampling layer after downsampling processing (maximum pooling downsampling processing).

[0104] From shallow to deep ( Figure 4The other two downsampling layers in the encoder network 401 are traversed in the order from top to bottom on the left side in the figure to obtain feature maps corresponding to the other two downsampling layers traversed.

[0105] Between the encoder network 401 and the decoder network 402, feature propagation of the feature graph is performed through feature propagation layers 4031, 4032, and 4033 respectively.

[0106] The output of the deepest downsampling layer is convolved ( Figure 4 After that, the feature map is directly transmitted to the deepest upsampling layer ( Figure 4 (shown by the upward hollow arrow and the upward solid arrow at the bottom right of the figure).

[0107] In the encoder network 401, the horizontal hollow arrows represent the convolution processing performed by the convolution layer, and the downward solid arrows represent the maximum pooling downsampling processing performed by the maximum pooling layer.

[0108] For the decoder network 402, not only is feature extraction of the 3D image data performed through several upsamplings, but feature fusion is also performed on the 3D image data.

[0109] Specifically, in this example, the first upsampling layer (the deepest upsampling layer) includes a transposed convolution layer and an interpolation module, and the other upsampling layers include a transposed convolution layer, an interpolation module, and two convolution layers. The deepest upsampling layer randomly selects half of the channels from the feature map obtained after convolution processing of the output of the deepest downsampling layer, and performs interpolation upsampling processing on the other half of the channels to obtain the first feature map to be fused.

[0110] The first feature map to be fused is input to the next deeper upsampling layer, and after being fused with the feature map corresponding to the deepest downsampling layer transmitted through the feature transmission layer 4033, a convolution process is performed (such as Figure 3 The figure shows two convolutions) to obtain the second feature map to be fused.

[0111] In order from deep to shallow ( Figure 3 The other two upsampling layers are traversed (wherein, each upsampling performs a random channel selection) to obtain the feature map corresponding to the upsampling layer traversed.

[0112] After the traversal is completed, the feature map corresponding to the three-dimensional image data is obtained from the feature map corresponding to the shallowest upsampling layer.

[0113] In the decoder network 402, the horizontal hollow arrows represent the convolution processing performed by the convolution layer, the upward solid arrows represent the transposed convolution upsampling processing performed by the transposed convolution layer, and the upward hollow arrows represent the interpolation upsampling processing performed by the interpolation module.

[0114] Through the above process, through the combination of the encoder network and the decoder network, not only the computational complexity of image segmentation is effectively reduced, which is beneficial to improving the segmentation efficiency, but also the accuracy of image segmentation is fully guaranteed, and the number of parameters of the network model can be effectively reduced, thereby improving the speed of image segmentation and target recognition.

[0115] exist Figure 3 and Figure 4 In the network structures shown in Figure 2, the convolutional layers can be Figure 5 The Inception model shown is the basic unit.

[0116] In a possible implementation, step S300 further includes: obtaining a recognition result for the image to be recognized according to the recognition result.

[0117] Therefore, by obtaining the recognition result of the original image based on the recognition result of the superpixel image, a more intuitive and precise recognition result can be obtained, which is convenient for subsequent analysis and verification.

[0118] In a specific example, the above steps S100-S300, that is, superpixel segmentation, calculation of depth information and deep neural network are connected in series to form an overall algorithm service, the camera intrinsic parameters of the binocular camera and the collected image to be identified are used as algorithm inputs, and the original image corresponding to the superpixel is output according to the segmentation result to output the target recognition result of the original image to be identified.

[0119] In some embodiments, the target recognition method provided in this embodiment further includes a step of training a deep neural network before executing step S300. The training step includes:

[0120] Obtaining a training sample with annotations, wherein the training sample is a superpixel image with annotated target edges;

[0121] A training set is constructed according to the training samples, and the deep neural network is obtained by training with the training set.

[0122] Through the above process, the trained deep neural network has the ability to perform image segmentation (or pixel classification) on the feature map corresponding to the input image to achieve target recognition. Then, after obtaining the collected image to be recognized, after the processing of steps S100 and S200, the obtained three-dimensional image data is input into the deep neural network, and the deep neural network can classify the pixels in the three-dimensional image data to achieve image segmentation, thereby achieving target recognition.

[0123] In a specific example, taking the application of this embodiment in an unmanned supermarket scenario to realize target recognition of commodities in the surveillance images of an unmanned supermarket as an example, this example takes five kinds of packaged beverages as examples for effect testing, including a first beverage in 500mL, a second beverage in 500mL, a second beverage in 330mL, a third beverage in 500mL and a fourth beverage in 500mL, and the annotated labels are 0-5 categories respectively. 200 targets are collected in each category (that is, there are several targets in each sample image, and there are a total of 200 targets of each category in all images to be identified) as training samples. All sample images are divided into training set, tuning set and test set in a ratio of 7:2:1, and the edge contour of each target is annotated. The resolution of a single sample image is 2592*1944.

[0124] For example, the training process is as follows: Cross-EntropyLoss is selected as the loss function of the deep neural network, and the sample images with the target contours and types marked are input for training. The training process uses the SGD+Momentum method for optimization, the initial learning rate is set to 0.1×(batch size / 256), and the cosine decay method is used to gradually reduce the learning rate.

[0125] Finally, the intersection over union (IoU) was used to measure the segmentation effect of the network model. On the test set, the IoUs of various targets were [0.93, 0.94, 0.91, 0.89, 0.89] respectively.

[0126] In summary, the target recognition method provided in this embodiment uses depth information as a new channel and uses a deep neural network to perform image segmentation (i.e., target edge extraction, or pixel classification operation) on three-dimensional image data to achieve target recognition, which can improve the recognition accuracy. In particular, it can accurately identify targets with similar appearances but different volumes. Moreover, due to the use of superpixel segmentation and interpolation and transposed convolution random splicing upsampling, it is possible to reduce the number of network model parameters and improve the recognition speed while ensuring the target recognition accuracy.

[0127] like Figure 6 As shown, another embodiment of the present application provides a target recognition system 600, including an image acquisition device 610 for acquiring an image to be recognized and a target recognition device 620, wherein the target recognition device 620 includes:

[0128] A calculation module 621 is used to calculate the depth information of the image to be identified, and perform super-pixel segmentation on the image to be identified to obtain a super-pixel image;

[0129] A generating module 622, configured to generate three-dimensional image data according to the image data of the superpixel image and the depth information;

[0130] The recognition module 623 is used to input the three-dimensional image data into a deep neural network for target recognition to obtain a recognition result.

[0131] The target recognition device 620 is an electronic device that provides background services for users, such as a personal computer, a server, etc. The background services include target recognition services.

[0132] Of course, according to actual operational needs, the target identification device 620 can be a server, a server cluster composed of multiple servers, or even a cloud computing center composed of multiple servers, so as to better provide backend services to a large number of users. This is not specifically limited here.

[0133] Furthermore, the target recognition device 620 deploys a deep neural network for target recognition.

[0134] A wireless or wired network connection is established between the target recognition device 620 and the image acquisition device 610 to achieve data transmission between the target recognition system 600 through the network connection. The network includes connection types such as wired, wireless communication links or optical fiber cables, etc. For example, this data transmission includes an image to be recognized, etc. Through the interaction between the image acquisition device 610 and the target recognition device 620, the image acquisition device 610 sends the collected image to be recognized to the target recognition device 620.

[0135] In some embodiments, the image acquisition device 610 is a binocular camera.

[0136] It should be noted that the principle and working process of the target recognition device 620 in the target recognition system 600 provided in this embodiment are similar to those of the target recognition method provided in the aforementioned embodiment. The relevant parts can be referred to the above description and will not be repeated here.

[0137] like Figure 7 As shown, a computer system suitable for implementing the target recognition device 620 in the target recognition system 600 provided in the above embodiment includes a central processing module (CPU), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage part into a random access memory (RAM). In the RAM, various programs and data required for the operation of the computer system are also stored. The CPU, ROM and RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0138] The following components are connected to the I / O interface: an input part including a keyboard, a mouse, etc.; an output part including a liquid crystal display (LCD), etc. and a speaker, etc.; a storage part including a hard disk, etc.; and a communication part including a network interface card such as a LAN card, a modem, etc. The communication part performs communication processing via a network such as the Internet. A drive is also connected to the I / O interface as needed. Removable media, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive as needed so that the computer program read therefrom is installed into the storage part as needed.

[0139] In particular, according to this embodiment, the process described in the flowchart above can be implemented as a computer software program. For example, this embodiment includes a computer program product, which includes a computer program tangibly contained on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium.

[0140] The flow chart and schematic diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product of the present embodiment. In this regard, each square box in the flow chart or schematic diagram can represent a part of a module, program segment or code, and a part of the above-mentioned module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the schematic diagram and / or flow chart, and the combination of the square boxes in the schematic diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0141] The modules involved in the description of this embodiment may be implemented by software or hardware. The modules described may also be arranged in a processor, for example, may be described as: a processor including a computing module, a generating module and an identifying module. The names of these modules do not constitute limitations on the modules themselves in certain circumstances. For example, the identifying module may also be described as a "target identifying module" or the like.

[0142] As another aspect, the present embodiment further provides a non-volatile computer storage medium, which may be the non-volatile computer storage medium included in the above-mentioned device in the above-mentioned embodiment, or may be a non-volatile computer storage medium that exists independently and is not installed in the terminal. The above-mentioned non-volatile computer storage medium stores one or more programs, and when the above-mentioned one or more programs are executed by a device, the above-mentioned device: calculates the depth information of the image to be identified, and performs super-pixel segmentation on the image to be identified to obtain a super-pixel image; generates three-dimensional image data based on the image data of the super-pixel image and the depth information; inputs the three-dimensional image data into a deep neural network for target recognition to obtain a recognition result.

[0143] It should be noted that, in the description of the present application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of further restrictions, the elements defined by the statement "comprise one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements.

[0144] Obviously, the above embodiments of the present application are merely examples for clearly illustrating the present application, and are not limitations on the implementation methods of the present application. For ordinary technicians in this field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present application are still within the scope of protection of the present application.

Claims

1. A target recognition method, characterized in that: include: Calculating the depth information of the image to be identified, and performing super-pixel segmentation on the image to be identified to obtain a super-pixel image; generating three-dimensional image data according to the image data of the superpixel image and the depth information; Inputting the three-dimensional image data into a deep neural network for target recognition to obtain a recognition result; The target recognition comprises: the deep neural network performs multiple downsampling on the input three-dimensional image data using multiple downsampling layers and then performs multiple upsampling using multiple upsampling layers; wherein the input of the first upsampling layer among the multiple upsampling layers is a feature map obtained by performing at least one convolution operation on the output of the last downsampling layer, and the input of other upsampling layers is a feature map formed by concatenating the feature map corresponding to the corresponding downsampling layer and the feature map output by the previous upsampling layer; Each of the multiple upsamplings includes performing interpolation upsampling on a portion of channels randomly selected from the feature map of the input upsampling layer and performing transposed convolution upsampling on another portion of the channels.

2. The method according to claim 1, characterized in that The super-pixel segmentation of the image to be identified includes: extracting texture features of pixel points of the image to be identified, and performing super-pixel segmentation on the image to be identified according to the texture features of the pixel points.

3. The method according to claim 1, characterized in that Each of the multiple upsamplings includes performing interpolation upsampling on half of the channels randomly selected from the feature map of the input upsampling layer and performing transposed convolution upsampling on the other half of the channels.

4. The method according to claim 1, characterized in that: The downsampling layer includes at least one convolution layer and one pooling layer; the first upsampling layer among the multiple upsampling layers includes an interpolation module and a transposed convolution layer, and the other upsampling layers include an interpolation module, a transposed convolution layer and at least one convolution layer.

5. The method according to claim 1, characterized in that The image to be identified is collected by a binocular camera, and calculating the depth information of the image to be identified includes: calculating the depth value of each coordinate point in the collected image to be identified according to the acquisition parameters of the binocular camera, normalizing the depth values ​​of all coordinate points, and obtaining the depth information of the image to be identified.

6. The method according to claim 1, characterized in that The method further includes: Obtaining a training sample with annotations, wherein the training sample is a superpixel image with annotated target edges; A training set is constructed according to the training samples, and the deep neural network is obtained by training with the training set.

7. A target recognition system, characterized in that: include: An image acquisition device, a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the image acquisition device is used to acquire an image to be identified, and when the processor executes the program, the following are achieved: calculating the depth information of the image to be identified, and performing super-pixel segmentation on the image to be identified to obtain a super-pixel image; generating three-dimensional image data according to the image data of the super-pixel image and the depth information; Inputting the three-dimensional image data into a deep neural network for target recognition to obtain a recognition result; The target recognition comprises: the deep neural network performs multiple downsampling on the input three-dimensional image data using multiple downsampling layers and then performs multiple upsampling using multiple upsampling layers; wherein the input of the first upsampling layer among the multiple upsampling layers is a feature map obtained by performing at least one convolution operation on the output of the last downsampling layer, and the input of other upsampling layers is a feature map formed by concatenating the feature map corresponding to the corresponding downsampling layer and the feature map output by the previous upsampling layer; Each of the multiple upsamplings includes performing interpolation upsampling on a portion of channels randomly selected from the feature map of the input upsampling layer and performing transposed convolution upsampling on another portion of the channels.

8. The system according to claim 7, characterized in that The image acquisition device is a binocular camera.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images