Target object recognition device, and method thereof

Abstract

To accurately perform target object recognition.SOLUTION: A target object recognition device recognizes a plurality of target objects from a plurality of static images. The target object recognition device comprises: a learning unit which forms color data and depth data from each image, and generates and learns a feature amount map with respect to all the color data and depth data; and a recognition unit which refers to a learning model constituted by learning using the color data and depth data, and recognizes the target object in the image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to multiple object recognition devices and methods. [Background technology]

[0002] For example, in Patent Document 1, an inference model generates labeled images with labels added on a pixel-by-pixel basis, so that each object can be recognized even if the input image data generated by the imaging unit contains multiple images each showing multiple objects. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-010970 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the system of Patent Document 1 is only suitable for cases where it is acceptable for the object to be deformed to some extent when picked, and is not suitable for cases where deformation or damage to the object is not permitted. [Means for solving the problem]

[0005] The invention according to a first aspect is an object recognition device that recognizes a plurality of objects from a plurality of still images, comprising: a learning unit that forms color data and depth data from each of the images and generates a feature map for all of the color data and depth data for learning; and a recognition unit that recognizes the objects in the images by referring to a learning model configured by learning using the color data and depth data, wherein the learning model is characterized in that, after acquiring the images, the color data and depth data from which background unnecessary for identifying the objects has been removed are input into a network structure that has previously been trained on recognition tasks for various objects, and the learning model is trained based on the features extracted from the network structure. [Effects of the Invention]

[0006] According to the present invention, object recognition is performed with high accuracy. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a schematic diagram showing a configuration of an object recognition device 20 according to an embodiment of the present invention. [Figure 2] 10A and 10B are schematic diagrams showing an example of video shooting according to the embodiment. [Figure 3] FIG. 2 is a schematic diagram showing an example of point cloud data according to the embodiment. [Figure 4] FIG. 2 is a schematic diagram showing an example of point cloud data according to the embodiment. [Figure 5] 10 is a schematic diagram showing an example of circumscribing rectangle data according to the embodiment; FIG. [Figure 6] 10A and 10B are schematic diagrams showing an example in which the influence of the image background according to the embodiment is reduced; [Figure 7] 10 is a graph showing an example of a Mahalanobis distance according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] (1) Configuration of the object recognition device

[0009] 1 is a schematic diagram showing the configuration of an object recognition device 20 according to this embodiment. For convenience, in the following description, the multiple objects 5 are assumed to be cooking oil, egg cartons, and milk cartons. However, the multiple objects 5 of the object recognition device 20 according to this embodiment are not limited to these and may be any objects.

[0010] The object recognition device 20 acquires color and depth from a plurality of still images of a plurality of objects 5 via the camera 15, and recognizes the objects from these images. The object recognition device 20 also transmits the recognition results to a user terminal device 30 operated by an observer. Such an object recognition device 20 can be realized by any computer. Here, the object recognition device 20 includes a storage unit 21, an input / output unit 22, a communication unit 23, and a processing unit 24. Note that the object recognition device 20 may also be realized as hardware using an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or the like.

[0011] The storage unit 21 stores color and depth information acquired from a plurality of still images showing the object 5, and is realized by any storage device such as a memory or a hard disk. Specifically, the storage unit 21 stores information for recognizing the object 5. For example, the recognition of the object 5 can be configured by a framework such as PaDiM using a network structure such as Wide-ResNet.

[0012] The input / output unit 22 is realized by a keyboard, a mouse, a touch panel, etc., and is used to input various information to the computer and output various information from the computer.

[0013] Furthermore, when the object 5 is recognized, the input / output unit 22 outputs its color and depth.

[0014] The communication unit 23 is realized by an arbitrary network card or the like, and enables communication with communication devices on a network via wire or wirelessly.

[0015] The processing unit 24 executes various information processes and is realized by a processor such as a CPU or a GPU (Graphics Processing Unit) and a memory. Here, the processing unit 24 functions as an image acquisition unit 24A, a recognition unit 24B, and a learning unit 24C by loading a program stored in the storage unit 21 into the CPU, GPU, etc. of the computer.

[0016] The image acquisition unit 24A acquires a plurality of still images of the object 5 via the camera 15 or the like, and stores the acquired colors and depths in the storage unit 21 as needed.

[0017] The recognition unit 24B recognizes the plurality of images of the object 5 that have been acquired.

[0018] The learning unit 24C calculates the average and variance of the feature amount and aspect ratio for recognizing the object 5 from a plurality of still images, and configures them as a profile.

[0019] The profile is constructed by deep learning using color data and depth data acquired from multiple still images of the object 5. In deep learning, images acquired from multiple still images of the object 5 placed on a plane, for example, are learned.

[0020] Deep learning is performed as follows: Color data and depth data are generated from multiple still images of a flat object. The image data is then divided and a feature map is generated for each image. The specific method for generating a feature map from each image data is to input the acquired image data into a network structure that has previously trained recognition models for various objects, and then use the feature maps extracted from the network structure. Note that Wide-ResNet or similar can be used as the network structure.

[0021] (2) Operation of the object recognition device The operation of the object recognition device 20 according to this embodiment will be described with reference to FIGS.

[0022] The object recognition device 20 captures multiple still images of multiple objects 5 using a camera 15 to form image data. At this time, color and depth information is obtained using, for example, a depth camera, as shown in Figure 2. Depth information is obtained because color information alone can only determine the position on the image, and the position of the object 5 in three-dimensional space cannot be determined.

[0023] Next, as shown in Figure 3, only the point clouds where the object 5 may exist are retained, using the distance from the reference plane as a threshold. Here, the reference plane is assumed to be a plane on which the object 5 is placed, such as a conveyor belt. Then, 3D point cloud clustering is performed, and, as shown in Figure 4, a color image and point cloud data of, for example, a milk carton are extracted.

[0024] Next, as shown in Figure 5, the circumscribing rectangles for the 2D image projection images corresponding to the point cloud clusters of each object are calculated, and each is cropped to obtain a vertically long image. For the depth map, the brightness of the valid area with depth information is normalized to 93-255, and then a 9x9 median filter is applied to fill in missing pixels. At this stage, the top and bottom of the object and the class to which it belongs (in this case, cooking oil, egg carton, or milk carton) are unknown, but they are cropped as objects.

[0025] Furthermore, as shown in Figure 6, to reduce the influence of background colors other than the milk carton, for example, a convex polygon containing the contour of the depth map is found and the background is roughly removed. During training, the image is masked with a random grayscale of brightness 32-223, and during object estimation, which involves comparing various objects with profiles, the image is masked with gray of brightness 128. Then, by combining this image with the color image, a milk carton color image is created, with the background unnecessary for object identification removed. The reason for removing the background in this way is to reduce the influence of the background, even when the shooting conditions are different. Now we have a color image and a depth map.

[0026] The entire image is cut out for color and depth independently, and the position is changed to cut out the top, middle, and bottom halves. The reason for cutting out the image for color and depth independently is that the discrimination ability is reduced when only color information is used, but it is expected that ensemble learning will be possible when images with both color and depth are used. The reason for cropping the image into the top, middle, and bottom halves is that spatial information would be lost in the subsequent global average pooling.

[0027] Next, these images are normalized to the input images of ResNet 18. Since the trained ResNet 18 treats the input images as squares, they are normalized to squares. Then, the normalized images are input to ResNet 18.

[0028] The output of global average pooling of the trained ResNet18 is used as a feature vector, and 512x8 features are output. A 512-dimensional vector is obtained for one image. For example, a square image and top, middle, and bottom images, along with their depth maps, are obtained, so the object reflected in the separated image area will be square, whitish, or bluish.

[0029] For each detected class, the average and deviation of the features are obtained by rotating the image 180° up and down. For data augmentation during training, changes are made to the hue, brightness, saturation, and contrast of the color information, and only contrast is changed for the depth information. Features with a large average / deviation are expected to be effective in identifying that class.

[0030] The Mahalanobis distance is evaluated for each detection class. The class to which the object belongs is estimated to be one whose distance is close to the average value of the detection class. For example, looking at the graph in Figure 7, cooking oil is between 40 and 60, and the solid and dotted lines are close to each other. The solid line is the Mahalanobis distance before data augmentation, and the dotted line is the Mahalanobis distance after data augmentation. Data augmentation is performed to improve robustness in a variety of shooting environments. Here, when predicting the class of an image (query image) that shows cooking oil, it is expected that the Mahalanobis distance to the cooking oil profile will be somewhere between 40 and 60 within the distribution range of the dotted line, and the Mahalanobis distance to the profiles of other classes will be greater than that value, at 70 or more. As a result, the class with the closest Mahalanobis distance is cooking oil, and therefore the class of the query image will be predicted to be cooking oil.

[0031] The following is a description of the detection class identification flow. (a) The aspect ratio of each image is evaluated using log2, compared with the aspect ratios of the registered classes, and classes within the range are extracted. (b) The Mahalanobis distance is evaluated for the filtered classes, and those below a threshold based on the quantile of the Mahalanobis distance distribution are extracted. (c) The class with the smallest Mahalanobis distance is output as the matching class. (d) Obtain a bounding rectangle that is normalized in the forward direction as seen from the camera. If it matches the backward direction class, rotate the bounding rectangle 180 degrees and normalize it in the forward direction.

[0032] (3) Effects According to this embodiment, objects that already have profiles are simply compared, and the neural network itself does not need to be trained. Only when a new object is introduced is the trained neural network model treated as a feature extractor, features are obtained for each image, and a profile of the average value and deviation is saved. For a new object, features are obtained from the training model, and their average value and deviation are output. Therefore, the parameters of the neural network model itself do not need to be updated.

[0033] By adding depth information, robustness to shooting conditions can be improved compared to using only color information, which is easily changed by light sources, etc.

[0034] <Other embodiments> The present disclosure is not limited to the above-described embodiments as they are. The present disclosure can be embodied by modifying the components within the scope of the gist of the disclosure in the implementation stage. Furthermore, the present disclosure can be formed into various disclosures by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be deleted from all the components shown in the embodiments. Furthermore, the components may be appropriately combined in different embodiments. [Explanation of symbols]

[0035] 5 Multiple Objects 15 Camera 20 Object Recognition Device 21 Memory section 22 Input / output section 23 Communications Department 24 Processing section 24A Image acquisition unit 24B Recognition part 24C Learning Department

Claims

1. An object recognition device that recognizes a plurality of objects from a plurality of still images, a learning unit that generates color data and depth data from each of the images, and generates and learns a feature map for all of the color data and depth data; a recognition unit that recognizes the object in the image by referring to a learning model configured by learning using the color data and the depth data; Equipped with The learning model is an object recognition device characterized in that after acquiring the image, the color data and depth data from which background unnecessary for identifying the object has been removed are input into a network structure that has been previously trained to recognize various objects, and the learning model is learned based on the features extracted from the network structure.

2. The object recognition device according to claim 1 , wherein the recognition unit compares the learned profiles of various objects to infer the object.

3. 2. The object recognition device according to claim 1, wherein after removing background that is unnecessary for identifying the object, the entire image is cut out independently of color and depth, and the upper, middle and lower halves are cut out by changing the position.

4. 4. The object recognition device according to claim 1, wherein the image data is a point cloud cluster of each object, a circumscribing rectangle is calculated for the corresponding two-dimensional image projection image, and each is cut out to obtain a vertically long image.

5. 5. The object recognition apparatus according to claim 4, wherein after the circumscribing rectangle is determined, a median filter is applied to fill in missing pixels.

6. 1. An object recognition method for recognizing a plurality of objects from a plurality of still images, generating color data and depth data from the image, and generating and learning feature maps for all of the color data and depth data; Recognizing the object in the image by referring to a learning model configured by learning using the color data and depth data; Equipped with The object recognition method is characterized in that the learning model is input into a network structure that has been previously trained to recognize various objects, and after acquiring the image, the color data and depth data from which background that is unnecessary for identifying the object has been removed are input, and the learning model is trained based on the features extracted from the network structure.

7. 7. The object recognition method according to claim 6, wherein the step of recognizing the object comprises comparing the object with the learned profiles of various objects to infer the object.

Images