method

Abstract

To provide a method for improving a technique for performing machine learning on the basis of an image or the like of an object.SOLUTION: The method includes acquiring training data, and training, using the acquired training data, an object detecting AI to output attribution information of an object included in an RGBD image or a depth image when the RGBD image or the depth image is input. The learning data includes an RGBD image or a depth image, and a correct label indicating attribute information of a target object included in the RGBD image or the depth image.SELECTED DRAWING: Figure 2B

Description

[Technical Field]

[0001] The present disclosure relates to computer-implemented methods. [Background technology]

[0002] Conventionally, there are known techniques for performing machine learning based on images of an object, etc. For example, Patent Document 1 discloses a technique for annotating an image of an object based on the difference between data relating to the depth of the object at a first time point and the depth of the object at a second time point measured by a depth sensor. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-058160 Summary of the Invention [Problem to be solved by the invention]

[0004] The above method automatically annotates an RGB image of an object by using the difference in depth between the first and second time points. That is, the RGB image annotated by the above method can be used, for example, as learning data for training an object detection AI that detects objects contained in the input RGB image. However, when RGB images of multiple objects photographed from above with an RGB camera are used as input to an object detection AI, the RGB image does not include depth (i.e., distance from the RGB camera). Therefore, when objects overlap vertically, there are issues such as reduced detection accuracy for lower objects or the inability to distinguish between lower and upper objects. Therefore, there is room for improvement in technology for performing machine learning based on object images, etc.

[0005] In view of the above circumstances, an object of the present disclosure is to improve the technology for performing machine learning based on images of objects, etc. [Means for solving the problem]

[0006] According to one embodiment of the present disclosure, a method comprises: 1. A computer-implemented method comprising: obtaining training data; training an object detection AI using the acquired learning data so that when an RGBD image or a Depth image is input, the AI ​​outputs attribute information of an object included in the RGBD image or the Depth image; Including, The training data includes an RGBD image or a depth image, and a correct label indicating the attribute information of an object included in the RGBD image or the depth image. [Effects of the Invention]

[0007] According to one embodiment of the present disclosure, a technique for performing machine learning based on an image of an object, etc., is improved. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating a schematic configuration of a computer according to an embodiment of the present disclosure. [Figure 2A] 10 is a flowchart illustrating an example of operation of a learning phase of a computer according to an embodiment of the present disclosure. [Figure 2B] 10 is a flowchart illustrating an example of operation of an inference phase of a computer according to an embodiment of the present disclosure. [Figure 2C] 10 is a flowchart illustrating an example of operation of a computer in an image creation and image output phase according to an embodiment of the present disclosure. [Figure 3] FIG. 10 is a diagram illustrating an example of learning data of a nail according to an embodiment of the present disclosure. [Figure 4] FIG. 10 is a diagram illustrating an example of learning data for individual packages according to an embodiment of the present disclosure. [Figure 5] FIG. 10 is a diagram illustrating an example of learning data for cardboard according to an embodiment of the present disclosure. [Figure 6] FIG. 10 is a diagram illustrating a display example according to an embodiment of the present disclosure. [Figure 7] FIG. 10 is a diagram illustrating a display example according to an embodiment of the present disclosure. [Figure 8] FIG. 10 is a diagram illustrating a display example according to an embodiment of the present disclosure. [Figure 9] FIG. 10 is a diagram illustrating a display example according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described.

[0010] (Outline of the embodiment) An overview of a computer according to an embodiment of the present disclosure will be described with reference to Fig. 1. A computer 10 includes a communication unit 11, an output unit 12, an input unit 13, a storage unit 14, and a control unit 15.

[0011] In the learning phase, the computer 10 acquires training data and uses the acquired training data to train the object detection AI so that, when an RGBD image is input, the AI ​​outputs attribute information of the object contained in the RGBD image. Here, the training data includes the RGBD image and a correct answer label indicating the attribute information of the object contained in the RGBD image. In the inference phase, the computer 10 inputs the RGBD image to the trained object detection AI and acquires the attribute information of the object output from the object detection AI. Then, the computer 10 creates and outputs an RGB image to be provided to the user based on the acquired attribute information of the object.

[0012] First, an outline of this embodiment will be described, and details will be described later. A computer 10 executes a step of acquiring training data and a step of training an object detection AI using the acquired training data so that when an RGBD image is input, the AI ​​outputs attribute information of an object included in the RGBD image. Here, the training data includes an RGBD image and a ground truth label indicating attribute information of the object included in the RGBD image.

[0013] As described above, according to this embodiment, the training data includes RGBD images of objects photographed from above and ground truth labels indicating attribute information of the objects contained in the RGBD images. Therefore, even when overlapping objects obscure a portion of the lower object, causing the lower object to appear partially missing in the RGB image, the object detection AI can be trained to detect overlapping objects based on a D value indicating the distance from the camera. Furthermore, even when overlapping objects of similar colors cause the upper and lower objects in the RGB image to appear as a single object, the object detection AI can be trained to detect overlapping objects based on a D value indicating the distance from the camera. Therefore, detecting overlapping objects improves the machine learning technology for object detection AI by enabling the accurate detection of the shape and quantity of objects.

[0014] Next, each component of the computer 10 will be described in detail.

[0015] (Computer Configuration) As shown in FIG. 1, the computer 10 includes a communication unit 11, an output unit 12, an input unit 13, a storage unit 14, and a control unit 15.

[0016] The communication unit 11 includes an interface for communicating with an external device, and the interface may be compatible with any wired or wireless communication standard.

[0017] The output unit 12 includes one or more output devices that output information. The output devices are, for example, but not limited to, a display or a speaker. Alternatively, the output unit 12 may include an interface for connecting an external output device.

[0018] The input unit 13 includes one or more input devices that detect user input. Examples of the input devices include, but are not limited to, physical keys, capacitive keys, pointing devices such as a mouse, a touch screen integrated with the display of the output unit 12, or a microphone that accepts voice input. Alternatively, the input unit 13 may include an input interface that detects user input via an external input device.

[0019] The storage unit 14 includes one or more memories. The memories may be, but are not limited to, semiconductor memories, magnetic memories, or optical memories. Each memory included in the storage unit 14 may function as, for example, a main memory device, an auxiliary memory device, or a cache memory. The storage unit 14 stores any information used in the operation of the computer 10. For example, the storage unit 14 may store system programs, application programs, embedded software, and the like.

[0020] In this embodiment, the storage unit 14 stores an object detection AI that detects objects included in an input image. The object detection AI may be, for example, Faster R-CNN or Mask R-CNN, or any anchor box-based object detection model. In addition, in this embodiment, in the inference phase, the storage unit 14 stores attribute information of objects detected by the object detection AI and the number of objects detected (count number).

[0021] The control unit 15 includes one or more processors, one or more programmable circuits, one or more dedicated circuits, or a combination thereof. The processor may be, for example, a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for a specific process, but is not limited to these. The programmable circuit may be, for example, an FPGA (Field-Programmable Gate Array), but is not limited to this. The dedicated circuit may be, for example, an ASIC (Application Specific Integrated Circuit), but is not limited to this. Hereinafter, the processor, programmable circuit, and dedicated circuit will be referred to as "processor, etc." without any particular distinction being made between them. The control unit 15 controls the overall operation of the computer 10.

[0022] (Operation flow of computer 10) 2A, the operation flow of the computer 10 according to this embodiment in the learning phase will be described. In summary, the operation of the computer 10 in the learning phase is to acquire training RGBD images and train the object detection AI. Training the object detection AI employs supervised learning, in which multiple training RGBD images, each of which is assigned a correct label indicating an object, serve as training data.

[0023] Step S100: The control unit 15 acquires learning data.

[0024] The control unit 15 acquires, via the communication unit 11, a training RGBD image to which a correct answer label has been previously assigned by annotation. The RGBD image is an image in which a D value is added to a still image (RGB image) of an object captured from above by the image capture device 20. The D value is a numerical value for each pixel of a depth image that represents depth. That is, the RGBD image may be an image having four independent parameters per pixel. In this embodiment, the D value is determined based on the image capture device. That is, the greater the distance from the image capture device to the object, the larger the D value. Also, the closer the distance from the image capture device to the object, the smaller the D value. Note that, based on the object, the greater the distance from the image capture device to the object, the smaller the D value may be, and the closer the distance from the image capture device to the object, the larger the D value may be. Any method can be used for annotation and annotation. For example, the annotated training RGBD image may be obtained by manual annotation or automatic annotation. The annotation method may be keypoint annotation, segmentation annotation, or bounding box annotation. The bounding box may be, for example, a rectangular area defined by a combination of center coordinates, a length in the x-axis direction (number of pixels), and a length in the y-axis direction (number of pixels). The depth image may be acquired by ToF or may be obtained from an RGB image by monocular depth estimation. When an RGB image capturing device and a depth image capturing device are separately installed, they are installed at approximately the same position above the object so that the size and number of pixels of the RGB image and the depth image, as well as the position and size of the object in the RGB image and the depth image, are the same. Examples of objects used as training data will be described later.

[0025] Step S101: The control unit 15 uses the acquired learning data to train the object detection AI so that when an RGBD image is input, the AI ​​outputs attribute information of an object included in the RGBD image.

[0026] The control unit 15 trains the object detection AI stored in the storage unit 14 based on the learning data. As a result, when an RGBD image is input to the trained object detection AI via the communication unit 11, the control unit 15 can acquire attribute information of the object from the object detection AI. Here, the attribute information of the object includes information indicating an area corresponding to the object on the RGBD image and information indicating one of multiple classes corresponding to the D value within the area of ​​the RGBD image. The attribute information may also include any information used to train the object detection AI. For example, when an object detection AI that also estimates the name or type of the detected object is employed, the correct label may further include information indicating the name or type of the object.

[0027] FIG. 3 shows an example of training data in which a nail 30 is used as an object. FIG. 3 illustrates a state in which multiple nails 30 are randomly placed inside a box. FIG. 3 also illustrates a state in which the object is being photographed by a camera 20 installed above the object. It also illustrates examples of RGB images and depth images acquired by the control unit 15 via the communication unit 11. In the RGB and depth images, the box is omitted and only the nails 30 are depicted. In this example, some nails 30 overlap each other. Therefore, with only the RGB image, the nails 30 in the lower row may be learned as two separated nails. However, with the depth image, it is possible to detect changes in the D value before and after the point where the nails 30 overlap each other. In the example of FIG. 3, the nails 30 in the depth image that overlap each other, i.e., the portions close to the camera 20, are white, while the portions of the nails 30 that are in contact with the bottom of the box, i.e., the portions far from the camera 20, are hatched. As described above, by training an object detection AI based on an RGBD image, it becomes possible to detect overlaps between nails 30 in the input RGBD image and the exact shapes of the nails 30.

[0028] FIG. 4 shows an example of learning data in which individual packages are used as the target object. FIG. 4 shows two layers of individually packaged objects stacked inside a box: an upper layer (hereinafter referred to as the second class) and a lower layer (hereinafter referred to as the first class). FIG. 4 also shows the object being photographed by a photographing device 20 installed above the object. It also shows examples of RGB images and depth images acquired by the control unit 15 via the communication unit 11. In the RGB and depth images, the box is omitted and only the individual packages are depicted. In this example, a first-class individual package 41 is visible through a gap between a second-class individual package 42. In the RGB image, the individual packages have the same shape and appearance. Therefore, it is not easy to determine and learn whether each individual package is a first-class individual package 41 or a second-class individual package 42 from the RGB image alone. However, the depth image allows detection of different D values ​​between the first-class individual package 41 and the second-class individual package 42. In the example of Figure 4, the individual packages in the Depth image are white for second-class individual packages 42, i.e., those close to the camera 20, and hatched for first-class individual packages 41, i.e., those far from the camera 20. As described above, by training the object detection AI based on the RGBD image, it is possible to detect whether each individual package in the input RGBD image is a first-class individual package 41 or a second-class individual package 42.

[0029] FIG. 5 shows an example of training data in which cardboard boxes are used as the object. FIG. 5 shows a state in which cardboard boxes 51 of the first class and cardboard boxes 52 of the second class are stacked two high on a pallet or the like. FIG. 5 also shows a state in which an image is captured by a camera 20 installed above the object. It also shows examples of RGB images and depth images acquired by the control unit 15 via the communication unit 11. In this example, eight cardboard boxes 51 of the first class are stacked, and four cardboard boxes 52 of the second class are stacked on top of them. In the RGB image, the cardboard boxes have the same shape and appearance. Therefore, it is not easy to determine and learn whether each cardboard box is a cardboard box 51 of the first class or a cardboard box 52 of the second class from the RGB image alone. However, in the depth image, different D values ​​can be detected for the cardboard boxes 51 of the first class and the cardboard boxes 52 of the second class. In the example of Figure 5, the cardboard boxes in the Depth image are classified as second class cardboard boxes 52, i.e., cardboard boxes that are close to the camera 20, and are white, while the first class cardboard boxes 51, i.e., cardboard boxes that are far from the camera 20, are hatched. As described above, by training an object detection AI based on an RGBD image, it becomes possible to detect whether each cardboard box in the input RGBD image is a first class cardboard box 51 or a second class cardboard box 52.

[0030] Next, the operation flow of the computer 10 according to this embodiment in the inference phase will be described with reference to Fig. 2B. In summary, the operation of the computer 10 in the inference phase is to cause the object detection AI to detect an object based on an RGBD image input via the communication unit 11 and acquire attribute information of the object.

[0031] Step S200: Input the RGBD image into the trained object detection AI.

[0032] In the inference phase, the RGB image and the RGBD image are assumed to be moving images. Specifically, the image capture device 20 begins capturing the RGB image. The control unit 15 of the computer 10 adds a D value to the RGB video acquired from the image capture device 20 via the communication unit 11 to acquire the RGBD image. The depth image may be acquired by ToF or may be obtained from the RGB image by monocular depth estimation. When the image capture device for the RGB image and the image capture device for the depth image are installed, they are installed at approximately the same position above the object so that the size and number of pixels of the RGB image and the depth image, as well as the position and size of the object in the RGB image and the depth image, are the same. The control unit 15 inputs the RGBD image to the object detection AI. Specifically, the control unit 15 inputs the latest frame of the moving RGBD image, i.e., the latest still image constituting the moving image, to the object detection AI.

[0033] Step S201: The control unit 15 checks whether an object included in the RGBD image has been detected by a trained object detection AI.

[0034] The control unit 15 checks whether the object has been detected by the object detection AI. If not, the process returns to step S200, and the control unit 15 again performs the operation of inputting the RGBD image into the trained object detection AI.

[0035] Step S202: The control unit 15 acquires attribute information of the object that is output from the trained object detection AI every time the trained object detection AI detects an object included in the RGBD image.

[0036] Each time an object included in the RGBD image is detected by the trained object detection AI, the control unit 15 acquires attribute information of the object output from the object detection AI and stores it in the storage unit 14. As described above, the attribute information of the object includes information indicating the region corresponding to the object in the RGBD image and information indicating one of multiple classes corresponding to the D value within the region of the RGBD image. The attribute information may further include any information used to train the object detection AI. For example, if an object detection AI that also estimates the name or type of the detected object is employed, the correct label may further include information indicating the name or type of the object. This allows the control unit 15 to cause the object detection AI to detect objects that have the same shape and appearance as objects belonging to a class corresponding to the D value.

[0037] Next, the operation flow of the computer 10 according to this embodiment in the image creation and image output phase will be described with reference to Fig. 2C. In summary, the operation of the computer 10 in the image creation and image output phase is an operation in which the control unit 15 creates an RGB image to be provided to the user based on the attribute information of the object stored in the storage unit 14 in the inference phase, and outputs the RGB image via the output unit 12.

[0038] Step S300: The control unit 15 checks whether a first class object has been detected by the object detection AI.

[0039] The control unit 15 checks whether the object detection AI has detected a first-class object on the RGBD image input from the image capture device 20 via the communication unit 11. Step S300 is repeatedly executed until the object detection AI detects a first-class object. When the object detection AI detects a first-class object, the process proceeds to the next step S301.

[0040] Step S301: The control unit 15 checks whether a second class object has been detected by the object detection AI.

[0041] The control unit 15 checks whether an object of the second class has been detected by the object detection AI on the RGBD image input from the image capturing device 20 via the communication unit 11.

[0042] Step S302: Each time a first class object with the largest D value is detected, the control unit 15 superimposes at least one of a first marker indicating the detected first class object (object marker 70 indicating the first class object) and a first count number of first class objects (number of first class object detections) on the RGB image that was the basis of the RGBD image input to the trained object detection AI.

[0043] The control unit 15 executes step S302 when the object detection AI detects first-class objects but not second-class objects in the RGBD image input from the camera device 20 via the communication unit 11. FIG. 6 illustrates an example of step S302. The example in FIG. 6 shows eight first-class cardboard boxes placed on a pallet. The example in FIG. 6 also shows the cardboard boxes being photographed by the camera device 20 installed above. The control unit 15 superimposes object markers 70 indicating first-class objects and the number of detected first-class cardboard boxes on the RGB image photographed by the camera device 20. The control unit 15 then provides a display image 60 to the user via the output unit 12. This improves the efficiency of tasks such as placing eight cardboard boxes one by one on a pallet. In other words, the user can quickly recognize the areas where first-class cardboard boxes exist and the number of first-class cardboard boxes during the task, thereby improving work efficiency. 6, the control unit 15 displays the object markers 70 and the number of detected cardboard boxes of the first class in a superimposed manner, but it is also possible to display only one of them in a superimposed manner. The control unit 15 can provide an image to the user via the output unit 12 by any method, such as a display.

[0044] Step S303: When an object of the second class, which has the next largest D value after the first class, is detected, the control unit 15 hides the first marker (object marker 70 indicating the object of the first class) and the first count number (the number of objects detected in the first class) superimposed on the RGB image.

[0045] The control unit 15 executes step S303 when the object detection AI detects a first-class object and a second-class object in the RGBD image input from the image capture device 20 via the communication unit 11. FIG. 7 illustrates an example of steps S303 and S304. The example in FIG. 7 shows eight first-class cardboard boxes placed on a pallet, with one second-class cardboard box placed on top of them. The example in FIG. 7 also illustrates a state in which an image is captured by the image capture device 20 installed above the cardboard boxes. In step S303, the control unit 15 hides the object markers 70 and the number of first-class detections superimposed on the first-class objects in the RGB image in step S302 and provides them to the user via the output unit 12. This improves efficiency in tasks such as placing eight cardboard boxes one by one on a pallet, and then placing another cardboard box on top of the eight cardboard boxes. In other words, the user does not need to pay attention to the first class object for which the task has been completed, which has the effect of improving work efficiency and reducing the user's workload. Note that the method by which the control unit 15 provides the image to the user via the output unit 12 can be any method, such as a display.

[0046] Step S304: Each time an object of the second class among the multiple classes is detected, the control unit 15 superimposes on the RGB image at least one of a second marker indicating the detected object of the second class (object marker 71 indicating the object of the second class) and a second count number of objects of the second class (number of objects detected of the second class).

[0047] As in step S303, the control unit 15 executes step S304 when the object detection AI detects a first-class object and a second-class object in the RGBD image input from the camera device 20 via the communication unit 11. FIG. 7 illustrates an example of step S303 and step S304. The example in FIG. 7 shows eight first-class cardboard boxes placed on a pallet, with one second-class cardboard box placed on top of them. The example in FIG. 7 also shows a state in which the image is captured by the camera device 20 installed above the cardboard boxes. In step S304, the control unit 15 creates a display image 60 in which object markers 71 indicating second-class objects and the number of detected second-class cardboard boxes are superimposed on the RGB image, and provides the display image 60 to the user via the output unit 12. This improves efficiency in tasks such as placing eight cardboard boxes one by one on a pallet, and then placing another cardboard box one by one on top of the eight cardboard boxes. That is, the user can quickly recognize the area where the second-class cardboard boxes are present and the number of second-class cardboard boxes during the work, which has the effect of improving work efficiency. Note that in FIG. 7, the control unit 15 superimposes object markers 71 indicating the second-class objects and the number of detected second-class cardboard boxes on the RGB image, but it is also possible to superimpose only one of them. Furthermore, the control unit 15 can use any method, such as a display, to provide the image to the user via the output unit 12.

[0048] Step S305: When the second marker or the second count number is superimposed on the RGB image, the control unit 15 superimposes a third marker (non-display marker 72) on at least a portion of the area on the RGB image where no objects of the second class exist.

[0049] As in steps S303 and S304, the control unit 15 executes step S305 when the object detection AI detects a first-class object and a second-class object in the RGBD image input from the image capture device 20 via the communication unit 11. FIG. 8 illustrates an example of step S305. The example in FIG. 8 shows eight first-class cardboard boxes placed on a pallet, with one second-class cardboard box placed on top of them. The example in FIG. 8 also illustrates a state in which an image is captured by the image capture device 20 installed above the cardboard boxes. In step S305, the control unit 15 creates a display image 60 in which non-display markers 72 are superimposed to hide or obscure areas other than the second-class object, and provides the display image 60 to the user via the output unit 12. This improves efficiency in tasks such as placing eight cardboard boxes one by one on a pallet, and then placing another cardboard box one by one on top of the eight cardboard boxes to complete the entire layout. In other words, the user does not need to pay attention to the first-class cardboard boxes for which the work has been completed, which has the effect of improving work efficiency and reducing the user's workload. Note that in FIG. 8, no hidden marker is superimposed on the palette portion, but the hidden marker may be superimposed on the palette portion as well. Also, in FIG. 8, the object marker 71 indicating the second-class object and the number of detected cardboard boxes of the second class are superimposed, but they do not need to be superimposed at the same time. That is, in step S305, either the object marker 71 indicating the second-class object or the number of detected cardboard boxes of the second class may be superimposed, or neither may be displayed. Furthermore, the control unit 15 may provide an image to the user via the output unit 12 using any method, such as a display.

[0050] Step S306: The control unit 15 superimposes and displays the total number of detected objects for each class on the RGB image.

[0051] The control unit 15 executes step S306 when the object detection AI detects a first-class object in the RGBD image input from the image capture device 20 via the communication unit 11. FIG. 9 illustrates an example of step S306. The example in FIG. 9 shows eight first-class cardboard boxes placed on a pallet, with one second-class cardboard box placed on top of them. The example in FIG. 9 also illustrates a state in which an image is captured by the image capture device 20 installed above the cardboard boxes. The control unit 15 counts the objects for each class stored in the memory unit 14, creates a display image 60 in which the total number of objects for each class is superimposed, and provides the display image 60 to the user via the output unit 12. This improves efficiency in tasks such as placing eight cardboard boxes one by one on a pallet, and then placing another cardboard box on top of the eight cardboard boxes. In other words, the user does not need to remember the number of first class cardboard boxes that have been completed and the number of second class cardboard boxes that are currently being worked on, which has the effect of improving work efficiency and reducing the workload on the user. Note that the method by which the control unit 15 provides the display image 60 to the user via the output unit 12 can be any method, such as a display.

[0052] As described above, the computer 10 according to this embodiment includes a step of acquiring training data and a step of training an object detection AI using the acquired training data so that when an RGBD image is input, the AI ​​outputs attribute information of an object included in the RGBD image. Here, the training data includes the RGBD image and a ground truth label indicating attribute information of the object included in the RGBD image.

[0053] With this configuration, the training data includes RGBD images of objects photographed from above and ground truth labels indicating attribute information of the objects contained in the RGBD images. Therefore, even when overlapping objects obscure a portion of the lower object, making it appear missing in the RGB image, it is possible to train an object detection AI that can detect overlapping objects based on the D value indicating the distance from the camera. Furthermore, even when overlapping objects of similar colors cause the upper and lower objects in the RGB image to appear as a single object, it is possible to train an object detection AI that can detect overlapping objects based on the D value indicating the distance from the camera. Therefore, detecting overlapping objects improves the machine learning technology for object detection AI by enabling accurate detection of the shape and quantity of objects.

[0054] Although the present disclosure has been described based on the drawings and examples, it should be noted that those skilled in the art may make various modifications and alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are included in the scope of the present disclosure. For example, the functions included in each component or step can be rearranged so as not to be logically inconsistent, and multiple components or steps can be combined, divided, or omitted.

[0055] In the above-described embodiment, the input image to be input to the trained object detection AI is described as being acquired from an external device, such as an image capture device, via the communication unit 11 of the computer 10. However, an embodiment in which the computer 10 includes an image capture device is also possible.

[0056] In the above-described embodiment, the control unit 15 performs steps S302 and S306 or all of steps S303 to S306. Here, some of the steps of superimposing object information on the RGB image in steps S302 to S306 may be omitted based on the user's work, etc. For example, while steps S303 and S304 are performed simultaneously in FIG. 7, only one of them may be performed. Furthermore, while steps S304 and S305 are performed simultaneously in FIG. 8, only one of them may be performed. Furthermore, although only step S306 is performed in FIG. 9, step S305 may be performed simultaneously to superimpose a non-display marker on objects other than those in the second class.

[0057] In the above-described embodiment, the control unit 15 superimposes information about predetermined objects on the RGB image in steps S302 to S306. An embodiment is also possible in which the user can select the information to be superimposed on the RGB image. For example, only information about objects of a specific class may be superimposed. In this case, when executing step S300, the control unit 15 may display a GUI (Graphical User Interface) on the display of the output unit 12, allowing the user to select the content to be superimposed. For example, the user can operate the GUI to select the class of objects to be superimposed, or select one of two items, markers and the number of detections, to be superimposed. Subsequently, the control unit 15 may identify the display content selected by the user upon detecting a user operation. Then, in step S300, the control unit 15 may superimpose the content selected by the user on the RGB image. Note that if there is no object to be superimposed, the control unit 15 may store the user's selection in the storage unit 14. Then, when the target object is detected, the control unit 15 may superimpose the content selected by the user on the RGB image.

[0058] In the above-described embodiment, examples of nails, individual packages, or cardboard boxes of the same type have been described. However, different types of objects may be mixed. For example, nails of different colors or lengths, individual packages of the same shape but with different packaging appearances, and cardboard boxes of different shapes may be mixed. In such a case, in step S100, the control unit 15 acquires learning data annotated with attribute information of the objects to be detected. In addition, in step S101, the control unit 15 trains the object detection AI. This enables the object detection AI to detect the objects the user wants to detect in the inference phase.

[0059] In the above-described embodiment, the case where there are first-class objects and second-class objects, i.e., two-tiered objects, has been described. However, an embodiment in which the objects are stacked in three or more tiers is also possible. For example, if there are three tiers of objects, in step S100, learning data of RGBD images with a correct answer label indicating that the object is a third-class object is acquired as attribute information of the object, and in step S101, the object detection AI is trained. This makes it possible to detect third-class objects in the inference phase. Furthermore, the image creation and image output phases can be realized by using the object class as a variable. Specifically, after detecting a second-class object, in step S300, it is confirmed whether a second-class object has been detected, and in step S301, it is confirmed whether a third-class object has been detected. In steps S302 to S305, the first-class object is treated as the second object, and the second-class object is treated as the third-class object. This enables the control unit 15 to create and output images of third-class objects as well.

[0060] In the above-described embodiment, an example has been described in which an image is captured by the image capturing device 20 installed above the object. However, an RGBD image captured by an image capturing device 20 installed to the side of the object may also be used. Such an embodiment can improve the efficiency of, for example, the work of sequentially loading cargo into a truck container from the back to the front.

[0061] Furthermore, in this embodiment, the RGBD image has been described as an image having four independent parameters per pixel, but it may also be an image having three parameters obtained by correcting an RGB image using a D value. For example, an image in which at least one value of R, G, and B is corrected depending on the amount of D value may be adopted as the RGBD image.

[0062] In addition, in this embodiment, an example has been described in which an object detection AI is trained so that, when an RGBD image is input, the AI ​​outputs attribute information of an object contained in the RGBD image. However, an embodiment in which a depth image is used instead of an RGBD image is also possible. In such a case, in step S302 described above, each time an object of the first class having the largest D value is detected, the control unit 15 superimposes at least one of a first marker and a first count number on the RGBD image or an RGB image corresponding to the depth image input to the trained object detection AI. Here, the "RGB image corresponding to the RGBD image" refers to the RGB image that is the basis of the RGBD image. Furthermore, the "RGB image corresponding to the depth image" refers to an RGB image captured within the same range as the depth image. [Explanation of symbols]

[0063] 10. Computers 11 Communications Department 12 Output section 13 Input section 14 Storage section 15 Control Unit 20 Imaging equipment 30 Nails of the object 41 First Class Individually Wrapped 42 Second class individually wrapped 51 First Class Cardboard 52 second-class cardboard 60 display images 70 Object markers indicating first class objects 71 Object markers indicating second class objects 72 invisible markers

Claims

1. 1. A computer-implemented method comprising: obtaining training data; training an object detection AI using the acquired learning data so that when an RGBD image or a Depth image is input, the AI ​​outputs attribute information of an object included in the RGBD image or the Depth image; Including, The learning data includes an RGBD image or a depth image, a correct answer label indicating the attribute information of an object included in the RGBD image or the depth image, and A method comprising:

2. 10. The method of claim 1, The attribute information of the object is Information indicating a region corresponding to the object on the RGBD image or the depth image; information indicating one class among a plurality of classes according to a D value in the region of the RGBD image or the Depth image; A method comprising:

3. 3. The method of claim 2, Inputting an RGBD image or a depth image into the trained object detection AI; A step of acquiring the attribute information of the object, which is output from the trained object detection AI, each time an object included in the RGBD image or the depth image is detected by the trained object detection AI; a step of superimposing and displaying at least one of a first marker indicating the detected object of the first class and a first count number of the object of the first class on an RGB image corresponding to the RGBD image or the Depth image input to the trained object detection AI each time an object of a first class having the largest D value is detected among the plurality of classes; The method further comprises:

4. 4. The method of claim 3, When an object of a second class having the second largest D value next to that of the first class is detected from among the plurality of classes, the first marker and the first count number superimposed on the RGB image are hidden. The method further comprises:

5. 5. The method of claim 4, a step of superimposing on the RGB image at least one of a second marker indicating the detected object of the second class and a second count number of the object of the second class each time the object of the second class is detected from among the plurality of classes; The method further comprises:

6. 6. The method of claim 5, a step of superimposing a third marker on at least a part of an area on the RGB image where no object of the second class exists, when the second marker or the second count number is superimposed on the RGB image; The method further comprises:

7. 7. The method according to any one of claims 3 to 6, comprising: a step of superimposing the total number of detected objects for each class on the RGB image; The method further comprises:

Images