Material detection system and method for generating learning data therefor

Abstract

To provide an object detection model capable of achieving high detection accuracy in a material detection system.SOLUTION: A material detection system for performing object recognition of materials based on photographed images of a plurality of materials, is configured to: detect objects in a sample image of the material using an object detection model corresponding to the type of the material; use a sample image of a bounding box area for the material as a reference image; post the reference image onto a predetermined background image to create learning data; and additionally learn the object detection model using the leaning data.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a material detection system that detects the number of materials using captured images, and a method for generating learning data for the same. [Background technology]

[0002] In response to the recent decline in the number of workers due to the declining birthrate and aging population, a technology has been proposed to replace visual inspection tasks such as checking inventory levels of materials. This technology inputs image data, detects each material contained in the image data using an object detection model, and counts the number of various objects.

[0003] Since materials have a wide variety of shapes and are irregular, it takes a lot of effort to prepare training data for an object detection model. For this reason, various methods for generating training data have been devised.

[0004] For example, Patent Document 1 describes a technology in which an object is detected using an object detection model, an image within the detection frame is cut out, and an image of the object is generated by penetrating the periphery of the object in the image within the detection frame.Then, the image of the object is pasted onto a scene image to synthesize the image, and the resulting image is used as learning data. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2020-149086 A Summary of the Invention [Problem to be solved by the invention]

[0006] According to the above-mentioned prior art, there is a problem that misdetected objects may be pasted into the learning data to create learning data, and when detecting objects using an object detection model trained using this learning model, high detection accuracy cannot be obtained. An object of the present invention is to provide an object detection model that can achieve high detection accuracy. [Means for solving the problem]

[0007] In order to solve the above problem, the material detection system of the present invention, which performs object recognition of multiple materials based on photographed images of the materials, performs object detection on a sample image of the material using an object detection model corresponding to the type of the material, uses a sample image of the bounding box area of ​​the material as a reference image, pastes the reference image onto a specified background image to use as learning data, and performs additional learning on the object detection model using the learning data. Effect of the Invention

[0008] According to the present invention, it is possible to provide an object detection model that can obtain high detection accuracy. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an overview of a material detection system. [Diagram 2] FIG. 10 is a diagram illustrating a method for handling an object detection model in a material detection system. [Diagram 3] FIG. 10 is a diagram illustrating a method for generating learning data. [Figure 4] FIG. 1 is a block diagram illustrating a material detection system. [Diagram 5] FIG. 11 is a flow diagram illustrating a method for generating training data for an object detection model of a material detection system. [Figure 6] 13 is a diagram showing an example of a display of the object detection results of materials and a display example in a step of selecting a reference image. [Figure 7] 13 is another example of a display in the step of displaying the object detection results of materials and selecting a reference image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a diagram for explaining an outline of a material detection system according to an embodiment.

[0011] The material detection system synchronously (and more preferably simultaneously) acquires two-dimensional image data (each frame of the video) of the object to be detected, which is video-recorded from multiple angles, and three-dimensional point cloud data of the object. It cross-references the recognition results (bounding boxes) of the objects to be recognized, detected based on the two-dimensional image data, with the three-dimensional point cloud information, and links the bounding boxes with the point cloud to recognize the number and positions of the objects to be recognized.

[0012] This enables the material detection system to accurately count materials even when they are hidden by other stacked materials, when they cannot fit into a single image no matter what angle they are taken from, or when there is so much material that all of it cannot fit into a single image.

[0013] FIG. 1 is a diagram illustrating an overview of a material detection system that counts the number of recognition target objects 31 in an object group 3 consisting of a plurality of recognition target objects 31, each of which is a cylindrical object (pipe) that is a scaffolding material used at a construction site, and estimates the position of the recognition target objects 31.

[0014] 1 is data obtained by performing point cloud accumulation processing on a 3D frame point cloud, which is a depth image of the object group 3 acquired in synchronization with a 2D frame image (moving image data) when shooting a moving image of the object group 3. In other words, the 3D point cloud data shows the surface of the object group 3 in three dimensions, and indicates the absolute coordinates of the shape and position of the recognition target object 31.

[0015] The two-dimensional frame image in Fig. 1 is information about two-dimensional frame images captured when a video of an object group 3 is shot. In each frame image, the end face of a recognition target object 31 (end face of a pipe) is detected, and the detected rectangular area is taken as a bounding box. Each frame of the two-dimensional frame image is an image of the object group 3 shot from a different angle, and even if the bounding box is for the same recognition target object, the position and size are different.

[0016] The material detection system performs an integrated analysis of the bounding boxes of each frame extracted from multiple 2D frame images using 3D point cloud data, maps them onto a representative image of a 2D surface, and generates overall image data that can grasp the entire object group 3. By performing an integrated analysis of the bounding boxes of each frame using 3D point cloud data, the material detection system can grasp and count the position of a recognition target object 31 that is hidden from a certain direction, simultaneously with other recognition target objects 31.

[0017] In the material detection system, a bounding box indicating a partial region including a recognition target object 31 in a two-dimensional frame image is obtained by an object detection model using deep learning such as a convolutional neural network. Next, a method of handling the object detection model in the material detection system will be described.

[0018] 2 is a diagram for explaining a method of handling an object detection model in a material detection system according to an embodiment. The object detection model is used in different ways in the learning phase and inference phase of the material detection system. Here, the inference phase is a state in which a video of a group of materials (object group 3) on which materials (objects to be recognized 31) are actually piled up is taken at a scaffolding material storage area at a construction site, and the object detection model detects the end faces of the materials to obtain bounding boxes, and the number of materials is counted based on the bounding boxes. The learning phase is a generation phase of an object detection model, and is composed of a learning data generation phase and an additional learning phase.

[0019] Here, there are many types and shapes of materials. Therefore, in order to obtain high counting accuracy, the object detection model of the material detection system of the embodiment adopts the form of an individual model in which an object detection model is prepared according to the type of material, such as pipes and scaffolding materials.

[0020] In the learning data generation phase, the material detection system first finds an object detection model corresponding to the specified type of material from among standard object detection models prepared in advance according to the type of material. Then, using this object detection model, learning data is generated using images of the material group (object group 3) that will actually be counted in the inference phase as sample images. The method of creating this learning data will be described later with reference to FIG. 3.

[0021] Next, in the additional learning phase, the material detection system learns an object detection model corresponding to the specified type of material based on the learning data generated in the learning data generation phase. In other words, additional learning of an object detection model according to the type of material is performed. This allows the environment of the material group (object group 3) that is actually counted in the inference phase to be reflected in the object detection model, thereby improving the accuracy of extracting the bounding box of the material (recognition target object 31).

[0022] In the above, images of the materials group (object group 3) to be actually counted in the inference phase are used as sample images to generate learning data, but images of an environment similar to the environment in the inference phase may also be used as sample images. In addition, the additionally trained object detection model may be made available for acquisition in another learning data generation phase as a standard object detection model for the object detection model corresponding to the specified type of material.

[0023] In the inference phase, the material detection system takes a video of a pile of materials (object group 3) with materials (recognition target objects 31) actually piled up in a scaffolding material storage area at a construction site, detects the end faces of the materials using the additionally trained object detection model to obtain a bounding box, and counts the number of materials based on the bounding box. At this time, when counting materials of the same type and shape and color as the material for which the sample image was obtained, the additionally trained object detection model may be used for counting.

[0024] Next, a method of generating learning data in the learning data generation phase of FIG. 2 will be described in detail with reference to FIG.

[0025] For a sample image, which is an image of the material group (object group 3) to be actually counted, an image of a bounding box detected by an object detection model corresponding to the specified material type is extracted and selected as a reference image. At this time, the bounding box is displayed on the sample image so that the detection result of the object detection model can be confirmed. Then, the selected reference image can be corrected based on the confirmation result of whether the recognition target object 31 has been detected.

[0026] For example, an operator checks the bounding box displayed on the sample image, and if it is determined to be a false positive, it is deleted from the reference image. Alternatively, an operator may specify an area on the sample image that corresponds to the bounding box to specify the reference image. Details will be described later with reference to Figures 6 and 7.

[0027] In addition, an arbitrary background image is prepared as the background of the learning data. This background image is preferably an image of the material storage area where counting will actually be performed, but may be a monochrome image.

[0028] The reference image is then pasted onto the background image to create the learning data. At this time, the reference image is transformed using arbitrary parameters such as rotation, inversion, and enlargement / reduction, and pasted at an arbitrary position to increase the variation of the reference image. At this time, the same reference image may be pasted multiple times.

[0029] When pasting a reference image, the index (Frame, No) of the reference image, its pasting position in the image (xpos, ypos) and size (xsize, ysize) are recorded as annotation information. This can reduce the annotation effort when using the training data for additional training of the object detection model.

[0030] Next, the material detection system will be described with reference to FIG. FIG. 4 is a block diagram illustrating a material detection system. The material detection system comprises a material detection device 1 which counts and recognizes the positions of multiple recognition objects based on a captured image of the object, and an object detection device 2 which detects the recognition objects from the two-dimensional image data of the object sent from the material detection device 1 and notifies the material detection device 1 of the detection as a bounding box.

[0031] For example, the material detection device 1 is a mobile terminal such as a smartphone of a worker who counts materials, and the object detection device 2 is a server device to which the mobile terminal is connected via a network. In the following description, a configuration will be described in which the material detection device 1 includes an object recognition processing unit 13 (details will be described later) that counts and recognizes the positions of objects to be recognized, but the object recognition processing unit 13 may be included in the server device.

[0032] The data acquisition unit 11 of the material detection device 1 includes a camera (not shown) that captures two-dimensional video data of the object group 3, and a distance image sensor or a three-dimensional sensor (not shown) such as LiDAR (Light Detection and Ranging) that acquires three-dimensional point cloud data of the object group 3. The data acquisition unit 11 may also acquire the three-dimensional point cloud data of the object group 3 by a stereo camera or monocular depth estimation.

[0033] The data transmission unit 12 is a communication unit that transmits the two-dimensional video data and three-dimensional point cloud data of the object group 3 acquired by the data acquisition unit 11 to the object detection device 2. The communication unit performs network communication such as a mobile communication network or a wireless LAN.

[0034] The object recognition processing unit 13 is a processing unit that cross-references the 2D image data (each frame of the video) of the object group 3 captured from multiple angles and the 3D point cloud data of the objects acquired by the data acquisition unit 11, and the bounding boxes of the objects to be recognized detected by the object detection device 2 described below based on the 2D image data, as described in Fig. 1, with the 3D point cloud information, and determines the number and positions of the objects to be recognized 31 by linking the bounding boxes with the point cloud. The object recognition processing unit 13 also controls the user interface in the learning phase of the material detection system.

[0035] The display unit 14 displays the number and positions of the recognition target objects 31 obtained by the object recognition processing unit 13, and is also a display unit used when selecting, correcting, and adding reference images in the object detection device 2 described later.

[0036] The data receiving unit 21 of the object detection device 2 is a communication unit that receives the two-dimensional image data and three-dimensional point cloud data of the object group 3 transmitted by the data transmitting unit 12 of the material detection device 1. The communication unit performs network communication such as a mobile communication network or a wireless LAN.

[0037] The object detection model selection unit 211 is a processing unit that selects an object detection model to be used in the learning phase (learning data generation phase and additional learning phase) and inference phase of the material detection system from among standard object detection models according to the designation of the type of material. The standard object detection models are prepared in advance in the object detection model storage unit 210 according to the type of material.

[0038] The inference unit 29 is a processing unit that performs object detection of the recognition target object 31 for the sample image for generating training data or the two-dimensional image data of the object group 3 using the object detection model selected by the object detection model selection unit 211, and obtains a bounding box as a detection result.

[0039] The editing unit 22 is a processing unit that, in the learning data generation phase, obtains a reference image from a sample image based on a bounding box obtained from two-dimensional image data that is a sample image by the inference unit 29. More specifically, the range of the bounding box is set as the reference image.

[0040] Specifically, a bounding box is displayed on a sample image, the detection result of the recognition target object 31 of the object detection model can be confirmed, and a valid bounding box can be selected. Then, an image of the area of ​​the selected bounding box is set as a reference image of the recognition target object 31. At this time, the size of the selected bounding box may be changed. Also, the reference image may be specified by the operator specifying an area on the sample image that corresponds to the bounding box.

[0041] The image conversion unit 23 is a processing unit that performs transformation processing such as rotation, inversion, enlargement, and reduction using arbitrary parameters on the reference image obtained by the editing unit 22. This increases the variation of the reference image.

[0042] The image synthesis unit 25 pastes the reference image obtained by the editing unit 22 and the reference image converted by the image conversion unit 23 at an arbitrary position in one of the background images stored in the background image storage unit 24, and stores the data in the learning data storage unit 27.

[0043] In addition, when the reference image is pasted, the index (Frame, No), the pasting position (xpos, ypos) and the size (xsize, ysize) of the reference image in the image are recorded in the learning data storage unit 27 as annotation information.

[0044] Furthermore, image synthesis unit 25 may paste a foreground image stored in foreground image storage unit 26 to create learning data.

[0045] The learning unit 28 performs additional learning on the object detection model selected by the object detection model selection unit 211 using the corresponding learning data and annotation information in the learning data storage unit 27, and either updates the object detection model in the object detection model storage unit 210 to the additionally learned object detection model, or stores it as an object detection model of a new classification.

[0046] In the inference phase of the material detection system, the inference unit 29 obtains a bounding box of the recognition target object 31 for the two-dimensional image data of the object group 3 by using a trained object detection model corresponding to the designation of the type of material in the object detection model storage unit 210.

[0047] Next, a method for generating learning data for an object detection model of the material detection system according to the embodiment will be described with reference to the flow chart of FIG.

[0048] In step S61, the object detection model selection unit 211 sets the type of material for the object detection model in response to a designation of the type of material for object detection. In step S62, the object detection model selection unit 211 refers to the object detection model storage unit 210 and selects an object detection model that corresponds to the set type of material.

[0049] In step S63, the inference unit 29 reads two-dimensional image data of a sample image for generating learning data. In step S64, the inference unit 29 performs object detection of the material (recognition target object 31) for the two-dimensional image data of the sample image, using the object detection model determined in step S62.

[0050] In step S65, the editing unit 22 obtains and displays a reference image from the sample image based on the bounding box that is the result of the object detection in step S64. In more detail, the bounding box is displayed on the sample image so that the detection result of the material of the object detection model can be confirmed.

[0051] In step S66, the editing unit 22 sets the image of the bounding box region selected by a user operation or the like as the reference image of the recognition target object 31. At this time, the size of the selected bounding box may be made changeable. Also, the operator may specify the reference image by specifying an area corresponding to the bounding box on the sample image.

[0052] In step S67, the image synthesis unit 25 pastes and synthesizes the reference image obtained in step S66 and the reference image converted by the image conversion unit 23 at arbitrary positions in one background image stored in the background image storage unit 24 to generate learning data, and stores the data in the learning data storage unit 27. At this time, the pasting position and size of the reference image are regarded as annotation information.

[0053] In step S68, the learning unit 28 additionally learns the object detection model selected in step S62 using the learning data and annotation information obtained in step S67. In step S69, the learning unit 28 stores the additionally trained object detection model in the object detection model storage unit 210.

[0054] In addition, in a method for generating learning data for an object detection model of a material detection system, when generating the learning data, the accuracy of the reference image can be improved by utilizing depth data of a sample image.

[0055] Specifically, the data acquisition unit 11 of the material detection device 1 acquires three-dimensional point cloud data of the object group 3 when acquiring a sample image, so that the editing unit 22 can refer to a range image of the object group 3.

[0056] Then, in step S65, when the editing unit 22 obtains a reference image from the sample image based on the bounding box, it determines whether the distance of the recognition target object 31 indicated by the bounding box is within a predetermined range, and displays the bounding box if it is within the predetermined range. In other words, if a material that is not within the predetermined distance is detected when the sample image is photographed, the bounding box is not displayed and it is excluded from the confirmation target of the detection result.

[0057] Next, examples of display of the material object detection results in steps S65 and S66 in FIG. 5 and display in the step of selecting a reference image will be described with reference to FIG. 6 and FIG.

[0058] FIG. 6 is a diagram showing an example of the display on the display unit 14 (FIG. 4) in which a bounding box is displayed over the sample image in step S65, allowing the detection result of the material of the object detection model to be confirmed.

[0059] In Fig. 6, the bounding box is shown as a black rectangle, but since the image of the bounding box area displayed outside the object group 3 or across the recognition target object 31 is not suitable for the reference image, touch the bounding box to select it and then double-tap the "Delete" button to delete the bounding box. Also, tap the bounding box to select it and then drag it to change the position and size of the bounding box and to change the position and size of the reference image.

[0060] When the “Confirm” button is double-tapped, the correction by the user operation or the like is terminated, and the image of the bounding box area is set as the reference image of the recognition target object 31 .

[0061] FIG. 7 is a diagram illustrating another example of correction of the bounding box. FIG. 7 shows an example of correction in which the user directly specifies an area of ​​the reference image corresponding to a bounding box with a filled-in rectangle, and adds the reference image.

[0062] In this case, a region similar to the directly specified region may be displayed as a dashed rectangle as a candidate region to be added to the reference image by image correlation, template matching, POC (Phase Only Correlation), etc. A candidate region is determined by tapping the dashed rectangle and then double-tapping the "Confirm" or "Delete" button. Candidates to be added are presented to the user with distinctions made by color, frame display method, etc.

[0063] Furthermore, the present invention is not limited to the above-mentioned examples, and various modified examples are included. The above-mentioned examples have been described in detail to easily explain the present invention, and the present invention is not necessarily limited to those having all of the configurations described. Furthermore, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. [Explanation of symbols]

[0064] 1 Material detection device 11 Data Acquisition Section 12 Data transmission section 13 Object recognition processing section 14 Display section 2. Object detection device 21 Data receiving section 22 Editorial Department 23 Image conversion section 24 Background image storage section 25 Image synthesis unit 26 Foreground image storage section 27 Learning data storage unit 28 Learning Department 29 Reasoning part 210 Object detection model storage unit 211 Object detection model selection unit 3 Object group 31 Recognition target object

Claims

1. 1. A material detection system that performs object recognition of a plurality of materials based on photographed images of the materials, Object detection is performed on a sample image of the material using an object detection model corresponding to the type of the material; A sample image of the region of the bounding box of the material is taken as a reference image; The reference image is pasted onto a predetermined background image to create learning data. additionally training the object detection model using the training data; Material detection system.

2. 2. The material detection system of claim 1, The size of the reference image and the position at which the reference image is pasted on the background image are defined as annotation information in the additional learning. Material detection system.

3. 2. The material detection system of claim 1, correcting the position or size of the bounding box to be used as the reference image; Material detection system.

4. 2. The material detection system of claim 1, a sample image corresponding to a rectangular area added as the bounding box to the sample image of the material in which the bounding box is displayed is set as a reference image; Material detection system.

5. 5. The material detection system according to claim 4, Based on the sample image corresponding to the added rectangular region, a similar image is obtained from the sample image by image correlation, template matching, or POC, and the similar image is set as a reference image. Material detection system.

6. 2. The material detection system of claim 1, When acquiring a sample image of the material, a distance image of the material is acquired; determining whether the distance of the material indicated by the bounding box is within a predetermined range, and if it is within the predetermined range, setting a sample image of the bounding box area of ​​the material as a reference image; Material detection system.

7. A method for generating learning data for an object detection model in a material detection system that performs object recognition of a plurality of materials based on photographed images of the materials, comprising: Object detection is performed on a sample image of the material using an object detection model corresponding to the type of the material; A sample image of the region of the bounding box of the material is taken as a reference image; The reference image is pasted onto a predetermined background image to obtain learning data. How to generate training data.

Images