Analysis apparatus and analysis method

Abstract

Estimating the posture of an object from a captured image of the object. [Solution] The analysis device 20A includes an object detection unit 26 that detects an object from 2D image data 23, a registration unit 28 that compares a 3DCG image 22, which is a rendering of a 3DCG model 21 based on posture parameters, with the 2D image data 23 and registers both images if they are similar, and a posture estimation unit 286 that estimates the posture state of the object in the 2D image data 23 based on the posture parameters of the registered 3DCG image 22. The registration unit 28 also narrows down the 3DCG images 22 with distance parameters similar to the estimated distance of the object in the 2D image data 23 as comparison targets.

Description

Technical Field

[0001] The present disclosure relates to an analysis device and an analysis method.

Background Art

[0002] In image processing, a technique for matching different coordinate systems and image systems is called registration. For example, Patent Document 1 describes a matching unit that performs "2D-3D matching" for detecting an object in a scene from the matching between a 2D image obtained by projecting a 3D model and a 2D image captured as a scene. This 2D-3D matching is an example of registration.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] On the other hand, estimating the posture state of an object from a captured image of the object is not described in conventional techniques such as Patent Document 1.

[0005] The present disclosure has been made in consideration of such circumstances, and the main problem is to estimate the posture state of an object from a captured image of the object.

Means for Solving the Problems

[0006] The analysis device of the present disclosure has the following features. The analysis device Photographed image includes an object detection unit that detects an object from and a rendering image obtained by rendering a 3DCG model based on pose parameters, and compares the Photographed image with the above, and in both images Features within the region where the object exists In similar cases, both Image location information A registration unit that registers the and A first object in the registered rendering image is placed in the captured image, and a same object determination unit integrates the placed first object and a second object in the captured image that overlaps with the first object as the same object. Object counting unit after the aforementioned identical object determination unit has been integrated counts the objects present in the captured image. It is characterized by having the following: Other methods will be described later. [Effects of the Invention]

[0007] According to this disclosure, the posture of an object can be estimated from a captured image of the object. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram showing the configuration of the image analysis system according to this embodiment. [Figure 2] This is a flowchart showing the processing of the 3DCG image conversion unit according to this embodiment. [Figure 3] This is an explanatory diagram showing an example of the 3DCG modeling process according to this embodiment. [Figure 4] This is an explanatory diagram showing an example of the posture parameter setting process according to this embodiment. [Figure 5] This is an explanatory diagram of the rotation angle in the attitude parameter table of Figure 4 relating to this embodiment. [Figure 6] This is an explanatory diagram showing an example of the 3DCG image generation process according to this embodiment. [Figure 7] This is an explanatory diagram showing an example of the process for registering 3DCG images in a database according to this embodiment. [Figure 8] This flowchart shows the processing mainly performed by the position narrowing unit of the registration unit according to this embodiment. [Figure 9] This is an explanatory diagram showing a specific example of the processing of distance information according to this embodiment. [Figure 10] This is an explanatory diagram illustrating a specific example of a filtering process that uses the distance to an object according to this embodiment. [Figure 11]It is a flowchart showing the process continued from FIG. 8 related to this embodiment. [Figure 12] It is an explanatory diagram showing a specific example of the process of FIG. 11 related to this embodiment. [Figure 13] It is a configuration diagram of the image analysis system related to this embodiment. [Figure 14] It is a flowchart showing the process of the analysis device related to this embodiment. [Figure 15] It is an explanatory diagram showing the outline of the process of FIG. 14 related to this embodiment. [Figure 16] It is a configuration diagram of the image analysis system related to this embodiment. [Figure 17] It is a histogram showing that the result integration unit related to this embodiment has integrated the output of the registration unit. [Figure 18] It is a hardware configuration diagram of each device of the image analysis system related to this embodiment.

Mode for Carrying Out the Invention

[0009] Hereinafter, this embodiment will be described with reference to the drawings.

[0010] FIG. 1 is a configuration diagram of an image analysis system 100A. The image analysis system 100A is configured such that a data acquisition device 10 and an analysis device 20A are connected by a network. The data acquisition device 10 includes a data acquisition unit 11 and a data transmission unit 12. The analysis device 20A has a storage unit that stores a 3DCG model 21, a 3DCG image 22, 2D image data 23, an object detection model 24, and a distance estimation model 25. The analysis device 20A includes, as a processing unit, an object detection unit 26, a 3DCG image conversion unit 27, and a registration unit 28.

[0011] The 3DCG image conversion unit 27 includes a model loading unit 271, a parameter setting unit 272, and a rendering unit 273. The registration unit 28 includes a distance estimation unit 281, a position narrowing unit 282, a feature extraction unit 283, a similarity calculation unit 284, a suitable image estimation unit 285, and a posture estimation unit 286.

[0012] The main features of the image analysis system 100A are as follows: [Feature 1] and [Feature 2]. [Feature 1] Based on the combination of the 3DCG image 22 registered by the registration unit 28 and the 2D image data 23, the posture estimation unit 286 estimates the posture state of the object from the captured image of the object in the 2D image data 23. [Feature 2] Based on the position information of the object obtained from the captured image of the object in the 2D image data 23, the position narrowing unit 282 narrows down the 3DCG images 22 to be registered. This shortens the registration time compared to the conventional brute-force method of 2D-3D registration.

[0013] Therefore, the data acquisition unit 11 acquires images of the target object from a camera (not shown) or the like. In this embodiment, the target object is exemplified by materials placed at a construction site. The data transmission unit 12 transmits the captured image of the object acquired by the data acquisition unit 11 to the analysis device 20A. The analysis device 20A saves the received captured image as 2D image data 23. Other aspects of the image analysis system 100A will be explained in the flowcharts in Figures 2, 8, and 11.

[0014] Figure 2 is a flowchart showing the processing of the 3DCG image conversion unit 27. The model loading unit 271 loads a 3DCG model 21 in which a pre-generated object has been modeled (S111). The parameter setting unit 272 sets the pose parameters of the 3DCG model 21 (S112). The pose parameters are parameters that change the distance or rotate the three-dimensional shape of the 3DCG model 21 in the space within the captured image. The rendering unit 273 generates 3DCG images 22 for each state in which the pose parameters of S112 are reflected for the 3DCG model 21 of S111 (S113). The rendering unit 273 registers the images generated in S113 in the 3DCG image 22 DB (database) (S114).

[0015] Figure 3 is an explanatory diagram showing an example of the 3DCG modeling process (S111). The user pre-models each material 201 placed at the construction site as a 3D CG model 21, specifically as a 3D component 202. The 3D components 202 are saved in a data format that can be processed by a computer, such as 3D-CAD (Computer-Aided Design).

[0016] Figure 4 is an explanatory diagram showing an example of the posture parameter setting process (S112). The posture parameter table 301 is generated for each 3D model part of the 3DCG model 21. The posture parameter table 301 associates rotation angles (roll angle, pitch angle, and yaw angle) and distances for each posture parameter number (No.). These combinations of rotation angles and distances are pre-generated as various different combinations. Therefore, the user can create an exhaustive set of combinations by, for example, preparing their own parameter file and defining the minimum, maximum, and step sizes for the following items. • The rotation angles of the object (roll angle, pitch angle, and yaw angle) • Spatial distance from the lens position to the front of the object.

[0017] Figure 5 is an explanatory diagram of the rotation angle in the attitude parameter table 301 of Figure 4. The roll angle is defined as one of the three axes indicated by the symbol 220. The pitch angle is defined as one of the three axes indicated by the symbol 230. The yaw angle is defined as one of the three axes indicated by the symbol 240.

[0018] Figure 6 is an explanatory diagram showing an example of the 3DCG image generation process (S113). The rendering unit 273 generates 3DCG images 22, indicated by ID=C1 to C5, for each posture parameter in the posture parameter table 301 in Figure 5. For example, the rendering unit 273 generates the following five 3DCG images 22. Image C1 is generated from attitude parameters [roll angle 90°, pitch angle 0°, yaw angle 0°, distance 30cm]. Image C2 is generated from attitude parameters [roll angle 0°, pitch angle 30°, yaw angle 0°, distance 30cm]. Image C3 is generated from attitude parameters [roll angle 0°, pitch angle 0°, yaw angle -90°, distance 30cm]. Image C4 is generated from attitude parameters [roll angle 0°, pitch angle 30°, yaw angle 0°, distance 30cm]. Image C5 is generated from attitude parameters [roll angle 0°, pitch angle 30°, yaw angle 0°, distance 100cm].

[0019] Figure 7 is an explanatory diagram showing an example of the process (S114) for registering the 3DCG image 22 in the database. As explained in Figure 6, the rendering unit 273 registers the images generated for each posture parameter as an image data table 302 in the database of the 3DCG images 22 (S114). Image data table 302 includes the items from posture parameter table 301, plus the image path where the generated image will be saved.

[0020] Figure 8 is a flowchart showing the processing mainly performed by the position narrowing unit 282 of the registration unit 28. The flowchart in Figure 8 is executed after the flowchart in Figure 2. The registration unit 28 acquires 2D image data 23 as image data captured by the camera (S121). The registration unit 28 selects the object to be detected (target object) from a candidate list of target objects, etc. (S122).

[0021] The object detection unit 26 performs object detection on the image data acquired in S121 using AI processing by the object detection model 24, and obtains the coordinates of the rectangular region of the object from the image (S123). The distance estimation unit 281 performs distance estimation on the entire image data (entire area) acquired in S121 using AI processing by the distance estimation model 25, and obtains distance information for each pixel (S124).

[0022] The distance estimation unit 281 obtains the distance information for each pixel acquired in S124 in list format (S125). The distance estimation unit 281 converts the distance information acquired in S125 into a histogram and obtains the average value of the most frequent band as the distance information of the object captured in the image data (S126). The position filtering unit 282 filters the DB containing the 3DCG images 22 using the distance to the object (S127). In other words, the position filtering unit 282 filters out the 3DCG images 22 so that only those 3DCG images 22 that match or are similar to the distance information of the object acquired in S126 remain.

[0023] Furthermore, the position narrowing unit 282 may also filter using angle information in addition to distance information as part of the filtering in S127. For example, the object detection model 24 detects not only the object's location within the image but also the object's surface from the 2D image data 23. The position narrowing unit 282 then further narrows down the 3DCG images 22 that have been filtered by distance information so that only those 3DCG images 22 that match or are similar to the angle information detected by surface detection remain. After processing S127, the process continues from terminal A to the processing shown in Figure 11.

[0024] Figure 9 is an explanatory diagram showing a specific example of the processing related to distance information (S123-S126). Image 251 shows the result of the process (S123) of obtaining the coordinates of a rectangular region of an object from the image. The object (material) is visible in the center of Image 251. Image 252 shows the results of acquiring distance information for each pixel (S124) relative to Image 251. The colored areas in Image 252 indicate regions that are close to the camera that took Image 251. Graph 253 shows the histogram of the distance information for each pixel in image 252. From this graph 253, the mean value of the mode, distance = 20 cm, is obtained (S126).

[0025] Figure 10 is an explanatory diagram illustrating a specific example of the filtering process (S127) that uses the distance to the object. Image data tables 303 and 304 have the same data format as image data table 302, which was explained in Figure 7. Image data table 303 shows the state before filtering is performed. In this state, 3DCG images 22 at various distances such as distance = 0cm, 10cm, 20cm, ... are registered. Image data table 304 shows the state after filtering. In this state, only data with a distance of 20 cm is included.

[0026] Figure 11 is a flowchart showing the process that continues from Figure 8. The process shown in Figure 11 begins from terminal A after the processing in S127 in Figure 8. The feature extraction unit 283 extracts the region in the 2D image data 23 where the object has been detected as a rectangle and converts the features within the rectangle into feature vectors (S131). The following describes the loop that compares the features of the 2D image data 23 with the features of the 3DCG image 22. The feature extraction unit 283 determines whether it has processed all the 3DCG images 22 in the image data table 304 that will be compared with the 2D image data 23 (S132). If the answer in S132 is YES, the process proceeds to S137. If the answer in S132 is NO, one 3DCG image 22 that is the subject of this comparison and for which similarity has not yet been calculated is selected from the image data table 304.

[0027] Here, instead of calculating the similarity for all combinations of 2D image data 23 and 3DCG image 22, some combinations may be pre-calculated, meaning that the similarity is not the highest value. For example, if the similarity of a 3DCG image 22 at a rotation angle of 0 degrees is calculated to be below a predetermined value (e.g., 50 / 100 or less), the registration unit 28 assumes that there is no image with the highest possible similarity around that rotation angle of 0 degrees. Therefore, the registration unit 28 omits the similarity calculation for 3DCG images 22 at rotation angles of 10 degrees, 20 degrees, 30 degrees, ... 90 degrees, and continues the similarity calculation from a rotation angle of 100 degrees.

[0028] The feature extraction unit 283 converts the feature quantities of the 3DCG image 22 into feature vectors (S133). The similarity calculation unit 284 calculates the similarity between the two combinations by comparing the feature vectors of the 2D image data 23 and the feature vectors of the 3DCG image 22 (S134). The matching image estimation unit 285 determines whether the current similarity has been updated from the previous similarity (i.e., whether the current similarity is the highest value) in order to find the combination with the highest similarity value (S135). If the answer in S135 is NO, the process returns to S132. If the answer in S135 is YES, the matching image estimation unit 285 stores an index (image number, image file name, etc.) for the 3DCG image 22 that is the subject of comparison in this case, in order to return it (S136). If the answer in S132 is YES, the matching image estimation unit 285 extracts the DB record of the 3DCG image 22 with the highest similarity (S137). For example, from the image data table 304 in Figure 10, DB record No. 3 is extracted as the most similar image.

[0029] The process of extracting the 3DCG image 22 that is most similar to the object depicted in the 2D image data 23 from the filtered image data table 304 has been explained in Figure 11. This eliminates the need for the similarity calculation unit 284 to comprehensively compare the entries in the image data table 303 before filtering. Therefore, the time required for registration can be reduced.

[0030] Furthermore, the posture estimation unit 286 reads posture parameters (distance, rotation angle) associated with the 3DCG image 22 most similar to the object from the image data table 304, and estimates these posture parameters as the posture parameters of the object captured in the 2D image data 23. This makes it possible to estimate the posture state of the object from the captured image of the object.

[0031] Figure 12 is an explanatory diagram showing a specific example of the process shown in Figure 11. Image vector 361 is a feature vector transformed by S131 from the detected rectangle in the 2D image data 23. Note that since the feature vector is a data sequence that is difficult to illustrate, Figure 12 shows the image from which the feature vector was extracted instead of the feature vector itself. Also, image vectors 371-373 are feature vectors transformed by S133 for each of the 3DCG images 22. The similarity calculation unit 284 calculates the similarity between the main portion 361A of the image vector 361 and the main portions 371A-373A of the image vectors 371-373. • Similarity between image vector 361 and image vector 371 = 98% • Similarity between image vector 361 and image vector 372 = 25% • Similarity between image vector 361 and image vector 373 = 30%

[0032] The analysis device 20A of the image analysis system 100A described above mainly has the following features. The object detection unit 26 detects an object from the 2D image data 23. The registration unit 28 compares the 3DCG image 22, which is a rendering of the 3DCG model 21 based on the pose parameters, with the 2D image data 23, and registers both images if they are similar. The posture estimation unit 286 estimates the posture state of the object captured in the 2D image data 23 based on the posture parameters of the registered 3DCG image 22.

[0033] Furthermore, the analysis device 20A has the following features. The posture parameters include a distance parameter for the object in the 2D image data 23 and a rotation angle parameter for the object in the 2D image data 23. The registration unit 28 narrows down the 3DCG images 22 with distance parameters similar to the estimated distance of the object captured in the 2D image data 23 as comparison targets.

[0034] Furthermore, the analysis device 20A has the following features: The registration unit 28 compares the 3DCG image 22 with the 2D image data 23 and, if they are dissimilar, it considers other 3DCG images 22 with similar rotation angle parameters to the compared 3DCG image 22 to be dissimilar to the 2D image data 23 and excludes them from the comparison.

[0035] Figure 13 is a diagram showing the configuration of the image analysis system 100B. The analysis device 20B of the image analysis system 100B has an additional identical material determination unit 31 and a material counting unit 32 compared to the analysis device 20A of the image analysis system 100A shown in Figure 1. The main features of the image analysis system 100B are as follows. The system includes a material counting unit 32 that counts the number of objects in the 2D image data 23, and a material identification unit 31 that performs pre-processing to prevent counting errors (double counting) in the counting process of the material counting unit 32. Double counting is the phenomenon in which one large physical material is counted as multiple small materials in the image.

[0036] First, let's explain the background behind the use of the image analysis system 100B. In the counting of temporary construction materials at construction sites, workers use object detection AI to detect the front (head) of the materials from captured images, and then integrate the detection results from multiple images using 3D point cloud information. Replacing manual counting with automated processing requires high counting accuracy. However, the appearance of the head portion of temporary construction materials to be counted varies greatly depending on their arrangement, and in some cases, similar-shaped parts may be counted multiple times (double counting), requiring countermeasures.

[0037] Therefore, in the image analysis system 100B, the identical material determination unit 31 determines whether the double-counted similar-shaped parts are made of the same material or not, as a method to counter double counting. For this purpose, the registration unit 28 registers the 2D image data 23 and the 3DCG model 21. Then, the posture estimation unit 286 estimates the detected object state (object posture).

[0038] Figure 14 is a flowchart showing the processing performed by the analysis device 20B. First, steps S121 to S123 are as explained in Figure 8. As a result, the object detection model 24 detects one or more object rectangles (rectangles containing the area in which an object is captured) from the 2D image data 23. The process by which the identical material determination unit 31 resolves double counting will be explained below in accordance with S141 to S150.

[0039] The identical material determination unit 31 determines whether all object rectangles detected in S121 to S123 have been processed (S141). If the result in S141 is YES, the process terminates. If the result in S141 is NO, the identical material determination unit 31 selects one of the unprocessed object rectangles as the object rectangle being processed in this instance. The identical material determination unit 31 executes the flowchart from Figure 8 to Figure 11 to extract a 3DCG image 22 that matches the 2D image data 23 acquired in S121 (S141B). The identical material determination unit 31 uses the coordinates of the object rectangle being processed to obtain coordinates for placing the circumscribing rectangle (interpolating it into the 2D image data 23) (S142). The circumscribing rectangle is the rectangle that contains (for example, circumscribs) the object rectangle being processed. In other words, in S142, the identical material determination unit 31 places the three-dimensional shape from the registered 3DCG image 22 into the 2D image data 23.

[0040] The identical material determination unit 31 calculates the overlap rate between the object rectangle being processed and the bounding rectangle (S143), and determines whether the overlap rate > α (a certain percentage such as 70%) (S144). If the answer in S144 is NO, the process returns to S141. The overlap ratio is also called IoU (Intersection over Union) and is calculated as (overlapping area of ​​object rectangle and circumscribing rectangle) ÷ (union area of ​​object rectangle and circumscribing rectangle). If the same material determination unit 31 determines YES in S144, it deletes the object rectangle being processed (S145) and integrates the smaller object rectangle with the larger bounding rectangle as the same material. In other words, the same material determination unit 31 integrates objects that overlap with the three-dimensional shape placed in S142 as the same material as the three-dimensional shape (integrated as the same object).

[0041] The identical material determination unit 31 determines whether there is another object rectangle whose area overlaps with the completed bounding rectangle (S146). If the answer in S146 is NO, the identical material determination unit 31 completes the bounding rectangle (S147). The interpolation process here may involve interpolating (overwriting) only the bounding rectangle within the 2D image data 23, or it may involve interpolating (overwriting) the image content of the 3DCG image 22 within the bounding rectangle region.

[0042] If the answer in S146 is YES, the identical material determination unit 31 uses the coordinates of the object rectangle being processed, which was determined to overlap in S146, to obtain the coordinates for placing the bounding rectangle (S148). The identical material determination unit 31 determines whether the overlap rate between the bounding rectangle obtained in S148 and the object rectangle being processed, which was determined to overlap in S146, is > α (S149). If the answer in S149 is NO, the process returns to S146. If the answer to S149 is YES, the object rectangle being processed is deleted (S150), and the smaller object rectangle is integrated with the larger bounding rectangle as the same material.

[0043] Figure 15 is an explanatory diagram showing an overview of the process in Figure 14. State 401 is the state in which two object rectangles 401A and 401B have been extracted from a single 2D image data 23 by the object detection unit 26 (S123). State 402 is the extraction result (S141B) of the 3DCG image 22 that matches the 2D image data 23 of state 401, and the areas 402A with a high degree of similarity.

[0044] State 403 is the circumscribing rectangle 403Z of the three-dimensional shape in the 3DCG image 22 of state 402. State 404 is the result of the same material determination unit 31 calculating the circumscribing rectangle 403Z of state 403 as the corresponding coordinates in the 2D image data 23 of state 401 and placing it as the circumscribing rectangle 404Z (S142). In other words, in S142, the same material determination unit 31 places the three-dimensional shape in the registered 3DCG image 22 into the 2D image data 23.

[0045] State 405 indicates that the overlap ratio between the object rectangle 401A and the circumscribing rectangle 404Z within the 2D image data 23 of state 404 is calculated (S143). Here, since the object rectangle 401A is completely contained within the circumscribing rectangle 404Z, the overlap ratio > α (YES in S144), and the object rectangle 401A is deleted (S145) and integrated into the circumscribing rectangle 404Z.

[0046] State 406 indicates that the overlap ratio between the object rectangle 401B and the bounding rectangle 404Z in the 2D image data 23 of state 404 is calculated (S149). Here again, since the object rectangle 401B is completely contained within the bounding rectangle 404Z, the overlap ratio > α (YES in S149), and the object rectangle 401B is deleted (S150) and integrated into the bounding rectangle 404Z.

[0047] The material counting unit 32 then counts the number of object rectangles 401A, 401B, and circumscribing rectangles 404Z. In other words, the material counting unit 32 counts the three-dimensional shapes and objects present in the 2D image data 23 after the same material determination unit 31 has integrated them. If counted as is, there would be three rectangles (three materials), but due to the processing in Figure 14, both object rectangle 401A and object rectangle 401B are integrated by the circumscribed rectangle 404Z, so the material counting unit 32 counts one circumscribed rectangle 404Z (one material).

[0048] Figure 16 is a diagram showing the configuration of the image analysis system 100C. The analysis device 20C of the image analysis system 100C has an additional result integration unit 41, a model modification unit 42, a learning image generation unit 43, and an AI model learning unit 44 compared to the analysis device 20A of the image analysis system 100A shown in Figure 1. The following describes the [first process] to [sixth process] performed by the analysis device 20C in order, and clarifies each processing unit added to the analysis device 20C.

[0049] [First Processing] The object detection unit 26 uses the object detection model 24, which has been trained by the AI ​​model learning unit 44, to perform object detection on the 2D image data 23 of the test image. Subsequently, the registration unit 28 selects the 3DCG image 22 that has the highest similarity to the image of the detected object rectangle. [Second Processing] The matching image estimation unit 285 selects 3DCG images 22 with high similarity for each object rectangle, and the result integration unit 41 counts how many times each 3DCG image 22 has been selected.

[0050] [Third Processing] After the selection of 3DCG images 22 for all object rectangles in all test images is complete, the posture estimation unit 286 extracts the posture parameters (rotation angle, distance) of the 3DCG images 22 that have been selected at least once. The result integration unit 41 creates a histogram by aggregating the frequency (number of times selected) of the 3DCG images 22 for each set of posture parameters. [Fourth Processing] The results integration unit 41 extracts the widths of intervals that exceed a certain frequency according to the histogram in list format.

[0051] [Process 5] If the model modification unit 42 has not incorporated the 3DCG images 22 corresponding to the list extracted in [Process 4] into the training data, it adds the extracted range of data to the training image dataset. The training images are the teaching material data that the AI ​​model training unit 44 uses for training. On the other hand, if the model modification unit 42 has incorporated the 3DCG images 22 corresponding to the list into the training, it edits (adjusts) the 3DCG images 22 in the training image dataset.

[0052] [Process 6] The learning image generation unit 43 generates learning images based on the 3DCG images 22 that were added to and edited in the learning image dataset in [Process 5]. Then, the AI ​​model learning unit 44 learns the machine learning models by inputting the learning images into various machine learning models such as the object detection model 24 and the distance estimation model 25.

[0053] Figure 17 is a histogram showing that the results integration unit 41 has integrated the output of the registration unit 28. The horizontal axis of the histogram divides the range of the attitude parameter (rotation angle) from 0 degrees to 360 degrees into 60-degree intervals, and plots the representative value for each interval (for example, the representative value for the interval from 0 degrees to 60 degrees = 0 degrees). Histogram 501 represents a case where the values ​​are detected evenly within a defined range of angles. In this case, the results integration unit 41 extracts the widths of the intervals with relatively high frequencies, namely 120, 180, and 240 degrees, in list format. Histogram 502 is a case where detection is difficult within a certain range of angles (typical value 120 degrees) due to its structure. In this case, the results integration unit 41 extracts the widths of intervals with relatively high frequencies, corresponding to typical values ​​of 60, 180, and 240 degrees, in list format. Alternatively, the results integration unit 41 may extract only the width of the interval with the highest frequency, which corresponds to 180 degrees, in list format.

[0054] In this way, the results integration unit 41 integrates the registration results, calculates the matching frequency (histogram) of the 3DCG model 21, and estimates the frequently detected regions (intervals with relatively high frequency). The model modification unit 42 removes areas of the 3DCG model 21 other than the frequently detected areas and generates a new 3DCG model 21. The learning image generation unit 43 estimates the effective feature regions during object detection, creates a 3DCG model 21 consisting only of the feature region portion, and generates training data from the 3DCG model 21. The AI ​​model learning unit 44 learns the object detection model 24 based on the learning data generated by the learning image generation unit 43.

[0055] The analysis device 20C of the image analysis system 100C described above has an AI model learning unit 44. The object detection unit 26 is built using a machine learning model. The AI ​​model learning unit 44 generates training images such that the more frequently the 3DCG images 22 registered by the registration unit 28 are reflected in the learning of the machine learning model, the greater the degree to which these images are incorporated.

[0056] As a result, the analysis device 20C prioritizes using 3DCG images 22 that are hit frequently (similar to 2D image data 23) for training the machine learning model. Therefore, 3DCG images 22 that are hit infrequently are actively excluded as noise from the training of the machine learning model, improving the accuracy of the machine learning model compared to using a dataset of randomly selected training images.

[0057] Figure 18 shows the hardware configuration diagrams for each device in the image analysis systems 100A to 100C. Each device in the image analysis system 100A to 100C (data acquisition device 10, analysis devices 20A to 20C) is configured as a computer 900 having a CPU 901, RAM 902, ROM 903, HDD 904, communication I / F 905, input / output I / F 906, and media I / F 907, respectively. Note that the HDD904 can be replaced or used in conjunction with an SSD or other storage device. Furthermore, a GPU may be used for AI and machine learning processing. The communication interface 905 is connected to an external communication device 915. The input / output interface 906 is connected to the input / output device 916. The media interface 907 reads and writes data to the recording medium 917. Furthermore, the CPU 901 controls each processing unit by executing a program (also called an application or app) loaded into the RAM 902. This program can also be distributed via a communication line or by recording it on a recording medium 917 such as a CD-ROM.

[0058] Furthermore, this disclosure is not limited to the embodiments described above, and various other applications and modifications are possible, provided that they do not deviate from the gist of this disclosure as described in the claims. For example, the embodiments described above describe the configurations of image analysis systems 100A, 100B, and 100C in detail and specifically in order to explain this disclosure clearly, and are not necessarily limited to those comprising all the components described. Also, it is possible to replace parts of the configuration of one embodiment with components of another embodiment. It is also possible to add components of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, replace, or delete other components for parts of the configuration of each embodiment.

[0059] Furthermore, some or all of the above configurations, functions, and processing units may be implemented in hardware, for example, by designing them as integrated circuits. Broadly defined processor devices such as FPGAs (Field Programmable Gate Arrays) and ASICs (Application Specific Integrated Circuits) may be used as hardware. Furthermore, each component of the image analysis systems 100A, 100B, and 100C according to the above-described embodiment may be implemented on any hardware, as long as the respective hardware can send and receive information from each other via a network. Also, the processing performed by a certain processing unit may be implemented by a single piece of hardware, or by distributed processing using multiple pieces of hardware. [Explanation of symbols]

[0060] 10 Data acquisition device 20A~20C analysis device 21 3DCG Models 22 3DCG images 23 2D image data 24 Object Detection Models 25 Distance Estimation Models 26 Object detection unit 27 3DCG Image Conversion Unit 28 Registration Department 31. Identical Material Determination Unit (Identical Object Determination Unit) 32. Material Counting Section (Object Counting Section) 41 Results Integration Department 42 Model Modification Section 43 Learning Image Generation Unit 44 AI Model Learning Department 100A~100C Image Analysis System 281 Distance Estimation Unit 282 Position narrowing section 283 Feature Extraction Unit 284 Similarity calculation unit 285 Suitable Image Estimation Unit 286 Posture estimation section

Claims

1. An object detection unit that detects an object from a captured image, A registration unit compares a rendered image obtained by rendering a 3DCG model based on posture parameters with the captured image, and registers the positional information of both images if the feature quantities within the region where the object exists are similar in both images. A unit for determining the same object places a first object in the registered rendering image into the captured image, and integrates the placed first object and a second object in the captured image that overlaps with the first object as the same object. The system is characterized by having an object counting unit that counts the objects present in the captured image after the aforementioned object determination unit has been integrated. Analysis device.

2. An object detection unit that detects an object from a captured image using a machine learning model, A registration unit compares a rendered image obtained by rendering a 3DCG model based on posture parameters with the captured image, and registers the positional information of both images if the feature quantities within the region where the object exists are similar in both images. The AI ​​model learning unit generates training images such that the rendering images that have been registered more frequently by the registration unit are reflected more heavily in the learning of the machine learning model. Analysis device.

3. The aforementioned posture parameter includes a distance parameter of the object captured in the image and a rotation angle parameter of the object captured in the image. The registration unit is characterized by narrowing down the rendering images with distance parameters similar to the estimated distance of the object captured in the image as comparison targets. The analysis apparatus according to claim 1 or claim 2.

4. The registration unit is characterized in that, if the rendered image and the captured image are found to be dissimilar, other rendered images that have similar rotation angle parameters to the compared rendered image are considered dissimilar to the captured image and are excluded from the comparison. The analysis apparatus according to claim 3.

5. The analysis device comprises an object detection unit, a registration unit, a unit for determining identical objects, and an object counting unit. The object detection unit detects an object from the captured image, The registration unit compares the rendered image obtained by rendering a 3DCG model based on the posture parameters with the captured image, and if the feature quantities within the region where the object exists are similar in both images, it registers the position information of both images. The identical object determination unit places the first object in the registered rendering image into the captured image, and integrates the placed first object and the second object in the captured image that overlaps with the first object as the same object. The object counting unit is characterized by counting the objects present in the captured image after the identical object determination unit has been integrated. Analysis method.

6. The analysis device comprises an object detection unit, a registration unit, and an AI model learning unit. The object detection unit detects objects from the captured image using a machine learning model, The registration unit compares the rendered image obtained by rendering a 3DCG model based on the posture parameters with the captured image, and if the feature quantities within the region where the object exists are similar in both images, it registers the position information of both images. The AI ​​model learning unit is characterized in that it generates training images such that rendering images that have been registered more frequently by the registration unit are reflected more heavily in the learning of the machine learning model. Analysis method.

Images