Training image generation system, image recognition system, image recognition method, and image recognition program
The learning image generation system uses gaze point visualization to efficiently generate CG images aligned with real images, addressing the inefficiencies of conventional random generation methods by focusing on specific correction points, thus enhancing DL learning data.
Patent Information
- Application Number
- JP2021116665
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-07-14
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-07-14
AI Technical Summary
Conventional methods for generating computer-generated (CG) images for deep learning (DL) require excessive computational resources and time due to random generation without specifying correction points, making it difficult to create effective learning data.
A learning image generation system that utilizes gaze point visualization technology to identify specific correction points by comparing the gaze area tendencies of CG and real images, regenerating CG images to minimize domain differences, thereby enhancing their effectiveness for DL learning.
The system efficiently generates CG images suitable for DL learning in a shorter time by correcting CG images to align with real image features, improving learning data quantity and quality.
Smart Images

Figure 0007586786000001 
Figure 0007586786000002 
Figure 0007586786000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a training image generation system, an image recognition system, an image recognition method, and an image recognition program. [Background technology]
[0002] In recent years, systems and services that utilize machine learning (e.g., deep learning) have been increasing in the field of image recognition. Deep learning (hereafter referred to as DL) can be divided into supervised learning, in which learning is done by attaching a correct label to the learning image data, and unsupervised learning, in which learning is done without attaching a correct label and extracting features from the learning image data. For example, supervised learning is often used to solve DL problems such as image classification problems that recognize "what" an object in an input image is, fault diagnosis that determines whether an object in an image is "normal or abnormal," and regression problems such as estimating age from a person's image.
[0003] In supervised learning, the more image data used for learning, the better the performance (recognition accuracy, generalization performance, etc.) of the DL model generated after learning. However, there are cases where it is not possible to collect enough image data for learning due to issues such as the cost of collecting it and privacy when the subjects are people. Therefore, a common method is to use generated images (CG images) to make up for the lack of learning data. However, because there are large differences between the domains of CG images and real images (such as subject tendencies, camera angle of view, background, and lighting), it is difficult to generate a highly accurate recognition model through DL by learning only with CG images.
[0004] In response to the above-mentioned problem, Patent Document 1 adopts a method of repeating CG image generation and learning until a CG image of a person that is effective for learning can be generated. [Prior art documents] [Patent documents]
[0005] [Patent Document 1] JP 2020-87310 A [Non-patent literature]
[0006] [Non-Patent Document 1] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, Dhruv Batra, “Grad-CAM:Visual Explanations from Deep Networks via Gradient-based Localization[v4],” [online], 3December 2019, [Retrieved June 1, 2021], Internet<URL:https: / / arxiv.org / abs / 1610.02391v4> [Non-Patent Document 2] Shan Yang, Tanya Ambert, ZherongPan, Ke Wang, Licheng Yu, Tamara Berg, Ming C. Lin, “Detailed Garment Recovery from a Single-View Image”, [online], 12 September 2016, [Retrieved June 1, 2021], Internet<URL:https: / / arxiv.org / abs / 1608.01250> Summary of the Invention [Problem to be solved by the invention]
[0007] However, in the conventional technology of Patent Document 1 mentioned above, generation and learning are repeated randomly without determining specific correction points, so it requires extremely large computational resources and a long generation processing time before a CG image of a person that is effective for learning can be generated.
[0008] The present invention has been made in consideration of the above-mentioned problems, and has an object to make it possible to generate CG images that are effective for DL learning in a shorter time than with conventional techniques. [Means for solving the problem]
[0009] In order to solve the above-mentioned problems, one aspect of the present invention is a training image generation system that generates image data for learning using an artificially generated image that is different from a real image acquired by an imaging device, and is characterized in that an input artificial image, which is an input artificial image, is accepted, and during an inference process that uses a recognition model to determine whether the input artificial image is a real image or an artificial image, a gaze area of the input artificial image is calculated, and the gaze area of the input artificial image is modified based on the result of comparing the tendency of the gaze area of the input artificial image with a predetermined standard to regenerate the artificial image. Effect of the Invention
[0010] According to the present invention, for example, CG images effective for DL learning can be generated in a shorter time than with the conventional techniques. [Brief description of the drawings]
[0011] [Figure 1] FIG. 1 is a block diagram showing an example of the overall configuration of an AI learning image generation system according to a first embodiment. [Diagram 2] FIG. 1 is a block diagram showing an example of the overall configuration of an AI learning image generation system according to a second embodiment. [Diagram 3] 11 is a flowchart showing an example of an AI learning image generation processing procedure according to the second embodiment. [Figure 4] FIG. 13 is an image (actual image) showing an example of the results of executing part division processing. [Diagram 5] An illustration (actual image) showing an example of the results of executing the gaze point visualization process. [Figure 6] An image (actual image) showing an example of the results of executing the domain difference detection process. [Figure 7] 13 is an example of a table configuration showing an example of a tendency of a fixation area of a real image. [Figure 8] FIG. 13 is an image (CG image) showing an example of the results of executing part division processing. [Figure 9] An illustration (CG image) showing an example of the results of executing the gaze point visualization process. [Figure 10] An illustration (CG image) showing an example of the results of executing the domain difference detection process. [Figure 11] 13 is an example of a table configuration showing an example of a tendency of a gaze area of a CG image. [Figure 12] An image showing an example of a CG image after correction. [Figure 13] FIG. 11 is a block diagram showing an example of the overall configuration of an AI learning image generation system according to a third embodiment. [Figure 14] 11 is a flowchart showing an example of an AI learning image generation processing procedure according to the third embodiment. [Figure 15] FIG. 1 is a block diagram showing an example of the configuration of an image recognition system including an AI learning image generation system according to first to third embodiments. [Figure 16] FIG. 1 is a diagram showing an example of the hardware configuration of a computer that realizes an AI learning image generation system and an image recognition system including the system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment is merely an example for explaining the present invention, and does not limit the present invention. For clarity of explanation, appropriate omissions and simplifications are made. The present invention can be implemented in various other forms or in a form combining a part or all of each form. Unless otherwise specified, each component may be singular or plural.
[0013] When there are multiple components having the same or similar functions, they may be described by using the same reference numerals with different subscripts, or when there is no need to distinguish between these multiple components, the subscripts may be omitted.
[0014] In the following embodiments, the differences from the above embodiments will be mainly described, and the description of the overlapping parts will be omitted as appropriate. In addition, the information represented in the table format in this specification may be represented in other data formats without being limited to the table format.
[0015] In the following embodiment, a CG (Computer Graphics) image generated by a person using a computer is given as an example of an artificial image that is artificially generated. The artificial image is not limited to a CG image, but may be a hand-drawn illustration, etc. An actual image is an image of an object such as a person included in a moving image or a still image captured by a camera, etc.
[0016] In the following embodiment, real images and artificial images including people are targeted, but the present invention can be targeted to real images and artificial images including a wide range of objects, not limited to people.
[0017] In the following embodiment, the difference between the CG image and the real image is clarified using the gaze point visualization technology, so that specific correction points can be specified and a correction policy can be devised, and a CG image effective for learning can be generated in a short time compared to the above-mentioned Patent Document 1. In the following embodiment, the difference between the CG image and the real image is determined from the tendency of the gaze area obtained by the gaze point visualization technology. For example, when a recognition model that infers whether an input image is a CG image or a real image analyzes the tendency of each part of an object that becomes the gaze area when inferring a CG image group and a real image group, if the tendency is different, it is highly likely that each gaze area has a feature that distinguishes between the CG image and the real image. Therefore, by applying a correction to the CG image that brings the tendency of the gaze area closer, it is possible to generate a CG image that is effective for learning and has a small difference between the CG image and the real image.
[0018] Specifically, when generating a CG image, there is a method of increasing the number of patterns of parts in the gaze area or close to the gaze area (for example, if the upper body is the gaze area, the number of types of clothing is increased) to weaken the characteristics unique to CG. There is also a method of generating a new CG image by generating parts close to the gaze area of the CG image from a real image and converting the CG image into a model with characteristics close to the real image.
[0019] [Embodiment 1] First, a first embodiment will be described with reference to FIG. The Deep Learning (DL) learning image generation system 1 uses CG images to augment learning data (data augmentation) in order to compensate for a shortage of learning image data used in model learning for machine learning such as Deep Learning (DL).
[0020] (Overall configuration of AI learning image generation system 1 according to embodiment 1) FIG. 1 is a block diagram showing an example of the overall configuration of an AI learning image generation system 1 according to the first embodiment. As shown in FIG. 1, the AI learning image generation system 1 includes an input unit 2, a CG image DB (Data Base) 3, a domain difference detection unit 4, a CG image correction method determination unit 6, and a CG image generation unit 7. The input unit 2 acquires real images from still image files, video files, etc. captured by a camera. The domain difference detection unit 4 detects the difference between the domain of the real image acquired by the input unit 2 and the CG image stored in the CG image DB3 from the tendency of the gaze area obtained when inference is performed to discriminate between the real image and the CG image using a recognition model 5. The CG image correction method determination unit 6 determines the correction method of the CG image according to a predefined rule or the like from the domain difference information detected by the domain difference detection unit 4. The CG image generation unit 7 regenerates a CG image based on the correction method determined by the CG image correction method determination unit 6 and stores it in the CG image DB3. The DB is an example of a storage unit.
[0021] [Embodiment 2] Next, a second embodiment will be described with reference to Figs. 2 to 12. In the second embodiment, a case where a CG image of a person is generated by the AI learning image generation system 1B will be described as an example. In the second embodiment, an example is given in which the real image and the CG image are images of a person, but the present invention is not limited to this, and the real image and the CG image may be images of an object.
[0022] (Overall configuration of AI learning image generation system 1B according to embodiment 2) Fig. 2 is a block diagram showing an example of the overall configuration of an AI learning image generation system 1B according to embodiment 2. A domain difference detection unit 4B in the AI learning image generation system 1B includes a person detection processing unit 41, a part division processing unit 42, a gaze point visualization processing unit 43, and a domain difference detection processing unit 44. An imaging device 8 that inputs an image file to the AI learning image generation system 1B includes a video acquisition unit 81 and a video transmission unit 82. The processing of each functional unit of the AI learning image generation system 1B and the imaging device 8 will be described later with reference to Figs. 3 to 12.
[0023] (AI learning image generation process according to the second embodiment) FIG. 3 is a flowchart showing an example of an AI learning image generation process procedure according to the second embodiment. FIG. 4 is an image (real image) showing an example of an execution result of a part division process. FIG. 5 is an image (real image) showing an example of an execution result of a gaze point visualization process. FIG. 6 is an image (real image) showing an example of an execution result of a domain difference detection process. FIG. 7 is an example of a table configuration showing an example of a tendency of a gaze area of a real image. FIG. 8 is an image (CG image) showing an example of an execution result of a part division process. FIG. 9 is an image (CG image) showing an example of an execution result of a gaze point visualization process. FIG. 10 is an image (CG image) showing an example of an execution result of a domain difference detection process. FIG. 11 is an example of a table configuration showing an example of a tendency of a gaze area of a CG image.
[0024] First, in S101, the imaging device 8 acquires real image data using the video acquisition unit 81, which is a surveillance camera or the like. Next, in S102, the imaging device 8 transmits the real image data captured by the video acquisition unit 81 using the video transmission unit 82 to the input unit 2 of the AI learning image generation system 1B.
[0025] Next, in S103, the AI learning image generation system 1B uses the person detection processing unit 41 to cut out a person area from the actual image data received from the imaging device 8 via the input unit 2. The process of cutting out the person area may utilize, for example, an object detection method using DL, or a conventional machine learning method such as extracting gradient strength using a Histogram of Oriented Gradients (HOG) feature and performing classification using a Support Vector Machine (SVM).
[0026] When the AI learning image generation system 1B detects a person from the real image data (S104 YES), the body part division processing unit 42 applies a DL region division method or the like to divide the image of the person into regions for each body part (S105). For example, when the processing of the body part division processing unit 42 is executed using the real image I101 (FIG. 4) after the person region has been cut out as input, the body of the person in the image is divided into multiple regions such as the head, shoulders, arms, and torso, as in image I102 (FIG. 4).
[0027] In parallel with the processing of the body part division processing unit (S105), the AI learning image generation system 1B uses the gaze point visualization processing unit 43 to input the real image I101 (Fig. 5) after the person area has been cut out and to determine whether the input image is a CG image or a real image using the recognition model 5 (S106). Then, the AI learning image generation system 1B uses the gaze point visualization processing unit 43 to calculate the gaze area that influenced the determination result of S106 using the gaze point visualization technology, as shown in image I103 (Fig. 5) (S107). The recognition model 5 used here is a recognition model that infers whether the input image is a CG image or a real image, and has been pre-trained by DL or the like.
[0028] Here, the gaze point visualization technology is a technology that uses an influence function to identify the feature parts that have influenced the inference result when a recognition model infers an image, as described in Non-Patent Document 1. For example, in the case of a recognition model that recognizes a person's hairstyle, the gaze area is displayed on the head, and in the case of a recognition model that recognizes the manufacturer of a vehicle, the gaze area is displayed on the vehicle's emblem. In this way, the gaze point visualization technology is characterized in that it allows visual confirmation of which area in an image the recognition model focused on when inferring an image and calculated the inference result.
[0029] Note that S105, S106, and S107 may be processed in parallel or sequentially.
[0030] In S108 after S105, S106, and S107 are completed, the AI learning image generation system 1B uses the domain difference detection processing unit 44 to synthesize the result of the part division process (S105) (image I102 (FIG. 4)) and the result of the gaze area calculation process (S107) (image I103 (FIG. 5)) into image I104 (FIG. 6). Then, it counts which parts of the person included in the actual image I101 (FIGS. 4 to 6) have the gaze area calculated (identification of gazed part). In the example of image I104, the head and shoulders are counted. The criterion for counting may be when the gaze area surrounds a certain percentage or more of the part area, or when the gaze area surrounds the entire part area. Alternatively, it may be freely set by the user.
[0031] After repeatedly performing the above processes from S101 to S108 on a certain number of real images (S109 YES), the AI learning image generation system 1B calculates the calculated rate of the gaze area for each part based on the result of counting the gaze areas for the real images. Then, the calculation results are stored in the real image column of table T101 (FIG. 7) as shown in FIG. 7 (S110). For example, in FIG. 7, when the head is calculated as the gaze area (gaze area) in 80 out of a total of 100 real images, the calculated rate of the head is 80%. Table T101 is stored in a memory area that can be referenced by the CG image modification method determination unit 6.
[0032] The calculation ratio of each part of the real image in table T101 may be given as a fixed value in advance. In this case, the processes from S103 to S110 are omitted for the real image and are executed only for the CG image. When the calculation ratio of each part of the real image in table T101 is given as a fixed value in advance, the extraction of the gaze area and part division are performed only for the CG image, so that the processing load can be reduced.
[0033] After the processes from S101 to S110 are performed on the real image, the processes from S103 to S110 are also performed on the CG image. For example, assume that the background of each CG image before correction is grayscale and monotonous as compared to the real image, as in image I201 (FIGS. 8 and 9). In this case, the processing result of the part division processing unit 42 is the same as the real image (image I102 (FIG. 4)) as in image I202 (FIG. 8), but the processing result of the gaze point visualization processing unit 43 is image I203 (FIG. 9), in which the monotonous background part (outside the person area) is a characteristic of the CG image and is calculated as the gaze area.
[0034] Thereafter, in the AI learning image generation system 1B, the domain difference detection processing unit 44 combines the result of the part division process (S105) (image I202 (Fig. 8)) with the result of the gaze area calculation process (S107) (image I203 (Fig. 9)) as in image I204 (Fig. 10), thereby counting which parts of the person included in the actual image I201 (Figs. 8, 9, and 10) have been calculated as gaze areas (gaze areas). In the example of image I204 (Fig. 10), the area outside the person area is counted, and the calculation ratio of the gaze area (gaze area) outside the person area is high at 70%, as shown in table T101 (Fig. 11).
[0035] The AI learning image generation system 1B repeatedly performs the processes from S103 to S108 as in the above example with a certain number of CG images or more (S109 YES), and then calculates the percentage of the gaze area calculated for each part based on the results of counting the gaze areas for the CG images, and stores this in the CG image column of table T101 (Figure 11) (S111).
[0036] Next, in S112, the CG image correction method determination unit 6 determines whether or not the CG image needs to be corrected. For example, if the calculated ratio of the gaze area for each part in the real image and the CG image is larger in the CG image than in the real image and the difference between them is equal to or greater than a threshold, the CG image correction method determination unit 6 determines that the CG image needs to be corrected, otherwise it determines that the CG image does not need to be corrected. If the CG image needs to be corrected (S112 YES), the CG image correction method determination unit 6 moves the process to S113, and if the CG image does not need to be corrected (S112 NO), it ends the AI learning image generation process. By repeating the loop from S103 to S112 YES, S113, and S114 and returning to S103, the tendency of the gaze area of the CG image approaches the real image.
[0037] In S113, the CG image correction method determination unit 6 determines a correction method for the CG image based on the tendency of the gaze area stored in the table T101 as shown in FIG. 11. The following will be described with reference to the result of the table T101 shown in FIG. 11. Comparing the tendency of the gaze area of the real image and the CG image in the table T101, it can be seen that many gaze areas are calculated for the head and shoulders in the real image, whereas many gaze areas are calculated outside the person area in the CG image. If the threshold value in the determination in S112 is 60%, the calculation ratio of the CG image is larger than that of the real image and the difference is equal to or greater than the threshold outside the person area. From this, a correction policy can be determined in which the characteristics unique to the CG image appear outside the person area (background part) of the CG image, and the background pattern of the CG image is increased to weaken the characteristics. There may be multiple parts to be corrected based on the comparison result of the calculation ratio of the gaze area of the real image and the CG image. The correction policy may be made into a rule in advance and automatically selected from this rule in advance. Or, a revision strategy may be considered based on the results stored by the user in table T101.
[0038] Next, in S114, the CG image generation unit 7 regenerates a certain number of CG images in which the backgrounds have been modified to various patterns, such as images I301, I302, and I303 (FIG. 12), in accordance with the modification policy determined in S113, and stores these in the CG image DB 3. When S114 ends, the process returns to S103, and the processes from S103 to S114 are repeated until modification of the CG images is no longer necessary (NO in S112).
[0039] (Effects of the second embodiment) In the above-mentioned second embodiment, during the inference process of determining whether an image is a real image or a CG image using a recognition model, the gaze area of the input CG image is calculated, and the gaze area of the input CG image is corrected based on the result of comparing the tendency of this gaze area with a predetermined standard, and the CG image is regenerated. Therefore, by correcting the characteristics of the CG image that are undesirable for learning by using the gaze area as a clue to bring them closer to the desired predetermined standard, it is possible to create a large amount of learning data of CG images that are effective for learning in a short time.
[0040] In the above-mentioned second embodiment, the gaze area of the input CG image is corrected based on the result of comparing the tendency of the gaze area of the CG image with that of the real image, and the CG image is regenerated. Therefore, by using the gaze area as a clue to correct the characteristics of the CG image that are undesirable for learning to be closer to the characteristics of the gaze area of the real image that are desirable, it is possible to create a large amount of learning data of CG images that are effective for learning in a short time.
[0041] In the above-mentioned second embodiment, the necessity of correcting the gaze area of the CG image is judged based on the difference in domain based on the comparison between the tendency of the gaze area of the CG image and the tendency of the gaze area of the real image, and if it is judged that correction is necessary, a correction method for correcting the gaze area of the CG image is determined. Therefore, by narrowing down the correction of the gaze area to CG images that require correction of the gaze area, it is possible to efficiently correct the gaze area of the CG image without performing unnecessary correction processing. In addition, by correcting the gaze area of the CG image into multiple patterns according to the determined correction method and regenerating multiple artificial images, it is possible to efficiently create a large amount of learning image data in a short time.
[0042] In the above-mentioned second embodiment, the difference between the domains of the real image and the CG image is detected based on the difference in the tendency of the gaze area to fall on each part of the object (human body) extracted from the real image and the CG image. Therefore, by looking at the tendency of the gaze area for each part, it is possible to increase the detection sensitivity of the domain difference and to grasp the correction part of the gaze area for each part.
[0043] In the above-mentioned second embodiment, there are multiple real images and multiple CG images. The tendency of the part corresponding to the gaze area of the real image is the ratio of input real images in which each part is a gaze part corresponding to the gaze area for all of the multiple input real images. Also, the tendency of the part corresponding to the gaze area of the CG image is the ratio of input artificial images in which each part is a gaze part corresponding to the gaze area for all of the multiple input CG images. Therefore, by quantitatively observing the tendency of the gaze area for each part, it is possible to improve the detection accuracy of the difference in domains and the correction part of the gaze area.
[0044] Furthermore, in the above-described second embodiment, the gaze area of the input CG image is modified according to the determined modification method to regenerate an artificial image until it is determined that modification of the gaze area of the input CG image is unnecessary. Therefore, each time a process for determining whether or not the gaze area of the input CG image needs to be modified, the characteristics of the CG image that are unfavorable for learning are modified, and a CG image having characteristics favorable for learning is generated.
[0045] In the above-mentioned second embodiment, when the gaze area includes a certain percentage or more of an object part, a process of setting a part as the gaze area is executed for all input real images and all input CG images. Then, a first percentage of the input real images in which each part is set as the gaze area for all input real images and a second percentage of the input CG images in which each part is set as the gaze area for all CG images are calculated. When the second percentage is greater than the first percentage and the difference between the first percentage and the second percentage is equal to or greater than a threshold, it is determined that the gaze area of the input CG image needs to be corrected. Therefore, since it is quantitatively determined that the gaze area of a CG image having a difference in characteristics from the real image by a certain percentage or more needs to be corrected, it is possible to accurately and efficiently determine whether or not the gaze area needs to be corrected based on a clear criterion.
[0046] [Embodiment 3] Next, a third embodiment will be described with reference to Figs. 13 and 14. In the third embodiment, the number of patterns of parts (the background of the person in the second embodiment) corresponding to the gaze area and positions near the gaze area of the CG image is increased to weaken the characteristics unique to the CG image. However, when there are only a few types of parts (CG model data), in order to generate a CG image in which the parts to be corrected are changed to various patterns, it is necessary to manually create CG parts (for example, model data of various types of clothes) and increase the number of patterns. In such a case, it may take a lot of man-hours to generate a CG image in accordance with the correction policy.
[0047] Therefore, the AI learning image generation system 1C of the third embodiment has a function of automatically increasing the number of part patterns.
[0048] FIG. 13 is a block diagram showing an example of the overall configuration of an AI training image generation system 1C according to embodiment 3. FIG. 14 is a flowchart showing an example of an AI training image generation processing procedure according to embodiment 3. The CG image generation unit 7C of the AI training image generation system 1C includes a 3D modeling processing unit 71, a model synthesis processing unit 72, and a 2D image conversion processing unit 73. Otherwise, the AI training image generation system 1C is similar to the AI training image generation system 1B of embodiment 2. The processing of each functional unit of the AI training image generation system 1C will be described later with reference to FIG. 14.
[0049] In the AI training image generation process of embodiment 3, steps up to S113 are the same as those in FIG. 3, and after S113 is completed, steps S201 to S205 (FIG. 14) are executed instead of S114, and after S205 is completed, processing returns to S103 (FIG. 3).
[0050] After execution of S113, in S201, the 3D modeling processing unit 71 converts the real image from two-dimensional image data into a three-dimensional solid model (3D model). For example, by applying Detailed Garment Recovery from a Single-View Image in Non-Patent Document 2, the clothing area and the body structure of the person are calculated from the input real image, and a 3D model of the clothing and human body is reconstructed. This makes it possible to extract parts necessary for increasing the pattern from the real image as 3D models. By extracting the parts as 3D models, it is possible to synthesize 3D models of the extracted parts for 3D human body models of various postures and body shapes.
[0051] Next, in S202, the 3D modeling processing unit 71 acquires a certain number of 3D models of parts (e.g., jacket parts) that need to be corrected from the actual images as described above. Next, in S203, the model synthesis processing unit 72 randomly synthesizes the 3D models of parts (e.g., jackets) acquired in S202 with 3D models of various human bodies generated by CG. Next, in S204, the 2D image conversion processing unit 73 performs conversion processing on the CG 3D model synthesized in S203 into two-dimensional image data as if it were photographed from a specific direction. Next, in S205, the CG image generating unit 7C executes the processing of S203 and S204 for a certain number of images or more to regenerate the CG images. When S205 ends, the CG image generating unit 7C returns the processing to S103 (FIG. 3).
[0052] In the above-mentioned third embodiment, the parts corresponding to the gaze area of the CG image to be corrected and positions near the gaze area are extracted from the actual image, and the CG image is regenerated by compositing the extracted parts at the same positions in the CG image. Therefore, the number of patterns of parts that need to be corrected is automatically increased, and it is possible to efficiently and quickly generate a large number of CG images for learning.
[0053] [Embodiment 4] The AI training image generation systems 1, 1B, and 1C of the above-mentioned embodiments 1 to 3 can be applied to an image recognition system. FIG. 15 is a block diagram showing a configuration example of an image recognition system 10 including any one of the AI training image generation systems 1, 1B, and 1C according to embodiments 1 to 3. The image recognition system 10 includes an AI training image generation system 1 (or 1B or 1C), a training image DB 11, a learning processing unit 12, a classifier 13, an inference image DB 14, and an inference processing unit 15.
[0054] The training image DB 11 stores training image data generated by the AI training image generation system 1 (or 1B, 1C) using CG images. The learning processing unit 12 learns the CG images generated by the training image generation system 1 (or 1B, 1C) to generate a classifier 13. The inference image DB 14 stores the inference image data. The inference processing unit 15 uses the classifier 13 generated by the learning processing unit to perform inference on the inference image data stored in the inference image DB 14, and outputs the inference result.
[0055] (Hardware of the computer 500 that realizes the AI learning image generation systems 1, 1B, and 1C and the image recognition system 10 including these) 16 is a diagram showing an example of a hardware configuration of a computer 500 that realizes each system of the AI learning image generation systems 1, 1B, and 1C and the image recognition system 10 including these. In the computer 500, a processor 510 such as a CPU (Central Processing Unit), a memory 520 such as a RAM (Random Access Memory), a storage 530 such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive), a network I / F (Inter / Face) 540, an input / output device 550 (e.g., a keyboard, a mouse, a touch panel, a display, etc.), and a peripheral device 560 are connected via a bus.
[0056] In computer 500, the programs for realizing each system of AI learning image generation systems 1, 1B, 1C and image recognition system 10 including these are read from storage 530 and executed by cooperation of processor 510 and memory 520, thereby realizing each system. Alternatively, the programs for realizing each system may be obtained from an external computer equipped with a non-transitory storage device by communication via network I / F 540. Alternatively, the programs for realizing each system may be obtained by being recorded on a non-transitory recording medium and read by a medium reading device.
[0057] The present invention is not limited to the above-described embodiments, and includes various modified examples. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those having all of the configurations described. In addition, it is also possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and to add the configuration of another embodiment to the configuration of one embodiment, as long as there is no contradiction. In addition, it is possible to add, delete, replace, integrate, or distribute a part of the configuration of each embodiment. In addition, the configurations and processes shown in the embodiments can be appropriately distributed, integrated, or replaced based on processing efficiency or implementation efficiency. [Explanation of symbols]
[0058] 1, 1B, 1C: AI learning image generation system, 4: domain difference detection unit, 41: human detection processing unit, 42: body part segmentation processing unit, 43: gaze point visualization processing unit, 44: domain difference detection processing unit, 5: recognition model, 6: CG image correction method determination unit, 7: CG image generation unit, 71: 3D modeling processing unit, 72: model synthesis processing unit, 73: 2D image conversion processing unit, 8: imaging device, 10: image recognition system, 11: learning image DB, 12: learning processing unit, 13: classifier, 14: inference image DB, 15: inference processing unit, 500: computer
Claims
1. A learning image generation system that generates learning image data using an artificial image that is artificially generated and is different from an actual image acquired by an imaging device, Accepting an input artificial image, the input artificial image being an input artificial image; A recognition model that has been pre-trained to infer whether an input image is a real image or an artificial image calculates, using a gaze point visualization technique, a gaze area of the input artificial image that the recognition model gazes at during an inference process for determining whether the input artificial image is a real image or an artificial image; Based on a result of comparing a tendency of the gaze area of the input artificial image with a predetermined criterion, the gaze area of the input artificial image is corrected and the input artificial image is regenerated. A training image generation system comprising:
2. The training image generation system according to claim 1 , an input unit that receives an input real image and the input artificial image, a domain difference detection unit that calculates a first gaze area of the input real image and a second gaze area of the input artificial image obtained during the inference process from the input real image and the input artificial image by a gaze point visualization technology, and detects a difference in domain between the input real image and the input artificial image based on a result of comparing the second gaze area with the first gaze area as the predetermined reference; an image generating unit that regenerates the input artificial image by modifying a gaze area of the input artificial image based on the domain difference detected by the domain difference detecting unit; A training image generation system comprising:
3. The training image generation system according to claim 2, a correction method determination unit that determines whether or not the second gaze area needs to be corrected based on the difference between the domains, and determines a correction method for correcting the second gaze area when it is determined that the correction is necessary; The image generating unit includes: The second gaze area is modified into a plurality of patterns according to the modification method determined by the modification method determination unit, and a plurality of the input artificial images are regenerated. A training image generation system comprising:
4. The training image generation system according to claim 3, The domain difference detection unit a detection processing unit that detects an object region from each of the input real image and the input artificial image; a part division processing unit that divides each object region detected by the detection processing unit into object parts; a gaze point visualization processing unit that calculates the first gaze area and the second gaze area obtained during the inference process from the input real image and the input artificial image by a gaze point visualization technique; a domain difference detection processing unit that detects a difference in domain between the input real image and the input artificial image based on a result of comparing a tendency of the part to which the second gaze area corresponds, calculated by the gaze point visualization processing unit, with a tendency of the part to which the first gaze area corresponds; A training image generation system comprising:
5. The training image generation system according to claim 4, The input real image and the input artificial image are each a plurality of images, the tendency of the part corresponding to the first gaze region is a ratio of the input real images in which the part is a gaze part corresponding to the first gaze region to all of the plurality of input real images; The tendency of the portion corresponding to the second gaze region is a ratio of the input artificial images in which the portion is a gaze portion corresponding to the second gaze region to all of the plurality of input artificial images. A training image generation system comprising:
6. The training image generation system according to claim 3, The image generating unit includes: The second fixation region is modified according to the modification method determined by the modification method determination unit until the modification method determination unit determines that modification of the second fixation region is unnecessary, and the input artificial image is regenerated. A training image generation system comprising:
7. The training image generation system according to claim 5 , The domain difference detection processing unit includes: A process is performed for all of the input real images and all of the input artificial images, in which a gaze area calculated by the gaze point visualization processing unit includes a certain percentage or more of a part of the object detected by the detection processing unit, and the part is set as a gaze area. A first percentage of the input real images in which each part is set as a gaze area for all of the input real images and a second percentage of the input artificial images in which each part is set as a gaze area for all of the artificial images are calculated. If the second percentage is greater than the first percentage and the difference between the first percentage and the second percentage is equal to or greater than a threshold value, it is determined that the gaze area of the input artificial image needs to be corrected. A training image generation system comprising:
8. The training image generation system according to claim 2, The image generating unit includes: From the real image, a part corresponding to the fixation area of the artificial image to be corrected by the image generation unit and a position near the fixation area are extracted, and the input artificial image is regenerated by synthesizing the extracted part at the corresponding position of the artificial image. A training image generation system comprising:
9. The training image generation system according to any one of claims 4, 5 and 7, The artificial image is a CG (Computer Graphics) image, The object is a person. A training image generation system comprising:
10. A training image generation system according to any one of claims 1 to 9; a learning processing unit that learns the artificial image generated by the learning image generation system and generates a classifier; an inference processing unit that performs inference on an image using the classifier generated by the learning processing unit; 1. An image recognition system comprising:
11. A training image generation method performed by a training image generation system that generates training image data using an artificially generated image different from an actual image captured by an imaging device, comprising: Accepting an input artificial image, the input artificial image being an input artificial image; A recognition model that has been pre-trained to infer whether an input image is a real image or an artificial image calculates, using a gaze point visualization technique, a gaze area of the input artificial image that the recognition model gazes at during an inference process for determining whether the input artificial image is a real image or an artificial image; Based on a result of comparing a tendency of the gaze area of the input artificial image with a predetermined criterion, the gaze area of the input artificial image is corrected and the input artificial image is regenerated. A learning image generating method comprising each process.
12. A learning image generation program for causing a computer to function as a learning image generation system that generates learning image data using an artificially generated image that is different from an actual image captured by an imaging device, The computer includes: Accepting an input artificial image, the input artificial image being an input artificial image; A recognition model that has been pre-trained to infer whether an input image is a real image or an artificial image calculates, using a gaze point visualization technique, a gaze area of the input artificial image that the recognition model gazes at during an inference process for determining whether the input artificial image is a real image or an artificial image; Based on a result of comparing a tendency of the gaze area of the input artificial image with a predetermined criterion, the gaze area of the input artificial image is corrected and the input artificial image is regenerated. A learning image generating program that executes each process.
Citation Information
Patent Citations
Image processing system and method therefor
JP2005286395A
Learning device, method for learning, and program
JP2019046390A
Image generation device, image generator, image discriminator, image generation program, and image generation method
JP2019220133A
Learning method, learning device, program and recording medium
JP2020087310A
Learning device, estimation device, and learning method
WO2021130888A1
