Information processing device, learning device, image recognition device, information processing method, learning method, and image recognition method

Abstract

To provide an information processing device, a learning device, an image recognition device, an information processing method, a learning method and an image recognition method for improving detection accuracy of an object area in an image.SOLUTION: A learning data generation device 200 as an information processing device comprises: a synthesis unit 204 which generates a synthetic image obtained by synthesizing a second image in a closed area in a first image; and a generation unit 206 which generates learning data including a label showing a corresponding area corresponding to the closed area in the synthetic image and the synthetic image.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to learning techniques. [Background technology]

[0002] Research and development in the field of image recognition has made remarkable progress, and it is not uncommon for it to be used in various everyday tools. In particular, with the development of deep learning, multi-object detection, which simultaneously detects various types of objects contained in a captured image, has become possible. Non-Patent Documents 1, 2, and 3 all disclose methods for performing multi-object detection from images using deep learning.

[0003] As an application example, deep learning can be used to obtain information to control various camera functions. One of the camera's shooting functions is the autofocus (AF) function, which detects object areas near a selected area and automatically focuses on the target object based on that object area. Possible methods for selecting the area include user-initiated selection using a touch panel, or automatic detection using object detection technology. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation., Ross Girshick et al., 2014 [Non-patent document 2] SSD: Single Shot MultiBox Detector, Wei Liu et al., 2015 [Non-patent document 3] You Only Look Once: Unified, Real-Time Object Detection, Joseph Redmon et al., 2015 [Non-patent document 4] Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization, Jonathan Tremblay, et al., 2018 Summary of the Invention [Problem to be solved by the invention]

[0005] However, there are a wide variety of objects that can be the subject of a camera, and in multitask detection that detects unspecified objects, it is difficult to prepare training data that covers all object features.

[0006] When detecting object regions with limited training data, the contours created by textures may be mistakenly detected as object contours. One way to prevent such misdetections is to synthesize new training data.

[0007] Non-Patent Document 4 discloses a technique for improving the accuracy of object detection by synthesizing training data. However, while the technique disclosed in Non-Patent Document 4 can improve the detection accuracy for a specific object, it is difficult to train object features with few textures. The present invention provides a technique for improving the detection accuracy of object regions in images. [Means for solving the problem]

[0008] One aspect of the present invention is characterized by comprising a first generation means for generating a composite image by combining a second image with a closed region in a first image, and a second generation means for generating learning data including the composite image and a label indicating a corresponding region in the composite image that corresponds to the closed region. [Effects of the Invention]

[0009] According to the present invention, it is possible to improve the accuracy of detecting an object region in an image. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 2 is a block diagram showing an example of the hardware configuration of a training data generation device 200. [Figure 2] FIG. 2 is a block diagram showing an example of the functional configuration of a training data generation device 200. [Figure 3] FIG. 2 is a block diagram showing an example of the functional configuration of an image recognition device 300. [Figure 4] FIG. 4 is a block diagram showing an example of the functional configuration of a learning device 400. [Figure 5] 4 is a flowchart of processing performed by the training data generation device 200 to generate training data. [Figure 6] FIG. 6 is a diagram showing a captured image 601 and closed regions 603a and 603b. [Figure 7] 7 is a diagram showing an image 701 including a texture and a partial image 702 thereof. [Figure 8] FIG. 2 is a block diagram showing an example of the functional configuration of a determination unit 202. [Figure 9] 1A is a diagram showing an example of a composite image, and FIGS. 1B and 1C are diagrams showing examples of object regions output by a detection unit 302. FIG. [Figure 10] 10 is a flowchart of a learning process of the detection unit 302 by the learning device 400. [Figure 11] 10 is a flowchart of processing performed by the image recognition device 300 to detect an object region in an input image. [Figure 12] FIG. 12 is a block diagram showing an example of the functional configuration of an image recognition device 1200. [Figure 13] 13 is a diagram showing an input image 1301, a texture pattern 1302, a texture region 1303, an object region 1304, and an object region 1305. FIG. [Figure 14] 12 is a flowchart showing the operation of the image recognition device 1200 for detecting an object region from an input image. [Figure 15]FIG. 15 is a block diagram showing an example of the functional configuration of a learning device 1500. [Figure 16] 15 is a flowchart of the learning process of the texture generating unit 1502 and the texture identifying unit 1504. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] [First embodiment] In this embodiment, a learning data generation device is described, which is an example of an information processing device that generates a composite image by combining a second image with a closed area in a first image, and outputs data including the composite image and a label indicating a corresponding area in the composite image that corresponds to the closed area as learning data.

[0013] First, an example of the hardware configuration of the training data generation device 200 according to this embodiment will be described using the block diagram in Fig. 1. Note that the hardware configuration applicable to the training data generation device 200 is not limited to the configuration shown in Fig. 1, and can be changed / modified as appropriate.

[0014] CPU 101 executes various processes using computer programs and data stored in memory 102. As a result, CPU 101 controls the overall operation of training data generation device 200, and also executes or controls various processes that will be described as being performed by training data generation device 200.

[0015] The memory 102 has an area for storing computer programs and data loaded from the storage unit 104, and an area for storing data received from the outside via the communication unit 106. The memory 102 also has a work area used by the CPU 101 when executing various processes. In this way, the memory 102 can provide various areas as needed.

[0016] The input unit 103 is a user interface such as a keyboard, a mouse, and a touch panel screen, and allows the user to input various instructions to the CPU 101 by operating it.

[0017] The storage unit 104 is a large-capacity information storage device such as a hard disk drive. The storage unit 104 stores an OS (operating system), computer programs and data for causing the CPU 101 to execute or control various processes described as being performed by the training data generation device 200. The computer programs and data stored in the storage unit 104 are loaded into the memory 102 as appropriate under the control of the CPU 101, and are then processed by the CPU 101.

[0018] The display unit 105 is a display device having a liquid crystal screen or a touch panel screen, and displays the processing results of the CPU 101 as images and characters, and receives operation inputs (touch operations, swipes, etc.) from the user.

[0019] The communication unit 106 is a communication interface for performing data communication with an external device via a wired and / or wireless network such as a LAN or the Internet. The CPU 101, the memory 102, the input unit 103, the storage unit 104, the display unit 105, and the communication unit 106 are all connected to a system bus 107.

[0020] An example of the functional configuration of the training data generation device 200 is shown in the block diagram of FIG. 2. In this embodiment, each functional unit shown in FIG. 2 is implemented by a computer program. In the following description, the functional units in FIG. 2 will be described as the main processing units, but in reality, the functions of the functional units are performed by the CPU 101 executing the computer programs corresponding to the functional units. Note that the functional units shown in FIG. 2 may also be implemented by hardware. The processing performed by the training data generation device 200 to generate training data will be described with reference to the flowchart in FIG. 5.

[0021] In step S501, the acquisition unit 201 acquires a first image (background image). The first image may be, for example, a captured image 601 of a landscape as shown in FIG. 6(a), or an image obtained by combining a captured image with another image (such as a background image that does not actually exist or a CG image). The acquisition unit 201 may acquire such a first image from the storage unit 104, or may receive and acquire the image from an external device via the communication unit 106. The acquisition unit 201 may also process an acquired image and acquire it as the first image. In this way, the method for acquiring the first image is not limited to a specific acquisition method. The same applies to various images that will be described later.

[0022] In step S502, the acquisition unit 203 acquires a second image (texture image). The second image is an image that includes an appropriate texture. For example, the acquisition unit 203 may acquire an image 701 that includes a zebra with a striped texture as shown in FIG. 7 as the second image, or may acquire a partial image 702 that is obtained by cutting out an image area of ​​the texture portion of the image 701 as the second image.

[0023] In step S503, the determination unit 202 sets one or more closed regions on the first image. For example, as shown in Fig. 6(b), the determination unit 202 sets an elliptical closed region 603a and a pentagonal closed region 603 on the background image 601. As shown in Fig. 8, the determination unit 202 has one or more of a generation unit 801 and an acquisition unit 802.

[0024] The generating unit 801 generates a closed region using a geometric figure having a shape such as a circle, an ellipse, or a polygon, and sets the generated closed region at a position on the first image (for example, the position may be a predetermined position or a position specified by the user using the input unit 103). The generating unit 801 may set a two-dimensional projection region in which a virtual object (three-dimensional model) having a three-dimensional shape is projected onto the first image as the closed region. The generating unit 801 may also set a two-dimensional region specified on the first image by the user operating the input unit 103 as the closed region.

[0025] The acquisition unit 802 acquires the contour (shape) of an object included in the first image and sets a region surrounding the acquired contour as a closed region. Note that there are various methods for setting a closed region on the first image based on the contour (shape) of an object included in the first image, and the method is not limited to a specific method.

[0026] In either case, by making the closed area set in step S503 similar in shape to an object that does not belong to an easily obtainable object category, it is expected that the detection accuracy of objects that do not belong to an easily obtainable object category will be improved.

[0027] In step S504, the synthesis unit 204 generates a synthesized image by synthesizing the second image in a closed region on the first image.

[0028] For example, when one second image is acquired in step S502, the composition unit 204 cuts out a partial image of the same shape and size as the closed region from an appropriate position in the second image and composes the partial image with the closed region. When multiple closed regions are set in the first image, the second image can be composited with each closed region by performing the same process for each closed region.

[0029] Furthermore, for example, if two or more second images are acquired in step S502, the compositing unit 204 cuts out partial images of the same shape and size as the closed region from appropriate positions in some or all of the two or more second images, and generates a composite partial image by combining the partial images. The compositing unit 204 then combines the composite partial image with the closed region. If multiple closed regions are set in the first image, the same process can be performed for each closed region, allowing the second image to be composited with each closed region.

[0030] For example, when one second image is acquired in step S502, the composition unit 204 cuts out multiple partial images of the same shape and size as the closed region from the single second image and generates a composite partial image by combining the multiple cut-out partial images. The composition unit 204 then combines the composite partial images into the closed region. When multiple closed regions are set in the first image, the second image can be composited into each closed region by performing the same process for each closed region.

[0031] Fig. 9(a) shows an example of a composite image obtained by combining the image 701 of Fig. 7 with the closed regions 603a and 603b in the background image 601 of Fig. 6(b). A partial image cut out from an appropriate position in the image 701 to fit the size and shape of the closed region 603a is combined with the closed region 603a in the composite image 901. Also, a partial image cut out from an appropriate position in the image 701 to fit the size and shape of the closed region 603b is combined with the closed region 603b in the composite image 901.

[0032] The method for combining images is not limited to a specific method, and for example, the pixel values ​​in the combined image may be the logical sum of the pixel values ​​of the images to be combined, or they may be combined using a method such as alpha blending.

[0033] In step S505, the assigning unit 205 generates a label for instructing the detection unit 302, described below, to treat the closed region in the composite image obtained by combining the second image as a region of a single object to be detected (object region). For example, when the assigning unit 205 determines that the closed region is a region of the object to be detected, the assigning unit 205 assigns 1 as a label to a region that corresponds to the object region to be output by the detection unit 302, and assigns 0 to other regions.

[0034] For example, the object region output by the detection unit 302 to which the composite image 901 is input is a rectangular region 902a circumscribing the closed region 603a and a rectangular region 902b circumscribing the closed region 603b, as shown in Fig. 9(b). Also, for example, the object region output by the detection unit 302 to which the composite image 901 is input is a polygonal region 903a inscribing or circumscribing the closed region 603a and a polygonal region 903b circumscribing the closed region 603b, as shown in Fig. 9(c).

[0035] Therefore, the assigning unit 205 outputs "1" as a label corresponding to pixels that form corresponding regions (rectangular regions 902a and 902b and polygonal regions 903a and 903b in the example of FIG. 9) that correspond to the closed regions in the composite image. The assigning unit 205 also outputs "0" as a label corresponding to pixels that form regions other than the corresponding regions.

[0036] In step S506, the generation unit 206 generates training data 207 including a composite image and a label map configured with labels corresponding to each pixel in the composite image, and stores the generated training data 207 in the storage unit 104. Note that the output destination of the training data 207 is not limited to the storage unit 104, and the training data 207 may be output to a device with which the learning device 400 (described later) can communicate, or may be output directly to the learning device 400.

[0037] In step S507, CPU 101 determines whether a termination condition for generating learning data has been met. The termination condition for generating learning data is not limited to a specific condition. For example, CPU 101 determines that the termination condition has been met when a predetermined number of composite images and corresponding label maps have been generated.

[0038] If the end condition for the generation of learning data is satisfied as a result of such determination, the processing according to the flowchart in Fig. 5 ends. On the other hand, if the end condition for the generation of learning data is not satisfied, the processing proceeds to step S501.

[0039] Next, a description will be given of a learning device 400 that uses the training data generated in this manner to train the detection unit 302. In this embodiment, the hardware configuration of the learning device 400 is the same as that of the training data generation device 200, as shown in Fig. 1, but it may have a configuration different from that shown in Fig. 1.

[0040] That is, CPU 101 controls the overall operation of learning device 400 by executing various processes using computer programs and data stored in memory 102, and also executes or controls various processes described as being performed by learning device 400. Storage unit 104 stores an OS (operating system), computer programs and data for causing CPU 101 to execute or control various processes described as being performed by learning device 400, and the like. The rest of the configuration is the same as that of learning data generation device 200.

[0041] Next, an example of the functional configuration of the learning device 400 is shown in the block diagram of Fig. 4. The learning process of the detection unit 302 by the learning device 400 will be described with reference to the flowchart of Fig. 10. In step S1001, the acquisition unit 401 acquires the learning data 207 stored in the storage unit 104. Note that in step S1001, the acquisition unit 401 is not limited to acquiring only the learning data 207 generated by the learning data generation device, and may also acquire learning data generated by other devices.

[0042] In step S1002, the learning unit 402 trains the detection unit 302 using the training data 207 acquired by the acquisition unit 401. The detection unit 302 may be various types, such as a neural network such as a CNN (Covolutional Neural Network), a ViT (Vision Transformer), or an SVM (Support Vector Machine) combined with a feature extractor. For the purpose of specific explanation, this embodiment will describe a case where the detection unit 302 is a CNN.

[0043] The learning unit 402 inputs the composite image included in the training data 207 into the CNN and performs arithmetic processing in the CNN, thereby obtaining the detection result of the object region in the composite image as the output of the CNN. The learning unit 402 then calculates the error between the detection result of the object region in the composite image and the label included in the training data 207, and updates the parameters (weights, etc.) of the CNN so as to reduce the error, thereby training the detection unit 302.

[0044] In step S1003, the learning unit 402 determines whether a learning termination condition has been satisfied. The learning termination condition is not limited to a specific condition. For example, the learning unit 402 may determine that the learning termination condition has been satisfied when the error described above becomes less than a threshold value. Alternatively, for example, the learning unit 402 may determine that the learning termination condition has been satisfied when the difference (change in error) between the previously calculated error and the currently calculated error becomes less than a threshold value. Alternatively, for example, the learning unit 402 may determine that the learning termination condition has been satisfied when the number of times learning has been performed (the number of times steps S1001 and S1002 have been repeated) exceeds a threshold value.

[0045] If the learning termination condition is met as a result of such determination, the processing according to the flowchart in Fig. 10 ends. On the other hand, if the learning termination condition is not met, the processing proceeds to step S1001, and the subsequent processing is performed on the next learning data.

[0046] Next, we will explain the image recognition device 300 that detects an object region from an input image using the thus trained detection unit 302. In this embodiment, the hardware configuration of the image recognition device 300 is the same as that of the training data generation device 200, as shown in Fig. 1, but it may have a different configuration from that shown in Fig. 1.

[0047] That is, CPU 101 executes various processes using computer programs and data stored in memory 102. As a result, CPU 101 controls the overall operation of image recognition device 300, and executes or controls various processes described as being performed by image recognition device 300. Storage unit 104 stores an OS (operating system), computer programs and data for causing CPU 101 to execute or control various processes described as being performed by image recognition device 300, and the like. Other configurations are the same as those of training data generation device 200.

[0048] Such an image recognition device 300 can be applied to, for example, an object detection circuit for autofocus control in a photographing device such as a digital camera, or to a program that performs object detection for use in image processing in a tablet terminal such as a smartphone. In this way, the image recognition device 300 is not limited to a specific form.

[0049] An example of the functional configuration of the image recognition device 300 is shown in the block diagram of Fig. 3. The process performed by the image recognition device 300 to detect an object region in an input image using the detection unit 302 that has been trained by the learning device 400 will be described with reference to the flowchart of Fig. 11.

[0050] In step S1101, acquisition unit 301 acquires an input image that is the target of object detection. In step S1102, detection control unit 310 inputs the input image to detection unit 302 and performs calculation processing on detection unit 302, thereby acquiring the output of detection unit 302 for the input image, i.e., the detection result of the object region in the input image. The output map obtained by forward propagation of CNN, which is detection unit 302, corresponds to the "detection result of the object region in the input image." The "detection result of the object region in the input image" is the object region expressed by the coordinates and likelihood of the object in the input image. The "coordinates of the object in the input image" is position information on the input image specified by a rectangle, ellipse, etc., and in the case of a rectangle, it can be expressed by the center position of the rectangle and the size of the rectangle.

[0051] In step S1103, the output unit 303 outputs the "detection result of the object region in the input image" acquired in step S1102. The output destination of the "detection result of the object region in the input image" is not limited to a specific output destination. For example, the output unit 303 may display the input image on the display unit 105 and superimpose a frame of the object region having the position and size indicated by the "detection result of the object region in the input image" on the input image. The output unit 303 may further display the position and size indicated by the "detection result of the object region in the input image" as text on the display unit 105. The output unit 303 may also transmit the "detection result of the object region in the input image" to an external device via the communication unit 106. In addition, if the image recognition device 300 is a device incorporated in an image capture device, the output unit 303 may output the "detection result of the object region in the input image (in this case, the captured image captured by the image capture device)" to a control circuit such as the CPU 101. In this case, the control circuit can focus on and track an object in the object region having the position and size indicated by the "detection result of the object region in the input image."

[0052] <Advantages of the First Embodiment> The training data generated by the training data generation device 200 is training data that includes objects with shapes and textures that are not actually photographed. By using labels to teach that the contours created by the textures are not the contours of the objects, the accuracy of detecting the object regions of objects that are not actually photographed as training data can be improved. Therefore, the accuracy can be improved in multitask detection that detects the object regions of any objects. Furthermore, the effect of suppressing erroneous detection of part or all of the contours created by the patterns as the contours of the objects when detecting objects with regular textures can be expected.

[0053] [Second embodiment] In each of the following embodiments, including this embodiment, differences from the first embodiment will be described, and unless otherwise specified below, it is assumed that the embodiments are the same as the first embodiment. In this embodiment, in addition to detecting an object region, a specific texture pattern is also detected. An example of the functional configuration of an image recognition device 1200 according to this embodiment is shown in the block diagram of FIG. 12. In FIG. 12, functional units that perform the same operations as those shown in FIG. 3 are assigned the same reference numerals.

[0054] The detection control unit 1210 inputs the input image acquired by the acquisition unit 301 to the detection unit 1203 and operates the detection unit 1203. The detection unit 1203 detects a texture area in which a specified texture pattern exists from the input image.

[0055] Forming unit 1204 acquires the object region detection result by detection unit 302 and the texture region detection result by detection unit 1203, and forms a new object region in the input image based on the object region and the texture region. Output unit 303 outputs information indicating the object region formed by forming unit 1204 (for example, the position and size of the object region in the input image).

[0056] The generation of training data used for training of the detection unit 302 to realize such an operation differs from the first embodiment in the following respects: The training data generation device 200 executes the process according to the flowchart of Fig. 5, and in step S505, the following process is performed.

[0057] In step S505, the assigning unit 205 designates an area having texture (part or all of the closed area) in the closed area in the composite image obtained by combining the second image as a texture area, and generates a texture label for instructing the detection unit 1203, described below, about the texture area. For example, assume that the closed areas 603a and 603b in the composite image 901 in FIG. 9 are both configured with a single texture pattern. In this case, the assigning unit 205 outputs "1" as a texture label corresponding to each pixel constituting an area equivalent to the texture area to be output by the detection unit 1203 (e.g., rectangular areas 902a and 902b or polygonal areas 903a and 903b). The assigning unit 205 also outputs "0" as a texture label corresponding to each pixel constituting an area other than the area equivalent to the texture area to be output by the detection unit 1203 (e.g., rectangular areas 902a and 902b or polygonal areas 903a and 903b).

[0058] In step S506, the generation unit 206 generates training data 207 including a composite image, a label map composed of labels corresponding to each pixel in the composite image, and a texture label map composed of texture labels corresponding to each pixel in the composite image, and stores the generated training data 207 in the storage unit 104.

[0059] The learning device 400 uses the learning data generated in this way to train the detection unit 302 and the detection unit 1203, but differs from the first embodiment in the following respects: The learning device 400 executes the process according to the flowchart in Fig. 10, but performs the following process in step S1002.

[0060] In step S1002, the learning unit 402 uses the training data generated as described above to train the detection unit 302 in the same manner as in the first embodiment. Furthermore, the learning unit 402 also uses the training data generated as described above to train the detection unit 1203. The detection unit 1203 may also be a neural network such as a CNN, a ViT, or an SVM combined with a feature extractor. The learning unit 1203 is trained by teaching the learning unit 1203 an area (texture area) in the synthetic image with a texture label of "1" to the learning unit 1203, causing the learning unit 1203 to learn the texture pattern of the area, and then training the learning unit 1203 to detect an area with a texture pattern similar to the texture pattern of the area. If the learning unit 1203 is a neural network, the learning unit 1203 is trained by updating parameters such as weights. Techniques for training a detection unit to detect an area having a predetermined feature in an input image are well known, and therefore a description of such training will be omitted.

[0061] In this case, by performing training of the detection unit 1203 using a texture pattern that would be erroneously detected even by the detection unit 302 of the first embodiment, the detection unit 1203 can detect a texture area that can correct the detection result of the object area. By using the texture area detected by the detection unit 1203, it becomes possible to correct the object area detected by the detection unit 302 to a more accurate object area.

[0062] Next, the operation of image recognition device 1200, which detects an object region from an input image using detection unit 302 and detection unit 1203 obtained by such learning processing, will be described with reference to the flowchart in Fig. 14. In Fig. 14, the same processing steps as those shown in Fig. 11 are assigned the same step numbers.

[0063] In step S1100, the acquisition unit 301 acquires an input image that is a target for object detection. In step S1102, the detection control unit 310 inputs the input image to the detection unit 302 and performs arithmetic processing in the detection unit 302, thereby acquiring a detection result of an object region in the input image.

[0064] In step S1401, the detection control unit 1210 inputs an input image to the detection unit 1203 and operates the detection unit 1203 to detect a "texture area having a texture pattern similar to the texture pattern learned by the detection unit 1203" from the input image.

[0065] For example, assume that the detection unit 1203 is learning using texture pattern 1302 in Fig. 13. In this case, when an input image 1301 shown in Fig. 13 is input to the detection unit 1203, the detection unit 1203 detects a texture region 1303 in the input image 1301 that has a texture pattern similar to texture pattern 1302. The detection unit 1203 then outputs a map that indicates the position and likelihood of the texture region 1303 in the input image 1301.

[0066] In step S1402, the forming unit 1204 forms a new object region in the input image based on the detection result of the object region by the detection unit 302 and the detection result of the texture region by the detection unit 1203.

[0067] Here, we will explain an example of a method for forming a new object region by the forming unit 1204. Below, we will explain a case where the detection unit 302 detects one or more rectangular object regions from the input image, and the detection unit 1203 divides the input image into a plurality of rectangular regions (divides the input image into a plurality of rectangular regions in a grid pattern) and outputs the likelihood (a real number from 0 to 1) that each rectangular region belongs to a texture region.

[0068] In this case, for each object region, forming unit 1204 calculates the sum S of likelihoods corresponding to rectangular regions belonging to that object region. If the sum S calculated for that object region is relatively large compared to the size of the object region, forming unit 1204 determines that the object region contains more texture patterns. For example, if the area (number of pixels) of the object region is A, forming unit 1204 determines that an object region where S / A is equal to or greater than a threshold contains more texture patterns. In the example of FIG. 13, all object regions 1304 in the input image are object regions where "the sum S calculated for that object region is relatively large compared to the size of the object region."

[0069] 13, if object region 1305 surrounding object region 1304 has been detected, it is more likely that object region 1305 surrounding it will be a more accurate object detection result that surrounds the entire object than object region 1304, which may contain a large amount of texture patterns. Therefore, from among the object regions detected by detection unit 302, formation unit 1204 excludes any object region that corresponds to "a smaller object region among object regions that are in an inclusion relationship with other object regions," even if "the sum S calculated for the object region is relatively large compared to the size of the object region." Then, formation unit 1204 designates the object region remaining as a result of this exclusion as a "new object region," thereby outputting a more accurate object region that surrounds the entire target object.

[0070] If an object region (target) in which "the sum S calculated for the object region is relatively large compared to the size of the object region" is not an "object region in an inclusion relationship with other object regions," the forming unit 1204 determines the target as a "new object region." Then, the output unit 303 outputs information indicating the "new object region" formed by the forming unit 1204 (for example, the position and size of the object region in the input image).

[0071] In this embodiment, the detection unit 302 and the detection unit 1203 are separate detection units, but the detection unit 302 and the detection unit 1203 may be implemented in a single neural network by operating a single neural network while switching parameters.

[0072] <Effects of the second embodiment> This embodiment makes it possible to detect texture pattern regions similar to learned texture patterns separately from object regions. This has the effect of making it less likely that the contours created by the texture will be mistakenly detected as the contours of the object, even for objects with unknown shapes that have not been learned. Therefore, it is possible to achieve improved accuracy in multitask detection that detects the object region of any object.

[0073] [Third embodiment] In this embodiment, the acquisition unit 203 generates a plausible texture image as the second image. As shown in Fig. 15, the acquisition unit 203 according to this embodiment includes a texture generation unit 1502 that is trained to output a plausible texture image corresponding to a random number or a random number vector. This training is performed by a training device 1500. The training device 1500 will be described below.

[0074] In this embodiment, the hardware configuration of the learning device 1500 is the same as that of the learning data generation device 200, as shown in Fig. 1, but may be a configuration different from that shown in Fig. 1. That is, the CPU 101 executes various processes using computer programs and data stored in the memory 102, thereby controlling the overall operation of the learning device 1500 and executing or controlling the various processes described as being performed by the learning device 1500. The storage unit 104 stores an OS (operating system), computer programs and data for causing the CPU 101 to execute or control the various processes described as being performed by the learning device 1500, and the like. The other configurations are the same as those of the learning data generation device 200.

[0075] 15 shows an example of the functional configuration of the learning device 1500. In addition to training the texture generation unit 1502 as described above, the learning device 1500 also trains the texture classification unit 104. A generative adversarial network (GAN) is used for training in the learning device 1500. The texture generation unit 1502 corresponds to the generator, and the texture classification unit 1504 corresponds to the discriminator.

[0076] The learning process of the texture generation unit 1502 and texture classification unit 1504 in the learning device 1500 will be described with reference to the flowchart in Fig. 16. In step S1601, the random number generation unit 1501 generates one or more random numbers or random number vectors.

[0077] In step S1602, the texture generation unit 1502 generates and outputs a texture image 1503 from the random number or random number vector generated in step S1601. The texture generation unit 1502 is configured by a CNN or ViT, and receives the random number or random number vector as input, performs arithmetic processing, and outputs a texture image 1503. The texture image 1503 corresponds to, for example, the output map output from the CNN, and is an image having the same number of channels as the training data 207 or a one-channel grayscale image.

[0078] In step S1603, the acquisition unit 1505 acquires an actually captured texture image having texture features that the texture generation unit 1502 is to learn, and outputs the acquired texture image.

[0079] In step S1604, the texture classification unit 1504 acquires the texture image output from the texture generation unit 1502 and the real texture image output from the acquisition unit 1505. Like the texture generation unit 1502, the texture classification unit 1504 is configured by a CNN or ViT.

[0080] The learning device 1500 uses the learning device 400 (learning unit 402) to train the texture generating unit 1502 and the texture identifying unit 1504, and in step S1605, the learning process for the texture identifying unit 1504 is performed.

[0081] The learning data used in training of texture classification unit 1504 includes texture image 1503, a teacher value (first teacher value) indicating that texture image 1503 is an image generated by texture generation unit 1502, a real-life texture image acquired by acquisition unit 1505, and a teacher value (second teacher value) indicating that the real-life texture image is an image acquired by acquisition unit 1505. Texture classification unit 1504 is trained using such learning data. That is, learning device 400 inputs a texture image or a real-life texture image as an input image to texture classification unit 1504, and trains texture classification unit 1504 by using the teacher value (specified by the first teacher value and the second teacher value, which is 0 or 1) indicating whether the input image is a texture image or a real-life texture image as teacher data. Such training improves the accuracy with which texture classification unit 1504 distinguishes whether the input texture image is a texture image generated by texture generation unit 1502 or a real-life texture image.

[0082] In step S1606, learning device 1500 determines whether the processes of steps S1601 to S1605 have been repeated K times (K is an integer greater than or equal to 2). If the result of this determination is that the processes of steps S1601 to S1605 have been repeated K times, the process proceeds to step S1607. On the other hand, if the processes of steps S1601 to S1605 have not been repeated K times (K is an integer greater than or equal to 2), the process proceeds to step S1601.

[0083] In step S1607, the random number generation unit 1501 generates a random number or a random number vector equal to or greater than 1. In step S1608, the texture generation unit 1502 generates and outputs a texture image 1503 from the random number or the random number vector generated in step S1607, in the same manner as in step S1602 above.

[0084] In step S1609, the texture classification unit 1504 receives the texture image 1503 output from the texture generation unit 1502 and performs calculations. As a result, the texture classification unit 1504 acquires a classification result indicating whether the texture image 1503 is an image generated by the texture generation unit 1502 or an actual texture image acquired by the acquisition unit 1505. For example, if the texture classification unit 1504 has determined that the texture image 1503 is an image generated by the texture generation unit 1502, it outputs "1" as the classification result. On the other hand, if the texture classification unit 1504 has determined that the texture image 1503 is an actual texture image acquired by the acquisition unit 1505, it outputs "0" as the classification result.

[0085] Then, in step S1610, the learning device 1500 performs a learning process on the texture generation unit 1502 using the above-mentioned learning device 400 (learning unit 402). The learning data used for learning the texture generation unit 1502 includes the random numbers or random number vectors generated in step S1607 and the classification result in step S1609. The texture generation unit 1502 is trained using such learning data. In other words, the learning device 400 trains the texture generation unit 1502 so that the classification result of the texture classification unit 1504 for the texture image generated by the texture generation unit 1502 based on the random numbers or random number vectors is a "real texture image." Through such learning, the texture generation unit 1502 learns to generate a texture image 1503 that the texture classification unit 1504 will mistakenly classify as a real texture image.

[0086] In step S1611, the learning device 1500 determines whether or not the termination condition (learning termination condition) for the processes in the above steps S1601 to S1610 has been satisfied. The learning termination condition is not limited to a specific condition, similar to the "learning termination condition" described in the first embodiment.

[0087] If the learning end condition is met as a result of such determination, the processing according to the flowchart in Fig. 16 ends. On the other hand, if the learning end condition is not met, the processing proceeds to step S1601.

[0088] When the processing according to the flowchart in FIG. 16 is completed, the texture generating unit 1502 can generate a plausible texture image 1503 corresponding to the given random number or random number vector.

[0089] <Advantages of the third embodiment> The acquisition unit 203 having such a trained texture generation unit 1502 can obtain new texture images having the characteristics of texture images, not limited to texture images actually captured. The training data generated by the training data generation device 200 can teach the detection unit 302 a wider variety of textures. This reduces the probability that contours created by a wider variety of textures will be erroneously detected as contours of an object when the detection unit 302 is trained. This therefore has the effect of improving the detection accuracy of the image recognition device 200.

[0090] The numerical values, processing timing, processing order, processing subject, data (information) structure / destination / source / storage location, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to these examples.

[0091] In addition, some or all of the above-described embodiments may be used in appropriate combination, and some or all of the above-described embodiments may be selectively used.

[0092] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0093] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0094] 201: Acquisition unit 202: Determination unit 203: Acquisition unit 204: Synthesis unit 205: Addition unit 206: Generation unit

Claims

1. a first generating means for generating a composite image by combining a second image having texture in a closed region of a first image; a second generation means for generating learning data including a label indicating a corresponding region in the composite image that corresponds to the closed region, and the composite image; An information processing device comprising:

2. 2. The information processing device according to claim 1, wherein the first generating means generates a closed area using a geometric figure, sets the generated closed area on the first image, and generates a composite image by combining the second image with the closed area.

3. 2 . The information processing apparatus according to claim 1 , wherein the first generating means generates a composite image by combining the second image with a two-dimensional projection area in which a virtual object having a three-dimensional shape is projected onto the first image.

4. The information processing apparatus according to claim 1 , wherein the first generating means generates a composite image by combining the second image with a closed region set in the first image in response to a user operation.

5. 2 . The information processing apparatus according to claim 1 , wherein the first generating means generates a composite image by combining the second image with a closed region that surrounds a contour of an object in the first image.

6. 6. The information processing apparatus according to claim 1, wherein the first generating means generates a composite image by combining the second image with each closed region in the first image.

7. 7. The information processing apparatus according to claim 1, wherein the first generating means generates a composite image by combining a plurality of the second images in a closed region in the first image.

8. 8. The information processing apparatus according to claim 1, wherein the second generation means generates learning data including the label, the synthetic image, and a texture label indicating an area having texture in the closed area of ​​the synthetic image.

9. moreover, An acquisition means for acquiring the second image, The information processing apparatus according to claim 1 , wherein the acquisition means acquires a texture image as the second image by using a generative adversarial network that generates a texture image.

10. The information processing device according to claim 9, characterized in that the acquisition means acquires as the second image a texture image generated by a generation unit that has been trained so that a texture image generated according to a random number or a random number vector can be identified as an actual texture image.

11. A learning device comprising: a learning means for learning a detection unit that detects an object region from an input image using a synthetic image contained in the learning data generated by the second generation means of the information processing device described in any one of claims 1 to 10 and a label contained in the learning data.

12. 12. An image recognition device comprising: a detection unit that detects an object region from an input image using a detection unit that has been trained by the learning device according to claim 11.

13. 9. A learning device comprising: a learning means for learning a first detection unit that detects an object region from an input image and a second detection unit that detects a region having texture from the input image using a synthetic image included in the learning data generated by the second generation means of the information processing device described in claim 8, a label included in the learning data, and a texture label included in the learning data.

14. An image recognition device comprising a formation means for forming a new object region using an object region detected from an input image using a first detection unit trained by the learning device described in claim 13, and a texture region detected from an input image using a second detection unit trained by the learning device.

15. An information processing method performed by an information processing device, a first generation step in which a first generation means of the information processing device generates a composite image by combining a first image with a second image having texture in a closed region of the first image; a second generation step in which a second generation means of the information processing device generates learning data including a label indicating a corresponding region in the composite image that corresponds to the closed region, and the composite image; An information processing method comprising:

16. A learning method performed by a learning device, comprising: A learning method characterized in that the learning means of the learning device includes a learning step of training a detection unit that detects an object region from an input image using a synthetic image contained in the learning data generated in the second generation step in the information processing method described in claim 15 and a label contained in the learning data.

17. An image recognition method performed by an image recognition device, 17. An image recognition method comprising: a detection step in which a detection means of the image recognition device detects an object region from an input image using a detection unit trained by the learning method according to claim 16.

18. A learning method performed by a learning device, comprising:

9. A learning method characterized in that the learning means of the learning device includes a learning step of training a first detection unit that detects an object area from an input image and a second detection unit that detects an area having texture from the input image using a synthetic image included in the training data generated by the second generation means of the information processing device described in claim 8, a label included in the training data, and a texture label included in the training data.

19. An image recognition method performed by an image recognition device, An image recognition method characterized in that the formation means of the image recognition device includes a formation step of forming a new object region using an object region detected from an input image using a first detection unit trained by the learning method described in claim 18, and a texture region detected from an input image using a second detection unit trained by the learning device.

20. A computer program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 10.

21. A computer program for causing a computer to function as the learning means of the learning device according to claim 11 or 13.

22. A computer program for causing a computer to function as each of the means of the image recognition device according to claim 12 or 14.

Images