Learning program, learning method, and information processing device

Abstract

To provide a learning program for detecting an object that does not exist in a background image but exists in a target image, a learning method, and an information processing apparatus.SOLUTION: An information processing apparatus includes: an acquisition unit configured to acquire training data that uses a background image obtained by imaging an area where an object is to be arranged and a target image obtained by imaging the object and the area as input data and uses information associated with the object that does not exist in the background image but exists in the target image as ground truth; and a learning processing unit configured to execute machine learning for a machine learning model on the basis of a result output from the machine learning model when the target image and the target image are input to the machine learning model, based on the training data, and the ground truth.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to a learning program and the like. [Background technology]

[0002] Self-checkout registers are becoming common in supermarkets, convenience stores, and other stores. A self-checkout register is a Point of Sale (POS) cash register system in which the user himself performs the entire process, from scanning the product's barcode to paying. For example, introducing self-checkout registers can reduce labor costs and prevent payment errors by store clerks.

[0003] On the other hand, at self-checkout registers, it is necessary to detect user fraud, such as not reading the barcode.

[0004] FIG. 21 is a diagram illustrating a conventional technique. In the example shown in FIG. 21, a user 1 picks up a product 2 placed on a temporary table 6, scans the barcode of the product 2 at a self-checkout register 5, and then wraps the product. In the conventional technique, image data from a camera 10 is analyzed to perform object detection of the products placed on the temporary table 6 and identify the number of products. By checking whether the identified number of products matches the number actually scanned, fraud can be detected. In the conventional technique, technology such as deep learning (hereinafter referred to as DL) is used to perform object detection as described in FIG. 21.

[0005] When performing object detection using DL, a large amount of label data is manually prepared, and this label data is used to perform machine learning on an object detection model for object detection. However, since the object detection model only detects objects that it has previously trained on, it is not realistic to repeatedly prepare label data and perform machine learning on the object detection model again in a store like the one described above, where there is a huge variety of products and products change daily.

[0006] There is a conventional technique that can identify the area of ​​an unknown object that has not been learned in advance. Fig. 22 is a diagram illustrating a conventional technique for identifying the area of ​​an unknown object. In the conventional technique described in Fig. 22, a DNN (Deep Neural Network) 20 that outputs a mask image 16 showing an area different from the background from a background image 15a and a target image 15b is acquired by machine learning using a large number of various images.

[0007] Although the photographed area of ​​background image 15a and the photographed area of ​​target image 15b are the same photographed area, background image 15a does not include objects 3a, 3b, and 3c that exist in target image 15b. Mask image 16 shows areas 4a, 4b, and 4c corresponding to objects 3a, 3b, and 3c. For example, "1" is set to the pixels of areas 4a, 4b, and 4c of mask image 16, and "0" is set to the pixels of other areas.

[0008] The conventional technology in Fig. 22 identifies an area that is different from the background, so it can identify the area even if it is an unknown object. Therefore, by applying the conventional technology in Fig. 22 to the conventional technology in Fig. 21, it is possible to identify the area of ​​an unknown product and determine the number of products. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Japanese Patent Application Publication No. 2019-153057 Summary of the Invention [Problem to be solved by the invention]

[0010] However, the above-mentioned conventional techniques have a problem in that they are unable to detect an object that exists in the target image but does not exist in the background image.

[0011] The conventional technology described in Fig. 22 is a technology for identifying the entire area that is different from the background, but it is not a technology for identifying the position, size, etc. of individual objects. Therefore, when multiple objects are close to each other, they become one mass.

[0012] FIG. 23 is a diagram illustrating the problems with the conventional technology. In the example shown in FIG. 23, when a background image 17a and a target image 17b are input to a machine-learned DNN 20, a mask image 18 is output. Target image 17b includes products 7a, 7b, and 7c. Because products 7a to 7c in target image 17b are close to each other, a single block region 8 is shown in mask image 18. It is difficult to identify the region corresponding to products 7a, 7b, and 7c and to identify the product number "3" based on region 8 in mask image 18.

[0013] In one aspect, the present invention aims to provide a learning program, a learning method, and an information processing device that can provide a model that can detect objects that exist in a target image but do not exist in a background image. [Means for solving the problem]

[0014] In the first proposal, the computer is made to execute the following process: The computer receives as input data a background image of an area where an object is to be placed and a target image of the object and the area, and acquires training data in which information related to an object that does not exist in the background image but does exist in the target image is used as correct answer data. The computer executes machine learning on the machine learning model based on the training data and the results output from the machine learning model when the target image and the target image are input to the machine learning model, and the correct answer data. [Effects of the Invention]

[0015] It is possible to provide a model that can detect objects that exist in the target image but do not exist in the background image. [Brief explanation of the drawings]

[0016] [Figure 1]FIG. 1 is a diagram illustrating a system according to the first embodiment. [Figure 2] FIG. 2 is a diagram illustrating the basic processing of the information processing device according to the first embodiment. [Figure 3] FIG. 3 is a diagram for explaining the characteristic process 1. In FIG. [Figure 4] FIG. 4 is a diagram for providing a supplementary explanation of the characteristic process 1. [Figure 5] FIG. 5 is a diagram for providing a supplementary explanation of the characteristic process 2. [Figure 6] FIG. 6 is a functional block diagram illustrating a configuration of the information processing device according to the first embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of a data structure of the learning data according to the first embodiment. [Figure 8] FIG. 8 is a diagram illustrating the process of the learning processing unit according to the first embodiment. [Figure 9] FIG. 9 is a diagram illustrating the processing of the inference processing unit according to the first embodiment. [Figure 10] FIG. 10 is a flowchart (1) showing the procedure of the learning process according to the first embodiment. [Figure 11] FIG. 11 is a flowchart (2) illustrating the processing procedure of the learning process according to the first embodiment. [Figure 12] FIG. 12 is a flowchart of the inference process according to the first embodiment. [Figure 13] FIG. 13 is a functional block diagram illustrating a configuration of an information processing device according to the second embodiment. [Figure 14] FIG. 14 is a diagram illustrating an example of a data structure of learning data according to the second embodiment. [Figure 15] FIG. 15 is a diagram illustrating the process of the learning processing unit according to the second embodiment. [Figure 16] FIG. 16 is a diagram illustrating the process of the inference processing unit according to the second embodiment. [Figure 17] FIG. 17 is a flowchart illustrating a processing procedure of the learning process according to the second embodiment. [Figure 18]FIG. 18 is a flowchart illustrating the procedure of the inference process according to the second embodiment. [Figure 19] FIG. 19 is a diagram illustrating a system according to the third embodiment. [Figure 20] FIG. 20 is a diagram illustrating an example of a hardware configuration of a computer that realizes the same functions as the information processing device. [Figure 21] FIG. 21 is a diagram for explaining the prior art. [Figure 22] FIG. 22 is a diagram for explaining a conventional technique for identifying an area of ​​an unknown object. [Figure 23] FIG. 23 is a diagram for explaining the problems of the conventional technology. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, embodiments of the learning program, learning method, and information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to these embodiments. [Example]

[0018] An example of a system according to the first embodiment will be described. FIG. 1 is a diagram showing the system according to the first embodiment. As shown in FIG. 1, the system includes a camera 10 and an information processing device 100. The camera 10 and the information processing device 100 are connected to each other via a network 11. The camera 10 and the information processing device 100 may be directly connected by wire or wirelessly.

[0019] The camera 10 may be a camera that captures images of the inside and outside of the store, or may be a camera that captures images of the temporary placement table 6 on which the products are placed, as shown in Fig. 21. The camera 10 transmits image data of the imaging range to the information processing device 100.

[0020] In the following description, image data captured by camera 10 that does not include an object to be detected will be referred to as "background image data." For example, background image data corresponds to background image 15a described in FIG. 22. Image data captured by camera 10 that includes an object to be detected will be referred to as "target image data." For example, target image data corresponds to target image 15b described in FIG. 22. Target image 15b includes objects 3a to 3c. The shooting area of ​​background image 15a is the same as the shooting area of ​​target image 15b, but background image 15a does not include objects 3a to 3c.

[0021] The information processing device 100 is a device that infers the area of ​​each object included in the target image data based on the background image data and the target image data. Before starting the inference, the information processing device 100 receives the background image data from the camera 10 in advance, and when starting the inference, it sequentially receives the target image data from the camera 10.

[0022] The following describes the basic processing of the information processing device 100, and characteristic processing 1 and 2 that are added to the basic processing.

[0023] 2 is a diagram illustrating the basic processing of the information processing device according to the present embodiment 1. As shown in FIG. 2, the information processing device 100 that executes the basic processing includes feature extraction units 50a and 50b, a synthesis unit 51a, and an estimation unit 52.

[0024] The feature extraction units 50a and 50b correspond to a general convolutional neural network (CNN). When the background image data 25a is input, the feature extraction unit 50a outputs image features to the synthesis unit 51a based on parameters trained by machine learning. When the target image data 25b is input, the feature extraction unit 50b outputs image features to the synthesis unit 51a based on parameters trained by machine learning.

[0025] The image feature amounts output from the feature extraction units 50a and 50b are values ​​before being converted into probability values ​​based on a softmax function or the like. In the following description, the image feature amount of the background image data 25a will be referred to as the "background image feature amount." The image feature amount of the target image data 25b will be referred to as the "target image feature amount." The background image feature amount and the target image feature amount correspond to intermediate feature amounts.

[0026] The same parameters are set for the feature extraction units 50a and 50b. For convenience of explanation, the feature extraction units 50a and 50b are shown separately in Fig. 2, but the feature extraction units 50a and 50b are the same CNN.

[0027] The synthesis unit 51 a synthesizes the background image feature amount and the target image feature amount, and outputs the synthesized feature amount to the estimation unit 52 .

[0028] The estimation unit 52 corresponds to a general convolutional neural network (CNN). When a feature obtained by combining a background image feature and a target image feature is input, the estimation unit 52 identifies a BBOX for each object based on parameters trained by machine learning. For example, a BBOX (Bounding Box) is area information surrounding an object and has position and size information. In the example shown in FIG. 2, three BBOXes 30a, 30b, and 30c are identified.

[0029] Next, a description will be given of "characteristic processing 1" that is added to the processing of the basic part of the information processing device 100 shown in Fig. 2. Fig. 3 is a diagram for explaining characteristic processing 1. In Fig. 3, in addition to the feature extraction units 50a and 50b, the synthesis unit 51a, and the estimation unit 52 described in Fig. 2, a position coordinate feature output unit 53 and a synthesis unit 51b are provided.

[0030] The description of the feature extraction units 50a and 50b is the same as the description of the feature extraction units 50a and 50b described with reference to FIG.

[0031] The synthesis unit 51a synthesizes the background image feature amount and the target image feature amount, and outputs the synthesized feature amount to the synthesis unit 51b.

[0032] The position coordinate feature output unit 53 outputs a plurality of coordinate feature values ​​arranged on an image plane. For example, as shown in Fig. 3, the position coordinate feature output unit 53 outputs an x-coordinate feature value 53a, a y-coordinate feature value 53b, and a distance feature value 53c to the synthesis unit 51b.

[0033] For each pixel of the x-coordinate feature 53a, a coordinate value ranging from "-1" to "+1" is set in ascending order from left to right in the row direction. The same coordinate value is set for each pixel in the column direction. For example, "-1" is set for each pixel in the leftmost column of the x-coordinate feature 53a.

[0034] Coordinate values ​​ranging from "-1" to "+1" are set in ascending order for each pixel in the y-coordinate feature 53b in the column direction from top to bottom. The same coordinate value is set for each pixel in the row direction. For example, "-1" is set for each pixel in the top row of the y-coordinate feature 53b.

[0035] Distance feature 53c is set with coordinate values ​​ranging from "0" to "+1" in ascending order from the central pixel outward. For example, "0" is set to the central pixel of distance feature 53c.

[0036] The synthesis unit 51b outputs to the estimation unit 52 information obtained by synthesizing the background image feature amount, the target image feature amount, the x-coordinate feature amount 53a, the y-coordinate feature amount 53b, and the distance feature amount 53c.

[0037] When information combining background image features, target image features, x-coordinate features 53a, y-coordinate features 53b, and distance features 53c is input, the estimation unit 52 identifies the BBOX of each object based on parameters trained by machine learning.

[0038] FIG. 4 is a diagram for providing additional explanation of characteristic process 1. For example, assume that image 21 shown in FIG. 4 is subjected to convolution using a neural network. In normal convolution processes, position is invariant, making it difficult to distinguish objects with the same appearance as separate objects. For example, objects 22 and 23 included in image 21 have the same appearance. Therefore, the result 22b obtained by performing convolution process on region 22a and the result 23b obtained by performing convolution process on region 23a will be the same.

[0039] In contrast, in Characteristic Process 1 described with reference to FIG. 3, convolution is performed on information obtained by combining an image feature with an x-coordinate feature 53a, a y-coordinate feature 53b, and a distance feature 53c. For example, when convolution is performed on region 22a, regions 53a-1, 53b-1, and 53c-1 are also convolution-performed. Similarly, when convolution is performed on region 23a, regions 53a-2, 53b-2, and 53c-2 are also convolution-performed. As a result, a result 22b obtained by performing convolution processing on region 22a and a result 23b obtained by performing convolution processing on region 23a are not identical, and it is possible to distinguish between objects 22 and 23.

[0040] Next, a description will be given of "characteristic processing 2" that is added to the processing of the basic part of the information processing device 100 shown in Fig. 2. Fig. 5 is a diagram for explaining characteristic processing 2. In Fig. 5, in addition to the feature extraction units 50a and 50b, synthesis units 51a and 51b, estimation unit 52, and position coordinate feature output unit 53 described in Fig. 3, a mask generation unit 54 is provided.

[0041] The description of the feature extraction units 50a and 50b and the position coordinate feature output unit 53 is the same as that given in FIGS.

[0042] The synthesis unit 51a synthesizes the background image feature amount and the target image feature amount, and outputs the synthesized feature amount to the synthesis unit 51b and the mask generation unit .

[0043] The mask generation unit 54 corresponds to a general convolutional neural network (CNN). When a feature obtained by combining background image feature amounts and target image feature amounts is input, the mask generation unit 54 generates a mask image 40 based on parameters trained by machine learning. The mask image 40 is information indicating an object region that does not exist in the background image data 25a but exists in the target image data 25b. For example, the mask image 40 is a bitmap, in which "1" is set to pixels corresponding to the object region and "0" is set to pixels corresponding to other regions.

[0044] The synthesis unit 51b outputs synthesis information 45 obtained by synthesizing the background image feature, the target image feature, the x-coordinate feature 53a, the y-coordinate feature 53b, the distance feature 53c, and the mask image 40 to the estimation unit 52.

[0045] When the synthesis information 45 is input, the estimation unit 52 identifies a BBOX of each object based on parameters trained by machine learning. For example, the synthesis information 45 is information in which a background image feature, a target image feature, an x-coordinate feature 53a, a y-coordinate feature 53b, a distance feature 53c, and a mask image 40 are superimposed. The estimation unit 52 places a kernel with set parameters on the synthesis information 45 in which each piece of information is superimposed, and performs convolution while moving the position of the kernel.

[0046] Here, a supplementary explanation will be given regarding Characteristic Process 2. For example, assuming machine learning without using the mask generation unit 54, machine learning is performed using training data in which background image data for training and target image data for training are used as input data, and the BBOX of an object included in the target image data for training is used as ground truth (GT).

[0047] When this type of machine learning is performed, the features of individual objects contained in the target image data may be memorized, and the BBOX of the object may be estimated from the target image data alone, without using background image data. In other words, the objects contained in the target image data used for learning are memorized as is, and unknown objects cannot be handled, which can be said to be overfitting.

[0048] To prevent the above-mentioned overfitting, tasks that cannot be solved without using background image data are machine-learned as auxiliary tasks, so that the NN can utilize the background image. For example, the machine learning process for the mask generation unit 54 shown in Figure 5 is an auxiliary task. For example, the estimation of the BBOX described above is the main task, and the task of generating a mask image is an auxiliary task.

[0049] Furthermore, the mask image 40 generated by the mask generation unit 54 is further input to the estimation unit 52, and machine learning is performed to estimate the BBOX of the object. This is expected to have the effect of limiting the object to be detected to the object region of the mask image.

[0050] In FIG. 5, the information processing device 100 inputs input data to feature extraction units 50a and 50b, and trains parameters of the feature extraction units 50a and 50b, the estimation unit 52, and the mask generation unit 54 so that the error between the BBOX output by the estimation unit 52 and the correct data (the correct value of the BBOX) and the error between the mask image output by the mask generation unit 54 and the correct data (the correct value of the mask image) are reduced.

[0051] Next, an example of the configuration of the information processing device 100 that executes the processes described with reference to Fig. 2 to Fig. 4 will be described. Fig. 6 is a functional block diagram showing the configuration of the information processing device according to the first embodiment. As shown in Fig. 6, the information processing device 100 includes a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0052] The communication unit 110 executes data communication with the camera 10 and an external device (not shown). For example, the communication unit 110 receives image data (background image data, target image data) from the camera 10. The communication unit 110 receives learning data 141 used for machine learning and the like from the external device.

[0053] The input unit 120 corresponds to an input device for inputting various types of information to the information processing device 100 .

[0054] The display unit 130 displays the output results from the control unit 150 .

[0055] The storage unit 140 includes training data 141, an image table 142, a feature extraction model 143, a generative model 144, and an estimation model 145. The storage unit 140 corresponds to a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as an HDD (Hard Disk Drive).

[0056] The learning data 141 is data used when machine learning is performed. FIG. 7 is a diagram illustrating an example of a data structure of the learning data according to the first embodiment. As illustrated in FIG. 7, the learning data 141 stores item numbers, input data, and correct answer data (GT) in association with each other. The input data includes background image data for learning and target image data for learning. The correct answer data includes the GT of the mask image and the GT of the BBOX (coordinates of the object area).

[0057] The image table 142 is a table that holds background image data and target image data used during inference.

[0058] The feature extraction model 143 is a machine learning model (CNN) executed by the feature extraction units 50a and 50b. When image data is input to the feature extraction model 143, an image feature amount is output.

[0059] The generative model 144 is a machine learning model (CNN) executed by the mask generating unit 54. When information obtained by combining background image features and target image features is input to the generative model 144, a mask image is output.

[0060] The estimation model 145 is a machine learning model (CNN) executed by the estimation unit 52. When the synthesis information 45 is input to the estimation model 145, a BBOX is output.

[0061] The control unit 150 includes an acquisition unit 151, a learning processing unit 152, and an inference processing unit 153. The control unit 150 corresponds to a CPU (Central Processing Unit) or the like.

[0062] When acquiring the learning data 141 from an external device or the like, the acquiring unit 151 registers the acquired learning data 141 in the storage unit 140.

[0063] The acquisition unit 151 acquires background image data from the camera 10 in advance and registers it in the image table 142. The acquisition unit 151 acquires target image data from the camera 10 and registers it in the image table 142.

[0064] The learning processing unit 152 executes machine learning of the feature extraction units 50a and 50b (feature extraction model 143), the mask generation unit 54 (generation model 144), and the estimation unit 52 (estimation model 145) based on the learning data 141.

[0065] 8 is a diagram illustrating the processing of the learning processing unit according to the first embodiment. For example, the learning processing unit 152 includes feature extraction units 50a and 50b, synthesis units 51a and 52b, an estimation unit 52, a mask generation unit 54, and a position coordinate feature output unit 53. The learning processing unit 152 also includes error calculation units 60a and 60b, a synthesis unit 61, and a weight update value calculation unit 62. In the following description, the feature extraction units 50a and 50b, the estimation unit 52, the position coordinate feature output unit 53, and the mask generation unit 54 will be collectively referred to as a "neural network" as appropriate.

[0066] The processing of the feature extraction units 50a and 50b is the same as that described in Fig. 5. For example, the feature extraction units 50a and 50b read and execute the feature extraction model 143. The feature extraction units 50a and 50b input image data to the feature extraction model 143 and calculate image feature amounts based on the parameters of the feature extraction model 143.

[0067] The explanation of the synthesis units 51a and 51b is the same as that given in FIG.

[0068] The processing of the position coordinate feature amount output unit 53 is the same as that explained in FIG.

[0069] The processing of the mask generation unit 54 is the same as that described in Fig. 5. For example, the mask generation unit 54 reads and executes the generative model 144. The mask generation unit 54 inputs a feature obtained by combining a background image feature and a target image feature into the generative model 144, and generates a mask image based on the parameters of the generative model 144. The mask generation unit 54 outputs the mask image to the synthesis unit 51b and the error calculation unit 60a.

[0070] The processing of the estimation unit 52 is the same as that described in FIG. 5. For example, the estimation unit 52 reads and executes the estimation model 145. The estimation unit 52 reads and executes the estimation model 145. The estimation unit 52 inputs synthesis information to the estimation model 145 and identifies a BBOX for each object based on parameters of the estimation model 145. The estimation model 145 outputs the BBOX to the error calculation unit 60b.

[0071] The learning processing unit 152 acquires background image data 26a for learning from the learning data 141 and inputs it to the feature extraction unit 50a. The learning processing unit 152 acquires target image data 26b for learning from the learning data 141 and inputs it to the feature extraction unit 50b. The learning processing unit 152 also acquires the GT of the mask image from the learning data 141 and inputs it to the error calculation unit 60a. The learning processing unit 152 acquires the GT of the BBOX from the learning data 141 and inputs it to the error calculation unit 60b.

[0072] The error calculation unit 60a calculates the error between the mask image 41 output from the mask generation unit 54 and the GT of the mask image of the training data 141. In the following description, the error between the mask image 41 and the GT of the mask image is referred to as a “first error.” The error calculation unit 60a outputs the first error to the synthesis unit 61.

[0073] The error calculation unit 60b calculates the error between the BBOX output from the estimation unit 52 and the GT of the BBOX in the training data 141. In the following description, the error between the BBOX output from the estimation unit 52 and the GT of the BBOX in the training data 141 is referred to as a “second error.” The error calculation unit 60b outputs the second error to the synthesis unit 61.

[0074] The combiner 61 calculates the sum of the first error and the second error. In the following description, the sum of the first error and the second error will be referred to as the “total error.” The combiner 61 outputs the sum to the weight update value calculator 62.

[0075] The weight update value calculation unit 62 updates the parameters (weights) of the neural network so as to reduce the total error. For example, the weight update value calculation unit 62 updates the parameters of the feature extraction units 50a and 50b (feature extraction model 143), the mask generation unit 54 (generation model 144), and the estimation unit 52 (estimation model 145) using an error backpropagation method or the like.

[0076] The learning processing unit 152 repeatedly executes the above process using each input data and correct answer data stored in the learning data 141. The learning processing unit 152 registers the machine-learned feature extraction model 143, the generation model 144, and the estimation model 145 in the storage unit 140.

[0077] Returning to the explanation of Fig. 6, the inference processing unit 153 uses the machine-learned feature extraction units 50a and 50b (feature extraction model 143), the mask generation unit 54 (generation model 144), and the estimation unit 52 (estimation model 145) to identify the area of ​​an object that is present in the target image data but not in the background image data.

[0078] 9 is a diagram for explaining the processing of the inference processing unit according to the present embodiment 1. For example, the inference processing unit 153 includes feature extraction units 50a and 50b, synthesis units 51a and 52b, an estimation unit 52, a mask generation unit 54, and a position coordinate feature output unit 53.

[0079] The processing of the feature extraction units 50a and 50b is the same as that described in Fig. 5. For example, the feature extraction units 50a and 50b read and execute the machine-learned feature extraction model 143. The feature extraction units 50a and 50b input image data to the feature extraction model 143 and calculate image features based on the parameters of the feature extraction model 143.

[0080] The explanation of the synthesis units 51a and 51b is the same as that given in FIG.

[0081] The processing of the position coordinate feature amount output unit 53 is the same as that explained in FIG.

[0082] The processing of the mask generation unit 54 is the same as that described in Fig. 5. For example, the mask generation unit 54 reads and executes the machine-learned generative model 144. The mask generation unit 54 inputs a feature obtained by combining a background image feature and a target image feature into the generative model 144, and generates a mask image based on the parameters of the generative model 144. The mask generation unit 54 outputs the mask image to the composition unit 51b.

[0083] The processing of the estimation unit 52 is the same as that described in Fig. 5. For example, the estimation unit 52 reads and executes the trained estimation model 145. The estimation unit 52 reads and executes the estimation model 145. The estimation unit 52 inputs the synthesis information 45 to the estimation model 145, and identifies the BBOX of each object based on the parameters of the estimation model 145.

[0084] The inference processing unit 153 obtains background image data 25a from the image table 142 and inputs it to the feature extraction unit 50a. The inference processing unit 153 obtains target image data 25b from the image table 142 and inputs it to the feature extraction unit 50b. The inference processing unit 153 may output information about the BBOX identified by the estimation unit 52 to the display unit 130 or to an external device.

[0085] Next, a description will be given of an example of a processing procedure of the information processing device 100 according to the first embodiment. The processing procedure of the learning processing and the processing procedure of the inference processing executed by the information processing device 100 will be described below in this order.

[0086] The processing procedure of the learning processing will be described. Fig. 10 and Fig. 11 are flowcharts showing the processing procedure of the learning processing according to the first embodiment. As shown in Fig. 10, the learning processing unit 152 of the information processing device 100 acquires background image data from the learning data 141 (step S101). The feature extraction unit 50a of the learning processing unit 152 extracts background image feature amounts based on the background image data (step S102).

[0087] The learning processing unit 152 acquires target image data from the learning data 141 (step S103). The feature extraction unit 50b of the learning processing unit 152 extracts target image feature amounts based on the target image data (step S104).

[0088] The synthesis unit 51a of the learning processing unit 152 synthesizes the background image feature amount and the target image feature amount (step S105). The mask generation unit 54 of the learning processing unit 152 generates a mask image based on the synthesized feature amount (step S106).

[0089] The position coordinate feature output unit 53 of the learning processing unit 152 generates the position coordinate feature (step S107). The synthesis unit 51b of the learning processing unit 152 generates synthesis information by synthesizing the feature amounts (step S108).

[0090] The estimation unit 52 of the learning processing unit 152 estimates the BBOX based on the combined information (step S109). The learning processing unit 152 proceeds to step S110 in FIG.

[0091] 11. The learning processing unit 152 acquires the GT of the mask image from the learning data 141 (step S110). The error calculation unit 60a of the learning processing unit 152 calculates a first error based on the mask image and the GT of the mask image (step S111).

[0092] The learning processing unit 152 acquires the GT of the BBOX from the learning data 141 (step S112). The error calculation unit 60b calculates a second error based on the BBOX and the GT of the BBOX (step S113).

[0093] The synthesis unit 61 of the learning processing unit 152 calculates the total error of the first error and the second error (step S114). The weight update value calculation unit 62 of the learning processing unit 152 calculates update values ​​for the neural network parameters (step S115). The learning processing unit 152 updates the neural network parameters (step S116).

[0094] If the learning processing unit 152 determines to continue the machine learning (step S117, Yes), the process proceeds to step S101 in Fig. 10. If the learning processing unit 152 determines not to continue the machine learning (step S117, No), the learning processing unit 152 ends the machine learning of the neural network.

[0095] Next, the processing procedure of the inference processing will be described. Fig. 12 is a flowchart showing the processing procedure of the inference processing according to the first embodiment. As shown in Fig. 12, the inference processing unit 153 of the information processing device 100 acquires background image data from the image table 142 (step S201). The feature extraction unit 50a of the inference processing unit 153 extracts background image feature amounts based on the background image data (step S202).

[0096] The inference processing unit 153 acquires the target image data from the image table 142 (step S203). The feature extraction unit 50b of the inference processing unit 153 extracts target image feature amounts based on the target image data (step S204).

[0097] The synthesis unit 51a of the inference processing unit 153 synthesizes the background image feature amount and the target image feature amount (step S205). The mask generation unit 54 of the inference processing unit 153 generates a mask image based on the synthesized feature amount (step S206).

[0098] The position coordinate feature output unit 53 of the inference processing unit 153 generates the position coordinate feature (step S207). The synthesis unit 51b of the inference processing unit 153 generates synthesis information by synthesizing the feature amounts (step S208).

[0099] The estimation unit 52 of the inference processing unit 153 estimates a BBOX based on the combined information (step S209).

[0100] Next, the effects of the information processing device 100 according to the first embodiment will be described. The information processing device 100 inputs background image data to the feature extraction unit 50a and inputs target image data to the feature extraction unit 50b, thereby extracting background image features and target image features. The information processing device 100 inputs a feature obtained by combining the background image features and the target image features to the mask generation unit 54, thereby generating a mask image. The information processing device 100 inputs the mask image and information obtained by combining the features to the estimation unit 52, thereby specifying the object region. This makes it possible to discriminate and detect each object even if the object included in the target image data is an unknown object that has not been learned in advance.

[0101] The information processing device 100 inputs information obtained by combining background image feature amounts, target image feature amounts, mask image, and coordinate feature amounts into the estimation unit 52, and identifies the object region. This makes it possible to perform convolution processing so that each object can be distinguished even if the target image data contains objects with the same appearance.

[0102] The information processing device 100 executes machine learning of the feature extraction units 50a and 50b, the mask generation unit 54, and the estimation unit 52 based on the training data 141. This allows for machine learning of a neural network that can discriminate and detect each object even if the object included in the target image data is an unknown object that has not been learned in advance.

[0103] The information processing device 100 inputs information obtained by further combining coordinate features in addition to each feature into the estimation unit 52 and executes machine learning. As a result, even if the target image data contains objects with the same appearance, the objects can be distinguished from each other and the neural network can be trained by machine learning.

[0104] The information processing device 100 inputs information obtained by further combining each feature amount with a mask image to the estimation unit 52 and executes machine learning. This is expected to have the effect of limiting the object to be detected to the object region of the mask image. [Example]

[0105] The configuration of the system according to the second embodiment is the same as the system described in the first embodiment. The information processing device according to the second embodiment is connected to a camera 10 via a network 11, similarly to the first embodiment.

[0106] The information processing device according to the second embodiment performs machine learning on the feature extraction units 50a and 50b and the estimation unit 52, which are the basic parts described in Fig. 2. The information processing device identifies each object using the feature extraction units 50a and 50b and the estimation unit 52 that have undergone machine learning.

[0107] Fig. 13 is a functional block diagram showing the configuration of an information processing device according to Example 2. As shown in Fig. 13, this information processing device 200 includes a communication unit 210, an input unit 220, a display unit 230, a storage unit 240, and a control unit 250.

[0108] The explanations regarding the communication unit 210, the input unit 220, and the display unit 230 are the same as the explanations regarding the communication unit 110, the input unit 120, and the display unit 130 explained in the first embodiment.

[0109] The storage unit 240 includes training data 241, an image table 242, a feature extraction model 243, and an estimation model 244. The storage unit 240 corresponds to a semiconductor memory element such as a RAM or a flash memory, or a storage device such as an HDD.

[0110] The learning data 241 is data used when machine learning is performed. FIG. 14 is a diagram illustrating an example of a data structure of the learning data according to the second embodiment. As illustrated in FIG. 14, the learning data 241 stores an item number, input data, and correct answer data (GT) in association with each other. The input data includes background image data for learning and target image data for learning. The correct answer data includes the GT (coordinates of the object area) of the BBOX.

[0111] The image table 242 is a table that holds background image data and target image data used during inference.

[0112] The feature extraction model 243 is a machine learning model (CNN) executed by the feature extraction units 50a and 50b. When image data is input to the feature extraction model 243, an image feature amount is output.

[0113] The estimation model 244 is a machine learning model (CNN) executed by the estimation unit 52. When a background image feature amount and a target image feature amount are input to the estimation model 244, a BBOX is output.

[0114] The control unit 250 includes an acquisition unit 251, a learning processing unit 252, and an inference processing unit 253. The control unit 250 corresponds to a CPU or the like.

[0115] When acquiring the learning data 241 from an external device or the like, the acquiring unit 251 registers the acquired learning data 241 in the storage unit 240.

[0116] The acquisition unit 251 acquires background image data from the camera 10 in advance and registers it in the image table 242. The acquisition unit 251 acquires target image data from the camera 10 and registers it in the image table 242.

[0117] The learning processing unit 252 executes machine learning of the feature extraction units 50a and 50b (feature extraction model 243) and the estimation unit 52 (estimation model 244) based on the learning data 241.

[0118] 15 is a diagram for explaining the processing of the learning processing unit according to the second embodiment. For example, the learning processing unit 252 includes feature extraction units 50a and 50b, a synthesis unit 51a, and an estimation unit 52. The learning processing unit 252 also includes an error calculation unit 80 and a weight update value calculation unit 81. In the following description, the feature extraction units 50a and 50b and the estimation unit 52 will be collectively referred to as a "neural network" as appropriate.

[0119] The processing of the feature extraction units 50a and 50b is the same as that described in Fig. 2. For example, the feature extraction units 50a and 50b read and execute the feature extraction model 143. The feature extraction units 50a and 50b input image data to the feature extraction model 243 and calculate image feature amounts based on the parameters of the feature extraction model 243.

[0120] The synthesis unit 51 a synthesizes the background image feature amount and the target image feature amount, and outputs the synthesized feature amount to the estimation unit 52 .

[0121] The estimation unit 52 reads out and executes the estimation model 244. The estimation unit 52 reads out and executes the estimation model 244. The estimation unit 52 inputs the combined feature amount to the estimation model 244, and identifies a BBOX for each object based on the parameters of the estimation model 244. The estimation model 244 outputs the BBOX to the error calculation unit 80.

[0122] The learning processing unit 252 acquires background image data 26a for learning from the learning data 241 and inputs it to the feature extraction unit 50a. The learning processing unit 252 acquires target image data 26b for learning from the learning data 241 and inputs it to the feature extraction unit 50b. The learning processing unit 252 acquires the GT of the BBOX from the learning data 241 and inputs it to the error calculation unit 80.

[0123] The error calculation unit 80 calculates the error between the BBOX output from the estimation unit 52 and the GT of the BBOX in the training data 241. The error calculation unit 80 outputs the calculated error to the weight update value calculation unit 81.

[0124] The weight update value calculation unit 81 updates the parameters (weights) of the neural network so as to reduce the error. For example, the weight update value calculation unit 81 updates the parameters of the feature extraction units 50a and 50b (feature extraction model 243) and the estimation unit 52 (estimation model 244) using an error backpropagation method or the like.

[0125] The learning processing unit 252 repeatedly executes the above process using each input data and correct answer data stored in the learning data 241. The learning processing unit 252 registers the machine-learned feature extraction model 243 and estimation model 244 in the storage unit 240.

[0126] Returning to the explanation of Fig. 13, the inference processing unit 253 uses the machine-learned feature extraction units 50a and 50b (feature extraction model 243) and the estimation unit 52 (estimation model 244) to identify the area of ​​an object that is present in the target image data but not in the background image data.

[0127] 16 is a diagram for explaining the processing of the inference processing unit according to the present embodiment 2. For example, the inference processing unit 253 includes feature extraction units 50a and 50b, a synthesis unit 51a, and an estimation unit 52.

[0128] The processing of the feature extraction units 50a and 50b is the same as that described in Fig. 2. For example, the feature extraction units 50a and 50b read and execute the machine-learned feature extraction model 243. The feature extraction units 50a and 50b input image data to the feature extraction model 243 and calculate image features based on the parameters of the feature extraction model 243.

[0129] The synthesis unit 51 a synthesizes the background image feature amount and the target image feature amount, and outputs the synthesized feature amount to the estimation unit 52 .

[0130] The processing of the estimation unit 52 is the same as that described in Fig. 2. For example, the estimation unit 52 reads and executes the trained estimation model 244. The estimation unit 52 reads and executes the estimation model 244. The estimation unit 52 inputs information obtained by combining background image features and target image features into the estimation model 244, and identifies a BBOX for each object based on the parameters of the estimation model 244.

[0131] The inference processing unit 253 obtains background image data 25a from the image table 242 and inputs it to the feature extraction unit 50a. The inference processing unit 253 obtains target image data 25b from the image table 242 and inputs it to the feature extraction unit 50b. The inference processing unit 253 may output information about the BBOX identified by the estimation unit 52 to the display unit 230 or to an external device.

[0132] Next, a description will be given of an example of a processing procedure of the information processing device 200 according to the second embodiment. The processing procedure of the learning processing and the processing procedure of the inference processing executed by the information processing device 200 will be described below in this order.

[0133] The processing procedure of the learning processing will be described. Fig. 17 is a flowchart showing the processing procedure of the learning processing according to the second embodiment. As shown in Fig. 17, the learning processing unit 252 of the information processing device 200 acquires background image data from the learning data 241 (step S301). The feature extraction unit 50a of the learning processing unit 252 extracts background image feature amounts based on the background image data (step S302).

[0134] The learning processing unit 252 acquires target image data from the learning data 241 (step S303). The feature extraction unit 50b of the learning processing unit 252 extracts target image feature amounts based on the target image data (step S304).

[0135] The synthesis unit 51a of the learning processing unit 252 synthesizes the background image feature amount and the target image feature amount (step S305). The estimation unit 52 of the learning processing unit 252 estimates a BBOX based on the synthesized feature amount (step S306).

[0136] The learning processing unit 252 acquires the GT of the BBOX from the learning data 241 (step S307). The error calculation unit 80 calculates an error based on the BBOX and the GT of the BBOX (step S308).

[0137] The weight update value calculation unit 81 of the learning processing unit 252 calculates the update values ​​of the parameters of the neural network (step S309). The learning processing unit 252 updates the parameters of the neural network (step S310).

[0138] If the learning processing unit 252 determines to continue the machine learning (Yes at step S311), it proceeds to step S301. If the learning processing unit 252 determines not to continue the machine learning (No at step S311), it ends the machine learning of the neural network.

[0139] Next, the processing procedure of the inference processing will be described. Fig. 18 is a flowchart showing the processing procedure of the inference processing according to the second embodiment. As shown in Fig. 18, the inference processing unit 253 of the information processing device 200 acquires background image data from the image table 242 (step S401). The feature extraction unit 50a of the inference processing unit 253 extracts background image feature amounts based on the background image data (step S402).

[0140] The inference processing unit 253 acquires the target image data from the image table 242 (step S403). The feature extraction unit 50b of the inference processing unit 253 extracts target image feature amounts based on the target image data (step S404).

[0141] The synthesis unit 51a of the inference processing unit 253 synthesizes the background image feature amount and the target image feature amount (step S405).

[0142] The estimation unit 52 of the inference processing unit 253 estimates a BBOX based on the combined feature amount (step S406).

[0143] Next, the effects of the information processing device 200 according to the second embodiment will be described. The information processing device 200 inputs background image data to the feature extraction unit 50a and inputs target image data to the feature extraction unit 50b, thereby extracting background image feature amounts and target image feature amounts. The information processing device 100 inputs a feature amount obtained by combining the background image feature amounts and the target image feature amounts to the estimation unit 52, thereby specifying the object region. This makes it possible to discriminate and detect each object even if the object included in the target image data is an unknown object that has not been learned in advance. [Example]

[0144] Next, an example of a system according to Example 3 will be described. Fig. 19 is a diagram showing a system according to Example 3. As shown in Fig. 19, this system includes a self-checkout register 5, a camera 10, and an information processing device 300. The self-checkout register 5, the camera 10, and the information processing device 300 are connected by wire or wirelessly.

[0145] The user 1 picks up the product 2 placed on the temporary table 6, scans the barcode of the product 2 at the self-checkout register 5, and wraps it.

[0146] The self-checkout register 5 is a point-of-sale (POS) register system in which a user 1 purchasing an item performs operations from reading the item's barcode to paying. For example, when the user 1 moves the item to be purchased into the scanning area of ​​the self-checkout register 5, the self-checkout register 5 scans the item's barcode. When the user 1 has completed scanning, the self-checkout register 5 notifies the information processing device 300 of the number of items scanned. In the following description, the information on the number of items scanned will be referred to as "scan information."

[0147] The camera 10 is a camera that captures an image of the temporary table 6 of the self-checkout 5. The camera 10 transmits image data of the capture range to the information processing device 300. The camera 10 transmits image data of the temporary table 6 on which no product is placed (background image data) to the information processing device 300 in advance. When a product to be purchased is placed on the temporary table 6, the camera 10 transmits image data of the temporary table 6 (target image data) to the information processing device 300.

[0148] The information processing device 300 performs machine learning of a neural network in the same manner as the information processing device 100 described in Example 1. The neural network includes feature extraction units 50a and 50b, synthesis units 51a and 51b, an estimation unit 52, a position coordinate feature output unit 53, and a mask generation unit 54.

[0149] The information processing device 300 inputs background image data and target image data into a machine-learned neural network to identify each object included in the target image data. The information processing device 300 counts the identified objects to identify the number of items. The information processing device 300 detects a missed scan when the identified number of items does not match the number of items included in the scan information.

[0150] For example, the information processing device 300 outputs the result of inputting background image data and target image data into a neural network as the output result 70. Because the output result 70 includes three BBOXes, BBOXes 70a, 70b, and 70c, the information processing device 300 identifies the number of products as "3." If the number of products included in the scan information is less than "3," the information processing device 300 detects a scan omission. The information processing device 300 may notify a management server (not shown) or the like of the scan omission.

[0151] As described above, by applying the information processing device 100 (200) described in the first and second embodiments to the system shown in FIG. 19, it is possible to detect user fraud such as not reading a barcode.

[0152] Next, an example of the hardware configuration of a computer that realizes the same functions as the information processing device 100 (200, 300) shown in the above embodiment will be described. Fig. 20 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the information processing device.

[0153] 20, the computer 400 includes a CPU 401 that executes various types of arithmetic processing, an input device 402 that accepts data input from a user, and a display 403. The computer 400 also includes a communication device 404 that receives distance image data from the camera 10, and an interface device 405 that connects to various devices. The computer 400 also includes a RAM 406 that temporarily stores various types of information, and a hard disk drive 407. The devices 401 to 407 are connected to a bus 408.

[0154] The hard disk drive 407 stores an acquisition program 407a, a learning processing program 407b, and an inference processing program 407c. The CPU 401 reads out the acquisition program 407a, the learning processing program 407b, and the inference processing program 407c and loads them into the RAM 406.

[0155] The acquisition program 407a functions as the acquisition process 406a. The learning processing program 407b functions as the learning processing process 406b. The inference processing program 407c functions as the inference processing process 406c.

[0156] The acquisition process 406a corresponds to the processing of the acquisition units 151 and 251. The learning process 406b corresponds to the processing of the learning processing units 152 and 252. The inference process 406c corresponds to the processing of the inference processing units 153 and 253.

[0157] It should be noted that each of the programs 407a to 407c does not necessarily have to be stored in the hard disk drive 407 from the beginning. For example, each of the programs may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD disk, magneto-optical disk, or IC card that is inserted into the computer 400. Then, the computer 400 may read and execute each of the programs 407a to 407c.

[0158] The following supplementary notes are further disclosed regarding the embodiments including the above examples.

[0159] (Supplementary Note 1) A background image of an area where an object should be placed and a target image of the object and the area are used as input data, and learning data is obtained in which information related to an object that does not exist in the background image but exists in the target image is used as correct answer data. Based on the learning data, machine learning is performed on the machine learning model based on the target image and a result output from the machine learning model when the target image is input into the machine learning model, and based on the correct answer data. A learning program that causes a computer to execute a process.

[0160] (Supplementary Note 2) The machine learning model includes a feature extraction model that outputs intermediate features when the background image and the target image are input, and an estimation model that outputs a region of an object that does not exist in the background image but exists in the target image when the intermediate features are input, The learning program described in Appendix 1, characterized in that the process of performing machine learning performs machine learning on the feature extraction model and the estimation model based on the correct answer data.

[0161] (Supplementary Note 3) The machine learning model further includes a generative model that outputs, when the intermediate features are input, a mask image that indicates an area of ​​an object that is not present in the background image but is present in the target image, and the correct answer data further includes correct answer data regarding the mask image; the estimation model outputs an object region when the intermediate feature and the mask image are input; The learning program described in Appendix 2, characterized in that the process of performing machine learning further performs machine learning on the generative model based on the correct answer data.

[0162] (Supplementary Note 4) The estimation model outputs an object region when the intermediate feature, the mask image, and coordinate feature in which coordinate values ​​are arranged on an image plane are input; The learning program according to claim 3, wherein the process of performing machine learning performs machine learning on the estimation model based on the correct answer data.

[0163] (Appendix 5) A background image of an area where an object should be placed and a target image of the object and the area are used as input data, and learning data is obtained in which information related to an object that does not exist in the background image but exists in the target image is used as correct answer data. Based on the learning data, machine learning is performed on the machine learning model based on the target image and a result output from the machine learning model when the target image is input into the machine learning model, and based on the correct answer data. A learning method characterized in that the processing is executed by a computer.

[0164] (Supplementary Note 6) The machine learning model includes a feature extraction model that outputs intermediate features when the background image and the target image are input, and an estimation model that outputs a region of an object that does not exist in the background image but exists in the target image when the intermediate features are input, The learning method described in Appendix 5, characterized in that the process of performing machine learning performs machine learning on the feature extraction model and the estimation model based on the correct answer data.

[0165] (Supplementary Note 7) The machine learning model further includes a generative model that outputs, when the intermediate features are input, a mask image that indicates an area of ​​an object that is not present in the background image but is present in the target image, and the correct answer data further includes correct answer data regarding the mask image; the estimation model outputs an object region when the intermediate feature and the mask image are input; The learning method described in Appendix 6, characterized in that the process of performing machine learning further performs machine learning on the generative model based on the correct answer data.

[0166] (Supplementary Note 8) The estimation model outputs an object region when the intermediate feature, the mask image, and coordinate feature in which coordinate values ​​are arranged on an image plane are input; The learning method according to claim 7, wherein the process of performing machine learning performs machine learning on the estimation model based on the correct answer data.

[0167] (Supplementary Note 9) An acquisition unit that acquires learning data using a background image of an area where an object is to be placed and a target image of the object and the target area as input data, and uses information related to an object that does not exist in the background image but exists in the target image as correct answer data; a learning processing unit that executes machine learning for the machine learning model based on the target image and a result output from the machine learning model when the target image is input to the machine learning model based on the learning data, and based on the correct answer data; An information processing device comprising:

[0168] (Supplementary Note 10) The machine learning model includes a feature extraction model that outputs intermediate features when the background image and the target image are input, and an estimation model that outputs a region of an object that does not exist in the background image but exists in the target image when the intermediate features are input, The information processing device according to claim 9, wherein the learning processing unit performs machine learning on the feature extraction model and the estimation model based on the correct answer data.

[0169] (Supplementary Note 11) The machine learning model further includes a generative model that outputs, when the intermediate feature is input, a mask image that indicates an area of ​​an object that is not present in the background image but is present in the target image, and the correct answer data further includes correct answer data regarding the mask image; the estimation model outputs an object region when the intermediate feature and the mask image are input; The information processing device described in Supplementary Note 10, wherein the learning processing unit further performs machine learning on the generative model based on the correct answer data.

[0170] (Supplementary Note 12) The estimation model outputs an object region when the intermediate feature, the mask image, and coordinate feature in which coordinate values ​​are arranged on an image plane are input; The information processing device according to claim 11, wherein the learning processing unit performs machine learning on the estimation model based on the correct answer data. [Explanation of symbols]

[0171] 50a, 50b Feature extraction unit 51a,51b,61 Synthesis section 52 Estimation part 53 Position coordinate feature output unit 54 Mask generation unit 60a,60b Error calculation section 62 Weight update value calculation unit 100, 200, 300 Information processing equipment 110,210 Communications Department 120,220 Input section 130,230 Display section 140,240 storage section 141 training data 142 Image Table 143 Feature Extraction Model 144 Generative Model 145 Estimation Model 150,250 Control unit 151,251 Acquisition Department 152,252 Learning processing unit 153,253 Inference processing unit

Claims

1. A background image of an area where an object is to be placed and a target image of the object and the area are used as input data, and learning data is acquired in which information related to an object that does not exist in the background image but exists in the target image is used as correct answer data; When a machine learning model including a generative model and an estimation model is machine-learned using the training data, By inputting the feature amounts of the background image and the target image into the generative model, a mask image showing the area of ​​an object that does not exist in the background image but exists in the target image is output from the generative model, and machine learning is performed on the machine learning model based on the result of inputting the feature amounts into the estimation model and the correct answer data. A learning program that causes a computer to execute a process.

2. the machine learning model further includes a feature extraction model that outputs intermediate features when the background image and the target image are input; the estimation model outputs a region of an object that does not exist in the background image but exists in the target image when the intermediate feature is input; The learning program according to claim 1 , wherein the process of executing machine learning executes machine learning for the feature extraction model and the estimation model based on the correct answer data.

3. The correct answer data further includes correct answer data regarding the mask image, the estimation model outputs an object region when the intermediate feature and the mask image are input; The learning program according to claim 2 , wherein the process of executing machine learning further executes machine learning on the generative model based on the correct answer data.

4. the estimation model outputs an object region when the intermediate feature, the mask image, and coordinate feature in which coordinate values ​​are arranged on an image plane are input; The learning program according to claim 3 , wherein the process of executing machine learning executes machine learning for the estimation model based on the correct answer data.

5. A background image of an area where an object is to be placed and a target image of the object and the area are used as input data, and learning data is acquired in which information related to an object that does not exist in the background image but exists in the target image is used as correct answer data; When a machine learning model including a generative model and an estimation model is machine-learned using the training data, By inputting the feature amounts of the background image and the target image into the generative model, a mask image showing the area of ​​an object that does not exist in the background image but exists in the target image is output from the generative model, and machine learning is performed on the machine learning model based on the result of inputting the feature amounts into the estimation model and the correct answer data. A learning method characterized in that the processing is executed by a computer.

6. an acquisition unit that acquires learning data using a background image of an area where an object is to be placed and a target image of the object and the area as input data, and uses information related to an object that does not exist in the background image but does exist in the target image as correct answer data; a learning processing unit that, when machine learning a machine learning model including a generative model and an estimation model using the learning data, executes machine learning on the machine learning model based on a mask image output from the generative model by inputting feature amounts of the background image and the target image into the generative model, the mask image indicating an area of ​​an object that does not exist in the background image but exists in the target image, a result of inputting the feature amounts into the estimation model, and the correct answer data; An information processing device comprising:

Images