Generation program, generation method, and information processing device.

Abstract

This information processing device acquires an image including a person, and analyzes the acquired image to extract an object used by the person in the image. The information processing device generates a composite image in which the extracted object is placed at a position satisfying a predetermined condition with the position of the object used by the person in the acquired image as a reference. The information processing device uses the generated composite image to generate a machine learning model trained to identify a person who uses the object.

Description

[Technical Field]

[0001] This invention relates to a generation program, etc. [Background technology]

[0002] If it's possible to detect specific customer actions towards physical objects such as merchandise in a store, this information can be used to analyze purchasing trends. For example, a customer picking up an item from a shelf is one way to gauge their purchasing intent.

[0003] The following describes conventional technologies 1 and 2 for detecting customer actions toward objects.

[0004] Let's explain Conventional Technology 1. Figure 17 is a diagram illustrating Conventional Technology 1. Here, the device that performs Conventional Technology 1 is referred to as "Conventional Device A". Conventional Device A estimates the relationship between people and objects using a rule-based approach.

[0005] As shown in Figure 17, for example, conventional device A identifies the region of a person 11a and the region of an object 11b by analyzing video data 11 captured by a camera. Conventional device A also identifies the skeletal information 11c of a person by analyzing the region of a person 11a. The skeletal information 11c contains the coordinate information of each joint of the person. By using the skeletal information 11c, the coordinates of parts of the person, such as the hands, can be identified.

[0006] Conventional device A detects that a person is grasping an object when, based on pre-set detection rules, it sequentially identifies that a person's hand has entered the product shelf 12, detected an object from the product shelf 12, and confirmed that a person's hand is touching the object.

[0007] In conventional technology 1, in order to improve detection accuracy, detailed detection rules are set according to the camera placement and the orientation of the person.

[0008] Let's explain Conventional Technology 2. Figure 18 is a diagram illustrating Conventional Technology 2. Here, the device that performs Conventional Technology 2 is referred to as "Conventional Device B". Conventional Device B utilizes HOID (Human-Object Interaction Detection). For example, HOID is a Transformer-based machine learning model.

[0009] In the example shown in Figure 18, the conventional device B utilizes a machine learning model 15. By inputting image data 16 into the machine learning model 15, the conventional device B outputs the region of a person, the region of an object, and the person's actions on the object.

[0010] The machine learning model 15 includes a Backbone 15a, an adder 15b, an encoder 15c, and a decoder 15d. When image data 16 is input, the Backbone 15a outputs the features of the image data 16. For example, conventional device B divides the image data 16 into multiple blocks and inputs them to the Backbone 15a.

[0011] The adder 15b receives the result of positional encoding for the image data 16 and the output result of the backbone 15a as input. The adder 15b adds the result of positional encoding and the output result of the backbone 15a and outputs the result to the encoder 15c. Positional encoding performs the process of encoding the positional information of each segment of the divided image data 16.

[0012] Encoder 15c converts the data input from adder 15b into vector data and inputs the vector data to Decoder 15d. When vector data is input, Decoder 15d outputs Bounding Box data, Object Category data, and Action data. Bounding Box data indicates the regions of people, objects, etc., contained in the image data 16. Object Category data indicates the attributes of the regions shown in each Bounding Box. Attributes include people, objects, etc. Action data indicates the actions of people on objects.

[0013] Conventional Technology 2 allows for the training of a machine learning model 15 by utilizing training data that defines the relationship between input data and the correct label, without having to set detailed detection rules as in Conventional Technology 1. Furthermore, Conventional Technology 2 allows for the simultaneous identification of human regions, object regions, and human actions on objects by inputting image data.

[0014] Figure 19 shows an example of the processing results of Conventional Technology 2. For example, Conventional Device B outputs a person region 18a, an object region 18b, and the action "grasp" by inputting image data 18 into a machine learning model 15. Conventional Device B outputs a person region 19a, an object region 19b, and the action "grasp" by inputting image data 19 into a machine learning model 15. [Prior art documents] [Patent Documents]

[0015] [Patent Document 1] Japanese Patent Publication No. 2018-15408 [Overview of the project] [Problems that the invention aims to solve]

[0016] However, in the above-described prior art 2, in a case where there are many similar objects like on the product shelves in a store, or in a scene where an object is included in many backgrounds, it is impossible to accurately estimate which object a person's action is acting on.

[0017] FIG. 20 is a diagram showing an example of image data in which the estimation accuracy of the prior art 2 deteriorates. For example, the image data 20 contains many similar products, and it is difficult to estimate which object a person's action is acting on by the prior art 2. The image data 21 contains many objects in the background, and similarly, it is difficult to estimate which object a person's action is acting on by the prior art 2.

[0018] In addition, even when detecting an action in which a person is holding an object based on a detection rule as in the prior art 1, if there are many similar products as in the image data 20 and 21 of FIG. 20, the detection accuracy deteriorates.

[0019] Therefore, there is a need to generate a machine learning model that can accurately estimate which object a person's action is acting on for image data containing many similar objects.

[0020] In one aspect, an object of the present invention is to provide a generation program, a generation method, and an information processing apparatus that can generate a machine learning model that can accurately estimate which object a person's action is acting on for image data containing many similar objects.

Means for Solving the Problem

[0021] In the first proposal, the computer executes the following processes. The computer acquires an image including a person, analyzes the acquired image, and extracts an object used by the person in the image. The computer generates a composite image in which the extracted object is arranged at a position that satisfies a predetermined condition based on the position of the object used by the person in the acquired image. The computer generates a machine learning model trained to identify a person using the object using the generated composite image.

Advantages of the Invention

[0022] The present invention can generate a machine learning model that accurately estimates which object a person's action is acting on with respect to image data containing many similar objects.

Brief Description of the Drawings

[0023] [Figure 1] FIG. 1 is a diagram showing an example of the system according to the present embodiment. [Figure 2] FIG. 2 is a diagram for explaining the machine learning model. [Figure 3] FIG. 3 is a diagram for explaining the learning data. [Figure 4] FIG. 4 is a diagram for explaining the process in which the information processing device generates composite image data. [Figure 5] FIG. 5 is a diagram for explaining the target learning executed by the information processing device. [Figure 6] FIG. 6 is a diagram for explaining the machine learning executed by the information processing device. [Figure 7] FIG. 7 is a diagram for explaining the process in the inference phase in the present embodiment. [Figure 8] FIG. 8 is a functional block diagram showing the configuration of the information processing device according to the present embodiment. [Figure 9] FIG. 9 is a diagram showing an example of the data structure of the learning data table. [Figure 10] FIG. 10 is a flowchart showing the processing procedure of the information processing device according to the present embodiment. [Figure 11] Figure 11 shows the results of the accuracy evaluation. [Figure 12] Figure 12 shows an example of annotation data expansion. [Figure 13] Figure 13 is a diagram illustrating other processes for generating composite image data. [Figure 14] Figure 14 shows an example of a human skeletal model. [Figure 15] Figure 15 shows an example of joint names. [Figure 16] Figure 16 shows an example of a computer hardware configuration that achieves similar functions to the information processing device in the embodiment. [Figure 17] Figure 17 is a diagram illustrating the prior art 1. [Figure 18] Figure 18 is a diagram illustrating the prior art 2. [Figure 19] Figure 19 shows an example of the processing results of the conventional technology 2. [Figure 20] Figure 20 shows an example of image data in which the estimation accuracy of the conventional technology 2 decreases. [Modes for carrying out the invention]

[0024] The following describes in detail, with reference to the drawings, embodiments of the generation program, generation method, and information processing device disclosed in this application. However, this invention is not limited by these embodiments. [Examples]

[0025] Figure 1 shows an example of a system according to this embodiment. As shown in Figure 1, this system includes cameras 30a, 30b, and 30c, and an information processing device 100. The cameras 30a to 30c and the information processing device 100 are interconnected via a network 35.

[0026] Cameras 30a to 30c are installed in a store that has shelves for storing goods. Cameras 30a to 30c capture images including the shelves installed in the store and transmit the captured image data to the information processing device 100. In the following explanation, the image data will be referred to as "image data". The image data includes time-series image data (still images). Cameras 30a to 30c will be collectively referred to as "camera 30".

[0027] The information processing device 100 performs various processes using the machine learning model 40. Figure 2 is a diagram illustrating the machine learning model. As shown in Figure 2, the machine learning model 40 includes a Backbone 41, an adder 42, an encoder 43, and a decoder 44. The Backbone 41 outputs feature quantities of the image data when image data is input. For example, the information processing device 100 divides the image data into multiple blocks and inputs them to the Backbone 41. The explanation is omitted below, but the image data input to the Backbone 41 is divided before being input to the Backbone 41.

[0028] The adder 42 receives the result of positional encoding for the image data and the output result of the backbone 41 as input. The adder 42 adds the result of positional encoding and the output result of the backbone 41 and outputs the result to the encoder 43. In positional encoding, the information processing device 100 performs a process to encode the positional information of each part of the image data that was divided when it was input to the backbone 41.

[0029] Encoder 43 converts the data input from adder 42 into vector data and inputs the vector data to Decoder 44. When vector data is input, Decoder 44 outputs Bounding Box data, Object Category data, and Action data. Bounding Box data indicates the regions of people, objects, etc., contained in the image data. Object Category data indicates the attributes of the regions shown in each Bounding Box. Attributes include people, objects, etc. Action data indicates the actions of people on objects. For example, information processing device 100 can identify the regions of people and objects in the image data by using the Bounding Box data and Object Category data.

[0030] The information processing device 100 performs the process of generating synthesized image data, the process in the learning phase, and the process in the inference phase. The processes of generating synthesized image data, the process in the learning phase, and the process in the inference phase will be described in order below.

[0031] First, the process by which the information processing device 100 generates composite image data will be explained. The information processing device 100 generates composite image data based on the training data.

[0032] Figure 3 is a diagram illustrating the training data. As shown in Figure 3, the training data 50 includes image data 51 and annotation data 52. For example, image data 51 includes images of people and images of objects. The input data for training the machine learning model 40 corresponds to the image data 51, and the correct labels correspond to the annotation data 52.

[0033] Annotation data 52 includes data for the person's region, data for the object's region, and data for the person's actions on the object. In the example shown in Figure 3, the data for the person's region is "Person1:{x1,y1,x2,y2}". This indicates that the coordinates of the upper left corner of the person's region 51a are "x1,y1", and the coordinates of the lower right corner of the person's region 51a are "x2,y2".

[0034] The data for the object (bottle) region is "Bottle1:{x1',y1',x2',y2'}". This indicates that the coordinates of the upper left corner of the object's region 51b are "x1',y1'", and the coordinates of the lower right corner of the object's region 51b are "x2',y2'".

[0035] The data for a person's action on an object is "Action:{Person1, Bottle1, Hold}". This indicates that the person in region 51a is holding the object (bottle) in region 51b.

[0036] The information processing device 100 generates composite image data using the training data 50 shown in Figure 3. Figure 4 is a diagram illustrating the process by which the information processing device generates composite image data. Based on the annotation data 52 included in the training data 50, the information processing device 100 extracts the regions of people 51a and objects 51b included in the image data 51. The information processing device 100 generates composite image data 54 by executing the processes in steps S1 to S4. As will be described later, the information processing device 100 generates composite image data 54 by combining image 51c with image data 51.

[0037] Step S1 will now be explained. The information processing device 100 identifies the center coordinates (xc1, yc1) of the region 51a of the person. The information processing device 100 identifies the center coordinates (xc2, yc2) of the region 51b of the object. The information processing device determines the synthesis direction from the positional relationship between the center coordinates (xc1, yc1) and the center coordinates (xc2, yc2). The synthesis direction indicates whether to synthesize the image 51c to the "left side" of the region 51a of the person, or to synthesize the image 51c to the "right side" of the region 51a of the person.

[0038] The information processing device 100 determines the direction of synthesis to be "left" if "xc1 - xc2 < 0". The information processing device 100 determines the direction of synthesis to be "right" if "xc1 - xc2 ≥ 0". In the example shown in Figure 4, "xc1 - xc2 < 0", so the direction of synthesis is "left".

[0039] Step S2 will now be explained. The information processing device 100 generates image data 53 by creating a margin around the image data 51.

[0040] Step S3 will now be explained. The information processing device 100 copies the image of the object region 51b contained in the image data 51. Since the synthesis direction determined in step S1 is "left side", the information processing device 100 places the copied image 51c in a region to the left of the person region 51a that does not overlap with region 51a. Furthermore, the information processing device 100 adjusts the coordinates of image 51c so that the distance Δd between the coordinates (x1', y1') of the upper left corner of the object region 51b and the coordinates (x3, y3) of the upper left corner of the placed image 51c becomes a pre-set hyperparameter.

[0041] Step S4 will now be explained. The information processing device 100 generates composite image data 54 by deleting the margins of the image data 53. In the example shown in Figure 4, the case where image 51c is included in the range of image data 51 was explained, but depending on the processing in step S3, a part of image 51c may also be included in the margins of image data 53. If a part of image 51c is included in the margins of image data 53, the part of image 51c that is included in the margins is deleted by the processing in step S4.

[0042] As explained in Figure 4, the information processing device 100 generates composite image data 54 by combining image data 51 and image 51c when it executes the processes in steps S1 to S4.

[0043] The information processing device 100 generates multiple composite image data by repeatedly executing the process described in Figure 4 above for training data other than the training data 50 described in Figure 3.

[0044] Next, we will describe the processing performed by the information processing device 100 during the learning phase. For example, the information processing device 100 performs controlled learning on the machine learning model 40 using image data and synthesized image data, and also performs machine learning on the machine learning model 40 using the training data.

[0045] The comparative learning performed by the information processing device 100 will be explained. Figure 5 is a diagram illustrating the comparative learning performed by the information processing device. In Figure 5, for the sake of explanation, image data 60, 61 and synthesized image data 62, 63 will be used for the explanation. Image data 60 and 61 are image data included in the training data. Image data 60 and image data 61 share the common feature that a person is grasping an object.

[0046] The composite image data 62 is generated by performing the process described in Figure 4 using the image data 60. The composite image data 63 is generated by performing the process described in Figure 4 using the image data 61. The composite image data 61 and the composite image data 63 share the common characteristic that a person is grasping an object, and other objects are placed near the person.

[0047] In the following explanation, image data 60, 61, and image data including an image of an object and an image of a person holding this object will be referred to as "positive examples" as appropriate. The composite image data obtained by performing the process described in Figure 4 will be referred to as "negative examples" as appropriate. As described above, positive examples have common characteristics, and negative examples also have common characteristics. In the comparative learning performed by the information processing device 100, the Encoder 43 is trained so that the outputs from the Encoder 43 when two positive examples are input become similar. The information processing device 100 trains the Encoder 43 so that the outputs from the Encoder 43 when two negative examples are input become similar.

[0048] Furthermore, since the information processing device 100 does not have any common features between a positive example and a negative example, the encoder 43 is trained so that the outputs from the encoder 43 when a positive example and a negative example are input are far apart.

[0049] The information processing device 100 inputs image data 60, 61 and synthesized image data 62, 63 to the Backbone 41 of the machine learning model 40, respectively, to obtain output f() from the encoder 43. For example, f() is vector data.

[0050] For example, the information processing device 100 calculates the error (cross-entropy error of cosine similarity) between the output f() when a positive or negative example is input to the Backbone 41 and the output f() when the other positive or negative example is input to the Backbone 41, using equation (1).

[0051]

number

[0052] l shown in equation (1) xi,xj This is calculated by equation (2). The f() shown in equation (2) is the output from Encoder43 as described above. g() is the cosine similarity.

[0053]

number

[0054] For example, the output from the Encoder 43 obtained by inputting image data 60 to the Backbone 41 is f 60 () is denoted as (). The output from the Encoder43 obtained by inputting the image data 61 to the Backbone41 is f 61 () is denoted as (). The output from the Encoder43 obtained by inputting the image data 62 to the Backbone41 is f 62 () is denoted as (). The output from the Encoder43 obtained by inputting the image data 63 to the Backbone41 is f 63 It is written as ().

[0055] The cross-entropy error decreases as the f() output from Encoder43 when a positive example is input to Backbone41 approaches the f() output from Encoder43 when another positive example is input to Backbone41. For example, information processing device 100 has f 60 () and f 61 Train the parameters of Encoder43 so that the value of () approaches the value of ().

[0056] The cross-entropy error decreases as the f() output from Encoder43 when a negative example is input to Backbone41 approaches the f() output from Encoder43 when another negative example is input to Backbone41. For example, information processing device 100 has f62 ( ) and f 63 Train the parameters of Encoder 43 so that ( ) and f( ) approach each other.

[0057] The cross-entropy error decreases as the f( ) output from Encoder 43 when a positive example is input to Backbone 41 and the f( ) output from Encoder 43 when a negative example is input to Backbone 41 move further apart. For example, information processing apparatus 100 trains the parameters of Encoder 43 so that f( ) and f( ) move further apart. 60 ( ) and f 62 Train the parameters of Encoder 43 so that ( ) and f( ) move further apart. Information processing apparatus 100 trains the parameters of Encoder 43 so that f( ) and f( ) move further apart. 60 ( ) and f 63 Train the parameters of Encoder 43 so that ( ) and f( ) move further apart. Information processing apparatus 100 trains the parameters of Encoder 43 so that f( ) and f( ) move further apart. 61 ( ) and f 62 Train the parameters of Encoder 43 so that ( ) and f( ) move further apart. Information processing apparatus 100 trains the parameters of Encoder 43 so that f( ) and f( ) move further apart. 61 ( ) and f 63 Train the parameters of Encoder 43 so that ( ) and f( ) move further apart.

[0058] Here, while performing the contrastive learning described in FIG. 5, information processing apparatus 100 executes machine learning of the parameters of Encoder 43 and Decoder 44 of machine learning model 40 based on the error backpropagation method.

[0059] FIG. 6 is a diagram for explaining the machine learning executed by the information processing apparatus. Image data 60, 61 are image data included in the learning data, and the correct labels (annotation data) corresponding to image data 60, 61 are set in advance. Synthetic image data 62 is generated based on image data 60. Synthetic image data 63 is generated based on learning data 61.

[0060] For example, the information processing device 100 inputs image data 60 into the backbone 41 of the machine learning model 40 and obtains output results from the decoder 44. The information processing device 100 compares the annotation data corresponding to the image data 60 with the output results from the decoder 44 and calculates the bounding box loss, object category loss, and action loss.

[0061] Furthermore, as explained in Figure 5, the information processing device 100 inputs pairs of positive examples, pairs of positive examples and negative examples, or pairs of negative examples into the Backbone 41 of the machine learning model 40, and calculates the cross-entropy error (contrastive loss) based on equation (1).

[0062] The information processing device 100 performs machine learning on the parameters of the Encoder 43 and Decoder 44 of the machine learning model 40, based on backpropagation, so that the overall error, which is the sum of the Bounding Box Loss, Object Category Loss, Action Loss, and cross-entropy error, is minimized.

[0063] As described above, the information processing device 100 according to this embodiment extracts images of objects used by a person from image data and generates composite image data by placing the extracted images near the person in the image data. The information processing device 100 can generate a machine learning model 40 that can identify a person using an object by performing machine learning on the machine learning model 40 using the composite image data.

[0064] Next, the processing in the inference phase of the information processing device 100 will be described. Figure 7 is a diagram illustrating the processing in the inference phase in this embodiment. The machine learning model 40 is a machine learning model trained in the learning phase processing described above.

[0065] The information processing device 100 inputs the image data 20 into the backbone 41 of the machine learning model 40 to obtain the output result 70. In the output result 70, the region of the person 70a and the region of the object 70b are identified, and the person's action on the object, "grasping," is indicated. Although the image data 20 contains multiple similar products, the device is able to accurately estimate which object the person's action is acting on.

[0066] The information processing device 100 inputs the image data 21 into the backbone 41 of the machine learning model 40 to obtain the output result 71. In the output result 71, the region of the person 71a and the region of the object 71b are identified, and the person's action on the object, "grasping," is indicated. Although the image data 21 has many objects in the background, it is possible to accurately estimate which object the person's action is acting on.

[0067] Next, an example of the configuration of the information processing device 100 that performs the above-described process will be explained. Figure 8 is a functional block diagram showing the configuration of the information processing device according to this embodiment. As shown in Figure 8, this information processing device 100 has a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0068] The communication unit 110 performs data communication with the camera 30, external devices, etc., via the network 35. The communication unit 110 is a NIC (Network Interface Card), etc. For example, the communication unit 110 receives video data from the camera 30.

[0069] The input unit 120 inputs various types of information to the control unit 150 of the information processing device 100. For example, the user may operate the input unit 120 to input execution commands for the learning phase and the inference phase. The user may also operate the input unit 120 to specify the hyperparameters described in Figure 4.

[0070] The display unit 130 displays the information output from the control unit 150.

[0071] The memory unit 140 includes a machine learning model 40, a training data table 141, and a video buffer 142. The memory unit 140 is a storage device such as memory.

[0072] Machine learning model 40 is a Transformer-based machine learning model. For example, machine learning model 40 is HOID. The description of machine learning model 40 is the same as the description of machine learning model 40 shown in Figure 2.

[0073] The training data table 141 is a table that holds multiple training data. Figure 9 shows an example of the data structure of the training data table. As shown in Figure 9, the training data table 141 has an item number, image data, annotation data, and composite image data. The item number is a number that identifies each record in the training data table 141. The image data and annotation data correspond to the image data and annotation data included in the training data, as explained in Figure 3. The image data includes images of people and images of objects such as products. The composite image data is the composite image data generated by performing the process shown in Figure 4. The composite image data is generated by the generation unit 152, which will be described later.

[0074] The video buffer 142 is a buffer that stores video data captured by the camera 30. The video buffer 142 may also store video data in association with camera identification information.

[0075] Returning to the explanation of Figure 8, the control unit 150 includes an acquisition unit 151, a generation unit 152, a learning processing unit 153, and an inference unit 154. The control unit 150 is a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc.

[0076] The acquisition unit 151 acquires video data from the camera 30 and stores the acquired video data in the video buffer 142. The acquisition unit 151 may also acquire data from the learning data table 141 from an external device or the like and store it in the storage unit 140.

[0077] The generation unit 152 generates composite image data by performing the processing described in Figure 4 based on the image data included in the training data. The generation unit 152 stores the generated composite image data in the training data table 141. An example of the processing performed by the generation unit 152 is shown below.

[0078] The generation unit 152 obtains pairs of image data and annotation data from the training data table 141. Based on the annotation data, the generation unit 152 extracts regions of people and regions of objects from the image data.

[0079] The generation unit 152 duplicates the image of the region of the object contained in the image data. As explained in step S1 of Figure 4, the generation unit 152 identifies the center coordinates of the region of the person and the center coordinates of the region of the object in the image data, and determines the direction of synthesis of the duplicated image of the object based on each center coordinate.

[0080] The generation unit 152 generates a margin around the image data, as described in step S2 of Figure 4. The generation unit 152 adjusts the coordinates of the duplicated image so that the distance Δd between the upper-left corner of the object region in the image data and the upper-left corner of the duplicated image becomes a pre-set hyperparameter, as described in step S3 of Figure 4. The generation unit 152 generates composite image data by removing the margin, as described in step 4 of Figure 4.

[0081] The learning processing unit 153 uses the learning data table 141 to train the machine learning model 40. For example, the learning processing unit 153 performs controlled learning as described in Figure 5 and machine learning as described in Figure 6.

[0082] The learning processing unit 153 obtains pairs of image data (positive examples) and image data (positive examples), pairs of image data (positive examples) and composite image data (negative examples), or pairs of composite image data (negative examples) and composite image data (negative examples) from the learning data table 141.

[0083] The learning processing unit 153 inputs pairs of positive examples, pairs of positive examples and negative examples, or pairs of negative examples into the backbone 41 of the machine learning model 40, and calculates the cross-entropy error (contrastive loss) based on equation (1).

[0084] The learning processing unit 153 inputs positive examples into the Backbone 41 of the machine learning model 40 and obtains output results from the Decoder 44. The learning processing unit 153 compares the annotation data corresponding to the image data with the output results from the Decoder 44 and calculates the Bounding Box Loss, Object Category Loss, and Action Loss.

[0085] The learning processing unit 153 performs machine learning on the parameters of the Encoder 43 and Decoder 44 of the machine learning model 40, based on backpropagation, so that the overall error, which is the sum of the Bounding Box Loss, Object Category Loss, Action Loss, and cross-entropy error, becomes smaller.

[0086] The inference unit 154 uses the machine learning model 40, trained by the learning processing unit 153, to infer which object a person's actions are affecting. The processing of the inference unit 154 corresponds to the processing described in Figure 7.

[0087] For example, the inference unit 154 acquires image data from the video buffer 142 and inputs the image data into the backbone 41 of the machine learning model 40 to infer the region of a person, the region of an object, and the person's actions toward the object.

[0088] As explained in Figure 1, the camera 30 is installed in a store that has shelves where products are stored, and the image data (video data) captured by the camera 30 includes the area of ​​the product shelves. Therefore, the inference unit 154 can identify the action of a customer taking a product from the shelf by inputting the image data into the backbone 41 of the machine learning model 40.

[0089] The inference unit 154 may output the inference results to the display unit 130 for display. The inference unit 154 outputs screen data to the display unit 130 in which the regions of people, the regions of objects, and the actions of people on objects are placed on the image data as inference results.

[0090] Next, an example of the processing procedure of the information processing device 100 according to this embodiment will be described. Figure 10 is a flowchart of the processing procedure of the information processing device according to this embodiment. As shown in Figure 10, the generation unit 152 of the information processing device 100 generates composite image data based on the image data of the learning data table 141 (step S101).

[0091] The learning processing unit 153 of the information processing device 100 inputs pairs of positive examples, pairs of positive examples, or pairs of negative examples into the backbone 41 of the machine learning model 40 and extracts features accordingly (step S102).

[0092] The learning processing unit 153 inputs each feature to the encoder 43 and calculates each output f() (step S103). The learning processing unit 153 calculates the cross-entropy error related to the controlled learning for each output f() (step S104).

[0093] The learning processing unit 153 inputs positive examples into the backbone 41 of the machine learning model 40 and calculates the loss based on the results output from the decoder 44 and the annotation data (step S105). The loss includes bounding box loss, object category loss, and action loss.

[0094] The learning processing unit 153 trains the machine learning model 40 so that the overall error, which is the sum of the Bounding Box Loss, Object Category Loss, Action Loss, and cross-entropy error, becomes smaller (step S106).

[0095] Next, the effects of the information processing device 100 according to this embodiment will be described. The information processing device 100 extracts images of objects used by a person from image data and generates composite image data by placing the extracted images near the person in the image data. By performing machine learning on the machine learning model 40 using the composite image data, the information processing device 100 can generate a machine learning model 40 that can identify a person using an object.

[0096] When generating composite image data, the information processing device 100 accepts hyperparameter settings as explained in Figure 4. Under the condition that image 51c does not overlap with the region 51a of the person, the information processing device 100 positions image 51c such that the distance Δd between the coordinates of the upper left corner of the region 51b of the object and the coordinates of the upper left corner of the positioned image 51c is a pre-set hyperparameter. This makes it possible to generate pseudo-composite image data that includes similar objects around the person holding the object.

[0097] In controlled learning, the information processing device 100 trains the Encoder 43 so that the outputs from the Encoder 43 become similar when two positive examples are input. The information processing device 100 also trains the Encoder 43 so that the outputs from the Encoder 43 become similar when two negative examples are input. Since the positive examples have common features, and the negative examples also have common features, the parameters of the Encoder 43 can be adjusted so that the outputs from the Encoder 43 become similar when image data with common features is input to the machine learning model 40.

[0098] In comparative learning, the information processing device 100 trains the Encoder 43 so that the outputs from the Encoder 43 become divergent when positive and negative examples are input. Since positive and negative examples have different features, the parameters of the Encoder 43 can be adjusted so that the outputs from the Encoder 43 become divergent when image data with different features is input to the machine learning model 40.

[0099] The information processing device 100 infers the area of ​​a person, the area of ​​an object, and the person's actions toward an object by inputting the image data captured by the camera 30 into the Backbone 41 of the machine learning model 40. For example, as explained in Figure 1, the camera 30 is installed in a store that has shelves where products are stored, and the image data (video data) captured by the camera 30 includes the area of ​​the product shelves. Therefore, by inputting the image data into the Backbone 41 of the machine learning model 40, the information processing device 100 can identify the action of a customer taking a product from the shelf.

[0100] Here, we present the results of the Mean Average Precision evaluation of the method of the information processing device 100 compared with conventional methods. Figure 11 shows the results of the precision evaluation. Dataset (1) is a public dataset containing images of people holding objects in various scenes. Dataset (2) is a closed dataset containing images of people reaching for products in stores.

[0101] With the conventional method, the accuracy evaluation for dataset (1) was "59.6" and for dataset (2) it was "24.1". On the other hand, with the information processing device 100, the accuracy evaluation for dataset (1) was "60.1" and for dataset (2) it was "26.2". In other words, for both datasets, the method of the information processing device 100 has a higher accuracy evaluation compared to the conventional method.

[0102] It should be noted that the processing details of the information processing device 100 and the data structure of each data item described above are merely examples. For example, in the annotation data 52 explained in Figure 3, the region of a person 51a is indicated by the coordinates of the upper left and lower right corners, and the region of an object 51b is indicated by the coordinates of the upper left and lower right corners, but this is not the only way to do so. For example, the annotation data can be expanded as shown in Figure 12.

[0103] Figure 12 shows an example of annotation data extension. In the annotation data 52a shown in Figure 12, multiple coordinates are set to trace the outline of a person. In addition, in annotation data 52a, multiple coordinates are set on the outline of an object. For example, in annotation data 52 explained in Figure 3, the region 51b of an object can be extracted, but in annotation data 52a, the outline 52b of an object can be extracted. The region 51b of an object includes areas that are not part of the object, but the outline 52b of an object contains an image of the object only. This makes it possible to use the image of the object to be composited into the composite image data as the image of the object itself.

[0104] Furthermore, while the generation unit 152 of the information processing device 100 generates composite image data by executing the process described in Figure 4, the process for generating composite image data is not limited to this. For example, the generation unit 152 of the information processing device 100 may generate composite image data by executing the process shown in Figure 13.

[0105] Figure 13 is a diagram illustrating other processes for generating composite image data. The generation unit 152 of the information processing device 100 performs preprocessing by extracting skeletal data from the region 51a of the person and extracting the segmentation of the person.

[0106] An example of the process by which the generation unit 152 extracts skeletal data from the region 51a of a person will be described. The generation unit 152 infers the skeletal data 80 of a person by inputting the region 51a of the person in the image data 51 into a skeletal inference model. The skeletal inference model is a trained model that takes the image data of the region of a person as input and outputs the skeletal data of a person. The skeletal inference model is a Neural Network (NN), etc.

[0107] Skeletal data is data in which two-dimensional or three-dimensional coordinates are assigned to multiple joints defined in a human skeletal model. Here, the coordinates of each joint in the skeletal data are assumed to be two-dimensional. Figure 14 shows an example of a human skeletal model. For example, as shown in Figure 14, the human skeletal model is defined by 21 joints ar0 to ar20.

[0108] The relationship between each joint ar0 to ar20 shown in Figure 14 and its name is shown in Figure 15. Figure 15 is a diagram showing an example of a joint name. For example, the name of joint ar0 is "SPINE_BASE". The names of joints ar1 to a20 are as shown in Figure 15, and their explanation is omitted.

[0109] The process by which the generation unit 152 extracts the segmentation of a person will be described. The generation unit 152 performs segmentation on the image data 51, grouping regions with similar features (color, texture, subject, etc.) within the image data 51 and dividing it into multiple regions. The generation unit 152 compares the divided regions with the region 51a of a person specified in the annotation data, and extracts the region that most overlaps with the region 51a of a person as the region of a person. In the example shown in Figure 13, the generation unit 152 extracts region 81 as the region of a person.

[0110] The generation unit 152 generates the composite image data 90 by performing the above preprocessing and then executing the processes in steps S11 to S14.

[0111] Step S11 will now be explained. The generation unit 152 identifies the coordinates (xc3, yc3) of the point of application based on the skeletal data 80. For example, the generation unit 152 identifies the coordinates of the left wrist joint ar19 among the joints ar0 to ar20 included in the skeletal data 80 as the coordinates of the point of application. The generation unit 152 identifies a predetermined range of area based on the coordinates (xc3, yc3) of the point of application as the object area 82. Alternatively, the generation unit 152 may compare the object area specified in the annotation data with the right wrist joint ar20 and the left wrist joint ar19, and identify the coordinates of the wrist joint closest to the object area as the coordinates of the point of application.

[0112] Step S12 will now be explained. The generation unit 152 generates image data 53 by generating a margin around the image data 51.

[0113] Step S13 will now be explained. The generation unit 152 copies the image of the object region 82 contained in the image data 51. The copied image will be referred to as image 83. The generation unit 152 places the copied image 83 in an area that does not overlap with the person region 81. Furthermore, the generation unit 152 adjusts the coordinates of image 83 so that the distance Δd between the coordinates (xc3, yc3) of the point of application of the object region 82 and the coordinates (xc4, yc4) of the upper left corner of the placed image 83 becomes a pre-set hyperparameter.

[0114] Step S4 will now be explained. The generation unit 152 generates composite image data 90 by removing the margins from the image data 53.

[0115] As explained in Figure 13, the information processing device 100 performs preprocessing and then executes steps S11 to S14 to generate composite image data 90 by combining image data 51 and image 83.

[0116] Next, an example of a computer hardware configuration that achieves the same functions as the information processing device 100 described above will be explained. Figure 16 is a diagram showing an example of a computer hardware configuration that achieves the same functions as the information processing device in the embodiment.

[0117] As shown in Figure 16, the computer 300 includes a CPU 301 that performs various calculations, an input device 302 that receives data input from the user, and a display 303. The computer 300 also includes a communication device 304 and an interface device 305 that exchange data with external devices via a wired or wireless network. Furthermore, the computer 300 includes a RAM 306 for temporarily storing various information and a hard disk drive 307. Each of these devices 301 to 307 is connected to a bus 308.

[0118] The hard disk drive 307 has an acquisition program 307a, a generation program 307b, a learning processing program 307c, and an inference program 307d. The CPU 301 reads each of the programs 307a to 307d and loads them into the RAM 306.

[0119] The acquisition program 307a functions as the acquisition process 306a. The generation program 307b functions as the generation process 306b. The learning process program 307c functions as the learning process 306c. The inference program 307d functions as the inference process 306d.

[0120] The processing of acquisition process 306a corresponds to the processing of acquisition unit 151. The processing of generation process 306b corresponds to the processing of generation unit 152. The processing of learning process 306c corresponds to the processing of learning processing unit 153. The processing of inference process 306d corresponds to the processing of inference unit 154.

[0121] Furthermore, programs 307a to 307d do not necessarily have to be stored in the hard disk drive 307 from the beginning. For example, each program could be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card inserted into the computer 300. Then, the computer 300 could read and execute each program 307a to 307d. [Explanation of Symbols]

[0122] 40 Machine Learning Models 100 Information Processing Devices 110 Communications Department 120 Input section 130 Display section 140 Storage section 141 Training Data Table 142 Video buffer 150 Control Unit 151 Acquisition Department 152 Generation part 153 Learning Processing Unit 154 Reasoning part

Claims

1. Obtain an image that includes a person, By analyzing the acquired image, the object used by the person in the image is extracted. A composite image is generated by placing the extracted object at a position that satisfies predetermined conditions, based on the position of the object used by the person in the acquired image. Using the generated composite image, a machine learning model is created that is trained to identify the person using the object. A generation program characterized by having a computer perform the processing.

2. The process for generating the aforementioned composite image is as follows: The system accepts parameter settings based on the distance between the coordinate position of the person and the coordinate position of the object the person is grasping. Based on the parameters set above, the coordinate positions of candidate placements for the extracted image of the object are generated. Determine whether the generated coordinate position is included within the region relating to the size of the object. The generation program according to claim 1, characterized in that it includes a process of generating a composite image by placing the image of the object on the acquired image based on the determined result.

3. The process for generating the composite image involves generating a first composite image based on a first object included in the first image, and generating a second composite image based on a second object included in the second image. The process of generating the aforementioned machine learning model is: The encoder is trained so that the output result when the first image is input to the encoder of the machine learning model is similar to the output result when the second image is input to the encoder. The encoder is trained so that the output result when the first image is input to the encoder and the output result when the first composite image is input to the encoder become far apart. The generation program according to claim 2, characterized in that it includes a process for training the encoder such that the output result when the first composite image is input to the encoder is similar to the output result when the second composite image is input to the encoder.

4. The generation program according to claim 1, characterized in that it inputs an image taken by a camera inside the store, which includes a product shelf containing the product and a person, into the machine learning model, thereby causing the computer to further perform a process to identify the action of the person taking a product from the product shelf.

5. The generation program according to claim 1, characterized in that the process of extracting the object involves analyzing the acquired image to extract skeletal information of the person in the image, and then extracting the object used by the person based on the skeletal information.

6. Obtain an image that includes a person, By analyzing the acquired image, the object used by the person in the image is extracted. A composite image is generated by placing the extracted object at a position that satisfies predetermined conditions, based on the position of the object used by the person in the acquired image. Using the generated composite image, a machine learning model is created that is trained to identify the person using the object. A generation method characterized by the processing being performed by a computer.

7. Obtain an image that includes a person, By analyzing the acquired image, the object used by the person in the image is extracted. A composite image is generated by placing the extracted object at a position that satisfies predetermined conditions, based on the position of the object used by the person in the acquired image. Using the generated composite image, a machine learning model is created that is trained to identify the person using the object. An information processing device having a control unit that performs processing.

Images