Generation program, inference program, generation method, and information processing device

Abstract

This information processing device acquires a video capturing an area including a product shelf on which products are arranged, and analyzes the acquired video to identify an action of grasping a product by a person. The information processing device identifies, from among a plurality of image frames constituting the acquired video, an image frame including products stored in the product shelf and the product grasped by the person on the basis of the identified action of grasping the product by the person. The information processing device uses the identified image frame to generate a machine learning model trained to identify a person who retrieves a product from a product shelf.

Description

Generation program, inference program, generation method, and information processing device

[0001] The present invention relates to a generation program and the like.

[0002] In a store, if it were possible to detect specific customer actions toward products or other objects, this could be used to analyze purchasing trends. For example, the act of a customer picking up a product from a shelf is one behavior that reveals the customer's intent to purchase.

[0003] Prior art techniques 1 and 2 for detecting customer actions on an object will be described below.

[0004] Conventional technique 1 will now be described. Fig. 21 is a diagram for explaining conventional technique 1. Here, a device that executes conventional technique 1 will be referred to as "conventional device A." Conventional device A estimates the relevance between a person and an object on a rule basis.

[0005] As shown in Fig. 21 , for example, conventional device A analyzes video data 11 captured by a camera to identify a person area 11a and an object area 11b. Furthermore, conventional device A analyzes the person area 11a to identify skeletal information 11c of the person. Coordinate information of each joint of the person is set in the skeletal information 11c. By using the skeletal information 11c, it is possible to identify the coordinates of parts of the person, such as the hands.

[0006] The conventional device A detects that a person is holding an object when it sequentially identifies that a person's hand has entered a product shelf 12, an object has been detected from the product shelf 12, and the person's hand is touching the object based on a preset detection rule.

[0007] In Prior Art 1, in order to improve the detection accuracy, detailed detection rules are set according to the placement of the camera and the direction of the person.

[0008] Conventional technique 2 will now be described. FIG. 22 is a diagram for explaining conventional technique 2. Here, a device that executes conventional technique 2 is referred to as "conventional device B." Conventional device B uses HOID (Human-Object Interaction Detection). For example, HOID is a Transformer-based machine learning model.

[0009] 22 , conventional device B uses machine learning model 15. By inputting image data 16 into machine learning model 15, conventional device B outputs a person area, an object area, and an action of the person with respect to the object.

[0010] The machine learning model 15 includes a backbone 15a, an adder 15b, an encoder 15c, and a decoder 15d. When image data 16 is input, the backbone 15a outputs feature quantities of the image data 16. For example, the conventional device B divides the image data 16 into multiple blocks and inputs the blocks to the backbone 15a.

[0011] The adder 15b receives the result of positional encoding of the image data 16 and the output result of the backbone 15a. The adder 15b adds the result of positional encoding to the output result of the backbone 15a and outputs the result to the encoder 15c. In positional encoding, a process is executed to encode each piece of position information of the divided image data 16.

[0012] The encoder 15c converts the data input from the adder 15b into vector data and inputs the vector data to the decoder 15d. When the decoder 15d receives the vector data, it outputs bounding box data, object category data, and action data. The bounding box data indicates the area of ​​a person, the area of ​​an object, etc. included in the image data 16. The object category data indicates the attributes of the area indicated by each bounding box. The attributes include person, object, etc. The action data indicates the action of a person with respect to an object.

[0013] Unlike in Prior Art 1, Prior Art 2 can train the machine learning model 15 by using training data that defines the relationship between input data and correct labels, without having to set detailed detection rules. Furthermore, Prior Art 2 can simultaneously identify the area of ​​a person, the area of ​​an object, and the person's action relative to the object by inputting image data.

[0014] 23 is a diagram showing an example of a processing result of Conventional Technique 2. For example, Conventional Device B inputs image data 18 into machine learning model 15, thereby outputting person region 18a, object region 18b, and action "grasp." Conventional Device B inputs image data 19 into machine learning model 15, thereby outputting person region 19a, object region 19b, and action "grasp."

[0015] JP 2018-15408 A

[0016] However, in the above-mentioned conventional technology 2, when there are many similar objects, such as product shelves in a store, or when there are many objects in the background, it is not possible to accurately estimate which object a person's action is acting on.

[0017] 24 is a diagram showing an example of image data in which the estimation accuracy of Conventional Technique 2 decreases. For example, image data 20 includes many similar products, making it difficult for Conventional Technique 2 to estimate which object the person's action is acting on. Image data 21 includes many objects in the background, making it similarly difficult for Conventional Technique 2 to estimate which object the person's action is acting on.

[0018] Even when detecting a person holding an object based on detection rules, as in prior art 1, the detection accuracy decreases if the image data contains many similar products, as in image data 20 and 21 in Figure 24.

[0019] For this reason, there is a need to generate a machine learning model that can accurately estimate which objects a person's actions are acting on when using image data that contains many similar objects.

[0020] In one aspect, the present invention aims to provide a generation program, an inference program, a generation method, and an information processing device that can generate a machine learning model that accurately estimates which objects a person's actions are acting on in image data that contains many similar objects.

[0021] In the first proposal, a computer performs the following process: The computer acquires video of an area including shelves on which products are placed, and identifies the behavior of a person grasping a product by analyzing the acquired video. Based on the identified behavior of the person grasping a product, the computer identifies, from among multiple image frames constituting the acquired video, image frames that include products stored on shelves and products being grasped by the person. The computer uses the identified image frames to generate a machine learning model trained to identify people who are taking products from shelves.

[0022] The present invention can generate a machine learning model that accurately estimates which object a person's action is acting on, for image data that contains many similar objects.

[0023] FIG. 1 is a diagram illustrating an example of a system according to a first embodiment. FIG. 2 is a diagram illustrating a machine learning model. FIG. 3 is a diagram illustrating training data. FIG. 4 is a diagram illustrating a process of generating synthetic image data by an information processing device. FIG. 5 is a diagram illustrating object learning performed by the information processing device. FIG. 6 is a diagram illustrating machine learning performed by the information processing device. FIG. 7 is a diagram illustrating a process of an inference phase in the first embodiment. FIG. 8 is a functional block diagram illustrating a configuration of an information processing device according to the first embodiment. FIG. 9 is a diagram illustrating an example of a data structure of a training data table. FIG. 10 is a flowchart illustrating a processing procedure of the information processing device according to the first embodiment. FIG. 11 is a diagram illustrating a result of accuracy evaluation. FIG. 12 is a diagram illustrating an example of extension of annotation data. FIG. 13 is a diagram illustrating another process of generating synthetic image data. FIG. 14 is a diagram illustrating an example of a skeletal model of a human body. FIG. 15 is a diagram illustrating an example of joint names. FIG. 16 is a diagram illustrating an example of a system according to a second embodiment. FIG. 17 is a diagram illustrating a process of the information processing device according to the second embodiment. FIG. 18 is a functional block diagram illustrating a configuration of an information processing device according to the second embodiment. FIG. 19 is a flowchart illustrating a processing procedure of the information processing device according to the second embodiment. Fig. 20 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the information processing device of the embodiment. Fig. 21 is a diagram for explaining Prior Art 1. Fig. 22 is a diagram for explaining Prior Art 2. Fig. 23 is a diagram showing an example of the processing result of Prior Art 2. Fig. 24 is a diagram showing an example of image data in which the estimation accuracy of Prior Art 2 decreases.

[0024] Hereinafter, embodiments of the generation program, inference program, generation method, and information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to these embodiments.

[0025] Fig. 1 is a diagram illustrating an example of a system according to the first embodiment. As shown in Fig. 1, the system includes cameras 30a, 30b, and 30c and an information processing device 100. The cameras 30a to 30c and the information processing device 100 are connected to each other via a network 35.

[0026] Cameras 30a to 30c are installed inside a store that has product shelves on which products are stored. Cameras 30a to 30c capture video including the product shelves installed inside the store and transmit the captured video data to information processing device 100. In the following description, the video data will be referred to as "video data." The video data includes time-series image data (still images). Cameras 30a to 30c will be collectively referred to as "camera 30."

[0027] The information processing device 100 executes various processes using a machine learning model 40. FIG. 2 is a diagram illustrating the machine learning model. As shown in FIG. 2, the machine learning model 40 includes a backbone 41, an adder 42, an encoder 43, and a decoder 44. When image data is input, the backbone 41 outputs feature quantities of the image data. For example, the information processing device 100 divides image data into multiple blocks and inputs the blocks to the backbone 41. Although not described below, the image data input to the backbone 41 is divided and input to the backbone 41.

[0028] The adder 42 receives the result of positional encoding of the image data and the output result of the backbone 41. The adder 42 adds the result of positional encoding and the output result of the backbone 41, and outputs the result to the encoder 43. In positional encoding, the information processing device 100 executes a process of encoding each piece of position information of the image data divided when it is input to the backbone 41.

[0029] The encoder 43 converts the data input from the adder 42 into vector data and inputs the vector data to the decoder 44. When the decoder 44 receives the vector data, it outputs bounding box data, object category data, and action data. The bounding box data indicates the area of ​​a person, the area of ​​an object, etc. included in the image data. The object category data indicates the attributes of the area indicated by each bounding box. The attributes include a person, an object, etc. The action data indicates the action of a person against an object. For example, the information processing device 100 can identify the area of ​​a person and the area of ​​an object in the image data by using the bounding box data and object category data.

[0030] The information processing device 100 executes a process for generating synthetic image data, a process in the learning phase, and a process in the inference phase. The process for generating synthetic image data, the process in the learning phase, and the process in the inference phase will be described below in this order.

[0031] First, a description will be given of a process for generating synthetic image data by the information processing device 100. The information processing device 100 generates synthetic image data based on learning data.

[0032] 3 is a diagram illustrating training data. As shown in FIG. 3, training data 50 includes image data 51 and annotation data 52. For example, the image data 51 includes an image of a person and an image of an object. Input data for training the machine learning model 40 corresponds to the image data 51, and correct labels correspond to the annotation data 52.

[0033] The annotation data 52 includes data on the area of ​​a person, data on the area of ​​an object, and data on the person's action with respect to the object. In the example shown in Fig. 3, the data on the area of ​​the person is "Person1: {x1, y1, x2, y2}". This indicates that the coordinates of the upper left corner of the person's area 51a are "x1, y1", and the coordinates of the lower right corner of the person's area 51a are "x2, y2".

[0034] The data for the object (bottle) region is "Bottle1: {x1', y1', x2', y2'}", which indicates that the coordinates of the upper left corner of object region 51b are "x1', y1'" and the coordinates of the lower right corner of object region 51b are "x2', y2'".

[0035] The data of the person's action on the object is "Action: {Person1, Bottle1, Hold}", which indicates that the person in area 51a is holding the object (bottle) in area 51b.

[0036] The information processing device 100 generates synthetic image data using the training data 50 shown in FIG. 3. FIG. 4 is a diagram for explaining the process by which the information processing device generates synthetic image data. The information processing device 100 extracts a person area 51a and an object area 51b included in image data 51 based on annotation data 52 included in the training data 50. The information processing device 100 generates synthetic image data 54 by executing the processes of steps S1 to S4. As will be described later, the information processing device 100 generates synthetic image data 54 by combining image data 51 with image 51c.

[0037] Step S1 will now be described. The information processing device 100 identifies the center coordinates (xc1, yc1) of the person region 51a. The information processing device 100 identifies the center coordinates (xc2, yc2) of the object region 51b. The information processing device identifies the compositing direction from the positional relationship between the center coordinates (xc1, yc1) and the center coordinates (xc2, yc2). The compositing direction indicates whether the image 51c is to be composited on the "left side" of the person region 51a or on the "right side" of the person region 51a.

[0038] The information processing device 100 determines that the composition direction is "left side" when "xc1-xc2<0". The information processing device 100 determines that the composition direction is "right side" when "xc1-xc2≧0". In the example shown in FIG. 4, "xc1-xc2<0" holds, so the composition direction is "left side".

[0039] Step S2 will now be described. The information processing apparatus 100 generates image data 53 by creating a margin around the image data 51.

[0040] Step S3 will now be described. The information processing device 100 copies an image of the object region 51b included in the image data 51. Because the composition direction determined in step S1 is "left side," the information processing device 100 places the copied image 51c to the left of the person region 51a in an area that does not overlap with the region 51a. Furthermore, the information processing device 100 adjusts the coordinates of the image 51c so that the distance Δd between the coordinates (x1', y1') of the upper left corner of the object region 51b and the coordinates (x3, y3) of the upper left corner of the placed image 51c becomes a preset hyperparameter.

[0041] Step S4 will now be described. The information processing device 100 generates composite image data 54 by deleting the margins of image data 53. In the example shown in Fig. 4, the case where image 51c is included within the range of image data 51 has been described, but depending on the processing of step S3, part of image 51c may be included in the margins of image data 53. If part of image 51c is included in the margins of image data 53, the part of image 51c that is included in the margins is deleted by the processing of step S4.

[0042] As described with reference to FIG. 4, the information processing device 100 executes the processes of steps S1 to S4 to generate composite image data 54 by combining the image data 51 and the image 51c.

[0043] The information processing device 100 generates a plurality of composite image data by repeatedly executing the process described above with reference to FIG. 4 for training data other than the training data 50 described with reference to FIG.

[0044] Next, a description will be given of the processing in the learning phase executed by the information processing device 100. For example, the information processing device 100 performs contrastive learning on the machine learning model 40 using image data and synthetic image data, and also performs machine learning on the machine learning model 40 using the learning data.

[0045] The contrastive learning performed by the information processing device 100 will be described. FIG. 5 is a diagram for explaining the contrastive learning performed by the information processing device. For convenience of explanation, FIG. 5 uses image data 60 and 61 and composite image data 62 and 63. The image data 60 and 61 are image data included in the learning data. The image data 60 and the image data 61 have a common feature in that they show a person holding an object.

[0046] The composite image data 62 is generated by executing the process described in Fig. 4 using the image data 60. The composite image data 63 is generated by executing the process described in Fig. 4 using the image data 61. The composite image data 61 and the composite image data 63 have a common feature in that they both show a person holding an object and another object being placed near the person.

[0047] In the following description, the image data 60, 61, and image data including an image of an object and an image of a person holding the object will be referred to as "positive examples" as appropriate. The composite image data obtained by executing the process described in FIG. 4 will be referred to as "negative examples" as appropriate. As described above, the positive examples share common characteristics, and the negative examples also share common characteristics. In the contrastive learning performed by the information processing device 100, the encoder 43 is trained so that the outputs from the encoder 43 when two positive examples are input are closer to each other. The information processing device 100 trains the encoder 43 so that the outputs from the encoder 43 when two negative examples are input are closer to each other.

[0048] Note that, since pairs of positive examples and negative examples do not have common features, the information processing device 100 trains the encoder 43 so that when a positive example and a negative example are input, the outputs from the encoder 43 move away from each other.

[0049] The information processing device 100 inputs the image data 60, 61 and the composite image data 62, 63 to the backbone 41 of the machine learning model 40, thereby obtaining an output f() from the encoder 43. For example, f() is vector data.

[0050] For example, the information processing device 100 calculates the error (cross-entropy error of cosine similarity) between the output f() when a positive example or a negative example is input to the Backbone 41 and the output f() when another positive example or negative example is input to the Backbone 41 using equation (1).

[0051]

[0052] l shown in formula (1) ｘｉ，ｘｊ is calculated by equation (2). f() shown in equation (2) is the output from the encoder 43. g() is the cosine similarity.

[0053]

[0054] For example, the output from the Encoder 43 obtained by inputting the image data 60 to the Backbone 41 is f ６０ The output from the Encoder 43 obtained by inputting the image data 61 to the Backbone 41 is expressed as f ６１ The output from the Encoder 43 obtained by inputting the image data 62 to the Backbone 41 is expressed as f ６２ The output from the Encoder 43 obtained by inputting the image data 63 to the Backbone 41 is expressed as f ６３ It is written as ().

[0055] The cross-entropy error has a smaller value as f() output from the Encoder 43 when a positive example is input to the Backbone 41 approaches f() output from the Encoder 43 when another positive example is input to the Backbone 41. For example, the information processing device 100 calculates f ６０ () and f ６１The parameters of the Encoder 43 are trained so that the values ​​of () and () approach each other.

[0056] The cross-entropy error becomes smaller as f() output from the Encoder 43 due to a negative example being input to the Backbone 41 and f() output from the Encoder 43 due to another negative example being input to the Backbone 41 become closer. ６２ () and f ６３ The parameters of the Encoder 43 are trained so that () and

[0057] The cross-entropy error becomes smaller as the difference between f() output from the Encoder 43 when a positive example is input to the Backbone 41 and f() output from the Encoder 43 when a negative example is input to the Backbone 41 increases. ６０ () and f ６２ The parameters of the encoder 43 are trained so that f ６０ () and f ６３ The parameters of the encoder 43 are trained so that f ６１ () and f ６２ The parameters of the encoder 43 are trained so that f ６１ () and f ６３ The parameters of the Encoder 43 are trained so that () and are moved away from each other.

[0058] Here, the information processing device 100 performs the contrastive learning described with reference to FIG. 5, and executes machine learning of the parameters of the Encoder 43 and the Decoder 44 of the machine learning model 40 based on the backpropagation method.

[0059] 6 is a diagram illustrating machine learning performed by an information processing device. Image data 60 and 61 are image data included in training data, and correct labels (annotation data) corresponding to the image data 60 and 61 are set in advance. Composite image data 62 is generated based on the image data 60. Composite image data 63 is generated based on the training data 61.

[0060] For example, the information processing device 100 inputs image data 60 into the backbone 41 of the machine learning model 40, and thereby obtains an output result from the decoder 44. The information processing device 100 compares the annotation data corresponding to the image data 60 with the output result from the decoder 44, and calculates the bounding box loss, the object category loss, and the action loss.

[0061] Furthermore, as described in FIG. 5 , the information processing device 100 inputs pairs of positive examples and positive examples, pairs of positive examples and negative examples, or pairs of negative examples and negative examples into the backbone 41 of the machine learning model 40, and calculates a cross-entropy error (contrastive loss) based on equation (1).

[0062] The information processing device 100 performs machine learning of the parameters of the Encoder 43 and Decoder 44 of the machine learning model 40 based on the backpropagation method so as to reduce the overall error obtained by adding the cross-entropy error to the Bounding Box Loss, Object Category Loss, and Action Loss.

[0063] As described above, the information processing device 100 according to the first embodiment extracts an image of an object used by a person from image data and generates synthetic image data in which the extracted image is positioned near the person in the image data. The information processing device 100 performs machine learning on the machine learning model 40 using the synthetic image data, thereby generating a machine learning model 40 that can identify a person using an object.

[0064] Next, a description will be given of the processing in the inference phase of the information processing device 100. Fig. 7 is a diagram for explaining the processing in the inference phase in this embodiment 1. The machine learning model 40 is a machine learning model trained in the processing in the learning phase described above.

[0065] The information processing device 100 inputs image data 20 into the backbone 41 of the machine learning model 40 to obtain an output result 70. In the output result 70, a person area 70a and an object area 70b are identified, and the person's action of "grasping" the object is indicated. The image data 20 includes multiple similar products, but it is possible to accurately estimate which object the person's action is acting on.

[0066] The information processing device 100 inputs the image data 21 into the backbone 41 of the machine learning model 40 to obtain an output result 71. In the output result 71, a person area 71a and an object area 71b are identified, and the person's action of "grasping" the object is indicated. Although the image data 21 includes many objects in the background, it is possible to accurately estimate which object the person's action is acting on.

[0067] Next, a configuration example of the information processing device 100 that executes the above-described processing will be described. Fig. 8 is a functional block diagram showing the configuration of the information processing device according to the first embodiment. As shown in Fig. 8, the information processing device 100 has a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0068] The communication unit 110 executes data communication with the camera 30, an external device, etc. via the network 35. The communication unit 110 is a network interface card (NIC), etc. For example, the communication unit 110 receives video data from the camera 30.

[0069] The input unit 120 inputs various types of information to the control unit 150 of the information processing device 100. For example, the user may operate the input unit 120 to input an execution command for a process in the learning phase and an execution command for a process in the inference phase. The user may also operate the input unit 120 to specify the hyperparameters described in FIG. 4 .

[0070] The display unit 130 displays information output from the control unit 150 .

[0071] The storage unit 140 includes a machine learning model 40, a learning data table 141, and a video buffer 142. The storage unit 140 is a storage device such as a memory.

[0072] The machine learning model 40 is a Transformer-based machine learning model. For example, the machine learning model 40 is HOID. The description of the machine learning model 40 is the same as the description of the machine learning model 40 described with reference to FIG. 2 .

[0073] The learning data table 141 is a table that holds multiple pieces of learning data. FIG. 9 is a diagram showing an example of the data structure of the learning data table. As shown in FIG. 9, the learning data table 141 has an item number, image data, annotation data, and composite image data. The item number is a number that identifies each record in the learning data table 141. The image data and annotation data correspond to the image data and annotation data included in the learning data described in FIG. 3. The image data includes an image of a person and an image of an object such as a product. The composite image data is composite image data generated by executing the process of FIG. 4. The composite image data is generated by the generation unit 152, which will be described later.

[0074] The video buffer 142 is a buffer that stores video data captured by the camera 30. The video buffer 142 may store the video data in association with the identification information of the camera.

[0075] Returning to the description of Fig. 8, the control unit 150 includes an acquisition unit 151, a generation unit 152, a learning processing unit 153, and an inference unit 154. The control unit 150 is a central processing unit (CPU), a graphics processing unit (GPU), or the like.

[0076] The acquisition unit 151 acquires video data from the camera 30 and stores the acquired video data in the video buffer 142. The acquisition unit 151 may also acquire data of the learning data table 141 from an external device or the like and store the data in the storage unit 140.

[0077] The generation unit 152 executes the process described in Fig. 4 based on the image data included in the training data to generate composite image data. The generation unit 152 stores the generated composite image data in the training data table 141. An example of the process performed by the generation unit 152 is shown below.

[0078] The generation unit 152 acquires a pair of image data and annotation data from the learning data table 141. The generation unit 152 extracts a person area and an object area from the image data based on the annotation data.

[0079] The generation unit 152 duplicates an image of an object area included in the image data. As described in step S1 of Fig. 4, the generation unit 152 identifies the center coordinates of the person area and the object area in the image data, and determines the synthesis direction of the duplicated object image based on the respective center coordinates.

[0080] The generation unit 152 generates white space around the image data, as described in step S2 of Fig. 4. The generation unit 152 adjusts the coordinates of the duplicated image so that the distance Δd between the upper left corner of the object area in the image data and the coordinates of the upper left corner of the duplicated image becomes a preset hyperparameter, as described in step S3 of Fig. 4. The generation unit 152 generates composite image data by deleting the white space, as described in step 4 of Fig. 4.

[0081] The learning processing unit 153 uses the learning data table 141 to train the machine learning model 40. For example, the learning processing unit 153 executes the contrastive learning described in FIG. 5 and the machine learning described in FIG. 6.

[0082] The learning processing unit 153 acquires from the learning data table 141 a pair of image data (positive example) and image data (positive example), a pair of image data (positive example) and composite image data (negative example), or a pair of composite image data (negative example) and composite image data (negative example).

[0083] The learning processing unit 153 inputs pairs of positive examples and positive examples, pairs of positive examples and negative examples, or pairs of negative examples and negative examples into the backbone 41 of the machine learning model 40, and calculates a cross-entropy error (contrastive loss) based on equation (1).

[0084] The learning processing unit 153 inputs the positive examples into the backbone 41 of the machine learning model 40, thereby acquiring output results from the decoder 44. The learning processing unit 153 compares the annotation data corresponding to the image data with the output results from the decoder 44, and calculates the bounding box loss, the object category loss, and the action loss.

[0085] The learning processing unit 153 performs machine learning of the parameters of the Encoder 43 and Decoder 44 of the machine learning model 40 based on the backpropagation method so that the overall error, which is the sum of the Bounding Box Loss, Object Category Loss, and Action Loss and the cross-entropy error, is small.

[0086] The inference unit 154 infers on which object the person's action acts, using the machine learning model 40 trained by the learning processing unit 153. The processing of the inference unit 154 corresponds to the processing described in FIG.

[0087] For example, the inference unit 154 acquires image data from the video buffer 142 and inputs the image data into the backbone 41 of the machine learning model 40 to infer the area of ​​the person, the area of ​​the object, and the person's action toward the object.

[0088] 1, the camera 30 is installed in a store having product shelves on which products are stored, and the product shelf area is included in the image data (video data) captured by the camera 30. Therefore, the inference unit 154 can identify the behavior of a customer taking a product from a product shelf by inputting the image data into the backbone 41 of the machine learning model 40.

[0089] The inference unit 154 may output and display the inference result to the display unit 130. The inference unit 154 outputs, as the inference result, screen data in which the area of ​​the person, the area of ​​the object, and the action of the person with respect to the object are arranged on the image data to the display unit 130.

[0090] Next, an example of a processing procedure of the information processing device 100 according to the present embodiment 1 will be described. Fig. 10 is a flowchart showing the processing procedure of the information processing device according to the present embodiment 1. As shown in Fig. 10, the generation unit 152 of the information processing device 100 generates synthetic image data based on image data in the learning data table 141 (step S101).

[0091] The learning processing unit 153 of the information processing device 100 inputs pairs of positive examples and positive examples, pairs of positive examples and negative examples, or pairs of negative examples and negative examples into the backbone 41 of the machine learning model 40 and extracts features from each (step S102).

[0092] The learning processing unit 153 inputs each feature amount to the Encoder 43 and calculates each output f() (step S103). The learning processing unit 153 calculates a cross-entropy error related to contrastive learning for each output f() (step S104).

[0093] The learning processing unit 153 inputs the positive examples into the backbone 41 of the machine learning model 40, and calculates losses based on the results output from the decoder 44 and the annotation data (step S105). The losses include bounding box loss, object category loss, and action loss.

[0094] The learning processing unit 153 trains the machine learning model 40 so that the overall error obtained by adding the cross-entropy error to the Bounding Box Loss, Object Category Loss, and Action Loss is small (step S106).

[0095] Next, the effects of the information processing device 100 according to the first embodiment will be described. The information processing device 100 extracts an image of an object used by a person from image data, and generates synthetic image data in which the extracted image is positioned near the person in the image data. The information processing device 100 executes machine learning of the machine learning model 40 using the synthetic image data, thereby generating a machine learning model 40 that can identify a person using an object.

[0096] When generating synthetic image data, the information processing device 100 accepts hyperparameter settings as described in Fig. 4. The information processing device 100 arranges the image 51c so that the distance Δd between the coordinates of the upper left corner of the object region 51b and the coordinates of the upper left corner of the arranged image 51c is a preset hyperparameter, under the condition that the image 51c does not overlap with the person region 51a. This makes it possible to generate pseudo-synthetic image data in which a similar object is included in the vicinity of the person holding the object.

[0097] In contrastive learning, the information processing device 100 trains the encoder 43 so that when two positive examples are input, the outputs from the encoder 43 approach each other. The information processing device 100 trains the encoder 43 so that when two negative examples are input, the outputs from the encoder 43 approach each other. Because the positive examples have common characteristics and the negative examples also have common characteristics, the parameters of the encoder 43 can be adjusted so that when image data having common characteristics are input to the machine learning model 40, the outputs from the encoder 43 approach each other.

[0098] In contrastive learning, the information processing device 100 trains the encoder 43 so that the outputs from the encoder 43 move away from each other when a positive example and a negative example are input. Because the features of positive examples and negative examples are not similar, the parameters of the encoder 43 can be adjusted so that the outputs from the encoder 43 move away from each other when image data having different features is input to the machine learning model 40.

[0099] The information processing device 100 infers the area of ​​a person, the area of ​​an object, and the person's action with respect to the object by inputting image data captured by the camera 30 into the backbone 41 of the machine learning model 40. For example, as described in FIG. 1 , the camera 30 is installed in a store having shelves on which products are stored, and the image data (video data) captured by the camera 30 includes the area of ​​the shelves. Therefore, by inputting the image data into the backbone 41 of the machine learning model 40, the information processing device 100 can identify the behavior of a customer removing a product from a shelf.

[0100] Here, we will show the results of an accuracy evaluation (Mean Average Precision) of the method of the information processing device 100 compared with the method of the conventional technology. FIG. 11 is a diagram showing the accuracy evaluation results. Dataset (1) is a public dataset that collects images of people holding objects in various scenes. Dataset (2) is a closed dataset that collects images of people reaching for products in a store.

[0101] With the conventional technique, the accuracy rating for dataset (1) was "59.6" and the accuracy rating for dataset (2) was "24.1." On the other hand, with the information processing device 100, the accuracy rating for dataset (1) was "60.1" and the accuracy rating for dataset (2) was "26.2." In other words, it can be seen that the technique of the information processing device 100 has a higher accuracy rating for both datasets compared to the conventional technique.

[0102] The above-described processing details of the information processing device 100 and the data structure of each piece of data are merely examples. For example, in the annotation data 52 described in Fig. 3, the person area 51a is indicated by the coordinates of the upper left corner and the lower right corner, and the object area 51b is indicated by the coordinates of the upper left corner and the lower right corner, but this is not limiting. For example, the annotation data can be extended as shown in Fig. 12.

[0103] FIG. 12 is a diagram showing an example of extended annotation data. In annotation data 52a shown in FIG. 12, multiple coordinates are set that trace the outline of a person. Furthermore, in annotation data 52a, multiple coordinates on the outline of an object are set. For example, in annotation data 52 described in FIG. 3, an object region 51b can be extracted, but in annotation data 52a, an object outline 52b can be extracted. While object region 51b includes an area that is not an object, object outline 52b includes an image of only the object. This allows the image of the object to be synthesized into synthetic image data to be the object image itself.

[0104] Although the generation unit 152 of the information processing device 100 generates the composite image data by executing the process described in Fig. 4, the process for generating the composite image data is not limited to this. For example, the generation unit 152 of the information processing device 100 may generate the composite image data by executing the process shown in Fig. 13.

[0105] 13 is a diagram illustrating another process for generating composite image data. As preprocessing, the generation unit 152 of the information processing device 100 executes a process for extracting skeletal data from the person region 51 a and a process for extracting a segmentation of the person.

[0106] An example of the process in which the generation unit 152 extracts skeletal data from the person's region 51 a will be described. The generation unit 152 infers the person's skeletal data 80 by inputting the person's region 51 a in the image data 51 into a skeletal inference model. The skeletal inference model is a trained model that receives image data of the person's region as input and outputs the person's skeletal data. The skeletal inference model is, for example, a neural network (NN).

[0107] Skeletal data is data in which two-dimensional or three-dimensional coordinates are set for multiple joints defined in a skeletal model of the human body. Here, the coordinates of each joint in the skeletal data are two-dimensional coordinates. FIG. 14 is a diagram showing an example of a skeletal model of the human body. For example, as shown in FIG. 14, the skeletal model of the human body is defined by 21 joints ar0 to ar20.

[0108] The relationship between the joints ar0 to ar20 shown in Fig. 14 and the joint names is as shown in Fig. 15. Fig. 15 is a diagram showing examples of joint names. For example, the joint name of joint ar0 is "SPINE_BASE". The joint names of joints ar1 to ar20 are as shown in Fig. 15, and explanations thereof will be omitted.

[0109] The process of extracting a person segmentation by the generation unit 152 will be described. The generation unit 152 performs segmentation on the image data 51, grouping regions in the image data 51 into groups having similar features (color, texture, subject), etc., and dividing the image data 51 into multiple regions. The generation unit 152 compares the multiple divided regions with the person region 51a specified in the annotation data, and extracts, as the person region, the region that most overlaps with the person region 51a among the multiple regions. In the example shown in FIG. 13 , the generation unit 152 extracts region 81 as the person region.

[0110] After performing the above pre-processing, the generating unit 152 performs the processes of steps S11 to S14 to generate the composite image data 90.

[0111] Step S11 will be described. The generation unit 152 identifies the coordinates (xc3, yc3) of the point of action based on the skeleton data 80. For example, the generation unit 152 identifies the coordinates of the left wrist joint ar19, among the joints ar0 to ar20 included in the skeleton data 80, as the coordinates of the point of action. The generation unit 152 identifies a predetermined range of area based on the coordinates (xc3, yc3) of the point of action as the object area 82. Note that the generation unit 152 may compare the object area specified by the annotation data with the right wrist joint ar20 and the left wrist joint ar19, and identify the coordinates of the wrist joint closest to the object area as the coordinates of the point of action.

[0112] Step S12 will now be described. The generating unit 152 generates image data 53 by generating a margin around the image data 51.

[0113] Step S13 will now be described. The generation unit 152 copies an image of the object region 82 included in the image data 51. The copied image is referred to as image 83. The generation unit 152 places the copied image 83 in an area that does not overlap with the person region 81. Furthermore, the generation unit 152 adjusts the coordinates of the image 83 so that the distance Δd between the coordinates (xc3, yc3) of the point of action of the object region 82 and the coordinates (xc4, yc4) of the upper left corner of the placed image 83 becomes a preset hyperparameter.

[0114] Step S4 will now be described. The generating unit 152 generates the composite image data 90 by deleting the white space of the image data 53.

[0115] As described in FIG. 13, the information processing device 100 executes preprocessing and executes the processes of steps S11 to S14, thereby generating composite image data 90 by combining the image data 51 and the image 83.

[0116] Next, an example of a system according to the second embodiment will be described. Fig. 16 is a diagram showing an example of a system according to the second embodiment. As shown in Fig. 1, this system includes cameras 30a, 30b, and 30c and an information processing device 200. The cameras 30a to 30c and the information processing device 200 are connected to each other via a network 35.

[0117] Cameras 30a to 30c are installed inside a store that has product shelves on which products are stored. Cameras 30a to 30c capture video including the product shelves installed inside the store and transmit the captured video data to information processing device 200. In the following description, the video data will be referred to as "video data." The video data includes time-series image data (still images). Cameras 30a to 30c will be collectively referred to as "camera 30."

[0118] The information processing device 200 executes various processes using the machine learning model 40. In addition to the processes of the information processing device 100 described above, the information processing device 200 generates learning data for training the machine learning model 40 based on the video data received from the camera 30.

[0119] For example, the information processing device 200 sequentially inputs time-series image data included in the video data acquired from the camera 30 into the identification model 45 to identify the area of ​​a person, the area of ​​a product that is the target of the person's behavior, and the person's behavior with respect to the product. The identification model 45 is a trained machine learning model that outputs the area of ​​a person, the area of ​​a product that is the target of the person's behavior, and the person's behavior with respect to the product when image data is input. For example, the identification model 45 is a HOID or the like.

[0120] The information processing device 200 repeatedly executes the above process to identify image data that meets a predetermined condition. The predetermined condition is a condition in which the area of ​​the product that is the target of the person's behavior is included in the area of ​​the product shelf, and the person's behavior with respect to the product is "grasping." Note that even if a portion of the area of ​​the product that is the target of the person's behavior is included in the area of ​​the product shelf, it may be determined that the area of ​​the product that is the target of the person's behavior is included in the area of ​​the product shelf. Also, although the person's behavior is described here as "grasping," the administrator may set other behaviors.

[0121] Furthermore, the information processing device 200 analyzes product areas in the image data that satisfy predetermined conditions and identifies the product type. The product types include electrical appliances, detergents, food, books, cosmetics, etc. The information processing device 200 may also use the identification model 45 to identify the product type included in the product area.

[0122] The information processing device 200 identifies image data that satisfies predetermined conditions from time-series image data, and generates training data based on the identified image data. As described with reference to FIG. 3 , the training data includes image data and annotation data. The information processing device 200 sets the image data that satisfies the predetermined conditions as image data for the training data. The information processing device 200 generates annotation data for the training data based on the person area, the product area that is the target of the person's behavior, the person's behavior with respect to the product, and the product type that are output when the image data that satisfies the predetermined conditions is input to the identification model 45.

[0123] FIG. 17 is a diagram for explaining the processing of an information processing device according to the second embodiment. The image data shown in FIG. 17 is image data that satisfies a predetermined condition. For example, the information processing device 200 inputs image data 22 into the identification model 45, and identifies a person area 22a, a product area 22b that is the target of the person's behavior, and the person's behavior of "grasping" the product. The information processing device 200 also identifies the type of product included in the product area 22b as "Book." The product area 22b (part of area 22b) is included in the area of ​​the product shelf 23. Therefore, the information processing device 200 identifies the image data 22 as image data that satisfies a predetermined condition.

[0124] The information processing device 200 generates learning data 25 based on information extracted from the image data 22. The information processing device 200 sets the image data 22 as image data of the learning data 25. Furthermore, the information processing device 200 sets "Person1: {x1, y1, x2, y2}" in the annotation data 26 based on the person region 22a. The coordinates of the upper left corner of the person region 22a are set to "x1, y1", and the coordinates of the lower right corner of the person region 22a are set to "x2, y2".

[0125] Based on the product area 22b and the product type "Book", the information processing device 200 sets "Book1: {x1', y1', x2', y2'}" in the annotation data 26. It is indicated that the coordinates of the upper left corner of the object area 22b are "x1', y1'", and the coordinates of the lower right corner of the object area 22b are "x2', y2'".

[0126] Based on the type of product and the person's action of "holding" the product, the information processing device 200 sets "Action: {Person1, Book1, Hold}" in the annotation data 26. This indicates that the person in area 22a is holding the object (book) in area 22b.

[0127] The information processing device 200 generates a plurality of pieces of training data by repeatedly executing the above process on image data that satisfies predetermined conditions. The information processing device 200 trains the machine learning model 40 based on the generated training data. The process by which the information processing device 200 trains the machine learning model 40 based on the training data is similar to the process by which the information processing device 100 in the first embodiment trains the machine learning model 40. For example, the information processing device 200 executes the process of generating synthetic image data, the process in the learning phase, and the process in the inference phase described in the first embodiment.

[0128] As described above, the information processing device 200 according to the second embodiment acquires video data (time-series image data) from the camera 30 and inputs the time-series image data into the identification model 45 to identify the area of ​​the person, the area of ​​the product that is the target of the person's behavior, and the person's behavior with respect to the product. The information processing device 200 identifies, from among multiple image data, image data in which the area of ​​the product that is the target of the person's behavior is included in the area of ​​the product shelf and which satisfies the condition that the person's behavior with respect to the product is "grasping." The information processing device 200 generates learning data based on the identified image data and trains the machine learning model 40 based on the generated learning data. This makes it possible to generate a machine learning model that accurately estimates which object a person's behavior is acting on when image data containing many similar objects is used.

[0129] Next, a configuration example of an information processing device 200 that executes the above-described processing will be described. Fig. 18 is a functional block diagram showing the configuration of an information processing device according to Example 2. As shown in Fig. 18, the information processing device 200 has a communication unit 210, an input unit 220, a display unit 230, a storage unit 240, and a control unit 250.

[0130] The descriptions of the communication unit 210, the input unit 220, and the display unit 230 are the same as the descriptions of the communication unit 110, the input unit 120, and the display unit 130 described with reference to FIG.

[0131] The storage unit 140 includes a machine learning model 40, a specific model 45, a learning data table 241, and a video buffer 242. The storage unit 140 is a storage device such as a memory.

[0132] The machine learning model 40 is a Transformer-based machine learning model. For example, the machine learning model 40 is HOID. The description of the machine learning model 40 is the same as the description of the machine learning model 40 described with reference to FIG. 2 .

[0133] The identification model 45 is a Transformer-based machine learning model that is a trained machine learning model. For example, the identification model 45 is HOID. When image data is input, the identification model 45 outputs a person area, a product area that is the target of the person's behavior, and the person's behavior with respect to the product. The identification model 45 may further output the type of product included in the product area.

[0134] The learning data table 241 is a table that holds multiple pieces of learning data. The data structure of the learning data table 241 is the same as the data structure of the learning data table 141 described in FIG. 9 . For example, the learning data table 241 includes an item number, image data, annotation data, and composite image data. A pair of image data and annotation data corresponds to learning data. The learning data set in the learning data table 241 includes learning data that is prepared in advance and learning data generated by the process described in FIG. 17 .

[0135] The video buffer 242 is a buffer that stores video data captured by the camera 30. The video buffer 242 may store the video data in association with the identification information of the camera.

[0136] Returning to the description of Fig. 18, the control unit 250 includes an acquisition unit 251, an identification unit 252, a generation unit 253, a learning processing unit 254, and an inference unit 255. The control unit 250 is a central processing unit (CPU), a graphics processing unit (GPU), or the like.

[0137] The acquisition unit 251 acquires video data from the camera 30 and stores the acquired video data in the video buffer 242. The acquisition unit 251 may also acquire data of the learning data table 241 from an external device or the like and store the data in the storage unit 240.

[0138] The identification unit 252 acquires video data (time-series image data) captured by the camera 30 from the video buffer 242, and generates learning data based on the acquired image data. The identification unit 252 registers the generated learning data in the learning data table 241. An example of a processing procedure of the identification unit 252 is shown below.

[0139] The identification unit 252 inputs image data into the learning data table 241, acquires the area of ​​the person, the area of ​​the product that is the target of the person's action, and the person's action with respect to the product, and identifies image data that satisfies a predetermined condition. The predetermined condition is that the area of ​​the product that is the target of the person's action is included in the area of ​​the product shelf, and the person's action with respect to the product is "grasping."

[0140] The identification unit 252 generates learning data based on image data that satisfies predetermined conditions. The process by which the identification unit 252 generates learning data based on image data is similar to the process described in FIG. 17 . For example, the identification unit 252 sets image data that satisfies predetermined conditions as learning data. The identification unit 252 generates annotation data based on the output result when the image data that satisfies the predetermined conditions is input to the identification model 45. The output result of the identification model 45 includes the area of ​​the person, the area of ​​the product that is the target of the person's behavior, and the person's behavior with respect to the product.

[0141] The identification unit 252 generates a plurality of pieces of learning data by repeatedly executing the above process on image data that satisfies predetermined conditions, and stores the generated learning data in the learning data table 241. Note that the process of the identification unit 252 is executed at a stage before the generation unit 253, the learning processing unit 254, and the inference unit 255 perform their processes.

[0142] The generating unit 253 generates composite image data based on the image data included in the learning data table 241. The process of generating the composite image data by the generating unit 253 is similar to the process by the generating unit 152 described in the first embodiment.

[0143] The learning processing unit 254 uses the learning data table 241 to train the machine learning model 40. The process by which the learning processing unit 254 trains the machine learning model 40 is similar to the process by the learning processing unit 153 described in the first embodiment.

[0144] The inference unit 255 infers on which object the person's action is acting, using the machine learning model 40 trained by the learning processing unit 254. The process by which the inference unit 255 infers on which object the person's action is acting, using the machine learning model 40, is similar to the process by the inference unit 154 described in the first embodiment.

[0145] Next, an example of a processing procedure of the information processing device according to the second embodiment will be described. Fig. 19 is a flowchart showing the processing procedure of the information processing device according to the second embodiment. Fig. 19 illustrates a processing procedure in which the identification unit 252 generates learning data from image data. As shown in Fig. 19, the acquisition unit 251 of the information processing device 200 acquires video data (time-series image data) from the camera 30 and stores it in the video buffer 242 (step S201).

[0146] The identification unit 252 of the information processing device 200 acquires image data from the video buffer 242 (step S202). The identification unit 252 inputs the image data to the identification model 45 and determines whether the image data satisfies a predetermined condition (step S203).

[0147] If the image data does not satisfy the predetermined condition (step S204, No), the specifying unit 252 proceeds to step S207. On the other hand, if the image data satisfies the predetermined condition (step S204, Yes), the specifying unit 252 proceeds to step S205.

[0148] The identifying unit 252 generates learning data based on image data that satisfies a predetermined condition (step S205), and registers the learning data in the learning data table 241 (step S206).

[0149] The identifying unit 252 determines whether or not unprocessed image data exists in the video buffer 242 (step S207). If unprocessed image data exists in the video buffer 242 (step S207, Yes), the identifying unit 252 proceeds to step S202. On the other hand, if unprocessed image data does not exist in the video buffer 242 (step S207, No), the identifying unit 252 ends the process.

[0150] For example, after executing the process of FIG. 19, the information processing apparatus 200 executes the process of FIG. 10 in the same manner as the information processing apparatus 100 in the first embodiment.

[0151] Next, the effects of the information processing device 200 according to the second embodiment will be described. The information processing device 200 acquires video data (time-series image data) from the camera 30 and inputs the time-series image data into the identification model 45 to identify the area of ​​a person, the area of ​​a product that is the target of the person's behavior, and the person's behavior with respect to the product. The information processing device 200 identifies, from among multiple image data, image data in which the area of ​​the product that is the target of the person's behavior is included in the area of ​​the product shelf and which satisfies the condition that the person's behavior with respect to the product is "grasping." The information processing device 200 generates learning data based on the identified image data and trains the machine learning model 40 based on the generated learning data. This makes it possible to generate a machine learning model that accurately estimates which object a person's behavior is acting on, even when image data contains many similar objects. The information processing device 200 can also automatically generate multiple pieces of learning data based on the time-series image data captured by the camera 30.

[0152] The information processing device 200 analyzes product areas in image data that satisfy predetermined conditions, identifies the type of product, sets this as annotation data for the learning data, and uses this learning data to perform machine learning on the machine learning model 40. This allows the machine learning model 40 to learn the behavior of selecting a specific product from among multiple types of products.

[0153] The processing details of the information processing device 200 in the second embodiment described above are merely an example. The information processing device 200 inputs image data into the identification model 45 to identify the behavior of a person with respect to a product, but the present invention is not limited to this. For example, the information processing device 200 may generate time-series skeletal data from time-series image data and identify the area of ​​the person based on the time-series skeletal data.

[0154] For example, the identification unit 252 of the information processing device 200 inputs image data into the identification model 45 to identify a person's area. The information processing device 200 infers skeletal data by inputting an image of the person's area included in the image data into the skeletal inference model described in Example 1. The identification unit 252 repeatedly executes the above process for each image data.

[0155] The identification unit 252 compares the transition of the wrist position in the time-series skeletal data with a rule table to identify the person's behavior. The rule table stores data associating the transition of the wrist position with the type of behavior. The identification unit 252 may also estimate the type of behavior of the person by further utilizing the presence or absence of an object at the wrist position.

[0156] Next, a description will be given of an example of the hardware configuration of a computer that realizes the same functions as the above-described information processing devices 100 and 200. Fig. 20 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the information processing devices of the embodiment.

[0157] 20, computer 300 has a CPU 301 that executes various types of arithmetic processing, an input device 302 that accepts data input from a user, and a display 303. Computer 300 also has a communication device 304 that exchanges data with external devices via a wired or wireless network, and an interface device 305. Computer 300 also has a RAM 306 that temporarily stores various types of information, and a hard disk drive 307. Each of devices 301 to 307 is connected to a bus 308.

[0158] The hard disk drive 307 stores an acquisition program 307 a, a specification program 307 b, a generation program 307 c, a learning processing program 307 d, and an inference program 307 e. The CPU 301 reads out each of the programs 307 a to 307 e and loads them into the RAM 306.

[0159] The acquisition program 307a functions as an acquisition process 306a. The identification program 307b functions as an identification process 306b. The generation program 307c functions as a generation process 306c. The learning processing program 307d functions as a learning processing process 306d. The inference program 307e functions as an inference process 306e.

[0160] The processing of the acquisition process 306a corresponds to the processing of the acquisition units 151 and 251. The processing of the identification process 306b corresponds to the processing of the identification unit 252. The processing of the generation process 306c corresponds to the processing of the generation units 152 and 253. The processing of the learning processing process 306d corresponds to the processing of the learning processing units 153 and 254. The processing of the inference process 306e corresponds to the processing of the inference units 154 and 255.

[0161] It should be noted that each of the programs 307a to 307e does not necessarily have to be stored in the hard disk drive 307 from the beginning. For example, each program may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 300. Then, the computer 300 may read and execute each of the programs 307a to 307e.

[0162] 40 Machine learning model 45 Identification model 100, 200 Information processing device 110, 210 Communication unit 120, 220 Input unit 130, 230 Display unit 140, 240 Storage unit 141, 241 Learning data table 142, 242 Video buffer 150, 250 Control unit 151, 251 Acquisition unit 152, 253 Generation unit 153, 254 Learning processing unit 154, 255 Inference unit 252 Identification unit

Claims

1. Acquire video footage of an area including product shelves where products are placed, By analyzing the acquired video, the behavior of the person grasping the product is identified; Identifying, from among a plurality of image frames constituting the acquired video, an image frame that includes the product stored on the product shelf and the product being held by the person, based on the identified person's behavior of holding the product; Using the identified image frames, a machine learning model is generated that is trained to identify a person who is taking the product from the shelf. A generating program that causes a computer to execute a process.

2. The process of acquiring the video acquires a video of an area including a product shelf on which a plurality of types of products are arranged, Identifying a specific product held by the person based on the action of the person with respect to the product identified by the process of identifying the action; Identifying an image frame including a product stored on the product shelf and the specific product held by the person from among a plurality of image frames constituting the acquired video based on the action of the person with respect to the product identified by the process of identifying the action; Using the identified image frame, a machine learning model is generated that is trained to identify a person who is taking the specific product from the product shelf.

2. The generation program according to claim 1, further comprising a step of causing a computer to execute a process.

3. extracting an image of a commodity being held by the person from the image frames identified by the process of identifying the image frames; further causing the computer to execute a process of generating a composite image by arranging the extracted product image at a predetermined position within the image frame; The generation program according to claim 1, wherein the process of generating the machine learning model uses the synthetic image to generate the machine learning model trained to identify a person who is taking the product from the product shelf.

4. The process of generating the composite image includes: accepting a parameter setting based on a distance between a coordinate position of the person and a coordinate position of an object held by the person; generating a coordinate position of a placement candidate for the extracted image of the object based on the set parameters; determining whether the generated coordinate position is included within a region related to the size of the object; 4. The generating program according to claim 3, further comprising a process for generating a composite image by arranging an image of the object in the acquired image based on the determination result.

5. The generation program of claim 1, further comprising inputting an image taken by a camera in a store, the image including the product shelf containing the product and the person, into the machine learning model, to cause the computer to execute a process of identifying the action of the person removing the product from the product shelf.

6. further causing the computer to execute a process of generating skeletal information of the person by analyzing the image acquired by the process of acquiring the image; 2. The generation program according to claim 1, wherein the process of identifying the behavior identifies the behavior of the person gripping the product based on the skeleton information.

7. Identifying a machine learning model that has been trained to identify a person who is taking a product from a product shelf, based on the action of a person grasping the product identified by analyzing a video captured of an area including a product shelf on which the product is placed, using image frames extracted from a plurality of image frames constituting the video, the image frames including the product stored on the product shelf and the product grasped by the person; Using the identified machine learning model, a person who takes the product from the shelf is identified. An inference program that causes a computer to execute processing.

8. Acquire video footage of an area including product shelves where products are placed, By analyzing the acquired video, the behavior of the person grasping the product is identified; Identifying, from among a plurality of image frames constituting the acquired video, an image frame that includes the product stored on the product shelf and the product being held by the person, based on the identified person's behavior of holding the product; Using the identified image frames, a machine learning model is generated that is trained to identify a person who is taking the product from the shelf. A generating method characterized in that the processing is carried out by a computer.

9. Acquire video footage of an area including product shelves where products are placed, By analyzing the acquired video, the behavior of the person grasping the product is identified; Identifying, from among a plurality of image frames constituting the acquired video, an image frame that includes the product stored on the product shelf and the product being held by the person, based on the identified person's behavior of holding the product; Using the identified image frames, a machine learning model is generated that is trained to identify a person who is taking the product from the shelf. An information processing device comprising a control unit that executes processing.