Generation program, generation method, and information processing device

Abstract

This information processing device acquires an image including a person, and analyzes the acquired image to extract an object used by the person in the image. The information processing device generates a composite image in which the extracted object is placed at a position satisfying a predetermined condition with the position of the object used by the person in the acquired image as a reference. The information processing device uses the generated composite image to generate a machine learning model trained to identify a person who uses the object.

Description

GENERATION PROGRAM, GENERATION METHOD, AND INFORMATION PROCESSING APPARATUS

[0001] The present invention relates to a generation program and the like.

[0002] In a store, if it were possible to detect specific customer actions toward products or other objects, this could be used to analyze purchasing trends. For example, the act of a customer picking up a product from a shelf is one behavior that reveals the customer's intent to purchase.

[0003] Prior art techniques 1 and 2 for detecting customer actions on an object will be described below.

[0004] Conventional technique 1 will now be described. Fig. 17 is a diagram for explaining conventional technique 1. Here, a device that executes conventional technique 1 will be referred to as "conventional device A." Conventional device A estimates the relevance between a person and an object on a rule basis.

[0005] 17 , for example, conventional device A analyzes video data 11 captured by a camera to identify a person region 11a and an object region 11b. Furthermore, conventional device A analyzes the person region 11a to identify skeletal information 11c of the person. Coordinate information of each joint of the person is set in the skeletal information 11c. By using the skeletal information 11c, it is possible to identify the coordinates of parts of the person, such as the hands.

[0006] The conventional device A detects that a person is holding an object when it sequentially identifies that a person's hand has entered a product shelf 12, an object has been detected from the product shelf 12, and the person's hand is touching the object based on a preset detection rule.

[0007] In Prior Art 1, in order to improve the detection accuracy, detailed detection rules are set according to the placement of the camera and the direction of the person.

[0008] Conventional technique 2 will now be described. FIG. 18 is a diagram for explaining conventional technique 2. Here, a device that executes conventional technique 2 is referred to as "conventional device B." Conventional device B uses HOID (Human-Object Interaction Detection). For example, HOID is a Transformer-based machine learning model.

[0009] 18 , conventional device B uses machine learning model 15. By inputting image data 16 into machine learning model 15, conventional device B outputs a person area, an object area, and an action of the person with respect to the object.

[0010] The machine learning model 15 includes a backbone 15a, an adder 15b, an encoder 15c, and a decoder 15d. When image data 16 is input, the backbone 15a outputs feature quantities of the image data 16. For example, the conventional device B divides the image data 16 into multiple blocks and inputs the blocks to the backbone 15a.

[0011] The adder 15b receives the result of positional encoding of the image data 16 and the output result of the backbone 15a. The adder 15b adds the result of positional encoding to the output result of the backbone 15a and outputs the result to the encoder 15c. In positional encoding, a process is executed to encode each piece of position information of the divided image data 16.

[0012] The encoder 15c converts the data input from the adder 15b into vector data and inputs the vector data to the decoder 15d. When the decoder 15d receives the vector data, it outputs bounding box data, object category data, and action data. The bounding box data indicates the area of ​​a person, the area of ​​an object, etc. included in the image data 16. The object category data indicates the attributes of the area indicated by each bounding box. The attributes include person, object, etc. The action data indicates the action of a person with respect to an object.

[0013] Unlike in Prior Art 1, Prior Art 2 can train the machine learning model 15 by using training data that defines the relationship between input data and correct labels, without having to set detailed detection rules. Furthermore, Prior Art 2 can simultaneously identify the area of ​​a person, the area of ​​an object, and the person's action relative to the object by inputting image data.

[0014] 19 is a diagram showing an example of a processing result of Conventional Technique 2. For example, Conventional Device B inputs image data 18 into machine learning model 15, thereby outputting person region 18a, object region 18b, and action "grasp." Conventional Device B inputs image data 19 into machine learning model 15, thereby outputting person region 19a, object region 19b, and action "grasp."

[0015] JP 2018-15408 A

[0016] However, in the above-mentioned conventional technology 2, when there are many similar objects, such as product shelves in a store, or when there are many objects in the background, it is not possible to accurately estimate which object a person's action is acting on.

[0017] 20 is a diagram showing an example of image data in which the estimation accuracy of Conventional Technique 2 decreases. For example, image data 20 includes many similar products, making it difficult for Conventional Technique 2 to estimate which object a person's action is acting on. Image data 21 includes many objects in the background, making it similarly difficult for Conventional Technique 2 to estimate which object a person's action is acting on.

[0018] Even when detecting a person holding an object based on detection rules, as in prior art 1, the detection accuracy decreases if the image data contains many similar products, as in image data 20 and 21 in Figure 20.

[0019] For this reason, there is a need to generate a machine learning model that can accurately estimate which objects a person's actions are acting on when using image data that contains many similar objects.

[0020] In one aspect, the present invention aims to provide a generation program, a generation method, and an information processing device that can generate a machine learning model that accurately estimates which objects a person's actions are affecting in image data that contains many similar objects.

[0021] In the first proposal, a computer performs the following processes: acquires an image including a person, and extracts an object used by the person in the image by analyzing the acquired image; generates a synthetic image in which the extracted object is placed in a position that satisfies a predetermined condition based on the position of the object used by the person in the acquired image; and uses the generated synthetic image to generate a machine learning model trained to identify people using the object.

[0022] The present invention can generate a machine learning model that accurately estimates which object a person's action is acting on, for image data that contains many similar objects.

[0023] FIG. 1 is a diagram illustrating an example of a system according to the present embodiment. FIG. 2 is a diagram illustrating a machine learning model. FIG. 3 is a diagram illustrating training data. FIG. 4 is a diagram illustrating a process in which an information processing device generates synthetic image data. FIG. 5 is a diagram illustrating object learning performed by the information processing device. FIG. 6 is a diagram illustrating machine learning performed by the information processing device. FIG. 7 is a diagram illustrating a process in the inference phase according to the present embodiment. FIG. 8 is a functional block diagram illustrating a configuration of an information processing device according to the present embodiment. FIG. 9 is a diagram illustrating an example of a data structure of a training data table. FIG. 10 is a flowchart illustrating a processing procedure of the information processing device according to the present embodiment. FIG. 11 is a diagram illustrating a result of accuracy evaluation. FIG. 12 is a diagram illustrating an example of extension of annotation data. FIG. 13 is a diagram illustrating another process for generating synthetic image data. FIG. 14 is a diagram illustrating an example of a skeletal model of a human body. FIG. 15 is a diagram illustrating an example of joint names. FIG. 16 is a diagram illustrating an example of a hardware configuration of a computer that realizes the same functions as the information processing device according to the embodiment. FIG. 17 is a diagram illustrating Prior Art 1. FIG. 18 is a diagram illustrating Prior Art 2. FIG. 19 is a diagram illustrating an example of a processing result of Prior Art 2. FIG. 20 is a diagram showing an example of image data in which the estimation accuracy of the conventional technique 2 decreases.

[0024] Hereinafter, embodiments of the generating program, generating method, and information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to these embodiments.

[0025] Fig. 1 is a diagram showing an example of a system according to this embodiment. As shown in Fig. 1, this system includes cameras 30a, 30b, and 30c and an information processing device 100. The cameras 30a to 30c and the information processing device 100 are connected to each other via a network 35.

[0026] Cameras 30a to 30c are installed inside a store that has product shelves on which products are stored. Cameras 30a to 30c capture video including the product shelves installed inside the store and transmit the captured video data to information processing device 100. In the following description, the video data will be referred to as "video data." The video data includes time-series image data (still images). Cameras 30a to 30c will be collectively referred to as "camera 30."

[0027] The information processing device 100 executes various processes using a machine learning model 40. FIG. 2 is a diagram illustrating the machine learning model. As shown in FIG. 2, the machine learning model 40 includes a backbone 41, an adder 42, an encoder 43, and a decoder 44. When image data is input, the backbone 41 outputs feature quantities of the image data. For example, the information processing device 100 divides image data into multiple blocks and inputs the blocks to the backbone 41. Although not described below, the image data input to the backbone 41 is divided and input to the backbone 41.

[0028] The adder 42 receives the result of positional encoding of the image data and the output result of the backbone 41. The adder 42 adds the result of positional encoding and the output result of the backbone 41, and outputs the result to the encoder 43. In positional encoding, the information processing device 100 executes a process of encoding each piece of position information of the image data divided when it is input to the backbone 41.

[0029] The encoder 43 converts the data input from the adder 42 into vector data and inputs the vector data to the decoder 44. When the decoder 44 receives the vector data, it outputs bounding box data, object category data, and action data. The bounding box data indicates the area of ​​a person, the area of ​​an object, etc. included in the image data. The object category data indicates the attributes of the area indicated by each bounding box. The attributes include a person, an object, etc. The action data indicates the action of a person against an object. For example, the information processing device 100 can identify the area of ​​a person and the area of ​​an object in the image data by using the bounding box data and object category data.

[0030] The information processing device 100 executes a process for generating synthetic image data, a process in the learning phase, and a process in the inference phase. The process for generating synthetic image data, the process in the learning phase, and the process in the inference phase will be described below in this order.

[0031] First, a description will be given of a process for generating synthetic image data by the information processing device 100. The information processing device 100 generates synthetic image data based on learning data.

[0032] 3 is a diagram illustrating training data. As shown in FIG. 3, training data 50 includes image data 51 and annotation data 52. For example, the image data 51 includes an image of a person and an image of an object. Input data for training the machine learning model 40 corresponds to the image data 51, and correct labels correspond to the annotation data 52.

[0033] The annotation data 52 includes data on the area of ​​a person, data on the area of ​​an object, and data on the person's action with respect to the object. In the example shown in Fig. 3, the data on the area of ​​the person is "Person1: {x1, y1, x2, y2}". This indicates that the coordinates of the upper left corner of the person's area 51a are "x1, y1", and the coordinates of the lower right corner of the person's area 51a are "x2, y2".

[0034] The data for the object (bottle) region is "Bottle1: {x1', y1', x2', y2'}", which indicates that the coordinates of the upper left corner of object region 51b are "x1', y1'" and the coordinates of the lower right corner of object region 51b are "x2', y2'".

[0035] The data of the person's action on the object is "Action: {Person1, Bottle1, Hold}", which indicates that the person in area 51a is holding the object (bottle) in area 51b.

[0036] The information processing device 100 generates synthetic image data using the training data 50 shown in FIG. 3. FIG. 4 is a diagram for explaining the process by which the information processing device generates synthetic image data. The information processing device 100 extracts a person area 51a and an object area 51b included in image data 51 based on annotation data 52 included in the training data 50. The information processing device 100 generates synthetic image data 54 by executing the processes of steps S1 to S4. As will be described later, the information processing device 100 generates synthetic image data 54 by combining image data 51 with image 51c.

[0037] Step S1 will now be described. The information processing device 100 identifies the center coordinates (xc1, yc1) of the person region 51a. The information processing device 100 identifies the center coordinates (xc2, yc2) of the object region 51b. The information processing device identifies the compositing direction from the positional relationship between the center coordinates (xc1, yc1) and the center coordinates (xc2, yc2). The compositing direction indicates whether the image 51c is to be composited on the "left side" of the person region 51a or on the "right side" of the person region 51a.

[0038] The information processing device 100 determines that the composition direction is "left side" when "xc1-xc2<0". The information processing device 100 determines that the composition direction is "right side" when "xc1-xc2≧0". In the example shown in FIG. 4, "xc1-xc2<0" holds, so the composition direction is "left side".

[0039] Step S2 will now be described. The information processing apparatus 100 generates image data 53 by creating a margin around the image data 51.

[0040] Step S3 will now be described. The information processing device 100 copies an image of the object region 51b included in the image data 51. Because the composition direction determined in step S1 is "left side," the information processing device 100 places the copied image 51c to the left of the person region 51a in an area that does not overlap with the region 51a. Furthermore, the information processing device 100 adjusts the coordinates of the image 51c so that the distance Δd between the coordinates (x1', y1') of the upper left corner of the object region 51b and the coordinates (x3, y3) of the upper left corner of the placed image 51c becomes a preset hyperparameter.

[0041] Step S4 will now be described. The information processing device 100 generates composite image data 54 by deleting the margins of image data 53. In the example shown in Fig. 4, the case where image 51c is included within the range of image data 51 has been described, but depending on the processing of step S3, part of image 51c may be included in the margins of image data 53. If part of image 51c is included in the margins of image data 53, the part of image 51c that is included in the margins is deleted by the processing of step S4.

[0042] As described with reference to FIG. 4, the information processing device 100 executes the processes of steps S1 to S4 to generate composite image data 54 by combining the image data 51 and the image 51c.

[0043] The information processing device 100 generates a plurality of composite image data by repeatedly executing the process described above with reference to FIG. 4 for training data other than the training data 50 described with reference to FIG.

[0044] Next, a description will be given of the processing in the learning phase executed by the information processing device 100. For example, the information processing device 100 performs contrastive learning on the machine learning model 40 using image data and synthetic image data, and also performs machine learning on the machine learning model 40 using the learning data.

[0045] The contrastive learning performed by the information processing device 100 will be described. FIG. 5 is a diagram for explaining the contrastive learning performed by the information processing device. For convenience of explanation, FIG. 5 uses image data 60 and 61 and composite image data 62 and 63. The image data 60 and 61 are image data included in the learning data. The image data 60 and the image data 61 have a common feature in that they show a person holding an object.

[0046] The composite image data 62 is generated by executing the process described in Fig. 4 using the image data 60. The composite image data 63 is generated by executing the process described in Fig. 4 using the image data 61. The composite image data 61 and the composite image data 63 have a common feature in that they both show a person holding an object and another object being placed near the person.

[0047] In the following description, the image data 60, 61, and image data including an image of an object and an image of a person holding the object will be referred to as "positive examples" as appropriate. The composite image data obtained by executing the process described in FIG. 4 will be referred to as "negative examples" as appropriate. As described above, the positive examples share common characteristics, and the negative examples also share common characteristics. In the contrastive learning performed by the information processing device 100, the encoder 43 is trained so that the outputs from the encoder 43 when two positive examples are input are closer to each other. The information processing device 100 trains the encoder 43 so that the outputs from the encoder 43 when two negative examples are input are closer to each other.

[0048] Note that, since pairs of positive examples and negative examples do not have common features, the information processing device 100 trains the encoder 43 so that when a positive example and a negative example are input, the outputs from the encoder 43 move away from each other.

[0049] The information processing device 100 inputs the image data 60, 61 and the composite image data 62, 63 to the backbone 41 of the machine learning model 40, thereby obtaining an output f() from the encoder 43. For example, f() is vector data.

[0050] For example, the information processing device 100 calculates the error (cross-entropy error of cosine similarity) between the output f() when a positive example or a negative example is input to the Backbone 41 and the output f() when another positive example or negative example is input to the Backbone 41 using equation (1).

[0051]

[0052] l shown in formula (1) ｘｉ，ｘｊ is calculated by equation (2). f() shown in equation (2) is the output from the encoder 43. g() is the cosine similarity.

[0053]

[0054] For example, the output from the Encoder 43 obtained by inputting the image data 60 to the Backbone 41 is f ６０ The output from the Encoder 43 obtained by inputting the image data 61 to the Backbone 41 is expressed as f ６１ The output from the Encoder 43 obtained by inputting the image data 62 to the Backbone 41 is expressed as f ６２ The output from the Encoder 43 obtained by inputting the image data 63 to the Backbone 41 is expressed as f ６３ It is written as ().

[0055] The cross-entropy error has a smaller value as f() output from the Encoder 43 when a positive example is input to the Backbone 41 approaches f() output from the Encoder 43 when another positive example is input to the Backbone 41. For example, the information processing device 100 calculates f ６０ () and f ６１The parameters of the Encoder 43 are trained so that the values ​​of () and () approach each other.

[0056] The cross-entropy error becomes smaller as f() output from the Encoder 43 due to a negative example being input to the Backbone 41 and f() output from the Encoder 43 due to another negative example being input to the Backbone 41 become closer. ６２ () and f ６３ The parameters of the Encoder 43 are trained so that () and

[0057] The cross-entropy error becomes smaller as the difference between f() output from the Encoder 43 when a positive example is input to the Backbone 41 and f() output from the Encoder 43 when a negative example is input to the Backbone 41 increases. ６０ () and f ６２ The parameters of the encoder 43 are trained so that f ６０ () and f ６３ The parameters of the encoder 43 are trained so that f ６１ () and f ６２ The parameters of the encoder 43 are trained so that f ６１ () and f ６３ The parameters of the Encoder 43 are trained so that () and are moved away from each other.

[0058] Here, the information processing device 100 performs the contrastive learning described with reference to FIG. 5, and executes machine learning of the parameters of the Encoder 43 and the Decoder 44 of the machine learning model 40 based on the backpropagation method.

[0059] 6 is a diagram illustrating machine learning performed by an information processing device. Image data 60 and 61 are image data included in training data, and correct labels (annotation data) corresponding to the image data 60 and 61 are set in advance. Composite image data 62 is generated based on the image data 60. Composite image data 63 is generated based on the training data 61.

[0060] For example, the information processing device 100 inputs image data 60 into the backbone 41 of the machine learning model 40, and thereby obtains an output result from the decoder 44. The information processing device 100 compares the annotation data corresponding to the image data 60 with the output result from the decoder 44, and calculates the bounding box loss, the object category loss, and the action loss.

[0061] Furthermore, as described in FIG. 5 , the information processing device 100 inputs pairs of positive examples and positive examples, pairs of positive examples and negative examples, or pairs of negative examples and negative examples into the backbone 41 of the machine learning model 40, and calculates a cross-entropy error (Contrastive Loss) based on equation (1).

[0062] The information processing device 100 performs machine learning of the parameters of the Encoder 43 and Decoder 44 of the machine learning model 40 based on the backpropagation method so as to reduce the overall error obtained by adding the cross-entropy error to the Bounding Box Loss, Object Category Loss, and Action Loss.

[0063] As described above, the information processing device 100 according to this embodiment extracts an image of an object used by a person from image data and generates synthetic image data in which the extracted image is positioned near the person in the image data. The information processing device 100 performs machine learning on the machine learning model 40 using the synthetic image data, thereby generating a machine learning model 40 that can identify a person using an object.

[0064] Next, a description will be given of the processing in the inference phase of the information processing device 100. Fig. 7 is a diagram for explaining the processing in the inference phase in this embodiment. The machine learning model 40 is a machine learning model trained in the processing in the learning phase described above.

[0065] The information processing device 100 inputs image data 20 into the backbone 41 of the machine learning model 40 to obtain an output result 70. In the output result 70, a person area 70a and an object area 70b are identified, and the person's action of "grasping" the object is indicated. The image data 20 includes multiple similar products, but it is possible to accurately estimate which object the person's action is acting on.

[0066] The information processing device 100 inputs the image data 21 into the backbone 41 of the machine learning model 40 to obtain an output result 71. In the output result 71, a person area 71a and an object area 71b are identified, and the person's action of "grasping" the object is indicated. Although the image data 21 includes many objects in the background, it is possible to accurately estimate which object the person's action is acting on.

[0067] Next, an example of the configuration of an information processing device 100 that executes the above-described processing will be described. Fig. 8 is a functional block diagram showing the configuration of an information processing device according to this embodiment. As shown in Fig. 8, the information processing device 100 has a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0068] The communication unit 110 executes data communication with the camera 30, an external device, etc. via the network 35. The communication unit 110 is a network interface card (NIC), etc. For example, the communication unit 110 receives video data from the camera 30.

[0069] The input unit 120 inputs various types of information to the control unit 150 of the information processing device 100. For example, the user may operate the input unit 120 to input an execution command for a process in the learning phase and an execution command for a process in the inference phase. The user may also operate the input unit 120 to specify the hyperparameters described in FIG. 4 .

[0070] The display unit 130 displays information output from the control unit 150 .

[0071] The storage unit 140 includes a machine learning model 40, a learning data table 141, and a video buffer 142. The storage unit 140 is a storage device such as a memory.

[0072] The machine learning model 40 is a Transformer-based machine learning model. For example, the machine learning model 40 is HOID. The description of the machine learning model 40 is the same as the description of the machine learning model 40 described with reference to FIG. 2 .

[0073] The learning data table 141 is a table that holds multiple pieces of learning data. FIG. 9 is a diagram showing an example of the data structure of the learning data table. As shown in FIG. 9, the learning data table 141 has an item number, image data, annotation data, and composite image data. The item number is a number that identifies each record in the learning data table 141. The image data and annotation data correspond to the image data and annotation data included in the learning data described in FIG. 3. The image data includes an image of a person and an image of an object such as a product. The composite image data is composite image data generated by executing the process of FIG. 4. The composite image data is generated by the generation unit 152, which will be described later.

[0074] The video buffer 142 is a buffer that stores video data captured by the camera 30. The video buffer 142 may store the video data in association with the identification information of the camera.

[0075] Returning to the description of Fig. 8, the control unit 150 includes an acquisition unit 151, a generation unit 152, a learning processing unit 153, and an inference unit 154. The control unit 150 is a central processing unit (CPU), a graphics processing unit (GPU), or the like.

[0076] The acquisition unit 151 acquires video data from the camera 30 and stores the acquired video data in the video buffer 142. The acquisition unit 151 may also acquire data of the learning data table 141 from an external device or the like and store the data in the storage unit 140.

[0077] The generation unit 152 executes the process described in Fig. 4 based on the image data included in the training data to generate composite image data. The generation unit 152 stores the generated composite image data in the training data table 141. An example of the process performed by the generation unit 152 is shown below.

[0078] The generation unit 152 acquires a pair of image data and annotation data from the learning data table 141. The generation unit 152 extracts a person area and an object area from the image data based on the annotation data.

[0079] The generation unit 152 duplicates an image of an object area included in the image data. As described in step S1 of Fig. 4, the generation unit 152 identifies the center coordinates of the person area and the object area in the image data, and determines the synthesis direction of the duplicated object image based on the respective center coordinates.

[0080] The generation unit 152 generates white space around the image data, as described in step S2 of Fig. 4. The generation unit 152 adjusts the coordinates of the duplicated image so that the distance Δd between the upper left corner of the object area in the image data and the coordinates of the upper left corner of the duplicated image becomes a preset hyperparameter, as described in step S3 of Fig. 4. The generation unit 152 generates composite image data by deleting the white space, as described in step 4 of Fig. 4.

[0081] The learning processing unit 153 uses the learning data table 141 to train the machine learning model 40. For example, the learning processing unit 153 executes the contrastive learning described in FIG. 5 and the machine learning described in FIG. 6.

[0082] The learning processing unit 153 acquires from the learning data table 141 a pair of image data (positive example) and image data (positive example), a pair of image data (positive example) and composite image data (negative example), or a pair of composite image data (negative example) and composite image data (negative example).

[0083] The learning processing unit 153 inputs pairs of positive examples and positive examples, pairs of positive examples and negative examples, or pairs of negative examples and negative examples into the backbone 41 of the machine learning model 40, and calculates a cross-entropy error (Contrastive Loss) based on equation (1).

[0084] The learning processing unit 153 inputs the positive examples into the backbone 41 of the machine learning model 40, thereby acquiring output results from the decoder 44. The learning processing unit 153 compares the annotation data corresponding to the image data with the output results from the decoder 44, and calculates the bounding box loss, the object category loss, and the action loss.

[0085] The learning processing unit 153 performs machine learning of the parameters of the Encoder 43 and Decoder 44 of the machine learning model 40 based on the backpropagation method so that the overall error, which is the sum of the Bounding Box Loss, Object Category Loss, and Action Loss and the cross-entropy error, is small.

[0086] The inference unit 154 infers on which object the person's action acts, using the machine learning model 40 trained by the learning processing unit 153. The processing of the inference unit 154 corresponds to the processing described in FIG.

[0087] For example, the inference unit 154 acquires image data from the video buffer 142 and inputs the image data into the backbone 41 of the machine learning model 40 to infer the area of ​​the person, the area of ​​the object, and the person's action toward the object.

[0088] 1, the camera 30 is installed in a store having product shelves on which products are stored, and the product shelf area is included in the image data (video data) captured by the camera 30. Therefore, the inference unit 154 can identify the behavior of a customer taking a product from a product shelf by inputting the image data into the backbone 41 of the machine learning model 40.

[0089] The inference unit 154 may output and display the inference result to the display unit 130. The inference unit 154 outputs, as the inference result, screen data in which the area of ​​the person, the area of ​​the object, and the action of the person with respect to the object are arranged on the image data to the display unit 130.

[0090] Next, an example of a processing procedure of the information processing device 100 according to this embodiment will be described. Fig. 10 is a flowchart showing the processing procedure of the information processing device according to this embodiment. As shown in Fig. 10, the generation unit 152 of the information processing device 100 generates synthetic image data based on image data in the learning data table 141 (step S101).

[0091] The learning processing unit 153 of the information processing device 100 inputs pairs of positive examples and positive examples, pairs of positive examples and negative examples, or pairs of negative examples and negative examples into the backbone 41 of the machine learning model 40 and extracts features from each (step S102).

[0092] The learning processing unit 153 inputs each feature amount to the Encoder 43 and calculates each output f() (step S103). The learning processing unit 153 calculates a cross-entropy error related to contrastive learning for each output f() (step S104).

[0093] The learning processing unit 153 inputs the positive examples into the backbone 41 of the machine learning model 40, and calculates losses based on the results output from the decoder 44 and the annotation data (step S105). The losses include bounding box loss, object category loss, and action loss.

[0094] The learning processing unit 153 trains the machine learning model 40 so that the overall error obtained by adding the cross-entropy error to the Bounding Box Loss, Object Category Loss, and Action Loss is small (step S106).

[0095] Next, the effects of the information processing device 100 according to this embodiment will be described. The information processing device 100 extracts an image of an object used by a person from image data, and generates synthetic image data in which the extracted image is positioned near the person in the image data. The information processing device 100 performs machine learning on the machine learning model 40 using the synthetic image data, thereby generating a machine learning model 40 that can identify a person using an object.

[0096] When generating synthetic image data, the information processing device 100 accepts hyperparameter settings as described in Fig. 4. The information processing device 100 arranges the image 51c so that the distance Δd between the coordinates of the upper left corner of the object region 51b and the coordinates of the upper left corner of the arranged image 51c is a preset hyperparameter, under the condition that the image 51c does not overlap with the person region 51a. This makes it possible to generate pseudo-synthetic image data in which a similar object is included in the vicinity of the person holding the object.

[0097] In contrastive learning, the information processing device 100 trains the encoder 43 so that when two positive examples are input, the outputs from the encoder 43 approach each other. The information processing device 100 trains the encoder 43 so that when two negative examples are input, the outputs from the encoder 43 approach each other. Because the positive examples have common characteristics and the negative examples also have common characteristics, the parameters of the encoder 43 can be adjusted so that when image data having common characteristics are input to the machine learning model 40, the outputs from the encoder 43 approach each other.

[0098] In contrastive learning, the information processing device 100 trains the encoder 43 so that the outputs from the encoder 43 move away from each other when a positive example and a negative example are input. Because the features of positive examples and negative examples are not similar, the parameters of the encoder 43 can be adjusted so that the outputs from the encoder 43 move away from each other when image data having different features is input to the machine learning model 40.

[0099] The information processing device 100 infers the area of ​​a person, the area of ​​an object, and the person's action with respect to the object by inputting image data captured by the camera 30 into the backbone 41 of the machine learning model 40. For example, as described in FIG. 1 , the camera 30 is installed in a store having shelves on which products are stored, and the image data (video data) captured by the camera 30 includes the area of ​​the shelves. Therefore, by inputting the image data into the backbone 41 of the machine learning model 40, the information processing device 100 can identify the behavior of a customer removing a product from a shelf.

[0100] Here, we will show the results of an accuracy evaluation (Mean Average Precision) of the method of the information processing device 100 compared with the method of the conventional technology. FIG. 11 is a diagram showing the accuracy evaluation results. Dataset (1) is a public dataset that collects images of people holding objects in various scenes. Dataset (2) is a closed dataset that collects images of people reaching for products in a store.

[0101] With the conventional technique, the accuracy rating for dataset (1) was "59.6" and the accuracy rating for dataset (2) was "24.1." On the other hand, with the information processing device 100, the accuracy rating for dataset (1) was "60.1" and the accuracy rating for dataset (2) was "26.2." In other words, it can be seen that the technique of the information processing device 100 has a higher accuracy rating for both datasets compared to the conventional technique.

[0102] The above-described processing details of the information processing device 100 and the data structure of each piece of data are merely examples. For example, in the annotation data 52 described in Fig. 3, the person area 51a is indicated by the coordinates of the upper left corner and the lower right corner, and the object area 51b is indicated by the coordinates of the upper left corner and the lower right corner, but this is not limiting. For example, the annotation data can be extended as shown in Fig. 12.

[0103] FIG. 12 is a diagram showing an example of extended annotation data. In annotation data 52a shown in FIG. 12, multiple coordinates are set that trace the outline of a person. Furthermore, in annotation data 52a, multiple coordinates on the outline of an object are set. For example, in annotation data 52 described in FIG. 3, an object region 51b can be extracted, but in annotation data 52a, an object outline 52b can be extracted. While object region 51b includes an area that is not an object, object outline 52b includes an image of only the object. This allows the image of the object to be synthesized into synthetic image data to be the object image itself.

[0104] Although the generation unit 152 of the information processing device 100 generates the composite image data by executing the process described in Fig. 4, the process for generating the composite image data is not limited to this. For example, the generation unit 152 of the information processing device 100 may generate the composite image data by executing the process shown in Fig. 13.

[0105] 13 is a diagram illustrating another process for generating composite image data. As preprocessing, the generation unit 152 of the information processing device 100 executes a process for extracting skeletal data from the person region 51 a and a process for extracting a segmentation of the person.

[0106] An example of the process in which the generation unit 152 extracts skeletal data from the person's region 51 a will be described. The generation unit 152 infers the person's skeletal data 80 by inputting the person's region 51 a in the image data 51 into a skeletal inference model. The skeletal inference model is a trained model that receives image data of the person's region as input and outputs the person's skeletal data. The skeletal inference model is, for example, a neural network (NN).

[0107] Skeletal data is data in which two-dimensional or three-dimensional coordinates are set for multiple joints defined in a skeletal model of the human body. Here, the coordinates of each joint in the skeletal data are two-dimensional coordinates. FIG. 14 is a diagram showing an example of a skeletal model of the human body. For example, as shown in FIG. 14, the skeletal model of the human body is defined by 21 joints ar0 to ar20.

[0108] The relationship between the joints ar0 to ar20 shown in Fig. 14 and the joint names is as shown in Fig. 15. Fig. 15 is a diagram showing examples of joint names. For example, the joint name of joint ar0 is "SPINE_BASE". The joint names of joints ar1 to ar20 are as shown in Fig. 15, and explanations thereof will be omitted.

[0109] The process of extracting a person segmentation by the generation unit 152 will be described. The generation unit 152 performs segmentation on the image data 51, grouping regions in the image data 51 into groups having similar features (color, texture, subject), etc., and dividing the image data 51 into multiple regions. The generation unit 152 compares the multiple divided regions with the person region 51a specified in the annotation data, and extracts, as the person region, the region that most overlaps with the person region 51a among the multiple regions. In the example shown in FIG. 13 , the generation unit 152 extracts region 81 as the person region.

[0110] After performing the above pre-processing, the generating unit 152 performs the processes of steps S11 to S14 to generate the composite image data 90.

[0111] Step S11 will be described. The generation unit 152 identifies the coordinates (xc3, yc3) of the point of action based on the skeleton data 80. For example, the generation unit 152 identifies the coordinates of the left wrist joint ar19, among the joints ar0 to ar20 included in the skeleton data 80, as the coordinates of the point of action. The generation unit 152 identifies a predetermined range of area based on the coordinates (xc3, yc3) of the point of action as the object area 82. Note that the generation unit 152 may compare the object area specified by the annotation data with the right wrist joint ar20 and the left wrist joint ar19, and identify the coordinates of the wrist joint closest to the object area as the coordinates of the point of action.

[0112] Step S12 will now be described. The generating unit 152 generates image data 53 by generating a margin around the image data 51.

[0113] Step S13 will now be described. The generation unit 152 copies an image of the object region 82 included in the image data 51. The copied image is referred to as image 83. The generation unit 152 places the copied image 83 in an area that does not overlap with the person region 81. Furthermore, the generation unit 152 adjusts the coordinates of the image 83 so that the distance Δd between the coordinates (xc3, yc3) of the point of action of the object region 82 and the coordinates (xc4, yc4) of the upper left corner of the placed image 83 becomes a preset hyperparameter.

[0114] Step S4 will now be described. The generating unit 152 generates the composite image data 90 by deleting the white space of the image data 53.

[0115] As described in FIG. 13, the information processing device 100 executes preprocessing and executes the processes of steps S11 to S14, thereby generating composite image data 90 by combining the image data 51 and the image 83.

[0116] Next, a description will be given of an example of the hardware configuration of a computer that realizes the same functions as the above-described information processing device 100. Fig. 16 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the information processing device of the embodiment.

[0117] 16, computer 300 has a CPU 301 that executes various types of arithmetic processing, an input device 302 that accepts data input from a user, and a display 303. Computer 300 also has a communication device 304 that exchanges data with external devices via a wired or wireless network, and an interface device 305. Computer 300 also has a RAM 306 that temporarily stores various types of information, and a hard disk drive 307. Each of devices 301 to 307 is connected to a bus 308.

[0118] The hard disk drive 307 stores an acquisition program 307 a, a generation program 307 b, a learning processing program 307 c, and an inference program 307 d. The CPU 301 reads out each of the programs 307 a to 307 d and loads them into the RAM 306.

[0119] The acquisition program 307a functions as an acquisition process 306a. The generation program 307b functions as a generation process 306b. The learning processing program 307c functions as a learning processing process 306c. The inference program 307d functions as an inference process 306d.

[0120] The processing of the acquisition process 306a corresponds to the processing of the acquisition unit 151. The processing of the generation process 306b corresponds to the processing of the generation unit 152. The processing of the learning process 306c corresponds to the processing of the learning processing unit 153. The processing of the inference process 306d corresponds to the processing of the inference unit 154.

[0121] It should be noted that each of the programs 307a to 307d does not necessarily have to be stored in the hard disk drive 307 from the beginning. For example, each program may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 300. Then, the computer 300 may read and execute each of the programs 307a to 307d.

[0122] 40 Machine learning model 100 Information processing device 110 Communication unit 120 Input unit 130 Display unit 140 Storage unit 141 Learning data table 142 Video buffer 150 Control unit 151 Acquisition unit 152 Generation unit 153 Learning processing unit 154 Inference unit

Claims

1. Obtain an image containing a person; Analyzing the acquired image to extract an object used by the person in the image; generating a composite image in which the extracted object is placed at a position that satisfies a predetermined condition based on the position of the object used by the person in the acquired image; Using the generated synthetic image, a machine learning model is generated that is trained to identify the person using the object. A generating program that causes a computer to execute a process.

2. The process of generating the composite image includes: accepting a parameter setting based on a distance between a coordinate position of the person and a coordinate position of an object held by the person; generating a coordinate position of a placement candidate for the extracted image of the object based on the set parameters; determining whether the generated coordinate position is included within a region related to the size of the object; 2. The generating program according to claim 1, further comprising a process for generating a composite image by arranging an image of the object in the acquired image based on the determination result.

3. The process of generating a composite image includes generating a first composite image based on a first object included in a first image, and generating a second composite image based on a second object included in a second image; The process of generating the machine learning model includes: training the encoder of the machine learning model so that an output result when the first image is input to the encoder approaches an output result when the second image is input to the encoder; training the encoder so that an output result when the first image is input to the encoder and an output result when the first composite image is input to the encoder move away from each other; The generation program according to claim 2, further comprising a process of training the encoder so that the output result when the first composite image is input to the encoder approaches the output result when the second composite image is input to the encoder.

4. The generation program of claim 1 further causes a computer to execute a process of identifying the behavior of a person taking a product from a product shelf by inputting an image taken by a camera in a store, the image including a product shelf containing products and a person into the machine learning model.

5. The generation program according to claim 1, characterized in that the process of extracting the object comprises analyzing the acquired image to extract skeletal information of the person in the image, and extracting objects used by the person based on the skeletal information.

6. Obtain an image containing a person; Analyzing the acquired image to extract an object used by the person in the image; generating a composite image in which the extracted object is placed at a position that satisfies a predetermined condition based on the position of the object used by the person in the acquired image; Using the generated synthetic image, a machine learning model is generated that is trained to identify the person using the object. A generating method characterized in that the processing is carried out by a computer.

7. Obtain an image containing a person; Analyzing the acquired image to extract an object used by the person in the image; generating a composite image in which the extracted object is placed at a position that satisfies a predetermined condition based on the position of the object used by the person in the acquired image; Using the generated synthetic image, a machine learning model is generated that is trained to identify the person using the object. An information processing device having a control unit that executes processing.