Behavior inference device, behavior inference method, and program

Abstract

In order to reduce the possibility of erroneously inferring the behavior of a person, a behavior inference device (1) comprises: an acquisition unit (11) that acquires a person feature amount indicating the feature of a person detected from an image, and an object feature amount indicating the feature of an object; an object feature amount string generation unit (12) that generates an object feature amount string by adding, to the object feature amount, an object feature amount of a dummy object; an information generation unit (13) that generates information indicating the relationship between the person and at least one of the object and the dummy object, on the basis of the person feature amount and each object feature amount included in the object feature amount string; and an inference unit (14) that infers the behavior of the person, on the basis of the relationship.

Description

Behavior estimation device, behavior estimation method, and program

[0001] The present invention relates to a behavior estimation device, a behavior estimation method, and a program for estimating human behavior.

[0002] 2. Description of the Related Art Techniques are known for detecting people and objects from images and estimating the behavior of people using the objects.

[0003] Patent Document 1 discloses an image processing device that detects people and objects from an input image, replaces the detected object recognition information with object recognition information from when the object was detected in the past, and determines human behavior based on the replaced object recognition information and the detected person recognition information.

[0004] Japanese Patent Application Publication No. 2022-187656

[0005] For example, at a construction site where many objects are installed, the object being used by a person is not necessarily included as a subject in the image. However, with the image processing device described in Patent Document 1, for example, if the object being used by a person is hidden behind an object, the image processing device may determine the person's behavior based on an object other than the object being used by the person. In other words, the image processing device described in Patent Document 1 has a problem in that it may erroneously determine (misestimate) the person's behavior.

[0006] One aspect of the present invention has been made in consideration of the above-mentioned problems, and one example of its purpose is to provide a technology for estimating human behavior that reduces the possibility of misestimating human behavior.

[0007] A behavior estimation device according to one aspect of the present invention includes an acquisition means for acquiring human features indicating features of a person detected from an image and object features indicating features of each of one or more objects detected from the image; an object feature sequence generation means for generating an object feature sequence by adding object features of dummy objects to the object features; an information generation means for generating information indicating a relationship between the person and at least one of the one or more objects and the dummy object based on the human features and each object feature included in the object feature sequence; and an estimation means for estimating the behavior of the person based on the relationship indicated by the information.

[0008] A behavior estimation method according to one aspect of the present invention includes a behavior estimation device acquiring human features indicating features of a person detected from an image and object features indicating features of each of one or more objects detected from the image, generating an object feature sequence by adding object features of dummy objects to the object features, generating information indicating a relationship between the person and at least one of the one or more objects and the dummy object based on the human features and each object feature included in the object feature sequence, and estimating the behavior of the person based on the relationship indicated by the information.

[0009] A program according to one aspect of the present invention causes a computer to execute an acquisition process for acquiring human features indicating the characteristics of a person detected from an image and object features indicating the characteristics of each of one or more objects detected from the image; an object feature sequence generation process for generating an object feature sequence by adding object features of dummy objects to the object features; an information generation process for generating information indicating a relationship between the person and at least one of the one or more objects and the dummy objects based on the human features and each object feature included in the object feature sequence; and an estimation process for estimating the behavior of the person based on the relationship indicated by the information.

[0010] According to one aspect of the present invention, the possibility of erroneously estimating a person's behavior can be reduced.

[0011] FIG. 1 is a block diagram showing the configuration of a behavior estimation device according to exemplary embodiment 1 of the present invention. FIG. 2 is a flow diagram showing the flow of a behavior estimation method according to exemplary embodiment 1 of the present invention. FIG. 3 is a block diagram showing the configuration of a behavior estimation device according to exemplary embodiment 2 of the present invention. FIG. 4 is a block diagram showing the configuration of a person-object relationship extraction unit according to exemplary embodiment 2 of the present invention. FIG. 5 is a diagram showing an example of processing executed by an object feature sequence generation unit according to exemplary embodiment 2 of the present invention. FIG. 6 is a diagram showing an example of processing executed by a person-object feature sequence comparison unit according to exemplary embodiment 2 of the present invention. FIG. 7 is a flow diagram showing the flow of a behavior estimation method according to exemplary embodiment 2 of the present invention. FIG. 8 is a flow diagram showing an example of processing executed by an information generation unit according to exemplary embodiment 2 of the present invention. FIG. 9 is a block diagram showing the configuration of a behavior estimation device according to modified embodiment 1 of the present invention. FIG. 10 is a block diagram showing the configuration of a person-object relationship extraction unit according to modified embodiment 1 of the present invention. FIG. 11 is a diagram showing an example of processing executed by an object feature sequence generation unit according to modified embodiment 1 of the present invention. FIG. 12 is a block diagram showing the configuration of a behavior estimation device according to modified embodiment 2 of the present invention. FIG. 13 is a block diagram showing the configuration of a person-object relationship extraction unit according to modified embodiment 2 of the present invention. FIG. 14 is a diagram showing an example of processing executed by an object feature sequence generation unit according to modified embodiment 2 of the present invention.

[0012] [First Exemplary Embodiment] A first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described below.

[0013] (Configuration of Behavior Inference Device 1) The configuration of the behavior inference device 1 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the behavior inference device 1 according to this exemplary embodiment.

[0014] 1, the behavior estimation device 1 includes an acquisition unit 11, an object feature sequence generation unit 12, an information generation unit 13, and an estimation unit 14. In this exemplary embodiment, the acquisition unit 11, the object feature sequence generation unit 12, the information generation unit 13, and the estimation unit 14 respectively implement an acquisition means, an object feature sequence generation means, an information generation means, and an estimation means.

[0015] The acquisition unit 11 acquires human feature amounts indicating features of a person detected from an image and object feature amounts indicating features of each of one or more objects detected from the image. The acquisition unit 11 supplies the acquired human feature amounts to the information generation unit 13. The acquisition unit 11 also supplies the acquired object feature amounts to the object feature sequence generation unit 12.

[0016] The object feature sequence generation unit 12 generates an object feature sequence by adding the object feature sequences of the dummy objects to the object feature sequences supplied from the acquisition unit 11. The object feature sequence generation unit 12 supplies the generated object feature sequence to the information generation unit 13.

[0017] The information generation unit 13 generates information indicating a relationship between a person and at least one of one or more objects and a dummy object, based on the person feature amount supplied from the acquisition unit 11 and each object feature amount included in the object feature amount sequence supplied from the object feature amount sequence generation unit 12. The information generation unit 13 supplies the generated information to the estimation unit 14.

[0018] The estimation unit 14 estimates the behavior of a person based on the relationship indicated by the information generated by the information generation unit 13.

[0019] As described above, the behavior estimation device 1 according to this exemplary embodiment employs a configuration including: an acquisition unit 11 that acquires human feature amounts indicating features of a person detected from an image, and object feature amounts indicating features of each of one or more objects detected from the image; an object feature sequence generation unit 12 that generates an object feature sequence by adding object feature amounts of a dummy object to the object feature amounts supplied from the acquisition unit 11; an information generation unit 13 that generates information indicating a relationship between the person and at least one of the one or more objects and the dummy object, based on the human feature amounts supplied from the acquisition unit 11 and each object feature included in the object feature sequence supplied from the object feature sequence generation unit 12; and an estimation unit 14 that estimates the behavior of the person based on the relationship indicated by the information generated by the information generation unit 13.

[0020] Therefore, the behavior estimation device 1 according to this exemplary embodiment has the effect of reducing the possibility of erroneously estimating a person's behavior.

[0021] (Flow of behavior estimation method S1) The flow of the behavior estimation method S1 according to this exemplary embodiment will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of the behavior estimation method S1 according to this exemplary embodiment.

[0022] (Step S11) In step S11, the acquisition unit 11 acquires human feature amounts indicating features of a person detected from an image and object feature amounts indicating features of each of one or more objects detected from the image. The acquisition unit 11 supplies the acquired human feature amounts to the information generation unit 13. The acquisition unit 11 also supplies the acquired object feature amounts to the object feature sequence generation unit 12.

[0023] (Step S12) In step S12, the object feature sequence generation unit 12 generates an object feature sequence by adding the object feature sequences of the dummy object to the object feature sequences supplied from the acquisition unit 11. The object feature sequence generation unit 12 supplies the generated object feature sequence to the information generation unit 13.

[0024] (Step S13) In step S13, the information generation unit 13 generates information indicating a relationship between the person and at least one of one or more objects and a dummy object, based on the person feature amount supplied from the acquisition unit 11 and each object feature amount included in the object feature amount sequence supplied from the object feature amount sequence generation unit 12. The information generation unit 13 supplies the generated information to the estimation unit 14.

[0025] (Step S14) In step S14, the estimation unit 14 estimates the behavior of a person based on the relationship indicated by the information generated by the information generation unit 13.

[0026] As described above, the behavior estimation method S1 according to this exemplary embodiment employs a configuration including: step S11 in which the acquisition unit 11 acquires person features indicating features of a person detected from the image, and object features indicating features of each of one or more objects detected from the image; step S12 in which the object feature sequence generation unit 12 generates an object feature sequence by adding object features of a dummy object to the object features supplied from the acquisition unit 11; step S13 in which the information generation unit 13 generates information indicating a relationship between the person and at least one of the one or more objects and the dummy object, based on the person features supplied from the acquisition unit 11 and each object feature included in the object feature sequence supplied from the object feature sequence generation unit 12; and step S14 in which the estimation unit 14 estimates the behavior of the person based on the relationship indicated by the information generated by the information generation unit 13.

[0027] Therefore, according to the behavior estimation method S1 according to this exemplary embodiment, the same effects as those of the behavior estimation device 1 described above can be obtained.

[0028]

[0033] A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first exemplary embodiment are denoted by the same reference numerals, and their description will be omitted as appropriate.

[0029] (Outline of Behavior Inference Device 2) The behavior inference device 2 according to this exemplary embodiment is a device that detects people and objects included as subjects in an image and infers the behavior of the people using the objects.

[0030] As one example, the behavior estimation device 2 may be configured to estimate, from an image of a construction site, an operation performed by a person working at the construction site using an object at the construction site. As another example, the behavior estimation device 2 may be configured to estimate, from an image of a manufacturing site, an operation performed by a person working at the manufacturing site using an object at the manufacturing site. As yet another example, the behavior estimation device 2 may be configured to estimate, from an image of a retail store, an operation performed by a person working at the retail store using an object at the retail store.

[0031] (Configuration of Behavior Inference Device 2) The configuration of the behavior inference device 2 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the behavior inference device 2 according to this exemplary embodiment.

[0032] As shown in FIG. 3, the behavior inference device 2 includes a control unit 20, an input unit 31, an output unit 32, a communication unit 33, and a storage unit .

[0033] The input unit 31 is an interface that accepts input of data. For example, the input unit 31 supplies information indicating the accepted data to the control unit 20. Examples of the input unit 31 include a keyboard, a mouse, a touchpad, a microphone, and an image capture device.

[0034] The output unit 32 is an interface that outputs data. For example, the output unit 32 outputs data supplied from the control unit 20. An example of the output unit 32 is a liquid crystal display.

[0035] The communication unit 33 is an interface that transmits and receives data via a network. For example, the communication unit 33 transmits data supplied from the control unit 20 to other devices, and supplies data received from other devices to the control unit 20. Examples of the communication unit 33 include, but are not limited to, communication chips for various communication standards such as Ethernet (registered trademark), Wi-Fi (registered trademark), and wireless communication standards for mobile data communication networks, and USB-compliant connectors.

[0036] Furthermore, the specific configuration of the network does not limit this exemplary embodiment, but as an example, a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public line network, a mobile data communication network, or a combination of these networks can be used.

[0037] The storage unit 34 stores data referenced by the control unit 20. Examples of the data stored in the storage unit 34 include an image pic, a person feature amount hfq, an object feature amount ofq, and a dummy feature amount dfq, which is a feature amount indicating the characteristics of a dummy object. The person feature amount hfq, the object feature amount ofq, and the dummy feature amount dfq will be described later. Examples of the storage unit 34 include, but are not limited to, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0038] (Control Unit 20) The control unit 20 controls each component included in the behavior inference device 2.

[0039] 3, the control unit 20 includes an image acquisition unit 21, a human region detection unit 22, a human feature extraction unit 23, an object region detection unit 24, an object feature extraction unit 25, a human-object relationship extraction unit 26, and an estimation unit 27. In this exemplary embodiment, the estimation unit 27 is configured to realize estimation means.

[0040] The image acquisition unit 21 acquires the image pic via the input unit 31 or the communication unit 33. The image acquisition unit 21 stores the acquired image pic in the storage unit .

[0041] The human region detection unit 22 detects a region including a person included as a subject in the image pic stored in the storage unit 34. The human region detection unit 22 supplies information indicating the detected region to the human feature extraction unit 23.

[0042] The method by which the human region detection unit 22 detects a human region included as a subject in the image pic is not limited, and known methods can be used. Examples include methods based on image features such as HOG (Histograms of Oriented Gradients), color histograms, and shape. Other examples include methods based on local features around feature points, such as SIFT (Scale-Invariant Feature Transform). Yet another example includes a method using a machine learning model, such as Faster R-CNN (Regions with Convolutional Neural Networks).

[0043] The human feature extraction unit 23 extracts the features of the person included in the area indicated by the information supplied from the human area detection unit 22. The human feature extraction unit 23 stores the human feature amount hfq indicating the extracted human feature in the storage unit 34.

[0044] The method by which the human feature extraction unit 23 extracts human features is not limited, and a known method may be used. As one example, the human feature extraction unit 23 may store image feature quantities such as HOG, color histogram, and shape of a person included in the area indicated by the information supplied from the human region detection unit 22 as human feature quantities hfq in the storage unit 34. As another example, the human feature extraction unit 23 may store local feature quantities around feature points such as SIFT in the area indicated by the information supplied from the human region detection unit 22 as human feature quantities hfq in the storage unit 34. As yet another example, the human feature extraction unit 23 may store feature quantities extracted using a machine learning model such as Faster R-CNN as human feature quantities hfq in the storage unit 34.

[0045] The object region detection unit 24 detects regions that each include one or more objects that are included as subjects in the image pic stored in the storage unit 34. The object region detection unit 24 supplies information indicating each detected region to the object feature extraction unit 25.

[0046] The method by which the object area detection unit 24 detects an area containing one or more objects included as subjects in the image pic is not limited, and a method similar to the method by which the above-mentioned person area detection unit 22 detects an area containing a person included as a subject in the image pic can be used.

[0047] Based on the information supplied from the object region detection unit 24, the object feature extraction unit 25 extracts the features of an object included in the region indicated by the information. The object feature extraction unit 25 stores the object feature amount ofq, which is the feature amount of the extracted object, in the storage unit 34. Note that if the object region detection unit 24 cannot detect an object included as a subject in the image pic, the object feature extraction unit 25 does not need to store the object feature amount ofq in the storage unit 34.

[0048] The method by which the object feature extraction unit 25 extracts the features of an object is not limited, and a method similar to the method by which the human feature extraction unit 23 extracts the features of a person described above can be used.

[0049] The person-object relationship extraction unit 26 extracts a relationship between a person included as a subject in the image pic and at least one of one or more objects based on the person feature amount hfq and the object feature amount ofq stored in the storage unit 34. The person-object relationship extraction unit 26 generates information indicating the relationship between the extracted person and at least one of the one or more objects. The person-object relationship extraction unit 26 supplies the generated information to the estimation unit 27.

[0050] As one example, the person-object relationship extraction unit 26 generates information indicating the degree of relationship between a person and at least one of one or more objects. As another example, the person-object relationship extraction unit 26 generates information indicating the degree of relationship between a person and each of one or more objects. As yet another example, the person-object relationship extraction unit 26 generates a feature obtained by integrating an object feature amount ofq indicating the characteristics of an object that has the strongest relationship with a person (or stronger than a predetermined degree) among multiple objects, and a person feature amount hfq indicating the characteristics of the person. Details of the person-object relationship extraction unit 26 will be described later.

[0051] The estimation unit 27 estimates the behavior of the person based on the relationship indicated by the information supplied by the person-object relationship extraction unit 26. The estimation unit 27 supplies the estimation result to at least one of the output unit 32 and the communication unit 33.

[0052] The method by which the estimation unit 27 estimates a person's behavior based on the relationship is not limited, and any known method may be used. As an example, when the information provided by the person-object relationship extraction unit 26 indicates that a certain person and a certain object have a relationship (for example, the degree of indicating the relationship is higher than a threshold), the estimation unit 27 estimates that the certain person is performing an action using the certain object.

[0053] As another example, assume that the person-object relationship extraction unit 26 generates, as information indicating the relationship, a feature amount obtained by integrating a person feature amount and an object feature amount that indicate the features of a person and an object, respectively, that have the strongest relationship (or that are stronger than a predetermined degree). In this case, the estimation unit 27 estimates, based on the feature amount generated by the person-object relationship extraction unit 26, that a person corresponding to the person feature amount integrated into the feature amount is performing an action using an object corresponding to the object feature amount integrated into the feature amount.

[0054] As an example of a behavior estimated by the estimation unit 27, if the information supplied by the person-object relationship extraction unit 26 indicates a strong relationship between a person and a cart, the estimation unit 27 estimates that the person is transporting the cart by pushing it. As another example, if the information supplied by the person-object relationship extraction unit 26 indicates a strong relationship between a person and a dummy object, the estimation unit 27 estimates that the person is not performing an activity using an object.

[0055] (Configuration of person-object relationship extraction unit 26) The configuration of the person-object relationship extraction unit 26 will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the configuration of the person-object relationship extraction unit 26 according to this exemplary embodiment.

[0056] 4, the person-object relationship extraction unit 26 includes an acquisition unit 261, an object feature sequence generation unit 262, and an information generation unit 263. In this exemplary embodiment, the acquisition unit 261, the object feature sequence generation unit 262, and the information generation unit 263 are configured to realize an acquisition means, an object feature sequence generation means, and an information generation means, respectively.

[0057] The acquisition unit 261 acquires the person feature amounts hfq and object feature amounts ofq stored in the storage unit 34. The acquisition unit 261 acquires at least one person feature amount hfq. That is, the acquisition unit 261 may be configured not to acquire the object feature amount ofq when the object feature amount ofq is not stored in the storage unit 34. The acquisition unit 261 supplies the acquired person feature amount hfq to the information generation unit 263. Furthermore, the acquisition unit 261 supplies the acquired object feature amount ofq to the object feature sequence generation unit 262.

[0058] The object feature sequence generation unit 262 generates an object feature sequence ofq_c by adding the dummy feature dfq stored in the storage unit 34 to the object feature ofq supplied from the acquisition unit 261. The object feature sequence generation unit 262 supplies the generated object feature sequence ofq_c to the information generation unit 263. Note that if the acquisition unit 261 does not acquire the object feature ofq, the object feature sequence generation unit 262 generates an object feature sequence ofq_c including only the dummy feature dfq.

[0059] An example of processing executed by the object feature sequence generation unit 262 will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of processing executed by the object feature sequence generation unit 262 according to this exemplary embodiment.

[0060] 5, a case will be described in which the object feature sequence generation unit 262 acquires an object 1 feature of q1, an object 2 feature of q2, and an object 3 feature of q3 from the acquisition unit 261. In this case, the object feature sequence generation unit 262 generates an object feature sequence of q_c including the object 1 feature of q1, the object 2 feature of q2, the object 3 feature of q3, and the dummy feature dfq by adding a dummy feature dfq, as shown in FIG.

[0061] Here, the dummy feature dfq is a vector having the same number of dimensions as the object 1 feature ofq1, the object 2 feature ofq2, and the object 3 feature ofq3. The dummy feature dfq may be a vector of predetermined values, or may be a vector derived using a machine learning model. As an example, if the dummy feature dfq is a 2048-dimensional vector, the machine learning model is a model that can manipulate each value of the 2048-dimensional vector.

[0062] When deriving the dummy feature dfq using a machine learning model, one example of a method for training the machine learning model is to train the machine learning model so that, when there is a strong relationship between a person and a dummy object, the dummy feature dfq will be such that the estimation unit 27 estimates that the person is not performing an action using the object.

[0063] The information generation unit 263 generates information indicating a relationship between a person corresponding to the person feature amount hfq and at least one of the objects corresponding to each object feature amount ofq, based on the person feature amount hfq supplied from the acquisition unit 261 and each object feature amount ofq included in the object feature amount sequence ofq_c supplied from the object feature amount sequence generation unit 262. The information generation unit 263 supplies the generated information to the estimation unit 27.

[0064] As shown in FIG. 4, the information generating unit 263 also includes a person-object feature comparison unit 264 and a feature integration unit 265 .

[0065] The person-object feature amount comparison unit 264 derives the relationship between the person and the object by referring to the feature amounts of each of the person and the object. As an example, the person-object feature amount comparison unit 264 sets a weight indicating the strength of the relationship between the person and the object to each object feature amount ofq included in the object feature amount sequence ofq_c supplied from the object feature amount sequence generation unit 262.

[0066] The method by which the person-object feature comparison unit 264 sets weights for each object feature of q included in the object feature sequence of q_c supplied from the object feature sequence generation unit 262 is not limited. As an example, the person-object feature comparison unit 264 uses a machine learning model that receives as input a person feature and an object feature and is trained to set weights for the object feature. In this case, the person-object feature comparison unit 264 inputs the person feature hfq supplied from the acquisition unit 261 and the object feature of q included in the object feature sequence of q_c supplied from the object feature sequence generation unit 262 to the machine learning model. Then, the person-object feature comparison unit 264 sets weights for the input object feature of q with reference to the output of the machine learning model. The person-object feature comparison unit 264 performs the same process for each object feature of q included in the object feature sequence of q_c.

[0067] Although the specific configuration of the machine learning model is not limited, for example, a neural network (NN) can be used. For example, the machine learning model may use Source-Target Attention used in a Transformer model.

[0068] An example of processing executed by the human-object feature amount comparison unit 264 will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of processing executed by the human-object feature amount comparison unit 264 according to this exemplary embodiment.

[0069] In the following, a case will be described in which the object feature sequence ofq_c includes an object 1 feature ofq1 which is a feature of object 1, an object 2 feature ofq2 which is a feature of object 2, and a dummy feature dfq which is a feature of a dummy object. In this case, the person-object feature comparison unit 264 sets a weight to each of the object 1 feature ofq1, the object 2 feature ofq2, and the dummy feature dfq. In FIG. 6 , the person-object feature comparison unit 264 sets a weight of "0.1" to the object 1 feature ofq1, a weight of "0.1" to the object 2 feature ofq2, and a weight of "0.8" to the dummy feature dfq.

[0070] The feature amount integrating unit 265 generates information in which the human feature amount and the object feature amount are integrated. As an example, when the relationship between a person and one or more objects is weak, the feature amount integrating unit 265 generates information in which the human feature amount hfq and the dummy feature amount dfq are integrated as information indicating at least the relationship between the person and the dummy object.

[0071] 6 , the feature integration unit 265 determines whether the weights set for the object 1 feature ofq1 and the object 2 feature ofq2 are greater than a threshold. If it is determined that the weights set for the object 1 feature ofq1 and the object 2 feature ofq2 are equal to or less than the threshold, the feature integration unit 265 generates an integrated feature by adding a dummy feature dfq to the person feature hfq. As another example, the feature integration unit 265 may generate a feature by combining the person feature hfq and the dummy feature dfq in a dimension direction.

[0072] With this configuration, when the relationship between the object detected from the image and the person is weak, the feature amount integration unit 265 can generate information indicating that the relationship between the person and the dummy object is strong.

[0073] As another example, the feature amount integrating unit 265 calculates a weighted average for each of one or more objects included in the image pic. Then, the feature amount integrating unit 265 generates information indicating a relationship between the person and at least one of the one or more objects based on the weighted average for each of the one or more objects.

[0074] With this configuration, the feature amount integration unit 265 can suitably derive the relationship between a person and at least one of one or more objects, and generate information indicating the relationship.

[0075] In this case, the feature amount integration unit 265 generates a feature amount obtained by integrating the object feature amount ofq, which indicates the characteristics of an object whose calculated weighted average value is greater than a threshold, and the person feature amount hfq, as information indicating the relationship between the person and at least one of the one or more objects. As one example, the feature amount integration unit 265 may generate the integrated feature amount by adding the object feature amount ofq to the person feature amount hfq. As another example, the feature amount integration unit 265 may generate a feature amount obtained by combining the person feature amount hfq and the object feature amount ofq in a dimensional direction.

[0076] With this configuration, the feature amount integration unit 265 can preferably indicate which object, out of multiple objects, has a strong association with a person.

[0077] As an example, a case will be described in which weights of "0.1," "0.1," and "0.8" are set for the object 1 feature ofq1, the object 2 feature ofq2, and the dummy feature dfq, respectively, as shown in Fig. 6. In this case, the feature integration unit 265 calculates a weighted average for each of object 1, object 2, and the dummy object.

[0078] In the case of Figure 6, each object has one object feature ofq, and therefore one weight is set for each object. In this case, the feature integration unit 265 sets the weight value to the weighted average value. That is, the feature integration unit 265 first determines whether the weight value for each object is greater than a threshold value. Next, the feature integration unit 265 generates a feature by integrating the object feature ofq indicating the features of an object greater than the threshold value and the human feature hfq.

[0079] For example, when the threshold is set to "0.5", the feature integration unit 265 generates a feature by integrating the dummy feature dfq, which is the feature of a dummy object among the objects in Figure 6, and the human feature hfq.

[0080] As another example, a case where there are multiple object features ofq for each object (e.g., a feature indicating a color feature of an object and a feature indicating a shape feature of the object) will be described. For example, a weight of "0.1" is assigned to the object feature indicating the color feature of a certain object, and a weight of "0.2" is assigned to the object feature indicating the shape feature of the certain object. In this case, the feature integration unit 265 first calculates a weighted average of the certain object (0.1 + 0.2) / 2 = 0.15. Then, the feature integration unit 265 determines whether the calculated weighted average is greater than a threshold. If it is determined that the calculated weighted average is greater than the threshold, the feature integration unit 265 generates a feature by integrating the person feature, the object feature indicating the color feature of the certain object, and the object feature indicating the shape feature of the certain object.

[0081] (Flow of behavior estimation method S2 executed by behavior estimation device 2) The flow of the behavior estimation method S2 executed by the behavior estimation device 2 according to this exemplary embodiment will be described with reference to Fig. 7. Fig. 7 is a flow diagram showing the flow of the behavior estimation method S2 according to this exemplary embodiment.

[0082] (Step S21) In step S21, the image acquisition unit 21 acquires the image pic via the input unit 31 or the communication unit 33. The image acquisition unit 21 stores the acquired image pic in the storage unit .

[0083] (Step S22) In step S22, the human region detection unit 22 detects a region including a person included as a subject in the image pic stored in the storage unit 34. The human region detection unit 22 supplies information indicating the detected region to the human feature extraction unit 23.

[0084] (Step S23) In step S23, the features of the person included in the area indicated by the information supplied from the human area detection unit 22 are extracted based on the information. The human feature extraction unit 23 stores the extracted human feature amount hfq indicating the human feature in the storage unit 34.

[0085] (Step S24) In step S24, the object region detection unit 24 detects regions including one or more objects included as subjects in the image pic stored in the storage unit 34. The object region detection unit 24 supplies information indicating each of the detected regions to the object feature extraction unit 25.

[0086] (Step S25) In step S25, the object feature extraction unit 25 extracts features of an object included in the area indicated by the information based on the information supplied from the object area detection unit 24. The object feature extraction unit 25 stores the object feature amount ofq, which is the feature amount of the extracted object, in the storage unit 34.

[0087] The order of steps S22 and S23 and steps S24 and S25 is not limited. For example, steps S22 and S23 may be performed after steps S24 and S25, or steps S22 and S23 and steps S24 and S25 may be performed in parallel.

[0088] (Step S26) In step S26, the acquisition unit 261 of the person-object relationship extraction unit 26 acquires the person feature amounts hfq and the object feature amounts ofq stored in the storage unit 34. The acquisition unit 261 supplies the acquired person feature amounts hfq to the information generation unit 263. The acquisition unit 261 also supplies the acquired object feature amounts ofq to the object feature sequence generation unit 262.

[0089] (Step S27) In step S27, the object feature sequence generation unit 262 generates an object feature sequence ofq_c by adding the dummy feature dfq stored in the storage unit 34 to the object feature ofq supplied from the acquisition unit 261. The object feature sequence generation unit 262 supplies the generated object feature sequence ofq_c to the information generation unit 263.

[0090] (Step S28) In step S28, the information generation unit 263 generates information indicating a relationship between the person corresponding to the person feature amount hfq and at least one of the objects corresponding to each object feature amount ofq and the dummy object corresponding to the dummy feature amount dfq, based on the person feature amount hfq supplied from the acquisition unit 261 and each object feature amount ofq included in the object feature amount sequence ofq_c supplied from the object feature amount sequence generation unit 262. The information generation unit 263 supplies the generated information to the estimation unit 27. An example of the processing of step S28 will be described later.

[0091] (Step S29) In step S29, the estimation unit 27 estimates the behavior of the person based on the relationship indicated by the information supplied by the person-object relationship extraction unit 26.

[0092] (Step S30) In step S30, the estimation unit 27 supplies the estimation result to at least one of the output unit 32 and the communication unit 33.

[0093] (Example of Processing in Step S28) An example of processing in step S28 executed by the information generating unit 263 will be described with reference to Fig. 8. Fig. 8 is a flow diagram showing an example of processing executed by the information generating unit 263 according to this exemplary embodiment.

[0094] (Step S281) In step S281, the person-object feature comparison unit 264 of the information generation unit 263 sets a weight indicating the strength of the relationship with a person to each object feature ofq included in the object feature sequence ofq_c supplied from the object feature sequence generation unit 262.

[0095] (Step S282) In step S282, the feature amount integration unit 265 calculates a weighted average for each of one or more objects included as subjects in the image pic.

[0096] (Step S283) In step S283, the feature integration unit 265 generates a feature by integrating the object feature ofq, which indicates the characteristics of an object whose calculated weighted average value is greater than a threshold, and the person feature hfq, as information indicating the relationship between the person and at least one of one or more objects.

[0097] (Step S284 ) The feature integration unit 265 outputs the generated feature to the estimation unit 27 .

[0098] (Effects of the Behavior Inference Device 2 According to this Exemplary Embodiment) In this way, the behavior inference device 2 according to this exemplary embodiment generates an object feature amount sequence ofq_c by adding dummy feature amounts dfq to object feature amounts ofq indicating features of objects detected from an image pic. Then, the behavior inference device 2 infers the behavior of a person based on information indicating a relationship between a person and an object, which is generated based on each object feature amount ofq and the person feature amount hfq included in the object feature amount sequence ofq_c.

[0099] With this configuration, when an object used for an action performed by a certain person is not included as a subject in the image pic, the activity estimation device 2 can select a dummy object as the object used for the action performed by the certain person. Therefore, when an object used for an action performed by a certain person is not included as a subject in the image pic, the activity estimation device 2 can reduce the possibility of making an erroneous estimation that a certain person is performing an action using an object not used for the action performed by the certain person.

[0100] Furthermore, when the relationship between the dummy object and the person is strong, the behavior estimation device 2 estimates that the person is not performing an action using the object. Therefore, the behavior estimation device 2 can suitably reduce the possibility of erroneously estimating the person's behavior.

[0101] (Modification 1) In this modification, a configuration in which multiple dummy features dfq are added will be described.

[0102] (Behavior Inference Device 2a) The configuration of the behavior inference device 2a according to this modification will be described with reference to Fig. 9. Fig. 9 is a block diagram showing the configuration of the behavior inference device 2a according to this modification.

[0103] The behavior inference device 2a is configured to include a control unit 20a instead of the control unit 20 included in the behavior inference device 2 described above. Furthermore, a plurality of dummy features dfq are stored in the storage unit 34. The configuration other than the control unit 20a and the storage unit 34 is as described above, and therefore description thereof will be omitted.

[0104] The multiple dummy feature values ​​dfq are vectors indicating the characteristics of each of the multiple dummy objects. As an example, the multiple dummy feature values ​​dfq may be vectors indicating the characteristics of objects that may be included in the area detected by the object area detection unit 24. For example, if the image pic acquired by the image acquisition unit 21 is an image pic of a construction site, the storage unit 34 stores a dummy feature value dfq indicating the characteristics of a rolling compactor, a dummy feature value dfq indicating the characteristics of a backhoe, a dummy feature value dfq indicating the characteristics of a cart, and the like.

[0105] The control unit 20a controls each component included in the behavior inference device 2a.

[0106] 9, the control unit 20a includes an image acquisition unit 21, a human region detection unit 22, a human feature extraction unit 23, an object region detection unit 24, an object feature extraction unit 25, a human-object relationship extraction unit 26a, and an estimation unit 27. The estimation unit 27 is configured to realize estimation means in this modification. The components other than the human-object relationship extraction unit 26a are as described above, and therefore will not be described again.

[0107] The person-object relationship extraction unit 26a extracts a relationship between a person included as a subject in the image pic and at least one of one or more objects based on the person feature amount hfq and the object feature amount ofq stored in the storage unit 34. The person-object relationship extraction unit 26a generates information indicating the relationship between the extracted person and at least one of the one or more objects. The person-object relationship extraction unit 26a supplies the generated information to the estimation unit 27.

[0108] The configuration of the person-object relationship extraction unit 26a will be described with reference to Fig. 10. Fig. 10 is a block diagram showing the configuration of the person-object relationship extraction unit 26a according to this modified example.

[0109] 10 , the person-object relationship extraction unit 26a includes an acquisition unit 261, an object feature sequence generation unit 262a, and an information generation unit 263. In this modification, the acquisition unit 261, the object feature sequence generation unit 262a, and the information generation unit 263 respectively function as an acquisition means, an object feature sequence generation means, and an information generation means. The functions of the acquisition unit 261 and the information generation unit 263 are as described above.

[0110] The object feature sequence generator 262a adds dummy feature values ​​dfq of dummy objects that are different from the object detected from the image pic, among the plurality of dummy objects.

[0111] An example of processing executed by the object feature sequence generation unit 262a will be described with reference to Fig. 11. Fig. 11 is a diagram showing an example of processing executed by the object feature sequence generation unit 262a according to this modification.

[0112] In the following, it is assumed that dummy 1 feature values ​​dfq1 to dummy 6 feature values ​​dfq6 indicating the features of dummy object 1 to dummy object 6, which are dummy objects for object 1 to object 6, respectively, are stored in storage unit 34. Dummy 1 feature values ​​dfq1 to dummy 6 feature values ​​dfq6 may be the same as object 1 feature values ​​ofq1 to object 6 feature values ​​ofq6 for object 1 to object 6, respectively, or may be different (similar).

[0113] When the acquisition unit 261 acquires the person feature amount hfq, the object 1 feature amount ofq1, the object 2 feature amount ofq2, and the object 3 feature amount ofq3, the object feature amount sequence generation unit 262a acquires the object 1 feature amount ofq1, the object 2 feature amount ofq2, and the object 3 feature amount ofq3 from the acquisition unit 261. Then, in step S27 described above, the object feature amount sequence generation unit 262a generates the object feature amount sequence ofq_c by adding dummy 4 feature amounts dfq4 to dummy 6 feature amounts dfq6 that indicate the features of dummy object 4 to dummy object 6, which are objects different from objects 1 to 3 and correspond to the acquired object 1 feature amount ofq1, object 2 feature amount ofq2, and object 3 feature amount ofq3, respectively.

[0114] (Effects of the Behavior Inference Device 2a According to This Modification) In this way, the behavior inference device 2a according to this modification generates the object feature sequence ofq_c by adding, among multiple dummy objects, dummy features dfq of dummy objects that are different from the object detected in the image pic. For example, when a person is performing an action using an object, even if the object is not included as a subject in the image pic, the dummy feature dfq of the dummy object for the object is included in the object feature sequence ofq_c. In this case, the relationship between the dummy object for the object and the person is stronger, and the behavior inference device 2a infers that the person is performing an action using the dummy object.

[0115] Therefore, when a person is performing an action using an object, the action estimation device 2a can estimate the action that the person is performing using the object even if the object is not included as a subject in the image pic.

[0116] (Modification 2) In this modification, a configuration will be described in which a dummy feature dfq based on a human feature hfq is added.

[0117] (Behavior Inference Device 2b) The configuration of the behavior inference device 2b according to this modification will be described with reference to Fig. 12. Fig. 12 is a block diagram showing the configuration of the behavior inference device 2b according to this modification.

[0118] The behavior inference device 2b is configured to include a control unit 20b instead of the control unit 20 included in the behavior inference device 2 described above. As in the above-described modified example, a plurality of dummy features dfq are stored in the storage unit 34. The configuration other than the control unit 20b is the same as described above, and therefore description thereof will be omitted.

[0119] The control unit 20b controls each component included in the behavior inference device 2b.

[0120] 12, the control unit 20b includes an image acquisition unit 21, a human region detection unit 22, a human feature extraction unit 23, an object region detection unit 24, an object feature extraction unit 25, a human-object relationship extraction unit 26b, and an estimation unit 27. The estimation unit 27 is configured to realize estimation means in this modification. The components other than the human-object relationship extraction unit 26b are as described above, and therefore will not be described again.

[0121] The person-object relationship extraction unit 26b extracts a relationship between a person included as a subject in the image pic and at least one of one or more objects based on the person feature amount hfq and the object feature amount ofq stored in the storage unit 34. The person-object relationship extraction unit 26b generates information indicating the relationship between the extracted person and at least one of the one or more objects. The person-object relationship extraction unit 26b supplies the generated information to the estimation unit 27.

[0122] The configuration of the person-object relationship extraction unit 26b will be described with reference to Fig. 13. Fig. 13 is a block diagram showing the configuration of the person-object relationship extraction unit 26b according to this modified example.

[0123] 13 , the person-object relationship extraction unit 26b includes an acquisition unit 261, an object feature sequence generation unit 262b, and an information generation unit 263. In this modification, the acquisition unit 261, the object feature sequence generation unit 262b, and the information generation unit 263 respectively function as an acquisition means, an object feature sequence generation means, and an information generation means. The functions of the acquisition unit 261 and the information generation unit 263 are as described above.

[0124] The object feature sequence generation unit 262b adds dummy feature values ​​dfq of dummy objects based on the human feature values ​​hfq acquired by the acquisition unit 261. As an example, the object feature sequence generation unit 262b uses a machine learning model that has been trained to use the human feature values ​​as input and estimate objects used in actions performed by a person corresponding to the human feature values. In this case, the object feature sequence generation unit 262b inputs the human feature values ​​hfq acquired by the acquisition unit 261 into the machine learning model, thereby adding dummy feature values ​​dfq of dummy objects of the estimated objects.

[0125] An example of processing executed by the object feature sequence generator 262b will be described with reference to Fig. 14. Fig. 14 is a diagram showing an example of processing executed by the object feature sequence generator 262b according to this modification.

[0126] When the acquisition unit 261 acquires the person feature amount hfq, the object 1 feature amount ofq1, the object 2 feature amount ofq2, and the object 3 feature amount ofq3, the object feature amount sequence generation unit 262a acquires the person feature amount hfq, the object 1 feature amount ofq1, the object 2 feature amount ofq2, and the object 3 feature amount ofq3 from the acquisition unit 261. Then, in step S27 described above, the object feature amount sequence generation unit 262b generates the object feature amount sequence ofq_c by adding a dummy feature amount dfq of a dummy object based on the acquired person feature amount hfq.

[0127] (Effects of the Behavior Inference Device 2b According to This Modification) In this way, the behavior inference device 2b according to this modification generates the object feature sequence ofq_c by adding a dummy feature dfq based on the person feature hfq. For example, when a person is performing an action using an object, even if the object is not included as a subject in the image pic, a dummy feature dfq based on the person's features (such as the person's posture or the color of the clothes the person is wearing) is included in the object feature sequence ofq_c. In this case, the relationship between the dummy object of the object and the person is strengthened, and the behavior inference device 2b infers that the person is performing an action using the dummy object.

[0128] Therefore, when a person is performing an action using an object, the action estimation device 2b can estimate the action that the person is performing using the object even if the object is not included as a subject in the image pic.

[0129] [Example of Software Implementation] Some or all of the functions of the behavior inference devices 1, 2, 2a, and 2b may be implemented by hardware such as an integrated circuit (IC chip), or may be implemented by software.

[0130] In the latter case, the behavior estimation devices 1, 2, 2a, and 2b are realized, for example, by a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in FIG. 15 . The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for operating the computer C as the behavior estimation devices 1, 2, 2a, and 2b. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing each function of the behavior estimation devices 1, 2, 2a, and 2b.

[0131] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0132] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0133] The program P can also be recorded on a non-transitory, tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0134] [Additional Note 1] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiments are also included in the technical scope of the present invention.

[0135] [Additional Note 2] Part or all of the above-described embodiment can also be described as follows: However, the present invention is not limited to the following described aspects.

[0136] (Supplementary Note 1) A behavior estimation device comprising: an acquisition means for acquiring human features indicating features of a person detected from an image, and object features indicating features of each of one or more objects detected from the image; an object feature sequence generation means for generating an object feature sequence by adding object features of dummy objects to the object features; an information generation means for generating information indicating a relationship between the person and at least one of the one or more objects and the dummy object based on the human features and each object feature included in the object feature sequence; and an estimation means for estimating a behavior of the person based on the relationship indicated by the information.

[0137] (Supplementary Note 2) The behavior inference device according to Supplementary Note 1, wherein the information generation means generates information indicating a relationship between at least the person and the dummy object when a relationship between the person and each of the one or more objects is weak.

[0138] (Supplementary Note 3) The behavior inference device according to Supplementary Note 1 or 2, wherein the information generation means sets a weight for each object feature included in the object feature sequence, and generates the information based on a weighted average for each of the one or more objects.

[0139] (Supplementary Note 4) The behavior estimation device according to Supplementary Note 3, wherein the information generation means generates, as the information, a feature amount obtained by integrating an object feature amount indicating a feature of an object whose weighted average value is greater than a threshold value and the person feature amount.

[0140] (Supplementary Note 5) The behavior estimation device according to any one of Supplementary Notes 1 to 4, wherein, when the information indicates a strong relationship between the dummy object and the person, the estimation means estimates that the person is not performing an action using an object.

[0141] (Supplementary Note 6) The behavior estimation device according to any one of Supplementary Notes 1 to 4, wherein the object feature sequence generation means adds features of a dummy object, of a plurality of dummy objects, that is different from the object detected from the image.

[0142] (Supplementary Note 7) The behavior estimation device according to any one of Supplementary Notes 1 to 4, wherein the object feature sequence generation means adds object features of a dummy object based on the human feature.

[0143] (Supplementary Note 8) A behavior estimation method including: a behavior estimation device acquiring human features indicating features of a person detected from an image, and object features indicating features of each of one or more objects detected from the image; generating an object feature sequence by adding object features of dummy objects to the object features; generating information indicating a relationship between the person and at least one of the one or more objects and the dummy object based on the human features and each object feature included in the object feature sequence; and estimating a behavior of the person based on the relationship indicated by the information.

[0144] (Supplementary Note 9) A program that causes a computer to execute an acquisition process that acquires human features indicating features of a person detected from an image and object features indicating features of each of one or more objects detected from the image; an object feature sequence generation process that generates an object feature sequence by adding object features of dummy objects to the object features; an information generation process that generates information indicating a relationship between the person and at least one of the one or more objects and the dummy objects based on the human features and each object feature included in the object feature sequence; and an estimation process that estimates the behavior of the person based on the relationship indicated by the information.

[0145] [Additional Note 3] Part or all of the above-described embodiment can also be expressed as follows.

[0146] (Supplementary Note 1) A behavior estimation device including at least one processor, wherein the at least one processor executes an acquisition process of acquiring human feature amounts indicating features of a person detected from an image and object feature amounts indicating features of each of one or more objects detected from the image; an object feature sequence generation process of generating an object feature sequence by adding object feature amounts of dummy objects to the object feature amounts; an information generation process of generating information indicating a relationship between the person and at least one of the one or more objects and the dummy object based on the human feature amounts and each object feature included in the object feature sequence; and an estimation process of estimating a behavior of the person based on the relationship indicated by the information.

[0147] (Supplementary Note 2) The behavior estimation device according to Supplementary Note 1, wherein, in the information generation process, when a relationship between the person and each of the one or more objects is weak, the at least one processor generates information indicating a relationship between the person and the dummy object.

[0148] (Supplementary Note 3) The behavior estimation device according to Supplementary Note 1 or 2, wherein, in the information generation process, the at least one processor sets a weight for each object feature included in the object feature sequence, and generates the information based on a weighted average for each of the one or more objects.

[0149] (Supplementary Note 4) The behavior estimation device according to Supplementary Note 3, wherein in the information generation process, the at least one processor generates, as the information, a feature amount obtained by integrating an object feature amount indicating a feature of an object whose weighted average value is greater than a threshold value and the person feature amount.

[0150] (Supplementary Note 5) The behavior estimation device according to any one of Supplementary Notes 1 to 4, wherein, when the information indicates a strong relationship between the dummy object and the person, the at least one processor estimates in the estimation process that the person is not performing an action using an object.

[0151] (Supplementary Note 6) The behavior estimation device according to any one of Supplementary Notes 1 to 4, wherein the at least one processor adds, in the object feature sequence generation process, features of a dummy object that is different from the object detected from the image, among a plurality of dummy objects.

[0152] (Supplementary Note 7) The behavior estimation device according to any one of Supplementary Notes 1 to 4, wherein the at least one processor adds object features of a dummy object based on the human features in the object feature sequence generation process.

[0153] The behavior inference device may further include a memory that stores a program for causing the processor to execute the acquisition process, the feature sequence generation process, the information generation process, and the inference process. The program may be recorded on a computer-readable, non-transitory, tangible recording medium.

[0154] 1, 2, 2a, 2b Behavior estimation device 11, 261 Acquisition unit 12, 262, 262a, 262b Object feature sequence generation unit 13, 263 Information generation unit 14, 27 Estimation unit 20, 20a, 20b Control unit 21 Image acquisition unit 22 Person region detection unit 23 Person feature extraction unit 24 Object region detection unit 25 Object feature extraction unit 26, 26a, 26b Object relationship extraction unit 264 Object feature comparison unit 265 Feature integration unit hfq Person feature ofq Object feature dfq Dummy feature pic Image

Claims

1. Acquisition means for acquiring human feature quantities that represent the characteristics of a person detected from an image, and object feature quantities that represent the characteristics of one or more objects detected from the image, An object feature sequence generation means generates an object feature sequence by adding the object feature sequence of a dummy object to the aforementioned object feature sequence, Information generation means that generates information indicating the relationship between the person and one or more objects and at least one of the dummy objects based on the person characteristics and each object characteristic included in the object characteristic sequence, An estimation means for estimating the actions of the person based on the relationships indicated by the aforementioned information, An action estimation device equipped with the following features.

2. The information generation means generates information indicating at least the relationship between the person and the dummy object if the relationship between the person and each of the one or more objects is weak. The behavior estimation device according to claim 1.

3. The information generation means is A weight is assigned to each object feature included in the aforementioned sequence of object features. Based on the weighted average of each of the one or more objects, the information is generated. The behavior estimation device according to claim 1 or 2.

4. The information generation means generates a feature quantity as the information which is an integrated feature quantity obtained by combining an object feature quantity that indicates the characteristics of an object whose weighted average value is greater than a threshold, and the human feature quantity. The behavior estimation device according to claim 3.

5. If the information indicates a strong relationship between the dummy object and the person, the estimation means estimates that the person did not use the object for any action. The behavior estimation device according to claim 1 or 2.

6. The object feature sequence generation means adds feature quantities of dummy objects that are different from the objects detected from the image, among a plurality of dummy objects. The behavior estimation device according to claim 1 or 2.

7. The object feature sequence generation means adds object features of a dummy object based on the human features. The behavior estimation device according to claim 1 or 2.

8. The behavior estimation device, Obtaining human feature quantities that represent the characteristics of a person detected from the image, and object feature quantities that represent the characteristics of one or more objects detected from the image, By adding the object features of a dummy object to the aforementioned object features, a sequence of object features is generated. Based on the aforementioned person features and each object feature included in the object feature sequence, information is generated that shows the relationship between the person and at least one of the one or more objects and the dummy object. Based on the relationships indicated by the aforementioned information, the actions of the person are estimated. A method for estimating behavior, including the following:

9. On the computer, An acquisition process to acquire human feature quantities that represent the characteristics of a person detected from an image, and object feature quantities that represent the characteristics of one or more objects detected from the image, The object feature sequence generation process generates an object feature sequence by adding the object feature sequence of a dummy object to the aforementioned object feature sequence, An information generation process that generates information indicating the relationship between the person and at least one of the one or more objects and the dummy object, based on the person characteristics and each object characteristic included in the object characteristic sequence. An estimation process that estimates the actions of the person based on the relationships indicated by the aforementioned information, A program that executes the command.