Action recognition device, action recognition method, and action recognition program

Abstract

This action recognition device is provided with: an acquisition unit that acquires image data and text describing an action; an extraction unit that extracts, from an image represented by the image data, a plurality of objects including an object that is an entity of action; an identification unit that identifies, from among the plurality of objects extracted by the extraction unit, an object related to the object that is the entity; a generation unit that generates a second feature amount using the feature amount of the object that is the entity and the feature amount of the object identified by the identification means; and a recognition unit that recognizes the action of the object that is the entity on the basis of the degree of similarity between the feature amount of the text and the second feature amount.

Description

Behavior recognition device, behavior recognition method, and behavior recognition program

[0001] The present disclosure relates to a behavior recognition device, a behavior recognition method, and a behavior recognition program.

[0002] Techniques for recognizing human actions from video are known. One example of a technique for recognizing actions is described in Non-Patent Document 1. Non-Patent Document 1 describes a technique called ActionCLIP that detects any action by utilizing a trained model that is pre-trained using a large amount of training data so that video features match features encoded from labeled text. Non-Patent Document 2 also describes an encoder that is trained so that linguistic features and video features are similar.

[0003] Mengmeng Wang et. al., "ActionCLIP: A New Paradigm for Video Action Recognition", 2021Alec Radford et. al., "Learning Transferable Visual Models From Natural Language Supervision", 2021

[0004] The technique described in Non-Patent Document 1 has a problem in that it is not possible to recognize behaviors by focusing on clues that distinguish between behaviors. The technique described in Non-Patent Document 2 also has a similar problem.

[0005] The present disclosure has been made in consideration of the above-mentioned problems, and an exemplary purpose thereof is to provide a technology capable of performing behavior recognition focusing on clues for distinguishing behaviors.

[0006] An information processing device according to an exemplary aspect of the present disclosure includes an acquisition means for acquiring image data and text describing an action, an extraction means for extracting a plurality of objects including an object that is the subject of the action from an image represented by the image data, an identification means for identifying an object related to the subject object from the plurality of objects extracted by the extraction means, a generation means for generating a second feature using a feature of the subject object and a feature of the object identified by the identification means, and a recognition means for recognizing the action of the subject object based on the similarity between the feature of the text and the second feature.

[0007] An information processing method according to an exemplary aspect of the present disclosure includes: an acquisition process in which at least one processor acquires image data and text describing an action; an extraction process in which the at least one processor extracts a plurality of objects, including an object that is the subject of the action, from an image represented by the image data; an identification process in which the at least one processor identifies an object related to the subject object from the plurality of objects extracted in the extraction process; a generation process in which the at least one processor generates second features using features of the subject object and features of the object identified in the identification process; and a recognition process in which the at least one processor recognizes the action of the subject object based on the similarity between the features of the text and the second features.

[0008] An information processing program according to an exemplary aspect of the present disclosure is a program that causes a computer to function as a behavior recognition device, and causes the computer to function as an acquisition means that acquires image data and text describing behavior, an extraction means that extracts multiple objects including an object that is the subject of the behavior from an image represented by the image data, an identification means that identifies an object related to the subject object from the multiple objects extracted by the extraction means, a generation means that generates second features using features of the subject object and features of the object identified by the identification means, and a recognition means that recognizes the behavior of the subject object based on the similarity between the features of the text and the second features.

[0009] According to an exemplary aspect of the present disclosure, an exemplary effect is achieved in that a technology that can recognize actions with higher accuracy without requiring a large amount of training data can be provided.

[0010] FIG. 1 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 2 is a flow diagram showing the flow of an information processing method according to the present disclosure. FIG. 3 is a diagram showing a specific example of behavior recognition processing of an information processing device according to the present disclosure. FIG. 4 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 5 is a diagram showing an example of the functional configuration of a control unit according to the present disclosure. FIG. 6 is a diagram showing an example of a target person and an object extracted by an object extraction unit according to the present disclosure. FIG. 7 is a diagram showing an outline of an example of processing of a relationship determination unit according to the present disclosure. FIG. 8 is a flow diagram showing an example of the flow of a behavior recognition method according to the present disclosure. FIG. 9 is a block diagram showing the configuration of a computer functioning as a behavior recognition device and an information processing device according to the present disclosure.

[0011] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, the effects mentioned in the exemplary embodiments shown below are examples of effects expected in the exemplary embodiments, and do not define the scope of the present invention. In other words, embodiments that do not exhibit the effects mentioned in the exemplary embodiments shown below may also be included in the scope of the present invention.

[0012] [First Exemplary Embodiment] A first exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is a basic form for each of the exemplary embodiments described below. Note that the scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technical means shown in the drawings referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0013] (Configuration of Behavior Recognition Device) The configuration of the behavior recognition device 1 will be described with reference to FIG. 1. FIG. 1 is a block diagram showing the configuration of the behavior recognition device 1. As shown in FIG. 1, the behavior recognition device 1 includes an acquisition unit 11, an extraction unit 12, an identification unit 13, a generation unit 14, and a recognition unit 15. The acquisition unit 11 acquires image data and text describing behavior. The extraction unit 12 extracts multiple objects including an object that is the subject of the behavior from an image represented by the image data. The identification unit 13 identifies an object related to the subject object from the multiple objects extracted by the extraction unit 12. The generation unit 14 generates second features using features of the subject object and features of the object identified by the identification unit 13. The recognition unit 15 recognizes the behavior of the subject object based on the similarity between the features of the text and the second features.

[0014] (Effects of the Behavior Recognition Device) As described above, the behavior recognition device 1 includes an acquisition unit 11 that acquires image data and text describing behavior, an extraction unit 12 that extracts multiple objects including an object that is the subject of the behavior from an image represented by the image data, an identification unit 13 that identifies an object related to the subject object from the multiple objects extracted by the extraction unit 12, a generation unit 14 that generates second features using features of the subject object and features of the object identified by the identification unit 13, and a recognition unit 15 that recognizes the behavior of the subject object based on the similarity between the features of the text and the second features. Therefore, the behavior recognition device 1 has the effect of being able to perform behavior recognition that focuses on clues for distinguishing behaviors.

[0015] (Flow of behavior recognition method) The flow of the behavior recognition method S1 will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of the behavior recognition method S1. As shown in Fig. 2, the behavior recognition method S1 includes an acquisition process S11, an extraction process S12, an identification process S13, a generation process S14, and a recognition process S15.

[0016] In an acquisition process S11, at least one processor acquires image data and text describing an action. In an extraction process S12, the at least one processor extracts a plurality of objects, including an object that is the subject of the action, from an image represented by the image data. In an identification process S13, the at least one processor identifies an object related to the subject object from the plurality of objects extracted in the extraction process S12. In a generation process S14, the at least one processor generates second features using features of the subject object and features of the object identified in the identification process S13. In a recognition process S15, the at least one processor recognizes the action of the subject object based on the similarity between the features of the text and the second features.

[0017] (Effects of Information Processing Method) As described above, the behavior recognition method S1 includes an acquisition process S11 in which at least one processor acquires image data and text describing a behavior, an extraction process S12 in which the at least one processor extracts, from an image represented by the image data, a plurality of objects including an object that is a subject of the behavior, an identification process S13 in which the at least one processor identifies an object related to the subject object from the plurality of objects extracted in the extraction process S12, a generation process S14 in which the at least one processor generates second features using features of the subject object and features of the object identified in the identification process S13, and a recognition process S15 in which the at least one processor recognizes the behavior of the subject object based on the similarity between the features of the text and the second features. Therefore, the behavior recognition method S1 has the effect of enabling behavior recognition that focuses on clues for distinguishing behaviors.

[0018] Second Exemplary Embodiment A second exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be denoted by the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs. Furthermore, each technical means shown in each drawing referenced to describe this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs.

[0019] (Overview of Information Processing Device) An information processing device 1A according to the present disclosure executes processing for recognizing behavior. The information processing device 1A is an example of a behavior recognition device according to the present disclosure. Here, the subject of the behavior may be, for example, a person, but the subject of the behavior is not limited to a person and may be, for example, an animal such as a pet or a wild animal, or a robot equipped with a robotic arm that grasps or moves an object.

[0020] Examples of actions recognized by the information processing device 1A include actions of workers in the construction, civil engineering, and manufacturing industries. Examples of actions of workers at construction sites include "heavy machinery excavation," "compaction," and "material transportation." Furthermore, the actions may be, for example, the actions of patients in medical facilities or the actions of medical personnel (medical procedures, etc.). Furthermore, when the subject of the action is an animal such as a pet, the action may be, for example, the behavior of animals (e.g., cats) playing with each other.

[0021] 3 is a diagram showing a specific example of behavior recognition processing by information processing device 1A. In the example of FIG. 3, information processing device 1A recognizes the behaviors of worker A, worker B, and worker C from image 300 captured by image capture device 200, and outputs information representing the recognition results. In the example of FIG. 3, information processing device 1A recognizes "heavy equipment digging," "rest," "compaction," etc. as the behaviors of worker A in chronological order. Furthermore, information processing device 1A recognizes "guidance work," "approaching heavy equipment (danger)," "material transportation," etc. as the behaviors of worker B in chronological order. Furthermore, information processing device 1A recognizes "rest," "material transportation," "compaction," etc. as the behaviors of worker C.

[0022] 3, the information processing device 1A can manage the work of workers in, for example, the construction industry, civil engineering industry, manufacturing industry, etc. This can support, for example, the decision-making of site supervisors regarding management and business innovation. More specifically, for example, the site supervisor can warn a worker who is recognized to be engaging in dangerous behavior such as "approaching heavy machinery," thereby preventing accidents from occurring.

[0023] (Configuration of information processing device) The configuration of the information processing device 1A will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the configuration of the information processing device 1A. The information processing device 1A includes a control unit 10A, a storage unit 20A, a communication unit 30A, an input unit 40A, and an output unit 50A.

[0024] (Communication Unit) The communication unit 30A communicates with devices external to the information processing device 1A via a communication line. While the specific configuration of the communication line does not limit the present exemplary embodiment, examples of the communication line include a wireless local area network (LAN), a wired LAN, a wide area network (WAN), a public line network, a mobile data communication network, or a combination thereof. The communication unit 30A transmits data supplied from the control unit 10A to other devices, and supplies data received from other devices to the control unit 10A.

[0025] (Input Unit) The input unit 40A is configured to receive input to the information processing device 1A, and includes, for example, input devices such as a keyboard, a mouse, a touch panel, a camera, a microphone, etc. The input unit 40A may also be configured to receive data from the input devices via an interface such as a USB (Universal Serial Bus).

[0026] (Output Unit) The output unit 50A is a component for performing output from the information processing device 1A, and includes, for example, output devices such as a display, a printer, a touch panel, a speaker, etc. The output unit 50A may be configured to include, for example, an interface such as a USB, and to output data to the output device via the interface.

[0027] (Storage Unit) The storage unit 20 A stores various types of information referenced by the control unit 10 A. Examples of such information include inference data 201 and description information 202.

[0028] (Inference Data) The inference data 201 is time-series image data that is the target of behavior recognition. Examples of the image data include, but are not limited to, data obtained by capturing images of workers working in the construction or civil engineering industry using a camera.

[0029] (Description Information) The description information 202 is text representing an action. One piece of text may be stored in the storage unit 20A for one action, or multiple pieces of description information may be stored in the storage unit 20A for one action. In other words, there may be a one-to-one correspondence between an action and description information, or a one-to-multiple correspondence between an action and description information. For example, as description information representing the action "compacting," the storage unit 20A may store the text "compacting the ground using a yellow roller" and the text "compacting the ground using a brown roller." Furthermore, as description information representing the action "compacting," the storage unit 20A may store the text "walking holding a roller in both hands with hands extended forward."

[0030] (Control Unit) FIG. 5 is a diagram illustrating an example of the functional configuration of the control unit 10A. The control unit 10A includes an object extraction unit 11A, a language feature extraction unit 12A, a person tracking unit 13A, a video feature extraction unit 14A, a relationship determination unit 15A, an attribute information integration unit 16A, a similarity calculation unit 17A, and a behavior classification output unit 18A. The object extraction unit 11A is an example of an acquisition means and an extraction means according to the present disclosure. The language feature extraction unit 12A is an example of an acquisition means according to the present disclosure. The relationship determination unit 15A is an example of a specification means according to the present disclosure. The attribute information integration unit 16A is an example of a generation means and an integration means according to the present disclosure. The similarity calculation unit 17A and the behavior classification output unit 18A are examples of a recognition means.

[0031] (Object Extraction Unit) The object extraction unit 11A acquires the inference data 201 and extracts multiple objects, including an object that is the subject of an action, from an image represented by the acquired inference data 201. Here, examples of objects extracted by the object extraction unit 11A include people and objects. The objects extracted by the object extraction unit 11A include the object that is the subject of an action. The objects extracted by the object extraction unit 11A may also include objects related to the subject's action and objects unrelated to the subject's action. For example, if the action is "compaction," an example of an object related to the subject's action is a compactor used by a worker performing the compaction work. On the other hand, examples of objects unrelated to the subject's action include a cart for transporting luggage, an excavator, etc. Furthermore, examples of objects related to the action of "walking" include a "handrail," a "cane," a "walker," and a "rolling frame." Furthermore, examples of objects related to the action of "playing" include a "ball," a "pet," and a "stuffed toy."

[0032] In the following explanation, a case where the subject of the action is a person will be described. The person who is the subject of the action will also be referred to as a "target person." In the following explanation, a case where an object related to the target person's action is an object will be described.

[0033] The number of target persons extracted by the object extraction unit 11A may be 1 or more. When the object extraction unit 11A extracts more than one target person, a person tracking unit 13A, a video feature extraction unit 14A, a relationship determination unit 15A, an attribute information integration unit 16A, a similarity calculation unit 17A, and a behavior classification output unit 18A, which will be described later, each perform their respective processes for each of the multiple target persons.

[0034] For example, the object extraction unit 11A may acquire the inference data by reading the inference data 201 from a storage destination (which may be a storage device within the information processing device 1A or a storage device external to the information processing device 1A) designated by the user of the information processing device 1A. Alternatively, the object extraction unit 11A may acquire the inference data 201 by receiving the inference data from another device via the communication unit 30A. Alternatively, the object extraction unit 11A may acquire the inference data 201 input to the input unit 40A.

[0035] For example, the object extraction unit 11A extracts objects using a pre-trained, existing deep learning-based object detection model. Examples of object detection models include, but are not limited to, YOLOX and DETR (End-to-End Object Detection with Transformers). In this case, the input of the object detection model includes, for example, inference data, and the output includes, for example, object position information and object class name.

[0036] The object extraction unit 11A generates position information indicating the position of a target person included in an image represented by the image data, and generates position information indicating the position of an object included in the image represented by the image data. The object extraction unit 11A supplies the position information of the target person to the person tracking unit 13A, and supplies the time-series position information of the extracted object to the video feature extraction unit 14A.

[0037] Fig. 6 is a diagram showing an example of a target person and an object extracted by the object extraction unit 11 A. In the example of Fig. 6, the object extraction unit 11 A extracts a target person 301, an object 302, and an object 303 from an image represented by image data.

[0038] (Person Tracking Unit) The person tracking unit 13A tracks the target person using the position information of the target person supplied from the object extraction unit 11A, and generates tracking information representing the tracking result. The tracking information includes, for example, rectangle information in which IDs are assigned to the same person in the time series direction. For example, the person tracking unit 13A performs tracking using a rectangle-based tracking method, a pose-based tracking method, a segmentation-based tracking method, or the like using an existing tracking engine. The person tracking unit 13A supplies the generated tracking information to the video feature extraction unit 14A.

[0039] (Video Feature Extraction Unit) The video feature extraction unit 14A extracts features of a target person using tracking information supplied from the person tracking unit 13A, and extracts features of an object using time-series position information of the object supplied from the object extraction unit 11A. Hereinafter, the features of a person and features of an object extracted from image data by the video feature extraction unit 14A are also referred to as "video features." The video features are, for example, vectors. The video feature extraction unit 14A generates feature-related information including the video features, and supplies the generated feature-related information to the relationship determination unit 15A. In addition to the video features, the feature-related information includes position information of the target person or object and a class name of the object.

[0040] As an example, the video feature extraction unit 14A performs the extraction process using a video feature extractor that has been trained to make linguistic features and video features similar. As an example, the video feature extractor may be a trained model generated by supervised machine learning using training data. In this case, the input to the video feature extractor is image data representing an image included in a rectangular area of ​​a target person or object, and the output is video features of the target person or object. Examples of training methods for the trained model include, but are not limited to, error propagation training, and more specifically, methods using cross-entropy loss, etc.

[0041] An example of a video feature extractor is, but is not limited to, the visual encoder described in Non-Patent Document 2. As an example, video feature extraction unit 14A generates video features of a target person by inputting an image included in the area of ​​the target person into the video feature extractor, and generates video features of an object by inputting an image included in the area of ​​the object into the video feature extractor.

[0042] (Language Feature Extraction Unit) The language feature extraction unit 12A acquires description information 202 and extracts features from the acquired description information 202. As an example, the language feature extraction unit 12A may acquire the description information 202 by reading the description information 202 from a storage location (which may be a storage device within the information processing device 1A or a storage device external to the information processing device 1A) specified by the user of the information processing device 1A. Alternatively, the language feature extraction unit 12A may acquire the description information 202 by receiving the description information 202 from another device via the communication unit 30A. Alternatively, the language feature extraction unit 12A may acquire the description information 202 input to the input unit 40A.

[0043] Hereinafter, the features extracted by the language feature extraction unit 12A from the description information 202 are also referred to as "language features." The language features are, for example, vectors. As an example, the language feature extraction unit 12A extracts language features using a language feature extractor generated by machine learning. For example, a language feature extractor trained to be similar to the output features of a video feature extractor is used. In this case, the input to the language feature extractor is the description information 202, and the output is the language features of the description information 202. An example of such a language feature extractor is, but is not limited to, the text encoder described in Non-Patent Document 2. The language feature extraction unit 12A supplies the extracted language features to the relationship determination unit 15A and the similarity calculation unit 17A.

[0044] The language feature extraction unit 12A does not need to perform a process of extracting language features each time an inference for behavior recognition is performed, and the inference may be performed using language features extracted in advance. In this case, for example, the language feature extraction unit 12A performs a process of extracting language features from the description information 202 prior to an inference for behavior recognition, and stores the extracted language features in the storage unit 20A. In this case, the relationship determination unit 15A and the similarity calculation unit 17A use the language features stored in the storage unit 20A in the processes described below.

[0045] The relationship determination unit 15A identifies an object related to the target person from among the objects extracted by the object extraction unit 11A, and generates related information representing the identification result. The relationship determination unit 15A also weights the video features of each object using the generated related information.

[0046] (Relationship Information) The relation information is information indicating the relationship of an object to a target person. The relation information may be, for example, a vector in which each component corresponds to an object, or may be, for example, a list of objects related to the target person.

[0047] (Example 1 of related information) As an example, the related information is a vector in which each component corresponds to an object. In this case, the number of components of the vector corresponds to the number of objects extracted from the image. As another example, the related information may be a vector in which each component corresponds to an object related to the target person among the objects extracted from the image. In this case, the number of components of the vector corresponds to the number of objects related to the target person.

[0048] More specifically, the related information may be, for example, a one-hot vector in which each component corresponds to an object. For example, when the related information u=[1, 0, ...], the related information u indicates that "the object corresponding to the first component corresponds to the target person, and the objects corresponding to components other than the first component do not correspond to the target person."

[0049] Furthermore, as an example, the related information may be a vector in which each component corresponds to an object and each component value indicates the probability that the component value is related to the target person. For example, when the related information u=[0.9, 0.5, 0.5, ...], the related information u indicates that "the probability that the object corresponding to the first component is related to the target person is 0.9, the probability that the object corresponding to the second component is related to the target person is 0.5, the probability that the object corresponding to the third component is related to the target person is 0.5, ...".

[0050] Alternatively, the related information may be a vector in which each component corresponds to an object, each component value indicates a probability that the component is related to the target person, and the sum of the components is 1. For example, when the related information u = [0.8, 0.2, 0.0, ...], the related information u indicates that there are two objects related to the target person, an object corresponding to the first component and an object corresponding to the second component, and that the probability that the object corresponding to the first component is related to the target person is 0.8, and the probability that the object corresponding to the second component is related to the target person is 0.2.

[0051] (Example 2 of Related Information) The related information may also be a list of objects related to the target person. For example, when the related information u = [[x1, y1, x2, y2, roller], ...], this indicates that an object with location information "x1, y1, x2, y2" and a class name "roller" is included in the objects related to the target person.

[0052] (Example 1 of Relevance Determination Method) The relationship determination unit 15A may determine the relevance between a target person and an object using a rule-based method. More specifically, as an example, the relationship determination unit 15A may identify at least one of an object whose at least a portion of an area in an image represented by image data overlaps with an area of ​​the target person and an object whose distance from the target person in the image is equal to or less than a threshold. In this case, as an example, the relationship determination unit 15A may determine the relevance using the feature amounts (video features, location information, etc.) of the target person and the feature amounts (video features, location information, etc.) of the object. In this case, the relationship determination unit 15A may determine, for example, that an object whose rectangular area overlaps with the rectangular area of ​​the target person in the image represented by the image data is related to the target person. As another example, the video feature extraction unit 14A may estimate the posture of the target person to generate posture information, and the relationship determination unit 15A may use the generated posture information to determine the relevance according to a rule prepared in advance. For example, the relationship determination unit 15A may determine that an object is related to the target person if the distance from the target person's hand to the rectangular center of the object is within a threshold. For example, in the example of Fig. 6, the relationship determination unit 15A determines that the object 302 is related to the target person if the distance between the center position of the rectangular region and the position of the target person's hand is equal to or less than a threshold.

[0053] (Example 2 of Relevance Determination Method) Furthermore, the relationship determination unit 15A may identify an object related to a target person based on information obtained by inputting video features of the target person and video features of the object into a trained model generated by machine learning. In this case, the trained model may, for example, be a trained model generated by supervised machine learning using training data. In this case, the input of the trained model includes, for example, video features of the target person and video features of the object, and the output is, for example, a vector in which each component value represents the degree of relevance between the object and the target person. Examples of training methods for the trained model include, for example, learning using an error propagation method, and more specifically, examples of methods using, for example, cross-entropy loss, but are not limited to these. Furthermore, the trained model may, for example, be a model having an attention mechanism.

[0054] 7 is a diagram illustrating an example of a process in which the relationship determination unit 15A generates related information using a trained model having an attention mechanism. In the example of FIG. 7, the relationship determination unit 15A generates related information based on the video feature V ｔ１ , and the image feature of the object V ｏ１ , …V ｏＮ , are input to a multilayer perceptron (MLP). The output from the MLP is converted into an output vector through attention calculation processing, and related information, which is a one-hot vector, is obtained from the output. In the attention calculation processing, MatMul calculates the dot product of Q and K, and Scale divides the output of MatMul by √d using d, which is the common dimensionality of Q and K.

[0055] (Example 3 of Relevance Determination Method) As another example, the relationship determination unit 15A may identify an object related to the target person by using the video features of each of the multiple objects extracted by the object extraction unit 11A and the linguistic features extracted by the linguistic feature extraction unit 12A. In this case, the relationship determination unit 15A narrows down candidates for objects related to the target person based on, for example, the description information 202.

[0056] 8 is a diagram illustrating an example of a process in which the relationship determination unit 15A generates related information. In the example of FIG. 8, the relationship determination unit 15A uses a first trained model M11 and a second trained model M12. The relationship determination unit 15A first uses the first trained model M11 to generate a description feature T １ , ...T Ｍ (M is the number of descriptive information) and the image features of the target person V ｔ１ Based on the similarity with ｔ１ Update the video feature V ｔ２ More specifically, the relationship determination unit 15A first generates the image feature V ｔ１ , description feature T １ , ...T Ｍ (M is the number of descriptive information) are input to the MLP. The output from the MLP is converted into an output vector through attention calculation processing. The relationship determination unit 15A adds descriptive features T1, ...T to the output vector. Ｍ The accumulated result is added to the image feature Vｔ１ The video feature V ｔ２ Let be the input of the second trained model M12.

[0057] Next, the relationship determination unit 15A determines the video feature V ｔ２ (an example of a third feature amount) and the image feature V ｏ１ , …V ｏＮ , and identifies an object related to the target person based on the similarity between the image feature V ｔ２ , the image feature of the object V ｏ１ , …V ｏＮ , are input to a multilayer perceptron (MLP). The output from the MLP is converted into an output vector through attention calculation processing, and related information, which is a one-hot vector, is obtained from the output.

[0058] (Specific example of weighting process using related information) The relationship determination unit 15A weights the video features of the objects using the related information and supplies the weighted video features to the attribute information integrating unit 16A. For example, if the related information is a vector, the relationship determination unit 15A multiplies each of the video features of the objects by a component value corresponding to each object and supplies the result to the attribute information integrating unit 16A. For example, if the related information is a one-hot vector, only the video features of the objects corresponding to components with a value of 1 are supplied to the attribute information integrating unit 16A.

[0059] (Attribute Information Integration Unit) The attribute information integration unit 16A calculates an integrated feature by integrating attribute information with the video feature supplied from the relationship determination unit 15A for each of the target person and object extracted by the object extraction unit 11A. The attribute information is information indicating the attributes of the target person and object, and is, for example, text. The attribute information can also be said to be information that explicitly describes the target person and object. Examples of person attribute information include information indicating gender, occupation (carpenter, supervisor, etc.), and basic behavior (running, sitting, holding, walking, etc.). Examples of object attribute information include the name, color, size, and class name of the object.

[0060] As an example, the attribute information integrating unit 16A uses text indicating the class names of the target person and object, the size of the object, the color of the object, etc. generated by the object extraction unit 11A as attribute information. The attribute information integrating unit 16A may also identify attribute information corresponding to the class name of the target person or object by referring to a table in which object class names and attribute information (occupation, basic behavior, etc.) are previously associated with each other. The attribute information integrating unit 16A may also identify attribute information by inputting image data of rectangular areas of the object into a trained model that has been trained in advance by machine learning to output attribute information.

[0061] (Specific Example of Attribute Information Extraction Processing) The attribute information integration unit 16A extracts features from the attribute information of a target person and integrates the extracted features into video features of the target person, and also extracts features from the attribute information of an object and integrates the extracted features into video features of the object. Hereinafter, the features extracted from the attribute information are also referred to as "attribute features." As an example, the attribute information integration unit 16A extracts attribute features from the attribute information using a linguistic feature extractor. As an example, the linguistic feature extractor may be a trained model generated by supervised machine learning using training data. In this case, the input of the linguistic feature extractor is the attribute information, and the output is the attribute features. Examples of training methods for the trained model include, but are not limited to, learning using an error propagation method, and more specifically, methods using cross-entropy loss, etc.

[0062] (Specific Example of Process for Integrating Attribute Information into Video Features) As an example, the attribute information integration unit 16A uses a weight parameter w ｉ (0≦w ｉ ≦1) to obtain the image feature F ｖｉｓ and the extracted N attribute features F ａｔｔ ｉ (0<i≦N, N is the number of attributes) is weighted to calculate the overall feature F. In formula (1), w ｉ (0≦w ｉ ≦1) is a weight parameter, The weight parameter w in equation (1) ｉmay be a parameter learned by machine learning.

[0063] The attribute information integrating unit 16A generates behavior classification features (an example of second features) using the integrated features F of the target person and the integrated features F of the object, and supplies the generated behavior classification features to the similarity calculating unit 17A. The behavior of the target person is recognized based on the similarity between the behavior classification features generated by the attribute information integrating unit 16A and the linguistic features of the description information 202. As an example, the attribute information integrating unit 16A performs statistical processing on the integrated features F of the target person and the integrated features F of each of the multiple objects to generate behavior classification features. More specifically, as an example, the attribute information integrating unit 16A may use an average vector of the integrated features F of the target person and the integrated features F of the multiple objects as the behavior classification features.

[0064] (Similarity Calculation Unit) The similarity calculation unit 17A calculates the similarity between the behavioral classification features supplied from the attribute information integration unit 16A and the language features supplied from the language feature extraction unit 12A. As an example, the similarity calculation unit 17A calculates the similarity using a trained model that has been pre-trained so that video features and text features match. In this case, the input of the trained model includes the behavioral classification features supplied from the attribute information integration unit 16A and the language features supplied from the language feature extraction unit 12A, and the output includes a score indicating the similarity. Examples of training methods for the trained model include, but are not limited to, the technology described in Non-Patent Document 1. Examples of similarities calculated by the similarity calculation unit 17A include, but are not limited to, cosine similarity.

[0065] (Behavior Classification Output Unit) The behavior classification output unit 18A classifies the behaviors using the similarities calculated by the similarity calculation unit 17A and outputs the classification results. As an example, the behavior classification output unit 18A identifies, among multiple behaviors, the behavior that has the highest similarity to the behavior classification feature as the behavior of the target person.

[0066] When multiple language features correspond to one behavior, the behavior classification output unit 18A may classify the behavior based on the average or maximum value of the similarity between the behavior classification feature and the language feature for each behavior. For example, the behavior classification output unit 18A may identify the behavior with the highest average value as the behavior of the target person.

[0067] As an example, the behavior classification output unit 18A may output the classification result by writing information indicating the classification result to a storage destination (which may be a storage device within the information processing device 1A or a storage device external to the information processing device 1A) specified by the user of the information processing device 1A. Furthermore, the behavior classification output unit 18A may output the classification result by transmitting the information indicating the classification result via the communication unit 30A, or may output the information indicating the classification result to an output device such as a display.

[0068] (Flow of Information Processing Method) Fig. 9 is a flowchart showing an example of the flow of a behavior recognition method S1A executed by the information processing device 1 A. The steps included in the flowchart in Fig. 9 may be executed in parallel or in a different order.

[0069] In step S101, the object extraction unit 11A extracts a target person and an object from an image represented by the image data. In step S102, the person tracking unit 13A tracks the target person using the position information of the target person supplied from the object extraction unit 11A, and generates tracking information indicating the tracking result.

[0070] In step S103, the image feature extraction unit 14A extracts the image feature F of the target person using the tracking information supplied from the person tracking unit 13A. ｖｉｓ , and extracts the image feature F of the object using the time-series position information of the object supplied from the object extraction unit 11A. ｖｉｓ Extract.

[0071] In step S104, the language feature extraction unit 12A extracts language features from the description information. In step S105, the relationship determination unit 15A determines the relationship between the object extracted by the object extraction unit 11A and the target person, and identifies an object related to the target person.

[0072] In step S106, the attribute information integration unit 16A calculates an integrated feature for each object extracted by the object extraction unit 11A by integrating the attribute feature of each object with the video feature supplied from the relationship determination unit 15A. The attribute information integration unit 16A also generates a feature for behavior classification using the integrated feature of the target person and the integrated feature of the object, and supplies the generated integrated feature to the similarity calculation unit 17A.

[0073] In step S107, the similarity calculation unit 17A calculates the similarity between the behavior classification features and the language features supplied from the attribute information integrating unit 16A. In step S108, the behavior classification output unit 18A classifies the behaviors using the similarity calculated by the similarity calculation unit 17A.

[0074] In step S109, the behavior classification output unit 18A outputs the classification result of the behavior of the target person. At this time, the behavior classification output unit 18A may output information indicating an object related to the target person. In other words, the behavior classification output unit 18A may output at least one of information indicating an object related to the target person and information indicating the behavior classification result (behavior recognition result).

[0075] Furthermore, the behavior classification output unit 18A may output information to support decision-making regarding the behavior of the target person based on the behavior recognition result. Here, examples of the information to support decision-making include information to warn workers who are engaging in dangerous behavior, information indicating what kind of care is needed for patients who require nursing care in medical facilities, etc.

[0076] (Use Case) The information processing device 1A according to the present disclosure can be utilized in various fields. For example, the information processing device 1A according to the present disclosure can be utilized in the medical / healthcare field. For example, it is conceivable to detect a person performing a specific action from video captured in a medical facility such as a hospital. For example, by having the information processing device 1A extract a person walking from the captured video using descriptive information corresponding to the action of "walking," the target person can be detected with higher accuracy even if the video includes objects unrelated to walking (e.g., a bed or a chair).

[0077] (Effects of Information Processing Device) However, the technology described in Non-Patent Document 1 may not be able to perform behavior recognition that focuses on clues for distinguishing behaviors. For example, if a video of a worker performing compaction work also shows heavy machinery used for leveling work, the worker may be recognized as performing leveling work, even though the worker is performing compaction work. In contrast, the information processing device 1A according to the present disclosure explicitly considers objects in text-based behavior recognition, thereby enabling arbitrary text-based recognition that focuses on clues for distinguishing behaviors. As a result, the information processing device 1A enables behavior recognition with higher accuracy.

[0078] Furthermore, in the information processing device 1A, the identification means is configured to identify an object related to the target person by using the image features of the target person, the image features of the object, and the linguistic features of the description information extracted by the object extraction unit 11A. Therefore, the information processing device 1A has the effect of being able to determine the relationship between the target person and the object by taking into account the description information of the behavior.

[0079] Furthermore, the information processing device 1A includes an attribute information integration unit 16A that generates an integrated feature by integrating attribute information of a target person with video features of the target person, and generates an integrated feature by integrating attribute information of an item with video features of the item, and the similarity calculation unit 17A is configured to generate features for behavior recognition using the integrated feature of the target person and the integrated feature of the item. Therefore, the information processing device 1A enables behavior recognition with higher accuracy by taking into account attribute information that explicitly describes the target person and object.

[0080] Furthermore, in the information processing device 1A, the relationship determination unit 15A is configured to update the video features of the target feature based on the similarity between the linguistic features of the descriptive information and the video features of the target person, and to identify an item related to the target person based on the similarity between the updated video features (third feature amount) and the video features of the object. Therefore, the information processing device 1A has the effect of being able to determine the relationship between the target person and the object in consideration of the descriptive information of the behavior.

[0081] Furthermore, the information processing device 1A employs a configuration in which the behavior classification output unit 18A outputs at least one of information indicating an item associated with the target person and information indicating a behavior recognition result. Therefore, the information processing device 1A has the effect of enabling a user of the information processing device 1A to understand the behavior recognition result and also to understand information that serves as the basis for behavior recognition (information indicating an item associated with the target person).

[0082] Furthermore, in the information processing device 1A, the relationship determination unit 15A is configured to identify at least one of an object having at least a partial area overlapping with an area of ​​the target person in the image represented by the image data and an object having a distance from the target person equal to or less than a threshold value in the image. Therefore, the information processing device 1A has the effect of being able to more accurately identify an object related to the target person.

[0083] In addition, in the information processing device 1A, the relationship determination unit 15A adds the image feature F of the target person to the trained model generated by machine learning. ｖｉｓ and the image feature F of the objectｖｉｓ The information processing device 1A is configured to identify an item related to the target person based on information obtained by inputting the above information. Therefore, the information processing device 1A has the effect of being able to identify an item related to the target person with higher accuracy.

[0084] Furthermore, in the information processing device 1A, the behavior classification output unit 18A is configured to output information for supporting decision-making regarding the target person based on the behavior recognition result. Therefore, the information processing device 1A has the effect of being able to support decision-making regarding the behavior of the target person.

[0085] [Example of Implementation by Software] Some or all of the functions of the behavior recognition device 1 and the information processing device 1A (hereinafter also referred to as "each of the above devices") may be implemented by hardware such as an integrated circuit (IC chip), or by software.

[0086] In the latter case, each of the above devices is realized by, for example, a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 10. Figure 10 is a block diagram showing the hardware configuration of computer C that functions as each of the above devices.

[0087] The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for causing the computer C to function as each of the above-mentioned devices. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing the functions of each of the above-mentioned devices.

[0088] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0089] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0090] The program P can also be recorded on a non-transitory, tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0091] [Appendix A] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0092] (Appendix A1) An action recognition device comprising: an acquisition means for acquiring image data and text describing an action; an extraction means for extracting a plurality of objects including an object that is the subject of an action from an image represented by the image data; an identification means for identifying an object related to the object that is the subject of an action from the plurality of objects extracted by the extraction means; a generation means for generating a second feature using a feature of the object that is the subject and a feature of the object identified by the identification means; and a recognition means for recognizing an action of the object that is the subject based on the similarity between the feature of the text and the second feature.

[0093] (Supplementary Note A2) The behavior recognition device according to Supplementary Note A1, wherein the specifying means specifies an object related to the subject by using feature amounts of each of the plurality of objects extracted by the extracting means and feature amounts of the text.

[0094] (Supplementary Note A3) The behavior recognition device according to Supplementary Note A1 or A2, further comprising: an integration means for generating an integrated feature by integrating attribute information of the object into a feature of the object, wherein the generation means generates the second feature using the integrated feature of the subject object and the integrated feature of the object identified by the identification means.

[0095] (Supplementary Note A4) The behavior recognition device according to Supplementary Note A2, wherein the identification means updates the features of the subject object based on a similarity between the features of the text and the features of the subject object to generate a third feature, and identifies an object related to the subject object based on a similarity between the third feature and each feature of an object other than the subject object among the plurality of objects.

[0096] (Supplementary Note A5) The behavior recognition device according to any one of Supplementary Notes A1 to A4, further comprising: an output unit configured to output at least one of information indicating the object identified by the identification unit and information indicating a recognition result by the recognition unit.

[0097] (Supplementary Note A6) The behavior recognition device according to Supplementary Note A1, wherein the identification means identifies at least one of an object whose at least a partial area in the image overlaps with an area of ​​the subject, and an object whose distance from the subject in the image is equal to or less than a threshold.

[0098] (Supplementary Note A7) The behavior recognition device according to Supplementary Note A1, wherein the identification means identifies an object related to the subject based on information obtained by inputting feature amounts of the plurality of objects into a trained model generated by machine learning.

[0099] (Supplementary Note A8) The behavior recognition device according to Supplementary Note A5, wherein the output means outputs information for supporting decision-making regarding the behavior based on a recognition result of the behavior.

[0100] [Appendix B] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0101] (Appendix B1) An action recognition method including: an acquisition process in which at least one processor acquires image data and text describing an action; an extraction process in which the at least one processor extracts a plurality of objects including an object that is the subject of an action from an image represented by the image data; an identification process in which the at least one processor identifies an object related to the object that is the subject of an action from the plurality of objects extracted in the extraction process; a generation process in which the at least one processor generates second features using features of the object that is the subject and features of the object identified in the identification process; and a recognition process in which the at least one processor recognizes the action of the object that is the subject based on the similarity between the features of the text and the second features.

[0102] (Supplementary Note B2) The behavior recognition method according to Supplementary Note B1, wherein in the identification process, the at least one processor identifies an object related to the subject by using features of each of the plurality of objects extracted in the extraction process and features of the text.

[0103] (Supplementary Note B3) The behavior recognition method described in Supplementary Note B1 or B2, further including an integration process in which the at least one processor generates an integrated feature by integrating attribute information of the object into a feature of the object, wherein in the generation process, the at least one processor generates the second feature using the integrated feature of the subject object and the integrated feature of an object identified in the identification process.

[0104] (Appendix B4) The behavior recognition method described in Appendix B2, wherein in the identification process, the at least one processor: updates the features of the subject object based on a similarity between the features of the text and the features of the subject object to generate a third feature; and identifies an object related to the subject object based on a similarity between the third feature and each feature of an object other than the subject object among the plurality of objects.

[0105] (Supplementary Note B5) The behavior recognition method according to any one of Supplementary Notes B1 to B4, further including an output process in which the at least one processor outputs at least one of information indicating an object identified in the identification process and information indicating a recognition result of the recognition process.

[0106] (Supplementary Note B6) The behavior recognition method according to Supplementary Note B1, wherein in the identification process, the at least one processor identifies at least one of an object whose at least a partial area overlaps with an area of ​​the subject in the image, and an object whose distance from the subject in the image is equal to or less than a threshold.

[0107] (Supplementary Note B7) The behavior recognition method according to Supplementary Note B1, wherein in the identification process, the at least one processor identifies an object related to the subject based on information obtained by inputting feature amounts of the plurality of objects into a trained model generated by machine learning.

[0108] (Supplementary Note B8) The behavior recognition method according to Supplementary Note B5, wherein in the output process, the at least one processor outputs information for supporting decision-making regarding the behavior based on a recognition result of the behavior.

[0109] [Appendix C] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0110] (Appendix C1) A behavior recognition program for causing a computer to function as a behavior recognition device, the behavior recognition program causing the computer to function as: an acquisition means for acquiring image data and text describing behavior; an extraction means for extracting multiple objects including an object that is the subject of behavior from an image represented by the image data; an identification means for identifying an object related to the subject object from the multiple objects extracted by the extraction means; a generation means for generating second features using features of the subject object and features of the object identified by the identification means; and a recognition means for recognizing the behavior of the subject object based on the similarity between the features of the text and the second features.

[0111] (Supplementary Note C2) The behavior recognition program according to Supplementary Note C1, wherein the identifying means identifies an object related to the subject by using feature amounts of each of the plurality of objects extracted by the extracting means and feature amounts of the text.

[0112] (Appendix C3) The behavior recognition program described in Appendix C1 or C2, further causing the computer to function as an integration means for generating an integrated feature by integrating attribute information of the object with a feature of the object, wherein the generation means generates the second feature using the integrated feature of the subject object and the integrated feature of the object identified by the identification means.

[0113] (Appendix C4) The behavior recognition program according to Appendix C2, wherein the identification means updates the features of the subject object based on a similarity between the features of the text and the features of the subject object to generate a third feature, and identifies an object related to the subject object based on a similarity between the third feature and each feature of an object other than the subject object among the plurality of objects.

[0114] (Appendix C5) The behavior recognition program according to any one of Appendices C1 to C4, further causing the computer to function as an output means for outputting at least one of information indicating an object identified by the identification means and information indicating a recognition result by the recognition means.

[0115] (Supplementary Note C6) The behavior recognition program according to Supplementary Note C1, wherein the identification means identifies at least one of an object whose at least a partial area overlaps with an area of ​​the subject in the image, and an object whose distance from the subject in the image is equal to or less than a threshold.

[0116] (Supplementary Note C7) The behavior recognition program according to Supplementary Note C1, wherein the identification means identifies an object related to the subject based on information obtained by inputting feature amounts of the plurality of objects into a trained model generated by machine learning.

[0117] (Supplementary Note C8) The behavior recognition program according to Supplementary Note C5, wherein the output means outputs information for supporting decision-making regarding the behavior based on a recognition result of the behavior.

[0118] [Appendix D] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0119] (Supplementary Note D1) A behavior recognition device including at least one processor, the at least one processor executing: an acquisition process for acquiring image data and text describing behavior; an extraction process for extracting, from an image represented by the image data, a plurality of objects including an object that is a subject of the behavior; an identification process for identifying an object related to the subject object from the plurality of objects extracted in the extraction process; a generation process for generating second features using features of the subject object and features of the object identified in the identification process; and a recognition process for recognizing the behavior of the subject object based on the similarity between the features of the text and the second features. The behavior recognition device may further include a memory. The memory may also store a program for causing the at least one processor to execute each of the processes.

[0120] (Supplementary Note D2) The behavior recognition device according to Supplementary Note D1, wherein in the identification process, the at least one processor identifies an object related to the subject by using feature amounts of each of the plurality of objects extracted in the extraction process and feature amounts of the text.

[0121] (Appendix D3) The behavior recognition device described in Appendix D1 or D2, wherein the at least one processor further executes an integration process to generate an integrated feature by integrating attribute information of the object with a feature of the object, and in the generation process, the at least one processor generates the second feature using the integrated feature of the subject object and the integrated feature of an object identified in the identification process.

[0122] (Appendix D4) The behavior recognition device described in Appendix D2, wherein in the identification process, the at least one processor: updates the features of the subject object based on a similarity between the features of the text and the features of the subject object to generate a third feature; and identifies an object related to the subject object based on a similarity between the third feature and each feature of an object other than the subject object among the plurality of objects.

[0123] (Supplementary Note D5) The behavior recognition device according to any one of Supplementary Notes D1 to D4, wherein the at least one processor further executes an output process of outputting at least one of information indicating an object identified in the identification process and information indicating a recognition result of the recognition process.

[0124] (Supplementary Note D6) The behavior recognition device according to Supplementary Note D1, wherein in the identification process, the at least one processor identifies at least one of an object whose at least a partial area overlaps with an area of ​​the subject in the image, and an object whose distance from the subject in the image is equal to or less than a threshold.

[0125] (Supplementary Note D7) The behavior recognition device according to Supplementary Note D1, wherein in the identification process, the at least one processor identifies an object related to the subject based on information obtained by inputting feature amounts of the plurality of objects into a trained model generated by machine learning.

[0126] (Supplementary Note D8) The behavior recognition device according to Supplementary Note D5, wherein in the output process, the at least one processor outputs information for supporting decision-making regarding the behavior based on a recognition result of the behavior.

[0127] [Appendix E] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0128] (Appendix E1) A non-transient recording medium having recorded thereon a behavior recognition program that causes a computer to function as a behavior recognition device, the behavior recognition program causing the computer to execute: an acquisition process that acquires image data and text describing behavior; an extraction process that extracts multiple objects including an object that is the subject of the behavior from an image represented by the image data; an identification process that identifies an object related to the subject object from the multiple objects extracted in the extraction process; a generation process that generates second features using features of the subject object and features of the object identified in the identification process; and a recognition process that recognizes the behavior of the subject object based on the similarity between the features of the text and the second features.

[0129] REFERENCE SIGNS LIST 1 Behavior recognition device 1A Information processing device 11A Object extraction unit 12A Language feature extraction unit 13A Person tracking unit 14A Video feature extraction unit 15A Relationship determination unit 16A Attribute information integration unit 17A Similarity calculation unit 18A Behavior classification output unit

Claims

1. An action recognition device comprising: an acquisition means for acquiring image data and text describing an action; an extraction means for extracting a plurality of objects including an object that is the subject of an action from an image represented by the image data; an identification means for identifying an object related to the subject object from the plurality of objects extracted by the extraction means; a generation means for generating second features using features of the subject object and features of the object identified by the identification means; and a recognition means for recognizing the action of the subject object based on the similarity between the features of the text and the second features.

2. The behavior recognition device according to claim 1, wherein the identification means identifies an object related to the subject by using the feature amounts of each of the plurality of objects extracted by the extraction means and the feature amounts of the text.

3. The behavior recognition device according to claim 1 or 2, further comprising an integration means for generating an integrated feature by integrating attribute information of the object with the feature of the object, wherein the generation means generates the second feature using the integrated feature of the subject object and the integrated feature of the object identified by the identification means.

4. The behavior recognition device according to claim 2, wherein the identification means updates the features of the subject object based on the similarity between the features of the text and the features of the subject object to generate a third feature, and identifies an object related to the subject object based on the similarity between the third feature and each feature of an object other than the subject object among the plurality of objects.

5. The behavior recognition device according to any one of claims 1 to 4, further comprising an output means for outputting at least one of information indicating the object identified by the identification means and information indicating the recognition result by the recognition means.

6. The behavior recognition device according to claim 1, wherein the identification means identifies at least one of an object whose area in the image overlaps with the area of ​​the subject, and an object whose distance from the subject in the image is equal to or less than a threshold.

7. The behavior recognition device according to claim 1, wherein the identification means identifies objects related to the subject based on information obtained by inputting feature quantities of the plurality of objects into a trained model generated by machine learning.

8. The behavior recognition device according to claim 5, wherein the output means outputs information to support decision-making regarding the behavior based on the behavior recognition result.

9. A behavior recognition method including: an acquisition process in which at least one processor acquires image data and text describing behavior; an extraction process in which the at least one processor extracts multiple objects including an object that is the subject of the behavior from an image represented by the image data; an identification process in which the at least one processor identifies an object related to the subject object from the multiple objects extracted in the extraction process; a generation process in which the at least one processor generates second features using features of the subject object and features of the object identified in the identification process; and a recognition process in which the at least one processor recognizes the behavior of the subject object based on the similarity between the features of the text and the second features.

10. A program for causing a computer to function as a behavior recognition device, the behavior recognition program causing the computer to function as: an acquisition means for acquiring image data and text describing behavior; an extraction means for extracting multiple objects including an object that is the subject of behavior from an image represented by the image data; an identification means for identifying an object related to the subject object from the multiple objects extracted by the extraction means; a generation means for generating second features using features of the subject object and features of the object identified by the identification means; and a recognition means for recognizing the behavior of the subject object based on the similarity between the features of the text and the second features.

Images