Behavior inference device, behavior inference method, and program

Abstract

This behavior inference device is provided with: an image acquisition unit that acquires an image; a first generation unit that generates posture information indicating the posture of a subject included in the image; a second generation means that generates a feature map representing a feature around the subject in the image; a filter unit that generates a second feature map by filtering the feature map on the basis of the posture information; and a behavior inference unit that infers the behavior of the subject on the basis of the posture information and the second feature map, and that outputs the inference result.

Description

Behavior estimation device, behavior estimation method, and program

[0001] The present disclosure relates to a behavior estimation device, a behavior estimation method, and a program.

[0002] Techniques for recognizing the behavior of people and the like from video are known. Patent Document 1, for example, describes an example of a technique for recognizing the behavior of people from video, which performs behavior recognition based on information about a person and objects surrounding the person. In Patent Document 1, peripheral features indicating the shape, position, or type of peripheral objects are vectors with a predetermined number of dimensions obtained by fully combining multiple intermediate calculation result data obtained by convolution processing. The peripheral feature filter unit described in Patent Document 1 filters the peripheral features by multiplying the peripheral features, which are vectors with a predetermined number of dimensions, by importance, which is a vector with the same number of dimensions as the peripheral features. Furthermore, the behavior discrimination unit described in Patent Document 1 estimates the behavior class of a person appearing in an image based on posture features and the peripheral features filtered by the peripheral feature filter unit.

[0003] International Publication No. 2018 / 163555

[0004] In the technology described in Patent Document 1, feature quantities are recombined through full coupling, and therefore spatial information of the image is missing from the vector obtained by full coupling. As a result, information on objects highly relevant to the recognition target and information on objects less relevant to the target are included in the same dimensional components. Even if such feature quantities are filtered in the dimensional direction of the vector, information on objects highly relevant to the target cannot be highlighted. As a result, information on objects less relevant to the target of behavior recognition is used in the behavior recognition process in the same way as information on objects more relevant, which poses a problem of not being able to improve the accuracy of behavior recognition.

[0005] The present disclosure has been made in view of the above-mentioned problems, and an exemplary purpose thereof is to provide a technology that can estimate the behavior of a target included in an image with higher accuracy.

[0006] A behavior estimation device according to an exemplary aspect of the present disclosure includes: image acquisition means for acquiring an image; first generation means for generating posture information representing the posture of a target included in the image; second generation means for generating a feature map representing features of the periphery of the target in the image; filter means for generating a second feature map by filtering the feature map based on the posture information; and behavior estimation means for estimating the behavior of the target based on the posture information and the second feature map, and outputting an estimation result.

[0007] An activity estimation method according to an exemplary aspect of the present disclosure includes, by at least one processor, acquiring an image, generating posture information representing a posture of an object included in the image, generating a feature map representing peripheral features of the object in the image, generating a second feature map by filtering the feature map based on the posture information, estimating an activity of the object based on the posture information and the second feature map, and outputting an estimation result.

[0008] A program according to an exemplary aspect of the present disclosure causes a computer to execute an image acquisition process to acquire an image, a first generation process to generate posture information representing the posture of a target included in the image, a second generation process to generate a feature map representing the features of the area around the target in the image, a filtering process to generate a second feature map by filtering the feature map based on the posture information, and a behavior estimation process to estimate the behavior of the target based on the posture information and the second feature map and output the estimation result.

[0009] According to an exemplary aspect of the present disclosure, an exemplary effect is achieved in that a technology can be provided that recognizes the behavior of a target included in an image with higher accuracy.

[0010] FIG. 1 is a block diagram showing a configuration of a behavior estimation device according to the present disclosure. FIG. 2 is a flow diagram showing a flow of a behavior estimation method according to the present disclosure. FIG. 3 is a diagram showing an overview of the behavior estimation method according to the present disclosure. FIG. 4 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 5 is a block diagram showing an example of a functional configuration and processing flow of an information processing device according to the present disclosure. FIG. 6 is a diagram showing an example of posture information detected by a posture detection unit according to the present disclosure. FIG. 7 is a block diagram showing an example of a functional configuration and processing flow of a filter unit according to the present disclosure. FIG. 8 is a diagram showing an outline of the content of processing performed by a filter unit according to the present disclosure. FIG. 9 is a diagram showing a specific example of filtering processing of a filter unit according to the present disclosure. FIG. 10 is a flow diagram showing an example of the flow of a behavior estimation method performed by an information processing device according to the present disclosure. FIG. 11 is a diagram showing an example in which an information processing device according to the present disclosure is applied to the medical / healthcare field. FIG. 12 is a block diagram showing an example of a configuration and processing flow of a filter unit according to the present disclosure. FIG. 13 is a block diagram showing an example of a configuration and processing flow of a filter unit according to the present disclosure.

[0011] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, the effects mentioned in the exemplary embodiments shown below are examples of effects expected in the exemplary embodiments, and do not define the scope of the present invention. In other words, embodiments that do not exhibit the effects mentioned in the exemplary embodiments shown below may also be included in the scope of the present invention.

[0012] [First Exemplary Embodiment] A first exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is a basic form for each of the exemplary embodiments described below. Note that the scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technical means shown in the drawings referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0013] (Configuration of Behavior Inference Device) The configuration of the behavior inference device 1 will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the behavior inference device 1. As shown in Fig. 1, the behavior inference device 1 includes an image acquisition unit 11, a first generation unit 12, a second generation unit 13, a filter unit 14, and a behavior inference unit 15.

[0014] The image acquisition unit 11 acquires an image. The first generation unit 12 generates posture information representing the posture of an object included in the image. The second generation unit 13 generates a feature map representing the features of the area around the object in the image. The filter unit 14 generates a second feature map by filtering the feature map based on the posture information. The behavior estimation unit 15 estimates the behavior of the object based on the posture information and the second feature map, and outputs the estimation result.

[0015] (Effects of the Behavior Estimation Device) As described above, the behavior estimation device 1 employs a configuration including an image acquisition unit 11 that acquires an image, a first generation unit 12 that generates posture information representing the posture of a target included in the image, a second generation unit 13 that generates a feature map representing features of the periphery of the target in the image, a filter unit 14 that generates a second feature map by filtering the feature map based on the posture information, and a behavior estimation unit 15 that estimates the behavior of the target based on the posture information and the second feature map and outputs the estimation result. Therefore, the behavior estimation device 1 has the effect of being able to estimate the behavior of a target included in an image with higher accuracy.

[0016] (Flow of Behavior Estimation Method) The flow of the behavior estimation method S1 will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of the behavior estimation method S1. As shown in Fig. 2, the behavior estimation method S1 includes an image acquisition process S11, a first generation process S12, a second generation process S13, a filter process S14, and a behavior estimation process S15.

[0017] In an image acquisition process S11, at least one processor acquires an image. In a first generation process S12, at least one processor generates posture information representing the posture of an object included in the image. In a second generation process S13, at least one processor generates a feature map representing the features of the periphery of the object in the image. In a filtering process S14, at least one processor generates a second feature map by filtering the feature map based on the posture information. In a behavior estimation process S15, at least one processor estimates the behavior of the object based on the posture information and the second feature map and outputs the estimation result.

[0018] (Effects of the Behavior Estimation Method) As described above, the behavior estimation method S1 employs a configuration including the following: at least one processor acquires an image, generates posture information representing the posture of a target included in the image, generates a feature map representing features of the area around the target in the image, generates a second feature map by filtering the feature map based on the posture information, estimates the behavior of the target based on the posture information and the second feature map, and outputs the estimation result. Therefore, the behavior estimation method S1 has the effect of being able to estimate the behavior of a target included in the image with higher accuracy.

[0019] Second Exemplary Embodiment A second exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be denoted by the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs. Furthermore, each technical means shown in each drawing referenced to describe this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs.

[0020] (Outline of Behavior Estimation Process) FIG. 3 is a diagram illustrating an overview of the behavior estimation method according to the present disclosure. The behavior estimation method is a method for estimating the behavior of an object included in an image. In the present disclosure, an object refers to the subject of the behavior, and is, for example, a person such as a worker, a patient, or hospital staff (including medical professionals), an animal such as a pet, or a moving object such as an autonomous mobile robot. In addition, in the present disclosure, the behavior of the object refers to the movement, action, or behavior of the object, or an event resulting from the movement, action, or behavior of the object. Examples of the behavior of the object include, but are not limited to, actions related to work content such as "heavy machinery excavation," "rest," "compaction," "guiding work," "approaching heavy machinery," or "material transportation." In addition, in the present disclosure, the behavior of the object may be, for example, an event or action that occurs in daily life, such as "collision," "fall," or "waking up."

[0021] In the example of Figure 3, the imaging device 200 captures an image of a workplace where multiple workers A, B, and C are working, and the information processing device 1A recognizes the actions of each worker in chronological order from the image Img1 obtained by the imaging. If the information processing device 1A recognizes, for example, a dangerous action by a worker (e.g., "approaching heavy machinery"), the information processing device 1A notifies a workplace manager or the like by outputting an alert or the like. This can, for example, support the decision-making of the workplace manager or the like. The behavior estimation method according to the present disclosure can be applied to various industries, such as the construction industry, civil engineering industry, manufacturing industry, and medical service industry.

[0022] 4 is a block diagram showing the configuration of an information processing device 1A. The information processing device 1A is a device that estimates the behavior of a person included in an image, and is an example of a behavior estimation device according to the present disclosure. The information processing device 1A includes a control unit 10A, a storage unit 20A, a communication unit 30A, and an input / output unit 40A.

[0023] The communication unit 30A communicates with devices external to the information processing device 1A via a communication line. While the specific configuration of the communication line does not limit the present exemplary embodiment, examples of the communication line include a wireless local area network (LAN), a wired LAN, a wide area network (WAN), a public line network, a mobile data communication network, or a combination thereof. The communication unit 30A transmits data supplied from the control unit 10A to other devices, and supplies data received from other devices to the control unit 10A.

[0024] Input / output devices such as a keyboard, a mouse, a display, a printer, and a touch panel are connected to the input / output unit 40A. The input / output unit 40A receives various types of information input to the information processing device 1A from the connected input devices. The input / output unit 40A also outputs various types of information to connected output devices under the control of the control unit 10A. An example of the input / output unit 40A is an interface such as a USB (Universal Serial Bus).

[0025] The control unit 10A controls each unit of the information processing device 1A in an integrated manner. In addition to the image acquisition unit 11, filter unit 14, and behavior estimation unit 15 provided in the behavior estimation device 1, the control unit 10A also includes a person area detection unit 16, an image feature extraction unit 17, a peripheral feature extraction unit 18, a posture detection unit 19, and a learning unit 21. The image feature extraction unit 17 is an example of an extraction means according to the present disclosure. The peripheral feature extraction unit 18 is an example of a second generation means according to the present disclosure. The posture detection unit 19 is an example of a first generation means according to the present disclosure.

[0026] 5 is a block diagram showing an example of the functional configuration and processing flow of the information processing device 1A. Note that the unidirectional arrows in Fig. 5 simply indicate the direction of the flow of a certain signal (data), and do not exclude bidirectionality.

[0027] (Image Acquisition Unit) The image acquisition unit 11 acquires image data D1. The image data D1 is, for example, image data representing an image captured by a photographing device. The image data D1 may be image data representing a two-dimensional image, or may be image data including depth information captured by a three-dimensional camera. The image acquisition unit 11 may acquire the image data D1 input via the input / output unit 40A, or may acquire the image data D1 from a storage location specified by the user of the information processing device 1A (which may be within the storage device of the information processing device 1A or may be a storage device external to the information processing device 1A). The image acquisition unit 11 may also acquire the image data D1 by receiving the image data D1 from another device via the communication unit 30A.

[0028] (Person Region Detection Unit) The person region detection unit 16 detects a person region from the image represented by the image data D1 and outputs region data D2 indicating the detected region. Examples of methods by which the person region detection unit 16 detects a person region include, but are not limited to, region detection methods used in object detection, such as R-CNN (Regions with CNN features), Fast R-CNN, Faster R-CNN, SSD (Single Shot Multibox Detector), and YOLO (You Only Look Once). As another example, the person region detection unit 16 may detect a person region based on the difference between a reference image captured in a workplace or the like without any people present and the image represented by the acquired image data D1.

[0029] When the image represented by the image data D1 includes multiple people, the person area detection unit 16 generates area data D2 for each of the multiple people. When the image data D1 is a moving image, the person area detection unit 16 generates area data D2 continuously for each frame or at intervals of multiple frames.

[0030] (Image Feature Extraction Unit) The image feature extraction unit 17 generates image feature data D3 representing image features of a person included in an image based on the image data D1 and the area data D2. The image feature data D3 is, for example, １ Dimension (d １ is an integer of 1 or more), but is not limited to this.

[0031] As an example, the image feature extraction unit 17 generates image feature data D3 using a trained model generated by machine learning. In this case, the trained model is a model that receives image data as input and outputs image feature data D3, and is generated by machine learning using training data including multiple pairs of image data and labels that represent the features of objects included in the image represented by the image data, for example. The machine learning method for the trained model may be, for example, linear regression, nonlinear regression, decision tree analysis, support vector machine, or neural network. For example, a convolutional neural network or a recurrent neural network may be used as the neural network. However, the machine learning method for the trained model is not limited to the above examples, and other methods may also be used.

[0032] (Peripheral Feature Extraction Unit) The peripheral feature extraction unit 18 generates a feature map M1 that represents image features around a person included in the image represented by the image data D1, based on the image data D1 and the region data D2. As an example, the peripheral feature extraction unit 18 extracts image features from an area that is obtained by enlarging (for example, enlarging by 1.4 times) the area detected by the person region detection unit 16, and generates the feature map M1.

[0033] The feature map M1 is data including a plurality of elements in the direction in which the pixels of the image data D1 are arranged. For example, the feature map M1 is an intermediate calculation result obtained by the convolution process of a convolutional neural network before being fully connected in a fully connected layer, and is, for example, h×w×d ２ (h, w, d ２ is an integer equal to or greater than 1, and is a tensor such that h+w>2. ２If the feature map M1 is a tensor of h × w (i.e., multiple) d ２ In this way, the feature map M1 is data that has multiple elements in the spatial direction of the image (the direction in which pixels are arranged), so it can also be said that the feature map M1 is data that holds spatial information about the image.

[0034] As an example, the peripheral feature extraction unit 18 generates the feature map M1 using a trained model generated by machine learning. In this case, the trained model is a model that receives image data as input and outputs the feature map M1, and is a model trained by machine learning using training data that includes multiple pairs of image data and labels that represent features of objects included in the image represented by the image data. Examples of machine learning methods for the trained model include, but are not limited to, neural networks such as convolutional neural networks and recurrent neural networks.

[0035] (Posture detection and extraction unit) The posture detection unit 19 generates posture information D4 representing the posture of the person included in the region data D2 based on the image data D1 and the region data D2. The posture information D4 includes, for example, coordinates (x coordinates, y coordinates) of the person's skeletal points (joint positions) and score values ​​indicating their reliability. However, the information included in the posture information D4 is not limited to the above example.

[0036] As an example, the posture detection unit 19 generates posture information D4 using a trained model generated by machine learning. In this case, the trained model is a model that receives image data as input and outputs posture information D4, and is generated by machine learning using training data including multiple pairs of image data and posture information, for example. The machine learning method for the trained model may be, for example, linear regression, nonlinear regression, decision tree analysis, support vector machine, or neural network. For example, a convolutional neural network or a recurrent neural network may be used as the neural network. However, the machine learning method for the trained model is not limited to the above examples, and other methods may also be used. For example, the posture detection unit 19 may detect a person's skeletal points by analyzing image data D1 including depth information captured by a three-dimensional camera.

[0037] Fig. 6 is a diagram showing an example of posture information D4 detected by posture detection unit 19. In the example of Fig. 6, posture detection unit 19 generates posture information D4 of a person included in person region A2 represented by region data D2. In Fig. 6, surrounding region A21 is an area obtained by enlarging person region A2 detected by person region detection unit 16 by a predetermined magnification.

[0038] (Filter Unit) The filter unit 14 generates a second feature map (see FIG. 7) using the posture information D4 and the feature map M1. The filter unit 14 also generates feature data D5 representing image features of the peripheral area of ​​the person using the second feature map M2.

[0039] The second feature map M2 is a feature map obtained by filtering the feature map M1 or a new feature map M1_3 (see FIG. 7) generated using the feature map M1. Like the feature map M1, the second feature map M2 is data including multiple elements in the spatial direction of the image data D1. For example, the second feature map M2 is a feature map of h×w×d ４ (d ４ is a tensor of (is an integer greater than or equal to 1).

[0040] The feature data D5 is, for example, ４The filter unit 14 performs processes such as pooling and full connection processing on the second feature map M2 to generate feature data D5. Details of the processes performed by the filter unit 14 will be described later with reference to other drawings.

[0041] (Behavior Estimation Unit) The behavior estimation unit 15 uses the image feature data D3, posture information D4, and feature data D5 to estimate the behavior of a person included in the area indicated by the area data D2 and outputs an estimation result. The estimation result includes, for example, information indicating the type of behavior of the person. Examples of behavior types include, but are not limited to, "heavy machinery excavation," "rest," "compaction," "guiding work," "approaching heavy machinery (danger)," "material transportation," "wake-up," "collision," and "fall." Because the feature data D5 is data obtained from the second feature map M2, the behavior estimation unit 15 can also be said to estimate the target's behavior based on the posture information D4, the second feature map M2, and the image features extracted by the image feature extraction unit 17.

[0042] The behavior estimation unit 15 estimates behavior using a trained model generated by machine learning, for example. In this case, the trained model is, for example, a model that receives image feature data D3, posture information D4, and feature data D5 as input and outputs information indicating the behavior estimation result. The output of the trained model may include information identifying a behavior class, or may include information indicating the confidence level for each behavior class. The trained model is generated by machine learning using training data including multiple pairs of image feature data D3, posture information D4, and feature data D5, and class labels indicating the type of behavior. As a machine learning method for the trained model, for example, a neural network technique such as a convolutional neural network or a recurrent neural network may be used. However, the machine learning method for the trained model is not limited to the above example, and other techniques may be used.

[0043] The behavior inferrer 15 may output information indicating the inference result to an output device (such as a display, printer, or speaker) connected via the input / output unit 40A, or may output the information by writing it to a storage destination (which may be in a storage device of the information processing device 1A or a storage device external to the information processing device 1A) designated by the user of the information processing device 1A. Furthermore, the behavior inferrer 15 may output the information by transmitting it to another device via the communication unit 30A.

[0044] (Learning Unit) The learning unit 21 performs a learning process using the teacher data D6 so as to optimize model parameters of at least one of the image acquisition unit 11, the person area detection unit 16, the image feature extraction unit 17, the peripheral feature extraction unit 18, the posture detection unit 19, the filter unit 14, and the behavior estimation unit 15. The model parameters include, but are not limited to, weight coefficients of a neural network, as an example.

[0045] The storage unit 20A stores image data D1, region data D2, image feature data D3, feature map M1, posture information D4, second feature map M2, feature data D5, model parameters MP, and teacher data D6. The model parameters MP include model parameters of a trained model used by at least one of the image acquisition unit 11, person region detection unit 16, image feature extraction unit 17, peripheral feature extraction unit 18, posture detection unit 19, filter unit 14, and behavior estimation unit 15.

[0046] (Configuration Example of Filter Unit) Fig. 7 is a block diagram showing an example of the functional configuration and processing flow of the filter unit 14. Fig. 8 is a diagram schematically showing the content of the processing performed by the filter unit 14. However, the unidirectional arrows in Figs. 7 and 8 simply indicate the direction of flow of a certain signal (data) and do not exclude bidirectionality. In the example of Fig. 7, the filter unit 14 includes a first preprocessing unit 141, a second preprocessing unit 142, a combining / adding unit 143, a feature map generating unit 144, a third preprocessing unit 145, a gaze map generating unit 146, a multiplying unit 147, and a feature amount generating unit 148.

[0047] (First Pre-Processing Unit) The first pre-processing unit 141 generates image feature data D3_1 having the same size in the spatial direction as the feature map M1 from the image feature data D3 (P141 in FIG. 8). More specifically, the first pre-processing unit 141 generates image feature data D3_1 having the same size in the spatial direction as the feature map M1 from the image feature data D3 (P141 in FIG. 8). １ The image feature data D3, which is a dimensional vector, is reduced in dimension using a neural network such as MLP (multilayer perceptron) to obtain d ３ Dimension (d ３ is an integer of 1 or more, and d １ >d ３ ) vector. ３ By copying the vector of dimension h×w into a grid of dimensions h×w×d ３ A tensor (image feature data D3_1) is generated.

[0048] (Second Pre-Processing Unit) The second pre-processing unit 142 performs a process P142 for adjusting the number of dimensions of the feature map M1. ２ The feature map M1, which is a tensor of h × w × d, is reduced in dimension using a neural network such as MLP. ３ is converted into a feature map M1_1, which is a tensor of

[0049] (Combining and Adding Unit) The combining and adding unit 143 combines or adds the image feature data D3_1 output by the first pre-processing unit 141 and the feature map M1_1 output by the second pre-processing unit 142, and outputs the feature map M1_2 (P143 in FIG. 8). At this time, the number of dimensions of the tensor generated by the first pre-processing unit 141 and the number of dimensions of the tensor generated by the second pre-processing unit 142 may be the same or different. Note that if the numbers of dimensions are different, the combining and adding unit 143 outputs the feature map M1_2 by combining the image feature data D3_1 and the feature map M1_1.

[0050] (Feature Map Generator) The feature map generator 144 performs processing such as adjusting the number of dimensions of the feature map M1_2 output by the combining and adding unit 143 to generate a feature map M1_3 (P144 in FIG. 8). For example, the feature map generator 144 generates a feature map M1_3 by using a neural network such as a CNN to generate a feature map M1_2.４ In other words, the feature map generating unit 144 generates a new feature map M1_3 using the image feature data D3 extracted by the image feature extracting unit 17 and the feature map M1 generated by the peripheral feature extracting unit 18.

[0051] (Third Pre-Processing Unit) The third pre-processing unit 145 executes a process P145 for generating, from the posture information D4, posture data D4_1 having a size in a spatial direction similar to that of the feature map M1. More specifically, the third pre-processing unit 145 performs a process P145 for generating posture data D4_1 from the posture information D4 using a neural network or positional encoding, for example. ５ Dimension (d ５ is an integer greater than or equal to 1).

[0052] (Gaze Map Generator) The gaze map generator 146 executes a process P146 to generate a gaze map Mtx having the same size in the spatial direction as the feature map M1 from the posture data D4_1. Since the posture data D4_1 is data generated from the posture information D4, it can also be said that the gaze map generator 146 generates the gaze map Mtx using the posture information D4.

[0053] The gaze map Mtx is, for example, an h×w matrix in which the value of each component falls within the range of 0 to 1. The gaze map generation unit 146 generates the gaze map d output by the third preprocessing unit 145 as an example. ５ The x-dimensional vector is reshaped into a lattice to generate an h×w matrix (P146-1 in FIG. 8). The gaze map generator 146 also generates a gaze map Mtx by converting the values ​​of each element of the h×w matrix obtained by reshaping using a predetermined function such as a Softmax function or a sigmoid function (P146-2 in FIG. 8).

[0054] As described above, the gaze map Mtx is a matrix generated from the posture data D4_1 that represents the posture characteristics of the person included in the region indicated by the region data D2, and is a matrix that includes multiple components in the spatial direction of the image. Therefore, by performing filtering using the gaze map Mtx, it is possible to highlight information in the surrounding region that corresponds to the posture characteristics of the person.

[0055] (Multiplication Unit) The multiplication unit 147 outputs a second feature map M2 obtained by multiplying the feature map M1_3 by the gaze map Mtx (P148 in FIG. 8). In other words, the multiplication unit 147 filters the feature map M1_3 by multiplying the feature map M1_3 by the gaze map Mtx. When the feature map M1_3 is h×w×d ４ When the attention map Mtx is an h×w×1 matrix, the second feature map M2 is obtained by filtering the feature map M1_3 in the row and column directions of the attention map Mtx. ４ In other words, the multiplication unit 147 generates the second feature map M2 by filtering the new feature map M1_3 based on the posture information D4.

[0056] (Feature Generator) The feature generator 148 generates feature data D5 from the second feature map M2. As an example, the feature generator 148 performs processing such as pooling and full connection processing on the second feature map M2 to generate the feature data D5.

[0057] 9 is a diagram showing a specific example of the filtering process of the filter unit 14. In the example of FIG. 9, a person area A31 is a person area detected by the person area detection unit 16 and includes a worker O31. A surrounding area A32 is a surrounding area that is a predetermined multiple (e.g., 1.4 times) the size of the person area A31 and includes objects O32 and O33. The object O32 is a cart used by the worker O31 to transport materials and is an object related to the actions of the worker O31. On the other hand, the object O33 is a piece of equipment reflected behind the worker O31 and is an object unrelated to the work being performed by the worker O31.

[0058] As described above, the filter unit 14 filters the feature map M1, which holds spatial information about the image, based on the posture information D4. Therefore, the filter unit 14 can filter out or reduce information about an object (object O33 in FIG. 9 ) located in a peripheral area that is less relevant to the posture of the worker O31, or can highlight information about an object (object O32 in FIG. 9 ) included in a peripheral area (area A33 in FIG. 9 ) that is more relevant to the posture of the worker O31.

[0059] 10 is a flow diagram showing an example of the flow of the behavior estimation method performed by the information processing device 1 A. Note that some steps may be executed in parallel or in a different order.

[0060] In S101, the image acquisition unit 11 acquires image data D1 and supplies it to the person area detection unit 16. In S102, the person area detection unit 16 detects a person area from the image represented by the image data D1 and generates area data D2 indicating the detected area. The person area detection unit 16 supplies the image data and area data D2 to the image feature extraction unit 17, the peripheral feature extraction unit 18, and the posture detection unit 19.

[0061] In S103, the image feature extraction unit 17 uses the image data D1 and the region data D2 to generate image feature data D3 representing the features of a person included in the region indicated by the region data D2. The image feature extraction unit 17 supplies the extracted image feature data to the behavior estimation unit 15 and the filter unit 14.

[0062] In S104, the posture detection unit 19 generates posture information D4 representing the posture of the person included in the region data D2 based on the image data D1 and the region data D2. The posture detection unit 19 supplies the generated posture information D4 to the filter unit 14.

[0063] In S105, the surrounding feature extraction unit 18 generates a feature map M1 representing the features of the surroundings of the person included in the image represented by the image data D1, based on the image data D1 and the region data D2. The surrounding feature extraction unit 18 supplies the generated feature map M1 to the filter unit 14.

[0064] In S106, the filter unit 14 generates a feature map M1_3 by adjusting the number of dimensions of the feature map M1_2 obtained by combining or adding the feature map M1 and the image feature data D3, and filters the generated feature map M1_3 using the gaze map Mtx obtained from the posture information D4. The filter unit 14 also performs processes such as pooling and full connection on the second feature map M2 obtained by filtering to generate feature data D5 and supplies the feature data D5 to the behavior estimation unit 15.

[0065] In S107, the behavior estimation unit 15 estimates the person's behavior using the image feature data D3, the posture information D4, and the feature data D5, and outputs the estimation result. The behavior estimation unit 15 may output information indicating the type of the person's behavior, or may output an alert if the estimated behavior is dangerous. In this case, the behavior estimation unit 15 may determine whether the estimated behavior is dangerous by, for example, referring to a table in which the type of behavior is associated with information indicating whether the behavior is dangerous. Furthermore, the behavior estimation unit 15 may output, for example, an image (such as the image shown in FIG. 9 ) indicating which peripheral areas were emphasized in the filtering process.

[0066] In S108, the behavior estimation unit 15 determines whether to end the process. For example, if image data to be processed remains, the behavior estimation unit 15 determines not to end the process. If the process is not to end (NO in step S108), the behavior estimation unit 15 returns to the process in step S101. On the other hand, if the process is to end (YES in step S108), the behavior estimation unit 15 ends the process.

[0067] (Effects of Information Processing Device) As described above, in the information processing device 1A, the feature map M1_3 having spatial information of the image is filtered using the gaze map Mtx generated from the posture information D4. The second feature map M2 obtained by filtering is also data having spatial features of the image. In other words, the information processing device 1A performs filtering while preserving the spatial information of the image. By filtering while preserving the spatial information of the image in this way, it is possible to highlight information about objects that are related to the person's behavior among objects included in the background. By estimating the person's behavior using the second feature map M2 obtained in this way, it is possible to estimate the person's behavior with higher accuracy.

[0068] The information processing device 1A also includes an image feature extraction unit 17 that extracts image features of a target from an image, and a filter unit 14 that generates a new feature map M1_3 using image feature data D3 extracted by the image feature extraction unit 17 and the feature map M1 generated by the peripheral feature extraction unit 18, and then generates a second feature map M2 by filtering the generated new feature map M1_3 based on posture information D4. Therefore, the information processing device 1A can generate a second feature map M2 that takes into account not only peripheral image features but also image features of the person, thereby enabling more accurate estimation of the behavior of the person included in the image.

[0069] Furthermore, in the information processing device 1A, the behavior estimation unit 15 is configured to estimate the behavior of a person based on the posture information D4, the second feature map M2, and the image feature data D3 extracted by the image feature extraction unit 17. Therefore, according to the information processing device 1A, by using the image feature data D3 in addition to the posture information D4 and the feature data D5 for behavior estimation, the behavior of a person can be estimated with higher accuracy.

[0070] Furthermore, in the information processing device 1A, the filter unit 14 generates a gaze map Mtx from the posture information D4, and filters the feature map M1_3 by multiplying the feature map M1_3 by the gaze map Mtx. Therefore, the information processing device 1A can filter the feature map M1_3 while preserving the spatial information of the image.

[0071] (Use Case) FIG. 11 is a diagram illustrating an example in which the information processing device 1A is applied to the medical / healthcare field. In this use case, the information processing device 1A estimates the behavior of a subject in a medical facility such as a hospital and outputs an alert based on the estimation result. In FIG. 11 , the system according to this application example is a video surveillance system aimed at ensuring the safety of people moving around within a hospital 100. In this system, the information processing device 1A acquires RGB images from an image capture device 200 installed in the hospital. Furthermore, the information processing device 1A detects dangerous behavior of a person 300, such as a patient, hospital staff (including medical professionals), or visitor, from the video acquired from the image capture device 200 installed in the hospital, and outputs an alert. Furthermore, in this system, the information processing device 1A provides medical professionals and others with appropriate countermeasures based on the detected dangerous behavior.

[0072] The specific flow of this use case will be described below, in which a series of steps will be described in order: (1) the inside of the hospital 100 is photographed by the imaging device 200, (2) the video captured by the imaging device 200 is acquired and risk estimation is performed, and (3) hospital staff checks for abnormalities and takes action.

[0073] ((1) Photographing the Inside of the Hospital with an Imaging Device) First, the inside of the hospital 100 is photographed with the imaging device 200. By installing the imaging device 200 in each room, corridor, elevator, etc. in the hospital 100, it becomes possible to acquire images of the inside of the hospital 100.

[0074] ((2) Obtaining video captured by imaging device and performing risk estimation) The information processing device 1A obtains video captured by the imaging device 200, estimates the behavior of the person 300 using the behavior estimation method described above, and estimates the risk associated with the behavior of the person 300. At this time, if the information processing device 1A determines that the behavior of the person 300 is risky, it transmits an alert to hospital staff or the like. Specifically, the information processing device 1A may, for example, transmit the alert to a terminal owned by the hospital staff via the communication unit 30A, or may output the alert to an output device (for example, a display, a speaker, etc.) connected to the input / output unit 40A.

[0075] In this use case, targets of behavior estimation include, for example, patients, medical staff (including medical professionals), visitors, etc. Furthermore, objects present around the targets include, for example, medical equipment, intravenous drips, wagons, wheelchairs, etc.

[0076] Examples of cases in which a risk is determined include when a patient is about to touch a specific medical instrument, when a patient with an IV drip is about to collide with a visitor pushing a wheelchair, or when a person using a white cane is about to collide with a patient with an IV drip. The information processing device 1A determines whether the estimated person's behavior is risky by, for example, referring to a table that stores a correspondence between the person's behavior and information indicating whether the behavior is risky. However, the method for determining whether a risk exists is not limited to the above example. The information processing device 1A may determine whether a risk exists by, for example, inputting information such as the estimated person's behavior into a trained model that has been trained by machine learning to input the person's behavior and environmental conditions, etc., and output information indicating whether a risk exists.

[0077] The information processing device 1A may estimate the presence or absence of risk based on video acquired in real time and issue an alert, or may estimate the presence or absence of risk collectively based on accumulated video and output it as a report.

[0078] (3) Hospital staff confirm the abnormality and take action) Upon receiving the alert, hospital staff will take appropriate action, such as rushing to the location where the abnormality occurred, providing appropriate care to the patient, or rearranging medical equipment.

[0079] At this time, the information processing device 1A may determine and present a measure to be proposed to the hospital staff. In this case, the information processing device 1A may determine a measure based on a predetermined rule, for example, based on the content and level of the estimated risk. Alternatively, the information processing device 1A may estimate a measure using a machine learning model that has been trained in advance by machine learning, based on the content and level of the estimated risk.

[0080] The information processing device 1A may issue an alert only to a corresponding target person among a plurality of hospital staff members, or may issue an alert by controlling broadcasting equipment within the hospital.The information processing device 1A may also generate a message for the target person and notify the target person of the message on a terminal owned by the target person.

[0081] (Effects of Use Case) The information processing device 1A according to this use case can improve the safety of patients in the hospital 100 and reduce the burden on hospital staff. In addition, the abnormal behavior detection system using video recognition technology can detect abnormalities more quickly than conventional manual monitoring, making it possible to prevent accidents and problems before they occur.

[0082] Furthermore, the information processing device 1A can optimize the actions of hospital staff by presenting countermeasures, and can also support the decision-making of hospital staff by presenting countermeasures.

[0083] Furthermore, accumulating and analyzing data will enable the development of more advanced anomaly detection algorithms and the prediction of anomaly occurrences.

[0084] [Third Exemplary Embodiment] A third exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be denoted by the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs. Furthermore, each technical means shown in each drawing referenced to describe this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs.

[0085] The information processing device 1B according to this exemplary embodiment includes a filter unit 14B instead of the filter unit 14 of the information processing device 1A.

[0086] 12 is a block diagram showing an example of the configuration and processing flow of the filter unit 14 B. The filter unit 14 B includes a first preprocessing unit 141, a second preprocessing unit 142, a combining and adding unit 143, a feature map generating unit 144, a third preprocessing unit 145, a gaze map generating unit 146, a multiplying unit 147, and a feature amount generating unit 148, as well as a combining and adding unit 149, a feature map generating unit 150, and a combining and adding unit 151.

[0087] The combining / adding unit 149 combines or adds the image feature data D3_1 output by the first preprocessing unit 141 and the posture data D4_1 output by the third preprocessing unit 145, and outputs a feature map M1_4. The feature map generating unit 150 performs processing such as adjusting the number of dimensions of the feature map M1_4 output by the combining / adding unit 149 to generate a feature map M1_5. The combining / adding unit 151 combines or adds the feature maps M1_3 and M1_5 to output a feature map M1_6. In this way, the filter unit 14B of the information processing device 1B generates a new feature map M1_6 using the image feature data D3 representing the image features extracted by the image feature extraction unit 17, the feature map M1 generated by the peripheral feature extraction unit 18, and the posture information D4. The multiplying unit 147 outputs a second feature map M2 obtained by multiplying the new feature map M1_6 by the gaze map Mtx.

[0088] As described above, in the information processing device 1B, the feature map generation unit 150 is configured to generate a new feature map M1_6 using image feature data D3 representing the image features extracted by the image feature extraction unit 17, the feature map M1 generated by the peripheral feature extraction unit 18, and posture information D4. Therefore, according to the information processing device 1B, by generating the feature map M1_6 using image feature data D3 in addition to the feature map M1 and posture information D4, the reliability of the feature map M1_6 can be increased, and the target's behavior can be estimated with higher accuracy.

[0089] [Fourth Exemplary Embodiment] A fourth exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be denoted by the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs. Furthermore, each technical means shown in each drawing referenced to describe this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs.

[0090] The information processing device 1C according to this exemplary embodiment includes a filter unit 14B instead of the filter unit 14 of the information processing device 1A.

[0091] 13 is a block diagram showing an example of the configuration and processing flow of the filter unit 14 C. The filter unit 14 C includes a first preprocessing unit 141, a second preprocessing unit 142, a combining and adding unit 143, a feature map generating unit 144, a third preprocessing unit 145, a gaze map generating unit 146, a multiplying unit 147, and a feature amount generating unit 148, as well as a combining and adding unit 152.

[0092] The combining / adding unit 152 outputs posture data D4_2 obtained by combining or adding image feature data D3 and posture data D4_1. The gaze map generating unit 146 generates a gaze map Mtx using posture data D4_2. In other words, the filter unit 14 generates the gaze map Mtx using posture information D4 and image feature data D3, and filters the feature map M1_3 by multiplying the feature map M1_3 by the gaze map Mtx.

[0093] As described above, the information processing device 1C employs a configuration in which the filter unit 14C generates the gaze map Mtx using the posture information D4 and the image feature data D3, and filters the feature map M1_3 by multiplying the feature map M1_3 by the gaze map Mtx. Therefore, the information processing device 1C can generate the gaze map Mtx that takes into account the image features indicated by the image feature data D3 in addition to the posture information D4, thereby enabling the target's behavior to be estimated with higher accuracy.

[0094] [Example of Implementation by Software] Some or all of the functions of the behavior inference device 1 and the information processing devices 1A, 1B, and 1C (hereinafter also referred to as "each of the above-mentioned devices") may be implemented by hardware such as an integrated circuit (IC chip), or by software.

[0095] In the latter case, each of the above devices is realized by, for example, a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 14. Figure 14 is a block diagram showing the hardware configuration of computer C that functions as each of the above devices.

[0096] The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for causing the computer C to function as each of the above-mentioned devices. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing the functions of each of the above-mentioned devices.

[0097] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0098] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0099] The program P can also be recorded on a non-transitory, tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0100] Furthermore, at least one or more of the functions of the above-mentioned behavior estimation device 1 and information processing devices 1A, 1B, and 1C may be executed by multiple different information processing devices installed and connected anywhere on the network, that is, they may be executed by so-called cloud computing.

[0101] [Appendix 1] The present disclosure includes the technologies described in the following appendices. However, the present invention is not limited to the technologies described in the following appendices, and various modifications are possible within the scope of the claims. (Appendix 1) A behavior estimation device comprising: image acquisition means for acquiring an image; first generation means for generating posture information representing a posture of an object included in the image; second generation means for generating a feature map representing features of the periphery of the object in the image; filter means for generating a second feature map by filtering the feature map based on the posture information; and behavior estimation means for estimating the behavior of the object based on the posture information and the second feature map, and outputting an estimation result.

[0102] (Supplementary Note 2) The behavior estimation device according to Supplementary Note 1, further comprising: an extraction means for extracting image features of the object from the image; wherein the filter means generates a new feature map using the image features extracted by the extraction means and the feature map generated by the second generation means, and generates the second feature map by filtering the generated new feature map based on the posture information.

[0103] (Supplementary Note 3) The behavior estimation device according to Supplementary Note 2, wherein the behavior estimation means estimates the behavior of the target based on the posture information, the second feature map, and the image features extracted by the extraction means.

[0104] (Supplementary Note 4) The behavior estimation device according to Supplementary Note 1 or 2, wherein the filter means generates a gaze map using the posture information, and filters the feature map by multiplying the feature map by the gaze map.

[0105] (Supplementary Note 5) The behavior estimation device according to Supplementary Note 2 or 3, wherein the filter means generates the new feature map using the image features extracted by the extraction means, the feature map generated by the second generation means, and the posture information.

[0106] (Supplementary Note 6) The behavior estimation device according to Supplementary Note 2 or 3, wherein the filter means generates a gaze map using the posture information and the image features, and filters the feature map by multiplying the feature map by the gaze map.

[0107] (Supplementary Note 7) The behavior inference device according to Supplementary Note 1 or 2, wherein the second generating means generates the feature map using a trained model trained by machine learning.

[0108] (Supplementary Note 8) The behavior estimation device according to Supplementary Note 1 or 2, wherein the behavior estimation means estimates a behavior of a subject present in a medical facility and outputs an alert based on an estimation result.

[0109] (Supplementary Note 9) The behavior estimation device according to Supplementary Note 1 or 2, wherein the behavior estimation means estimates a behavior of a subject present in a medical facility, and outputs information for supporting decision-making of a medical professional based on an estimation result.

[0110] (Supplementary Note 10) A behavior estimation method including: at least one processor acquiring an image; generating posture information representing a posture of an object included in the image; generating a feature map representing features of the periphery of the object in the image; generating a second feature map by filtering the feature map based on the posture information; and estimating a behavior of the object based on the posture information and the second feature map, and outputting an estimation result.

[0111] (Supplementary Note 11) The behavior estimation method according to Supplementary Note 10, further comprising: the at least one processor extracting image features of the object from the image; and in the step of generating the second feature map, the at least one processor generates a new feature map using the image features extracted by the extraction means and the feature map generated by the second generation means, and filters the generated new feature map based on the posture information to generate the second feature map.

[0112] (Supplementary Note 12) The behavior estimation method according to Supplementary Note 11, wherein in the step of estimating the behavior of the target, the at least one processor estimates the behavior of the target based on the posture information, the second feature map, and the image features extracted by the extraction means.

[0113] (Supplementary Note 13) The activity estimation method according to Supplementary Note 10 or 11, wherein in the step of generating the second feature map, the at least one processor generates a gaze map using the posture information and filters the feature map by multiplying the feature map by the gaze map.

[0114] (Supplementary Note 14) The behavior estimation method according to Supplementary Note 11 or 12, wherein in the step of generating the second feature map, the at least one processor generates the new feature map using the image features extracted by the extraction means, the feature map generated by the second generation means, and the posture information.

[0115] (Supplementary Note 15) The activity estimation method according to Supplementary Note 11 or 12, wherein in the step of generating the second feature map, the at least one processor generates a gaze map using the posture information and the image features, and filters the feature map by multiplying the feature map by the gaze map.

[0116] (Supplementary Note 16) The behavior estimation method according to Supplementary Note 10 or 11, wherein in the step of generating the feature map, the at least one processor generates the feature map using a trained model trained by machine learning.

[0117] (Supplementary Note 17) The behavior estimation method according to Supplementary Note 10 or 11, wherein in the step of estimating the behavior of the subject, the at least one processor estimates the behavior of a subject present in a medical facility and outputs an alert based on an estimation result.

[0118] (Supplementary Note 18) The behavior estimation method according to Supplementary Note 10 or 11, wherein in the step of estimating the behavior of the subject, the at least one processor estimates the behavior of the subject present in a medical facility and outputs information to support decision-making by medical personnel based on the estimation result.

[0119] (Supplementary Note 19) A program for causing a computer to execute: an image acquisition process for acquiring an image; a first generation process for generating posture information representing the posture of an object included in the image; a second generation process for generating a feature map representing features of the periphery of the object in the image; a filtering process for generating a second feature map by filtering the feature map based on the posture information; and a behavior estimation process for estimating the behavior of the object based on the posture information and the second feature map and outputting an estimation result.

[0120] (Supplementary Note 20) The program according to Supplementary Note 19, further causing the computer to execute an extraction process to extract image features of the object from the image, wherein in the filtering process, the computer generates a new feature map using the image features extracted in the extraction process and the feature map generated in the second generation process, and generates the second feature map by filtering the generated new feature map based on the posture information.

[0121] (Supplementary Note 21) The program according to Supplementary Note 20, wherein in the behavior estimation process, the computer estimates the behavior of the target based on the posture information, the second feature map, and the image features extracted in the extraction process.

[0122] (Supplementary Note 22) The program according to Supplementary Note 19 or 20, wherein in the filtering process, the computer generates a gaze map using the posture information, and filters the feature map by multiplying the feature map by the gaze map.

[0123] (Supplementary Note 23) The program according to Supplementary Note 20 or 21, wherein in the filtering process, the computer generates the new feature map using the image features extracted in the extraction process, the feature map generated in the second generation process, and the posture information.

[0124] (Supplementary Note 24) The program according to Supplementary Note 20 or 21, wherein in the filtering process, the computer generates a gaze map using the posture information and the image features, and filters the feature map by multiplying the feature map by the gaze map.

[0125] (Supplementary Note 25) The program according to Supplementary Note 19 or 20, wherein in the second generation process, the computer generates the feature map using a trained model trained by machine learning.

[0126] (Supplementary Note 26) The program according to Supplementary Note 19 or 20, wherein in the behavior estimation process, the computer estimates a behavior of a subject present in a medical facility and outputs an alert based on an estimation result.

[0127] (Supplementary Note 27) The program according to Supplementary Note 19 or 20, wherein in the behavior estimation process, the computer estimates the behavior of a subject present in a medical facility and outputs information to support decision-making by a medical professional based on an estimation result.

[0128] (Supplementary Note 28) A behavior estimation system comprising: an image acquisition means for acquiring an image; a first generation means for generating posture information representing the posture of an object included in the image; a second generation means for generating a feature map representing features of the periphery of the object in the image; a filter means for generating a second feature map by filtering the feature map based on the posture information; and a behavior estimation means for estimating the behavior of the object based on the posture information and the second feature map, and outputting an estimation result.

[0129] [Appendix 2] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0130] (Supplementary Note 1) A behavior estimation device comprising at least one processor, the at least one processor executing: an image acquisition process for acquiring an image; a first generation process for generating posture information representing a posture of an object included in the image; a second generation process for generating a feature map representing features of the periphery of the object in the image; a filtering process for generating a second feature map by filtering the feature map based on the posture information; and a behavior estimation process for estimating behavior of the object based on the posture information and the second feature map and outputting an estimation result.

[0131] The behavior inference device may further include a memory, and the memory may store a program for causing the at least one processor to execute each of the processes.

[0132] (Supplementary Note 2) The behavior estimation device according to Supplementary Note 1, wherein the at least one processor further executes an extraction process to extract image features of the target from the image, and in the filtering process, the at least one processor generates a new feature map using the image features extracted in the extraction process and the feature map generated in the second generation process, and generates the second feature map by filtering the generated new feature map based on the posture information.

[0133] (Supplementary Note 3) The behavior estimation device according to Supplementary Note 2, wherein in the behavior estimation process, the at least one processor estimates the behavior of the target based on the posture information, the second feature map, and image features extracted in the extraction process.

[0134] (Supplementary Note 4) The behavior estimation device according to Supplementary Note 1 or 2, wherein in the filtering process, the at least one processor generates a gaze map using the posture information and filters the feature map by multiplying the feature map by the gaze map.

[0135] (Supplementary Note 5) The behavior estimation device according to Supplementary Note 2 or 3, wherein in the filtering process, the at least one processor generates the new feature map using the image features extracted in the extraction process, the feature map generated in the second generation process, and the posture information.

[0136] (Supplementary Note 6) The behavior estimation device according to Supplementary Note 2 or 3, wherein in the filtering process, the at least one processor generates a gaze map using the posture information and the image features, and filters the feature map by multiplying the feature map by the gaze map.

[0137] (Supplementary Note 7) The behavior inference device according to Supplementary Note 1 or 2, wherein in the second generation process, the at least one processor generates the feature map using a trained model trained by machine learning.

[0138] (Supplementary Note 8) The behavior estimation device according to Supplementary Note 1 or 2, wherein in the behavior estimation process, the at least one processor estimates a behavior of a subject present in a medical facility and outputs an alert based on an estimation result.

[0139] (Supplementary Note 9) The behavior estimation device according to Supplementary Note 1 or 2, wherein in the behavior estimation process, the at least one processor estimates the behavior of a subject present in a medical facility and outputs information to support decision-making of a medical professional based on an estimation result.

[0140] REFERENCE SIGNS LIST 1 Behavior estimation device 11 Image acquisition unit 12 First generation unit 13 Second generation unit 14, 14B, 14C Filter unit 15 Behavior estimation unit

Claims

1. Image acquisition means for acquiring images, A first generation means for generating posture information representing the posture of an object included in the aforementioned image, A second generation means for generating a feature map representing the features of the area surrounding the object in the aforementioned image, A filter means for generating a second feature map by filtering the feature map based on the posture information, An action estimation device comprising: action estimation means for estimating the action of the target based on the posture information and the second feature map, and outputting the estimation result.

2. The system further comprises extraction means for extracting image features of the target from the aforementioned image, The behavior estimation device according to claim 1, wherein the filtering means generates a new feature map using the image features extracted by the extraction means and the feature map generated by the second generation means, and generates the second feature map by filtering the generated new feature map based on the posture information.

3. The behavior estimation device according to claim 2, wherein the behavior estimation means estimates the behavior of the target based on the posture information, the second feature map, and the image features extracted by the extraction means.

4. The behavior estimation device according to claim 1 or 2, wherein the filtering means generates a gaze map using the posture information and filters the feature map by multiplying it by the gaze map.

5. The behavior estimation device according to claim 2 or 3, wherein the filtering means generates the new feature map using the image features extracted by the extraction means, the feature map generated by the second generation means, and the posture information.

6. The behavior estimation device according to claim 2 or 3, wherein the filtering means generates a gaze map using the posture information and the image features, and filters the feature map by multiplying the feature map by the gaze map.

7. The behavior estimation device according to claim 1 or 2, wherein the second generation means generates the feature map using a trained model learned by machine learning.

8. The behavior estimation device according to claim 1 or 2, wherein the behavior estimation means estimates the behavior of a subject in a medical facility and outputs an alert based on the estimation result.

9. At least one processor, Acquiring an image and, To generate posture information representing the posture of the object included in the aforementioned image, To generate a feature map representing the features of the area surrounding the object in the aforementioned image, A second feature map is generated by filtering the feature map based on the posture information, A method for estimating behavior, comprising estimating the behavior of the target based on the posture information and the second feature map, and outputting the estimation result.

10. On the computer, Image acquisition process to obtain an image, A first generation process that generates posture information representing the posture of the object included in the aforementioned image, A second generation process generates a feature map representing the features of the area surrounding the object in the aforementioned image, A filtering process that generates a second feature map by filtering the feature map based on the posture information, A program for performing an action estimation process that estimates the action of the target based on the posture information and the second feature map, and outputs the estimation result.