Behavior recognition device, behavior recognition method, and program

Abstract

This behavior recognition device selects a candidate behavior to be a recognition candidate from behaviors of interest on the basis of location information of an image sensor and floor plan information, acquires image data detected by the image sensor, determines one or more recognizers corresponding to the candidate behavior, calculates a feature amount of the image data using the one or more recognizers, and recognizes the candidate behavior on the basis of the feature amount.

Description

[Technical Field]

[0001] The present disclosure relates to a technology for recognizing user behavior in a building. [Background technology]

[0002] In recent years, research into recognizing human behavior from video images has been progressing. For example, Patent Document 1 discloses a technology for detecting a human region including a person from a video image and recognizing the human behavior from a combination of the posture of the person in the region and surrounding objects.

[0003] For example, Patent Document 2 discloses a technology for extracting skeleton information based on human joints from a moving image in time series, extracting an enclosed area of ​​the skeleton information, and recognizing human behavior from the extracted enclosed area.

[0004] However, the technologies in Patent Documents 1 and 2 do not take into account the floor plan information of the building, and therefore improvements were needed to accurately recognize human behavior that depends on the space within the building. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Publication 2018-206321 [Patent Document 2] Publication 2019-144830 Summary of the Invention

[0006] A behavior recognition device according to one aspect of the present disclosure is a behavior recognition device that recognizes human behavior in a building, and includes: a first acquisition unit that acquires target behavior information including one or more target behaviors that are predetermined targets for recognition, floor plan information of the building, and location information of an image sensor installed in the building; a behavior selection unit that selects a candidate behavior to be recognized from the one or more target behaviors included in the target behavior information based on the location information of the image sensor and the floor plan information; a second acquisition unit that acquires image data detected by the image sensor; a first behavior recognition unit that determines one or more recognizers according to the candidate behaviors and calculates feature amounts of the image data using the one or more recognizers; a second behavior recognition unit that recognizes the candidate behaviors based on the feature amounts; and an output unit that outputs the recognition result by the second behavior recognition unit.

[0007] According to the present disclosure, it is possible to accurately recognize the behavior of people who depend on the space within a building. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram illustrating an example of a configuration of a behavior recognition device according to an embodiment. [Figure 2] FIG. 1 is an explanatory diagram of a CNN that constitutes a recognizer. [Figure 3] FIG. 2 is a diagram illustrating an example of the configuration of a first behavior recognition unit. [Figure 4] FIG. 10 is a diagram illustrating an example of the configuration of a behavior selection table. [Figure 5] FIG. 10 is a diagram illustrating an example of the configuration of a recognizer selection table. [Figure 6] FIG. 10 is a diagram illustrating an example of a weight table that is referred to when a combiner sets weight coefficients for feature quantities. [Figure 7] 10 is a flowchart illustrating an example of a process for generating list information in the behavior recognition device according to the embodiment. [Figure 8] 8 is a flowchart showing details of the process of step S106 in FIG. 7. [Figure 9] 10 is a flowchart illustrating an example of a behavior recognition process in the behavior recognition device. [Figure 10] FIG. 10 is a diagram showing the data structure of a prior knowledge table that summarizes information about recognizers. [Figure 11] This is a table summarizing the processing time per frame when existing recognizers are run individually. [Figure 12] 10 is a table summarizing input image data. [Figure 13] 10 is a table summarizing the results of a simulation performed to evaluate the recognition accuracy of the behavior recognition device. [Figure 14] FIG. 10 is a block diagram illustrating an example of a configuration of a behavior recognition device according to a modified example of the present disclosure. [Figure 15] FIG. 10 is a block diagram illustrating an example of the configuration of a behavior recognition device according to a modified example (3) of the present disclosure. [Figure 16] FIG. 10 is a diagram showing a scene in which an image sensor is installed at an entrance. [Figure 17] FIG. 10 is a diagram showing an example of interaction between a user and a display terminal in a scene where an image sensor is installed at an entrance. [Figure 18] FIG. 18 shows the interaction following FIG. 17. [Figure 19] FIG. 10 is a diagram illustrating an example of a setting screen on which an annotation image is superimposed. [Figure 20] FIG. 1 is a diagram showing a scene in which an image sensor is installed in a kitchen. [Figure 21] FIG. 10 is a diagram showing an example of interaction between a user and a display terminal in a scene where an image sensor is installed in a kitchen. [Figure 22] FIG. 22 shows the interaction following FIG. 21. [Figure 23] FIG. 10 is a diagram illustrating an example of a setting screen on which an annotation image is superimposed. [Figure 24] FIG. 10 is a diagram illustrating an example of interaction between a user and a display terminal when correcting an annotation image after installing an image sensor. [Figure 25] FIG. 10 is a diagram showing an example of a setting screen on which an annotation image is superimposed. DETAILED DESCRIPTION OF THE INVENTION

[0009] (Findings that led to the present disclosure) Previously, methods have been proposed for estimating human behavior from sensor data, such as video or still images. In these methods, a label is assigned in advance to the behavior to be recognized, and the sensor data is then determined to which label it corresponds. When the sensor data is video or still images, high recognition accuracy can be achieved by using a deep neural network (DNN) with convolutional layers and pooling layers. However, current methods using DNNs only recognize behaviors that are independent of the space in which they are performed, such as human walking. Therefore, in order to recognize behaviors that are dependent on space, it is necessary to train the DNN from scratch, taking into account spatial information, which is time-consuming and costly.

[0010] In the above-mentioned Patent Document 1, a person's behavior is recognized by taking into consideration a combination of the person's posture and objects around the person, but if an object unrelated to the person's behavior is reflected around the person, the object does not provide useful information for estimating the person's behavior. Therefore, Patent Document 1 needs further improvement to recognize space-dependent human behavior.

[0011] In the above-mentioned Patent Document 2, as is clear from the fact that human behavior is estimated from the enclosed area of ​​the skeleton information, behavior that is not dependent on the background is assumed as the behavior to be recognized. Therefore, Patent Document 2 needs further improvement in order to estimate behavior that is highly dependent on space, such as cooking behavior in a kitchen.

[0012] Therefore, the inventors have discovered that by taking into account the floor plan information of a building, it is possible to accurately recognize human behavior that depends on the space within the building, and have come up with the various aspects of the present disclosure described below.

[0013] A behavior recognition device according to one aspect of the present disclosure is a behavior recognition device that recognizes human behavior in a building, and includes: a first acquisition unit that acquires target behavior information including one or more target behaviors that are predetermined targets for recognition, floor plan information of the building, and location information of an image sensor installed in the building; a behavior selection unit that selects a candidate behavior to be recognized from the one or more target behaviors included in the target behavior information based on the location information of the image sensor and the floor plan information; a second acquisition unit that acquires image data detected by the image sensor; a first behavior recognition unit that determines one or more recognizers according to the candidate behaviors and calculates feature amounts of the image data using the one or more recognizers; a second behavior recognition unit that recognizes the candidate behaviors based on the feature amounts; and an output unit that outputs the recognition result by the second behavior recognition unit.

[0014] According to this configuration, candidate actions to be recognized are selected from target actions based on the position information of the image sensor and the floor plan information of the building, and the recognition results for the candidate actions are calculated. This makes it possible to accurately recognize human actions that depend on the space within the building.

[0015] Furthermore, with this configuration, the first behavior recognition unit determines a recognizer according to the candidate behavior, calculates features of the image data using the determined recognizer, and the second behavior recognizer recognizes the candidate behavior from the calculated features. This makes it possible to use an existing recognizer as the recognizer, making it easier to build a behavior recognition device. Furthermore, because features are calculated using one or more recognizers according to the candidate behavior, it is possible to calculate features suitable for recognizing the candidate behavior, thereby improving the recognition accuracy of the target behavior.

[0016] In the above behavior recognition device, when the candidate behavior is a predetermined behavior, the first behavior recognition unit may determine a plurality of recognizers, and the second behavior recognition unit may combine feature amounts calculated by the plurality of recognizers and recognize the candidate behavior based on the combined feature amounts.

[0017] According to this configuration, when the target behavior is a predetermined behavior, multiple recognizers are determined. Therefore, in a case where the predetermined behavior depends on an object around a person, for example, it is possible to calculate the features using a recognizer that calculates the features of the person and a recognizer that calculates the features of the object, thereby making it possible to improve the recognition accuracy of candidate behaviors.

[0018] Furthermore, with this configuration, the features calculated by each recognizer are combined, and the target behavior is recognized based on the combined features. Therefore, when multiple features are calculated by multiple recognizers, they can be combined into one feature and input to the second behavior recognition unit, thereby simplifying the configuration of the behavior recognition device.

[0019] In the above-described behavior recognition device, the predetermined behavior may be cleaning, brushing teeth, cooking, laundry, using a computer, reading, or eating.

[0020] According to this configuration, it is possible to accurately calculate, from image data, the feature amount of a person's behavior, such as cleaning or brushing their teeth, which depends on objects around the person.

[0021] In the above behavior recognition device, each recognizer may be configured with a convolutional neural network (hereinafter referred to as CNN), and the second behavior recognition unit may recognize the candidate behavior using a classifier that uses any one of logistic regression, support vector machine, decision tree, random forest, k-nearest neighbor algorithm, Gaussian Naive Bayes, perceptron, and probabilistic descent.

[0022] According to this configuration, each recognizer is configured with a CNN, and the second behavior recognizer is configured with a classifier using logistic regression or the like, so it is possible to configure the second behavior recognizer with a classifier that has a lower processing cost than the first behavior recognizer.

[0023] In the behavior recognition device, the second behavior recognition unit may recognize the candidate behavior using a classifier that has been machine-learned using the feature amounts as explanatory variables and the target behavior as a target variable.

[0024] According to this configuration, the second behavior recognition unit can be configured using a classifier generated by machine learning using feature quantities calculated by each recognizer as explanatory variables and candidate behaviors corresponding to those feature quantities as objective variables. For example, if the classifier is configured with a classifier that has a smaller processing load than CNN, such as the above-mentioned logistic regression, the classifier can be trained in a short time. Furthermore, by configuring the first behavior recognition unit with an existing recognizer configured with CNN, it is possible to configure a behavior recognition device by performing machine learning on only the second behavior recognition unit, without performing machine learning on the first behavior recognition unit.

[0025] In the behavior recognition device, the second behavior recognition unit may weight each feature amount using a weighting coefficient determined in advance according to the candidate behavior, and recognize the candidate behavior based on each weighted feature amount.

[0026] According to this configuration, each feature amount is weighted in accordance with the target behavior, and candidate behaviors are recognized based on each weighted feature amount, so that candidate behaviors can be recognized accurately.

[0027] In the above behavior recognition device, one or more objects installed in the building are extracted from the floor plan information, and the one or more objects are classified into a first object that is a movable object, a second object that is a plumbing facility, and a third object that is a structure of the building. For each of the one or more objects, floor plan features that associate classification information indicating the classification result with an installation position are extracted. A behavior selection table is generated based on the floor plan features. The behavior selection table is a table that associates one or more spaces of the building with the target behavior corresponding to each space, and the first acquisition unit may acquire the behavior selection table as the target behavior information.

[0028] According to this configuration, floor plan features that associate classification information with installation locations for each object installed in a building are extracted based on floor plan information. This makes it possible to understand what objects are installed in each space within the building, and extracts information useful for creating a behavior selection table. Then, based on the floor plan features, a behavior selection table that associates each space in the building with a target behavior corresponding to each space is generated, and this behavior selection table is acquired as target behavior information. This makes it possible to quickly understand the relationship between each space and the target behavior, and easily select candidate behaviors.

[0029] In the above-described behavior recognition device, the behavior recognition device may further include an installation support unit that is communicatively connected to a display terminal, acquires via the display terminal the name of the space in the building in which the image sensor is to be installed, and outputs to the display terminal installation guidance for installing the image sensor so that a specific device or specific facility related to the space is included in the field of view of the image sensor.

[0030] According to this configuration, installation guidance for installing the image sensor so that a specific device or facility related to the space is included in the field of view of the image sensor is output to the display terminal, allowing the user to install the image sensor appropriately. Furthermore, because the name of the space in the building is acquired via the display terminal, the image sensor can be associated with the space in which the image sensor is installed. As a result, the user can easily select a candidate action by referring to the installation position of the image sensor and the action selection table.

[0031] In the above-mentioned behavior recognition device, the installation support unit may acquire image data captured by the image sensor, detect the specific device or specific equipment included in the image data, and superimpose an annotation image indicating the detection result of the specific device or specific equipment on an image indicated by the image data.

[0032] According to this configuration, an annotation image that is the detection result of the specific device or specific facility detected from the image data is superimposed on an image represented by the image data, making it easy to confirm whether the specific device or specific facility has been correctly detected from the image data.

[0033] In the behavior recognition device, the installation support unit may acquire an instruction to correct the annotation image via the display terminal, and store annotation information indicated by the corrected annotation image in a memory.

[0034] According to this configuration, since an instruction to correct the annotation information is acquired via the display terminal, for example, if the recognizer is unable to correctly detect a specific device or specific equipment, the annotation information can be corrected to accurately indicate the position of the specific device or specific equipment. Furthermore, the recognizer can be made to grasp the position on the image of the specific device or specific equipment, which is important for recognizing the target behavior.

[0035] The present disclosure can also be realized as a behavior recognition method in which a computer executes each characteristic configuration included in such a behavior recognition device, and as a behavior recognition program that causes a computer to execute each characteristic configuration. Needless to say, such behavior recognition program can be distributed on a computer-readable non-transitory recording medium such as a CD-ROM or via a communication network such as the Internet.

[0036] Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, components, steps, and step orders shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concept are described as optional components. Furthermore, in all of the embodiments, the respective contents can be combined.

[0037] (Embodiment) Computer-based methods for recognizing human behavior can be classified into three types based on the relationship between the person and the space in which they are located. The first type recognizes behaviors unrelated to the space in which a person is located. For example, behaviors such as walking or standing are expressed only by the movement of the person in question and are unrelated to the space in which the person is located. The second type recognizes behaviors involving objects in the space in which a person is located or objects located near the person. For example, the behavior of riding a bicycle can be recognized by combining human posture detection and object detection. When recognizing human behaviors involving objects outdoors, it is difficult to grasp in advance the structure of the building or the condition of the road. Therefore, such behaviors can be recognized by performing recognition processing that utilizes the person and the objects around the person. The third type recognizes behaviors by understanding both the person and the space in which the person is located. For example, the behavior of starting to cook in the kitchen can be recognized by the person's presence in the kitchen and facing the microwave.

[0038] In this way, actions such as "start cooking", which are difficult to recognize without using information about the space in which a person is present, cannot be recognized by the first and second types of conventional recognition methods. In this embodiment, a method for recognizing human actions using floor plan information, which is information about the person and the space in which the person is present, is disclosed. Hereinafter, this embodiment will be described with reference to the drawings. Human actions that are the target of recognition in this embodiment include, for example, actions that involve movement, such as starting cooking or cleaning, as well as actions that do not involve movement, such as watching TV while lying down.

[0039] 1 is a block diagram showing an example of the configuration of a behavior recognition device 1 according to an embodiment. The behavior recognition device 1 is a device that recognizes a user's behavior in the user's home (an example of a building).

[0040] The behavior recognition device 1 is configured by a computer including, for example, a processor, a memory, an interface circuit, etc. The behavior recognition device 1 does not need to be realized by a single computer, but may be realized by a distributed processing system (not shown) including a terminal device and a server. For example, the behavior recognition device 1 may be configured by providing a memory for storing image data 204 in a terminal device in the home, and providing some or all of the blocks constituting the processor 100 in a server. This form will be described in a modified example below.

[0041] The behavior recognition device 1 includes a processor 100 and a memory 200. The memory 200 is configured with a non-volatile storage device such as an SSD or an HHD, and stores target behavior information 201, floor plan information 202, location information 203, image data 204, and list information 205. The memory 200 may store image data 204 of a certain period of time (for example, one minute) going back in time from the present among the image data 204 acquired from the image sensor 2 by the second acquisition unit 103 at a predetermined frame rate.

[0042] The image sensor 2 is configured, for example, by a camera installed inside the house. The image sensor 2 captures images of the space inside the house at a predetermined frame rate to acquire image data 204, which is input to the second acquisition unit 103. There may be multiple image sensors 2. The image data 204 may be, for example, color image data or monochrome image data.

[0043] The processor 100 is composed of electrical circuits such as a CPU. The processor 100 includes a first acquisition unit 101, a behavior selection unit 102, a second acquisition unit 103, a first behavior recognition unit 104, a second behavior recognition unit 105, and an output unit 106. Each of these blocks is realized by the processor 100 executing, for example, a behavior recognition program. However, this is just one example, and each of these blocks may also be realized by a dedicated hardware circuit such as an ASIC.

[0044] The first acquisition unit 101 acquires target behavior information 201 indicating a predetermined target behavior to be recognized, and stores the target behavior information 201 in the memory 200. For example, the first acquisition unit 101 may acquire the target behavior information 201 input through a registration operation using an input device (not shown). The input device may be, for example, a keyboard, a mouse, or the like. However, this is just an example, and the target behavior information 201 may be stored in the memory 200 in advance. The target behavior is, for example, various behaviors such as wiping and sweeping registered in a behavior selection table T1 shown in FIG. 4. The target behavior information 201 may be stored in the memory 200 in advance. The behavior selection table T1 is an example of the target behavior information 201.

[0045] Furthermore, the first acquisition unit 101 acquires floor plan information 202 of the home and stores it in the memory 200. The floor plan information 202 is two-dimensional or three-dimensional information that represents room components such as a living room, dining room, and kitchen, as well as the shapes and positional relationships of each room. The floor plan information 202 is composed of, for example, two-dimensional drawing data describing the floor plan, three-dimensional design data (CAD data) used for house design, three-dimensional point cloud information (point cloud) of the home measured with a three-dimensional laser scanner, trajectory information of a device moving around the home such as a cleaning robot, video data of the home captured with a calibrated camera, or information acquired from a device having information on the room in which the device is installed.

[0046] Furthermore, the first acquisition unit 101 acquires position information 203 of the image sensor 2 installed in the house and stores it in the memory 200. The position information 203 is composed of coordinate data expressed in a coordinate system of a two-axis or three-axis coordinate space included in the floor plan information 202, for example. The first acquisition unit 101 acquires the position information 203 inputted by, for example, a registration operation using an input device (not shown).

[0047] The behavior selection unit 102 selects a candidate behavior to be recognized from the target behaviors based on the location information 203 of the image sensor 2 and the floor plan information 202. For example, the behavior selection unit 102 identifies a space in the house where the image sensor 2 is installed based on the location information 203 and the floor plan information 202, and selects a predetermined behavior that is estimated to be performed by the user in the identified space. Specifically, the behavior selection unit 102 may determine the behavior corresponding to the identified space by referring to the behavior selection table T1 shown in FIG. 4. The space is, for example, a room, an entrance, a kitchen, or other space that constitutes the house.

[0048] The second acquisition unit 103 acquires image data 204 captured by the image sensor 2 at a predetermined frame rate, and stores the image data 204 in the memory 200 .

[0049] The first behavior recognition unit 104 determines one or more recognizers according to the candidate behaviors selected by the behavior selection unit 102, and calculates feature quantities of the image data 204 using the determined recognizers. The first behavior recognition unit 104 includes a recognizer selection unit 110 and N (N is an integer equal to or greater than 1) recognizers 111_1, 111_2, ... 111_N. Hereinafter, the recognizers 111_1, 111_2, ... 111_N will be collectively referred to as recognizers 111.

[0050] The recognizer selection unit 110 selects the recognizer 111 to be used for recognizing the candidate behavior selected by the behavior selection unit 102. For example, when generating list information 205 (described later), the recognizer selection unit 110 may select the recognizer 111 corresponding to the candidate behavior by referring to the recognizer selection table T2 shown in Fig. 5, and when executing the recognition process, the recognizer selection unit 110 may select the recognizer 111 for the candidate behavior by referring to the list information 205 (described later).

[0051] The recognizer 111 is a recognizer configured by, for example, CNN. In this embodiment, the first behavior recognition unit 104 includes recognizers 111 that individually recognize each of posture estimation, object detection, face detection, head direction estimation, age / gender estimation, individual estimation, and tracking, as shown in Fig. 5. These recognizers 111 are reused or partially modified from existing recognizers whose source codes are publicly available.

[0052] The feature amount differs depending on the recognizer 111. The feature amount calculated by the posture estimation recognizer 111 includes, for example, two-dimensional coordinate data representing each of a plurality of feature points (for example, 17 points such as the right shoulder and right elbow) that make up the skeletal information.

[0053] The feature values ​​calculated by the object detection recognizer 111 include, for example, coordinate data of a circumscribing rectangle surrounding an object and a label of the recognized object. The objects detected by object detection are given, and for example, 80 types of household objects (such as a microwave oven and a refrigerator) are detected. The feature values ​​calculated by the face detection recognizer 111 include, for example, image data of a face region surrounded by a circumscribing rectangle. Here, the feature values ​​calculated by each recognizer 111 are not data that cannot be understood by humans (for example, tensors), such as those output from the intermediate layer of a DNN.

[0054] The first behavior recognition unit 104 includes a recognizer 111 that depends on another recognizer 111 depending on the candidate behavior. The dependent recognizer 111 is a recognizer 111 that receives a feature calculated by another recognizer 111 and calculates a feature using the feature. For example, the head direction estimation recognizer 111 calculates a feature indicating the head direction using the feature (image data of the face region) calculated by the face detection recognizer 111.

[0055] The second behavior recognition unit 105 recognizes each candidate behavior selected by the behavior selection unit 102 based on the feature amounts calculated by each recognizer 111. The second behavior recognition unit 105 includes a combiner 121 and a classifier 122. The combiner 121 combines the feature amounts calculated by each combiner 121 and inputs the combined feature amount to the classifier 122. Here, the combiner 121 may weight each feature amount using a weighting coefficient determined in advance according to the candidate behavior, and connect the weighted feature amounts and input them to the classifier 122.

[0056] The classifier 122 recognizes a candidate action by calculating the likelihood of the candidate action based on the feature input from the combiner 121. The classifier 122 is configured with a classifier that performs class classification. The classifier 122 is configured with a classifier that uses, for example, any one of logistic regression, support vector machine, decision tree, random forest, k-nearest neighbor algorithm, Gaussian naive Bayes, perceptron, and stochastic descent. When there are multiple candidate actions, the classifier 122 calculates the likelihood of each candidate action.

[0057] The classifier 122 is machine-learned using the feature values ​​output from the recognizer 111 as explanatory variables and the target behavior as a response variable. This machine learning is performed individually for each target behavior. For example, if the target behavior is "start cooking," machine learning is performed using the combined results of the feature values ​​output from each recognizer 111 used for this target behavior as explanatory variables and "start cooking" as a response variable.

[0058] The output unit 106 outputs the recognition result of the second behavior recognition unit 105. Specifically, the output unit 106 outputs the recognition result for the human behavior based on the likelihood calculated by the classifier 122. For example, when there are multiple candidate behaviors, the classifier 122 may output the label with the maximum likelihood as the recognition result.

[0059] The recognition result is output to an external device (not shown), for example, via a communication circuit. The external device may be, for example, a home appliance installed in the home and performing control based on the recognition result. This home appliance may be, for example, a display device installed in the home and displaying the recognition result.

[0060] FIG. 2 is an explanatory diagram of the CNN that constitutes the recognizer 111. The CNN includes a convolutional layer and a fully connected layer. In this example, the CNN is composed of nine convolutional layers and two fully connected layers. In FIG. 2, the subscripts of various symbols indicate the layer number, where the input layer of the input image data D1 is layer 0, the convolutional layers are layers 1 to 9, the fully connected layers are layers 10 to 11, and the output layer is layer 12. The input image data D1 is expressed as H0 × W0 × C0. Here, H represents the height of the data in each layer, W represents the width of the data in each layer, and C represents the number of channels in each layer. If the input image data D1 is color image data, C0 = 3. In the convolutional layer, the features of the data in each layer are calculated by convolutional calculation. For example, the size of the data output from the fifth layer, i.e., H5 × W5 × C5, is expressed as 64 × 64 × 256 = 1,048,576.

[0061] In the fully connected layer, the data output from the convolutional layer is compiled and output data is generated. For example, in the case of a CNN for posture estimation that detects a maximum of P people from M skeletal information points, the fully connected layer outputs P × M × 2(x,y) pieces of data as output data.

[0062] CNN has extremely high recognition accuracy for image data. However, because CNN is composed of many layers, such as convolutional layers and fully connected layers, it imposes a high processing load and takes a long time to learn. In contrast, the classifier 122 is a classifier such as a support vector machine, which has a significantly lower processing load than CNN. Therefore, in this embodiment, an existing recognizer is used as the recognizer 111 composed of CNN, and the classifier 122, which has a low processing load, learns the target behavior. This shortens the learning time and makes it possible to easily build the behavior recognition device 1.

[0063] 3 is a diagram illustrating an example of the configuration of the first behavior recognition unit 104. In this example, the first behavior recognition unit 104 includes a recognizer 111_1 that performs posture estimation, a recognizer 111_2 that performs object detection, a recognizer 111_3 that performs face detection, and a recognizer 111_4 that performs head direction estimation. Input image data D1 is input to each of the recognizers 111_1 to 111_3. The input image data D1 is image data of the inside of a house captured by the image sensor 2.

[0064] The following description will be given taking an example where the target behavior is "start cooking." For example, input image data D1 includes a scene in which a person is standing in a kitchen facing a refrigerator. When input image data D1 of a person standing is input, recognizer 111_1 outputs coordinate data constituting skeletal information of the person standing as features. Recognizer 111_2 detects a given object from input image data D1 and outputs coordinate data of the vertices of a circumscribing rectangle surrounding the detected object and the label of the detected object as features. In this example, a refrigerator is detected as the object.

[0065] The recognizer 111_3 detects a human face from the input image data D1 and outputs image data of a face region surrounding the detected face with a circumscribing rectangle as a feature. The recognizer 111_4 estimates the direction of the person's head from the image data of the face region output from the recognizer 111_3. The head direction is represented by a direction vector starting from the center of gravity of the face region, for example.

[0066] For example, if the center of gravity of a circumscribed rectangle surrounding the refrigerator is in the direction of the direction vector, the second behavior recognition unit 105 determines that the person is facing the refrigerator. For example, the second behavior recognition unit 105 rotates the direction vector on the image from the origin within a range of plus or minus a threshold angle (for example, plus or minus 30 degrees), and if an extension of the direction vector passes through the center of gravity of the refrigerator, the second behavior recognition unit 105 determines that the person is facing the refrigerator.

[0067] By combining the information on the posture of the person, the object position, and the head direction, the second behavior recognition unit 105 can determine the behavior of "start cooking." Note that the recognizer 111_4 may detect the direction of the line of sight, the direction of the nose, and the normal direction of the body instead of the head direction.

[0068] FIG. 4 is a diagram showing an example of the configuration of the behavior selection table T1. The behavior selection table T1 stores a plurality of spaces and target behaviors to be recognized for each space, in association with each other. The target behaviors include mopping, sweeping, brushing teeth, starting to cook, making coffee, using a laptop computer (notebook PC), doing laundry, reading, eating, and walking. The spaces include the entrance, kitchen, living room, dining room, bedroom, bathroom, and washroom. However, these are merely examples, and other spaces and target behaviors may also be used. For example, a target behavior such as lying on the sofa and watching TV may be assigned to the living room.

[0069] In the behavior selection table T1, a circle mark indicates that the corresponding target behavior is a recognition target for the corresponding space, and an × mark indicates that the corresponding target behavior is not a recognition target for the corresponding space.

[0070] For example, the mopping action shown in the first row is a recognition target in the entrance, kitchen, living room, dining room, bathroom, and washroom.

[0071] Mopping is likely to be performed in all spaces, so it is targeted for recognition in all spaces. Sweeping is unlikely to be performed in the bathroom, so it is targeted for recognition in spaces other than the bathroom. Toothbrushing is an action performed in spaces around water, so it is targeted for recognition in the kitchen, bathroom, and washroom.

[0072] Since the act of starting cooking is not performed anywhere other than the kitchen, only the kitchen is considered to be the target space. Since the act of brewing coffee may have a coffee maker placed not only in the kitchen but also in the dining room, the kitchen and dining room are considered to be the target spaces.

[0073] Laptop use is being recognized in dining rooms, living rooms, and bedrooms as laptops may be used in these spaces.

[0074] Laundry is recognized in the bathroom, assuming that a washing machine will be installed in the bathroom. Reading is likely to be done in a space where people sit, so it is recognized in the living room, dining room, and bedroom. Eating may take place not only in the dining room but also in the living room, so it is recognized in the living room and dining room. Walking is recognized in all spaces, so it is recognized in all spaces.

[0075] The behavior selection unit 102 can select a target behavior corresponding to a space by referring to such a behavior selection table T1, and can therefore easily select a target behavior appropriate for the space.

[0076] 5 is a diagram showing an example of the configuration of the recognizer selection table T2. The recognizer selection table T2 stores a plurality of target actions and the recognizer 111 used to recognize each target action in association with each other. In the recognizer selection table T2, a circle mark indicates a recognizer 111 used for the corresponding target action, and an cross mark indicates a recognizer not used for the corresponding target action. The target actions are the same as the target actions shown in FIG. 4. Pose estimation, object detection, face detection, head direction estimation, age / gender estimation, individual estimation, and tracking are each performed individually by the corresponding recognizer 111.

[0077] For example, since there is no need to identify an individual when wiping, the recognizers 111 for face detection, head orientation estimation, age / gender determination, and individual estimation are not used, and the recognizers 111 for pose estimation, object detection, and tracking are used.

[0078] For sweeping, for the same reasons as for mopping, recognizers 111 for pose estimation, object detection, and tracking are used. For tooth brushing, because individual identification and movement are not important, recognizers 111 other than age / gender estimation, individual estimation, and tracking are used.

[0079] In the "start cooking" step, all of the recognizers 111 are used. For example, since it is assumed that cooking is often done by mothers, in the "start cooking" step, the age / gender estimation and individual estimation recognizers 111 are used to estimate whether the person is the mother or not.

[0080] Brewing coffee is different from starting a meal, and it is assumed that a father or a child may be making it, and movement is not important, so recognizers 111 other than age / gender estimation, individual estimation, and tracking are used.

[0081] When using a laptop, a recognizer 111 other than tracking is used because people rarely move while using the PC. When doing laundry, a recognizer 111 other than tracking is used because the washing machine is installed in a fixed location. When reading, there is no need to identify an individual and there is no movement, so a recognizer 111 other than age / gender estimation, individual estimation, and tracking is used.

[0082] For eating, anyone can do it, head direction is not important, and movement is not expected, so the posture estimation and object detection recognizer 111 is used. For walking, no object is involved, head direction, and individual identification are not important, so the posture estimation and tracking recognizer 111 is used. When detecting unknown behaviors other than those mentioned above, all recognizers 111 are used.

[0083] Next, the effectiveness of the recognizer 111 used to recognize each target behavior will be described.

[0084] Pose estimation is effective for recognizing all target actions because it can detect the posture corresponding to the target action. Furthermore, face detection is required for head orientation estimation. In other words, head orientation detection depends on face detection.

[0085] When wiping, object detection is effective for detecting cleaning tools such as vacuum cleaners, and tracking is effective because cleaning is done while moving around multiple places.

[0086] Sweeping, like mopping, benefits from object detection and tracking.

[0087] When brushing teeth, object detection is effective for detecting the toothbrush and mirror, and head orientation estimation is effective because the head tends to be oriented toward the mirror.

[0088] In the case of "Start cooking," object detection is effective for detecting cooking utensils and ingredients, and head orientation estimation is effective because people tend to face the kitchen or sink. In addition, in the case of "Start cooking," age / gender estimation and individual estimation are effective for identifying individuals who cook frequently, such as mothers who often cook, and tracking is effective because people may move from the sink to the stove in the kitchen.

[0089] In brewing coffee, object detection is effective for detecting the coffee maker and coffee cup, and head orientation estimation is effective for detecting whether the head is facing the coffee maker.

[0090] When using a laptop, object detection is effective for detecting the laptop, head orientation estimation is effective for detecting whether the head is facing the laptop, and age, gender, and individual estimation are effective because laptops are sometimes used for work.

[0091] In the case of laundry, object detection is effective for detecting the washing machine, head orientation estimation is effective for detecting whether the head is facing the washing machine, and age / gender estimation and individual estimation are effective for identifying individuals, such as mothers who often do the laundry.

[0092] When reading, object detection is effective for detecting the book, and head orientation estimation is effective for detecting whether the head is facing the book.

[0093] When it comes to meals, object estimation is effective for detecting food.

[0094] When walking, tracking is effective for capturing continuous movement.

[0095] For actions other than these target actions, all recognizers 111 are selected because it is unclear whether they are effective or not.

[0096] The target behaviors recognized by the behavior recognition device 1 are related to space. For example, "start cooking" is strongly related to the space of a kitchen, and information about this space is useful for behavior recognition. In other words, weighting the target behavior with respect to a specific object or head direction, etc., further improves the detection accuracy of the target behavior. Therefore, in this embodiment, the combiner 121 sets a weight coefficient corresponding to the target behavior to the feature.

[0097] 6 is a diagram showing an example of the weight table T3 that the combiner 121 refers to when setting weight coefficients for feature quantities. The weight table T3 stores a plurality of target actions and the class index, label, space information, object information, head direction information, weight coefficient Wr for the space information, weight coefficient We for the object information, and weighting details for each target action, in association with each other. Here, in addition to starting cooking as exemplified in FIG. 4, mopping is also exemplified as the target actions. Although not shown here, records for each target action as exemplified in FIG. 4 are also registered in the weight table T3.

[0098] The class index is an index uniquely assigned to each target behavior. Here, the class index is assigned as a consecutive number. The label is the label of the target behavior output by the output unit 106. Here, "start cooking" and "mopping" are shown as examples of labels. Although not shown in the figure, the weight table T3 has registered therein the labels of each target behavior shown as an example in FIG. 4. The space information is information about the space in which the target behavior is performed. Here, the space for the target behavior specified in the behavior selection table T1 is registered, such as "kitchen" for "start cooking."

[0099] The object information is information indicating an object related to the corresponding target behavior. For example, the object information for "start cooking" corresponds to at least one of a refrigerator, a microwave oven, a gas stove, and an oven. The head direction information is information indicating the relationship between the object described in the object information and the head direction. For example, the head direction information for "start cooking" means that the head is facing at least one of a refrigerator, a microwave oven, a gas stove, and an oven. Whether the head is facing these objects is determined based on the relationship between the above-mentioned direction vector and the center of gravity of the object.

[0100] The weighting coefficient Wr of the space information is a weighting coefficient for the space in which the target behavior is performed. Here, the weighting coefficient is set to "1" for spaces marked with a circle in the behavior selection table T1, and the weighting coefficient is set to "0" for spaces marked with an x. For example, for "start cooking," since the kitchen is marked with a circle in the behavior selection table T1, the weighting coefficient Wr for the kitchen is "1," and the weighting coefficient Wr for other spaces is "0."

[0101] The weighting coefficient We for object information is a weighting coefficient for the detected object, and is "1" if the corresponding object is detected, and "0" otherwise. If multiple feature amounts are extracted for one object, 1 or 0 is set for each of the multiple feature amounts. The weighting content indicates the content of the weighting coefficient that is set according to the relationship between the detected object and the head direction.

[0102] Weighting by the combiner 121 will be described below using as an example the feature amounts output from the three recognizers 111 shown in FIG. 4 for posture estimation, object detection, and head direction estimation. Here, the vector finally output by the combiner 121 is designated Vo. Also, the feature amounts output by each recognizer 111 are also designated as vectors. Note that the output vector Vo is not limited to a one-dimensional vector, and may be a multidimensional tensor. When no weighting is performed, the output vector Vo is expressed by equation (1).

[0103]

number

[0104] On the other hand, when weighting is performed, the output vector Vo is expressed by equation (2). When weighting is performed, as in the case of no weighting, the output vector Vo' is generated by concatenating the vectors of each recognizer 111. At this time, weighting is performed on the feature amount of each recognizer 111. Therefore, the number of elements of the output vector Vo' is the same as when no weighting is performed.

[0105]

number

[0106] Wr, Wp, Wob, and Wh are the weighting coefficients for the room, posture estimation, object detection, and head direction estimation, respectively. The weighting coefficient Wp is broadcast on the feature Vp, the weighting coefficient Wob is broadcast on the Hadamard product of the weighting coefficient We and the feature Vob, and the weighting coefficient Wh is broadcast on the feature Vh. The broadcast operation is an operation that takes the product of a weighting coefficient (scalar value) and each element of a vector. The weighting coefficient We is the weighting coefficient for the object, and has the same number of elements as the feature Vob.

[0107] For example, suppose the target behavior is starting cooking, the input image data D1 is image data captured in a kitchen, the feature value Vob includes a refrigerator label, and the feature value Vh includes a direction vector pointing toward the refrigerator. In this case, the combiner 121 refers to the weighting table T3 and sets the weighting factor Wr to "1," the weighting factor We to "1," the weighting factor Wh to "5," and the weighting factor Wob to "1."

[0108] For simplicity, the weighting in equation (2) has been described using an example in which there is one target action. When there are two or more target actions, the combiner 121 performs weighting in equation (2) for each target action and inputs the weighted feature amount to the classifier 122. The classifier 122 calculates the likelihood individually for each target action. For example, when recognizing two actions, "start cooking" and "mopping the floor," the combiner 121 sets each weight coefficient in equation (2) to a value corresponding to "start cooking," and then sets the weight coefficient in equation (2) to a value corresponding to "mopping the floor."

[0109] As an alternative method when there are two or more target actions, it is possible to combine the weighting factors by taking the arithmetic mean, arithmetic sum, or logical sum of the weighting factors for each of the cases, for example, "start cooking" and "sweeping."

[0110] Fig. 7 is a flowchart showing an example of a process of generating the list information 205 in the behavior recognition device 1 according to the embodiment. Details of the list information 205 will be described later. The flowchart in Fig. 7 is executed, for example, when the behavior recognition device 1 is installed in a house. The flowchart in Fig. 7 may also be executed, for example, when the floor plan information 202 or the location information 203 is changed.

[0111] First, the first acquisition unit 101 acquires the behavior selection table T1 (target behavior information 201) and stores it in the memory 200 (step S101). Next, the first acquisition unit 101 acquires the floor plan information 202 and stores it in the memory 200 (step S102).

[0112] Next, the second acquisition unit 103 acquires the position information 203 of the image sensor 2 and stores it in the memory 200 (step S103). If there are multiple image sensors 2, the second acquisition unit 103 only needs to acquire the position information 203 of each image sensor 2.

[0113] Next, the behavior selection unit 102 identifies the space in which the image sensor 2 is installed using the position information 203 and the floor plan information 202 (step S104). For example, the behavior selection unit 102 may identify the space by checking in which space, among the multiple spaces included in the floor plan information 202, the coordinate data indicated by the position information 203 is located. Furthermore, when the position information 203 for multiple image sensors 2 is acquired, the behavior selection unit 102 may identify the space corresponding to each piece of position information 203.

[0114] Next, the behavior selection unit 102 refers to the behavior selection table T1 and selects candidate behaviors that are target behaviors corresponding to the identified space (step S105). If multiple spaces are identified, one or multiple candidate behaviors corresponding to each space are selected. If multiple candidate behaviors are selected, a recognizer 111 corresponding to each candidate behavior is selected.

[0115] Next, the recognizer selection unit 110 selects a recognizer 111 corresponding to the selected candidate behavior (step S106). Details of this process will be described later with reference to FIG.

[0116] Next, the recognizer selection unit 110 generates list information 205 that associates the identifier of each image sensor 2, the candidate behavior recognized from the image data 204 captured by each image sensor 2, and the recognizer 111 used to recognize the candidate behavior, and stores the list information 205 in the memory 200 (step S107). In this way, the list information 205 is generated.

[0117] 8 is a flowchart showing details of the process of step S106 in FIG. 7. First, the recognizer selection unit 110 acquires the label of the target action acquired in step S101 (step S201). Next, the recognizer selection unit 110 refers to the recognizer selection table T2 and selects a recognizer 111 to be used to recognize the target action (step S202). Here, if there are multiple target actions, a recognizer 111 to be used for recognition is selected for each target action. Next, the recognizer selection unit 110 selects a dependent recognizer 111 (step S203). Here, which recognizer 111 depends on which other recognizer 111 is determined by referring to the item of recognizer dependency relationship in the prior knowledge table T4, which will be described later.

[0118] Fig. 9 is a flowchart showing an example of the behavior recognition process in the behavior recognition device 1. Fig. 9 shows the process when one frame of image data 204 is captured from one image sensor 2. Therefore, when there are multiple image sensors 2, the process in Fig. 9 is executed in parallel for the image data 204 captured by each image sensor 2. Furthermore, the process in Fig. 9 may be executed each time one frame of image data 204 is captured, or each time multiple frames of image data 204 are captured.

[0119] First, the second acquisition unit 103 acquires the image data 204 captured by the image sensor 2 and stores it in the memory 200 (step S301).

[0120] Next, the recognizer selection unit 110 refers to the list information 205 and selects one candidate action from among the candidate actions associated with the identifier of the image sensor 2 that captured the image data 204 (step S302). As a result, a candidate action corresponding to the space is selected.

[0121] Next, the recognizer selection unit 110 refers to the list information 205, inputs the image data 204 to the recognizer 111 used to recognize one candidate behavior, and causes the recognizer 111 to calculate a feature amount (step S303). The calculated feature amount is input to the combiner 121.

[0122] Next, the combiner 121 weights the input feature amount with a weighting coefficient corresponding to one candidate behavior (step S304).

[0123] Next, the combiner 121 concatenates the weighted features (step S305). The concatenated features are input to the classifier 122. Next, the classifier 122 calculates the likelihood of the input features (step S306).

[0124] Next, the recognizer selection unit 110 determines whether all candidate actions have been selected (step S307). If all candidate actions have not been selected (NO in step S307), the process returns to step S302, and the likelihood of the next selected candidate action is calculated.

[0125] On the other hand, if all candidate actions have been selected (YES in step S307), the output unit 106 outputs the label of the candidate action with the maximum likelihood among the likelihoods calculated for each candidate action as the recognition result (step S308). Note that in step S308, instead of just the label of the candidate action with the maximum likelihood, labels of k candidate actions in descending order of likelihood may be output. For example, if k is 5, the top 5 classes are output.

[0126] As described above, according to this embodiment, a candidate behavior to be recognized is selected from the target behaviors based on the position information 203 of the image sensor 2 and the floor plan information 202, and a recognition result for the candidate behavior is calculated. Therefore, it becomes possible to accurately recognize human behavior that depends on space.

[0127] (Selection of recognizer) Next, we will explain how to select a recognizer. Conventional behavior recognition methods using DNNs can be broadly divided into two types of approaches.

[0128] The first approach is an activity recognition method that uses multiple general-purpose feature extraction layers, such as convolutional layers and pooling layers, to extract activity features from input data. In this activity recognition method, the likelihood of a given activity is calculated from the extracted features. However, this activity recognition method is not an excellent approach in terms of both computational cost and recognition accuracy.

[0129] The second approach is an activity recognition method that heuristically designs features that contribute to the activity to be recognized, and calculates the likelihood of a given activity from the extracted features, as in the first approach. A detailed explanation of this heuristic design is provided below.

[0130] In the field of image processing, skeletal information (representation of joints such as shoulders and knees with straight lines) is used to represent people, and bounding boxes are widely used to represent the position of objects. For this reason, heuristic action recognition methods using the aforementioned skeletal information or bounding boxes have been proposed. However, neither skeletal information nor bounding box representations were designed as features for action recognition. Therefore, action recognition methods are determined based on the results of heuristic trial and error. Furthermore, because training and evaluation of DNNs requires a significant amount of time, the conditions under which they are tried are limited. The reason for implementing such heuristic trial and error is that the actions to be recognized are the same or similar to conventional recognition targets, and the features used in conventional recognition can be used as reference. In particular, when using public datasets for training, the actions to be recognized are classes defined in the dataset.

[0131] Furthermore, conventional DNN-based action recognition methods are not effective even when the action to be recognized is not included in the existing recognition targets. Therefore, feature extraction that is appropriate for the action to be recognized is necessary for efficient action recognition. To achieve this, a configuration is required that can determine the feature extraction method when given training data for the action to be recognized.

[0132] DNN-based action recognition calculates the likelihood of a given action by processing sensor data with a DNN. When obtaining a label for a single action, the action label with the highest likelihood is selected. When calculating the likelihood of each action from multiple DNNs or feature extractors, there is no guarantee that the recognition accuracy will be higher as the amount of feature data increases. This is because the features are more likely to contain components that behave like noise and do not contribute to recognition accuracy. Similarly, increasing the number of DNNs or feature extractors used does not necessarily improve accuracy. In other words, it is important to set the number of DNNs or feature extractors to a number that will obtain information appropriate for the action to be recognized.

[0133] When considering computational cost, it is important to select the configuration with the lowest computational cost from among the configurations that achieve the desired recognition accuracy. However, because typical DNNs or feature extractors include nonlinear transformations in their internal processing, recognition accuracy cannot be calculated analytically. In other words, the only way to obtain recognition accuracy for a certain configuration is to prepare a dataset and actually perform the recognition process. The computational cost required to achieve such recognition accuracy is high, and only a limited number of conditions can be tried.

[0134] As mentioned above, in order to use DNN-based action recognition, which has a high computational cost, for real-time processing, the configuration of the DNN or feature extractor used is important.

[0135] The computational cost of a DNN layer (convolutional layer) that processes convolution is explained below. Similar to convolution processing in mathematics, a convolutional layer obtains output by applying a given kernel to input image data. There are two types of convolutional layers: two-dimensional convolution, which is applied to one frame of two-dimensional image data, and three-dimensional convolution, which is applied to N frames of two-dimensional image data (N is the number of frames). The number of parameters Wc for the convolution weighting coefficients can be expressed by equation (3), where k is the kernel size, d is the number of convolution dimensions (2 for two-dimensional convolution, 3 for three-dimensional convolution), Ci is the number of input channels, and Co is the number of output channels.

[0136]

number

[0137]

number

[0138]

number

[0139]

number

[0140] The high computational cost of conventional DNNs is due to the fact that the layers constituting the network are primarily arranged in series, with most of them being convolutional layers. The intermediate layers of the network also have a large number of weight coefficients and an increasing number of multiplications. The output of the intermediate layers is a tensor that is difficult for humans to interpret, and the output of the intermediate layers is not necessarily an optimal representation for recognition. In other words, the intermediate layers contain more weight coefficients than necessary, and the number of multiplications is also increasing. In other words, by effectively representing human behavior or spatial states in the intermediate layers with fewer parameters, it is possible to reduce the number of weight coefficients and the number of multiplications.

[0141] When considering identifying human behavior, it is possible to infer behavior from a person's skeletal information rather than from input image data, which is dense data. For example, the action of cutting ingredients with a knife involves repeated, characteristic up and down movements of the hand. The number of data bytes required to represent input image data is the product of the vertical and horizontal lengths of the image data and the number of channels, in the case of 8-bit image data. On the other hand, two-dimensional skeletal information can be represented by 34, which is the product of the number of vertices of the skeleton (for example, 17 points) and the number of components of the coordinate data (x, y) of the vertices of the skeleton.

[0142] Considering the above, a method for selecting the recognizer 111 in the embodiment of the present invention will be described below. Human actions to be recognized include, for example, actions such as walking, running, standing, talking, cleaning, and starting to cook, as well as stationary actions such as sleeping, lying down, sitting, and watching television.

[0143] First, the large number of options for the recognizer 111 will be described. The number of recognizers 111 used in the first behavior recognition unit 104 is at least one. When the first behavior recognition unit 104 holds N recognizers 111 and selects M recognizers 111, the combinations of the recognizers 111 will be on the factorial order of N. A factorial order of N diverges faster than an exponential function. Considering the learning time, if N is greater than about 5, it becomes difficult to learn all combinations of the recognizers 111.

[0144] Next, the problem regarding the combination of recognizers 111 is expressed as a mathematical problem. For each combination of selected recognizers 111, learning is performed to obtain the recognition accuracy and computational cost. Given a required recognition accuracy (for example, 70%), searching for a combination that minimizes the computational cost while still satisfying that recognition accuracy becomes a conditional combinatorial optimization problem. Such a combinatorial optimization problem cannot be solved analytically when the number of combinations becomes too large to perform an exhaustive search. Therefore, a suboptimal solution is sought. Note that there is no universal algorithm for combinatorial optimization problems.

[0145] Therefore, in order to efficiently search for the selection of the recognizer 111, the recognizer 111 is selected based on prior knowledge and a greedy method.

[0146] First, the prior knowledge will be explained. FIG. 10 shows the data structure of a prior knowledge table T4 that summarizes information about the recognizer 111. This table is stored in the memory 200. The prior knowledge table T4 has items related to the identification number of the recognizer 111, the recognition content, the input sensor data, the relative calculation cost, and the dependency relationship between the recognizers. Each item will be explained. The identification number is a unique number (e.g., a consecutive number) assigned to each registered recognizer 111, and serves as an index or hash in combination search. The recognition content is the content recognized by the recognizer 111. The input sensor data is sensor data required for the recognizer 111 to make inference, and is 0 or more. The number includes 0 because a recognizer 111 that makes inference using only features calculated by other dependent recognizers 111 does not require sensor data. The relative calculation cost is a relative value of the calculation cost of the recognizer 111. The relative calculation cost is a relative value calculated based on benchmark results executed on a reference computer. The relative calculation cost may be calculated based on the benchmark results of a plurality of computers, or the absolute value of the benchmark result itself may be used.

[0147] The dependency relationship of a recognizer is information indicating whether or not it depends on other recognizers 111. For example, recognizers 111 with identification numbers 4 to 6 depend on recognizer 111 with identification number 3. This is because the results of face detection are used when head direction estimation, age / gender estimation, and individual estimation are performed.

[0148] Note that a recognizer 111 that detects a specific region of a person, such as a recognizer 111 that detects a hand or a foot, may be used. Also, a recognizer 111 that outputs dense data such as a mask that indicates the range of an image may be used.

[0149] Next, the greedy method will be described. The greedy method is a technique for making the best selection based on partial information. When selecting a recognizer 111, first, for each recognizer 111 included in the prior knowledge table T4, the recognition accuracy when only one recognizer 111 is selected is calculated using training data. If the recognition accuracy when using only one recognizer 111 is high, it is expected that the recognition accuracy when that recognizer 111 is used in combination with other recognizers 111 will also be high. However, this is not guaranteed. In the greedy method, if the recognition accuracy when using only one recognizer 111 is high, it is assumed that the recognition accuracy when that recognizer 111 is used in combination with other recognizers 111 will also be high, and the priority of the combination of recognizers 111 to be evaluated is determined. Specifically, the recognizers 111 are sorted in descending order of the recognition accuracy when using only one recognizer 111, and the top recognizer 111 is evaluated first. For example, if the first recognizer 111 is placed at the top and indexes are assigned in order, the indexes closest to the top are used preferentially, such as 1 and 2, 1 and 3, 1 to 3, and 1 to 4. A more detailed index selection method can be expressed by a parameter X that takes into account the first to X-th indexes, and a parameter Y that allows the selection of up to Y indexes. The parameters X and Y are determined according to the performance of the computer that executes the inference, and one example is X=4 and Y=3. The recognizer selection table T2 shown in FIG. 5 is created based on the selection results of the recognizers 111 as described above, for example.

[0150] (Classifier selection) Next, the selection of the classifier 122 will be described. The feature quantities output by the first behavior recognition unit 104 are data that vectorizes a small number of parameters, such as skeletal information, and therefore can be recognized without using a CNN. The classifier 122 requires a significantly smaller amount of computational processing than a CNN. Cross-validation is performed on multiple classifier candidates, and the classifier 122 that maximizes accuracy or F1 score can be selected. The candidate classifier 122 is, for example, a classifier that uses logistic regression, support vector machine (SVM), decision tree, random forest, k-nearest neighbor algorithm, Gaussian naive Bayes, perceptron, or stochastic descent.

[0151] (Classifier training) Next, learning of the classifier 122 will be described. The classifier 122 can be trained using training data including labels of target behaviors. For example, video image data that serves as training data is prepared for each of the labels "start cooking" and "not started cooking." Using the prepared image data, the behavior recognition device 1 executes the behavior recognition process shown in FIG. 9, compares the obtained label with the correct label of the training data, and updates the weights of the classifier 122 so as to minimize the error.

[0152] (simulation) Next, a simulation performed to confirm that the behavior recognition device 1 requires less computational processing than existing behavior estimation AI will be described.

[0153] In this simulation, a CNN was used as the existing recognizer 111. Specifically, "1: PoseNet" was used as the pose estimation recognizer 111, "2: SSD" was used as the object detection recognizer 111, "3: RetinaFace" was used as the face detection recognizer 111, and "4: DeepHeadPose" was used as the head direction estimation recognizer 111. A probabilistic descent classifier was used as the classifier 122. Meanwhile, "5: RepresentationFlowNet" was used as the existing behavior estimation AI.

[0154] The input image data used was a single color image data containing three people. This image data had a resolution of 640 x 480, three channels, and an 8-bit resolution. The calculation cost of the classifier 122 was ignored because it was sufficiently small compared to the existing recognizer 111, regardless of the number of classes to be identified.

[0155] The computer used was a desktop personal computer with calculation assistance provided by a graphics processing unit (product model number: Geforce GTX 1080Ti).

[0156] FIG. 11 is a table summarizing the processing time per frame when the existing recognizers 111 are executed individually. In the simulation, the processing time was measured when the recognizers 111 of process A (pose estimation), process B (object detection), and process C (head direction detection) in FIG. 11 were executed. The processing time of process C includes the processing time for head direction estimation and face detection. Processes A, B, and C are independent of each other, so they can be executed in parallel. Ignoring overhead, the maximum processing time of processes A, B, and C was 0.0455 seconds for process C. On the other hand, the total processing time when processes A, B, and C were processed sequentially was 0.0725 seconds.

[0157] In contrast, the processing time for process D (existing behavior estimation AI) was 0.1429 seconds. Therefore, the processing speed when processes A, B, and C were processed in parallel was three times faster than process D. The processing speed when processes A, B, and C were processed sequentially was twice as fast as process D. In this way, it was confirmed that using multiple recognizers 111 results in significantly faster processing speed than existing behavior estimation AI.

[0158] Next, a simulation performed to evaluate the recognition accuracy of the behavior recognition device 1 will be described. In this simulation, image data of five videos shown in FIG. 12 was used as input image data. FIG. 12 is a table summarizing the input image data. Video IDs are identifiers for identifying the five videos. FIG. 12 shows the total number of frames of "start cooking" and "not started cooking" for each video.

[0159] FIG. 13 is a table summarizing the results of a simulation performed to evaluate the recognition accuracy of the behavior recognition device 1. In FIG. 13, for example, Process A+B+C indicates a case in which the first behavior recognition unit 104 is configured with a recognizer 111 for Process A, a recognizer 111 for Process B, and a recognizer 111 for Process C. For example, Process A+B indicates a case in which the first behavior recognition unit 104 is configured with a recognizer 111 for Process A and a recognizer 111 for Process B. In this simulation, the recognition accuracy and processing time of each of eight types of classifiers 122 were measured for each combination of Processes A to C shown in FIG. 13. The eight types of classifiers 122 were logistic regression, support vector machine, decision tree, random forest, k-nearest neighbor algorithm, Gaussian naive Bayes, perceptron, and stochastic descent. In this simulation, the processing time was shortest and the recognition accuracy was highest when the classifier 122 for the stochastic descent method was used among the eight types of classifiers 122.

[0160] Furthermore, when the probabilistic descent classifier 122 was used, the average value of the recognition accuracy for each combination of processes A to C shown in Fig. 13 was 88.5%. This result indicates that the recognition accuracy of the behavior recognition device 1 is higher than the expected value of 50% of the recognition accuracy when the two-class classification of "start cooking" and "not started cooking" is randomly estimated. Therefore, this simulation result confirmed that the behavior recognition device 1 can recognize behaviors.

[0161] Of the combinations shown in Figure 13, the best recognition accuracy was 88.8% for Process A+C. However, Process A alone also achieved a recognition accuracy of 88.7%. If the recognition accuracy threshold is set to 88.7%, using only Process A minimizes the computational cost. Process A+B+C is not the best because it includes a recognizer 111 that does not contribute to improving accuracy, and this recognizer 111 behaves like a noise source.

[0162] The above simulation results are merely an example, and the recognizers 111 constituting the first behavior recognition unit 104 are not limited to those described above, and an optimal recognizer 111 may be adopted as appropriate depending on the target behavior. Also, a classifier 122 other than a probabilistic descent classifier may be adopted as the classifier 122.

[0163] (Variation)

[0164] (1) In the above embodiment, the behavior recognition device 1 is configured as a single device. However, the present disclosure is not limited to this and may be configured as a plurality of devices. FIG. 14 is a block diagram showing an example of the configuration of a behavior recognition device 1A according to a modified example of the present disclosure. The behavior recognition device 1A is configured as a server. The behavior recognition device 1A further includes a communication unit 500. The communication unit 500 transmits the recognition result output by the output unit 106 to the home appliance 600 via the network NT and the gateway 700. The communication unit 500 receives image data captured by the image sensor 300 installed in the home. The network NT is, for example, a wide area communication network such as the Internet. The gateway 700 is installed in the home and connects the image sensor 300 and the home appliance 600 to the network NT. The home appliance 600 is a washing machine, a microwave oven, a television, or the like. The home appliance 600 executes control using the behavior recognition result transmitted from the behavior recognition device 1A and displays the recognition result. In this way, even when the behavior recognition device 1A is configured as a server, it can recognize the behavior of people in the home.

[0165] (2) In the above embodiment, image data was used as input data. In addition, at least one of thermal imaging data, depth image data, audio data, room temperature data, humidity data, illuminance data, and radio wave data may also be used.

[0166] (3) When recognizing human behavior within a home using cameras installed within the home, the camera's installation position and angle are expected to be fixed or changed infrequently. Furthermore, the camera's viewing angle is approximately 110° for wide-angle lenses and less for narrow-angle lenses, and the area captured by a given camera is a portion of the space in which it is installed. Therefore, it is expected that the camera will be installed so as to capture a space where the behavior to be recognized occurs frequently or where the occurrence is of high importance. Under these assumptions, if the space in which the camera is installed and the objects that are or may be captured by the camera can be known in advance, the accuracy of behavior recognition can be improved. For example, behaviors with a zero occurrence frequency can be excluded from recognition targets. Furthermore, if a sofa, which is infrequently moved, is captured on the camera, it is expected that the behavior of sitting on it occurs frequently. Furthermore, if a movable chair is present in the space, it is expected that the behavior of sitting on the chair occurs frequently. This makes it possible to determine the values ​​of the weight coefficients in the weight table T3 shown in Figure 6.

[0167] In indoor activity recognition, the interaction between a person and an object, such as direct contact between the person and the object or a person being in the vicinity of the object, is important. When the location of an object (the first object described below) that involves movement, opening / closing, or manipulation is known, the positional relationship or orientation of the person relative to the object becomes a basis for activity recognition. This is based on the fact that people often move, open / close, and manipulate objects when they are in action.

[0168] Immovable equipment in the home, such as plumbing fixtures (the second object described below), is also important for activity recognition. Actions that require water, such as cooking or washing, usually involve the use of equipment installed in the home. Therefore, the relative positions of plumbing fixtures and people are important factors for activity recognition. This is based on the fact that actions that require water are frequent in people's lives.

[0169] A home is generally divided into about 4 to 10 rooms by doors or fixed or movable walls. Furthermore, multi-story homes are equipped with elevators such as stairs. Therefore, a home has multiple spaces and objects containing numerous doors (third objects, described below). The names of these spaces, the locations of the doors or elevators, and the relative positions of people are used as information for activity recognition. This is based on the fact that some activities, such as cooking or bathing, can only be performed in specific rooms.

[0170] As described above, information about where objects or facilities are located and how spaces are divided is important for behavior recognition. Conventionally, there have been techniques for performing behavior recognition that take into account space names such as "living room" or "bedroom." However, the same space names are used in a variety of ways, and the space names "living room" or "bedroom" alone are insufficient as information for behavior recognition. Therefore, it is important to consider what kind of interactions between people and objects may occur in which spaces. Taking these points into consideration, a modified example (3) of the present disclosure will be described below.

[0171] 15 is a block diagram showing an example of the configuration of a behavior recognition device 1B according to Modification (3) of the present disclosure. In this modification, floor plan features are extracted based on floor plan information 202, and a behavior selection table T1 is generated based on the extracted floor plan features. In this modification, the floor plan information 202 includes information on spaces in the building as well as information (type information and location information) on objects (facilities and equipment) installed in each space.

[0172] The behavior recognition device 1B is communicatively connected to the display terminal 400 via a predetermined communication path. Wireless LAN or Bluetooth (registered trademark) can be used as the predetermined communication path. The predetermined communication path may also be the Internet. The display terminal 400 is configured, for example, by a smartphone or a tablet computer. The display terminal 400 is owned, for example, by a user. The user is, for example, a person who installs the image sensor 2. The installer is, for example, a resident of the residence or an installer.

[0173] 1, the processor 100B of the behavior recognition device 1B further includes an installation support unit 301 and a floor plan feature extraction unit 302. Based on the floor plan information 202, the floor plan feature extraction unit 302 extracts objects installed in a building, classifies the extracted objects into one of a first object that is movable, a second object that is a plumbing fixture, and a third object that is a building structure, and extracts, for each classified object, floor plan features that associate the installation position with classification information that indicates the classification result. The floor plan features are configured, for example, as a two-dimensional table.

[0174] The first object includes, for example, movable objects such as furniture and electrical appliances, etc. Specifically, the first object includes a vacuum cleaner, a coffee maker, a laptop computer, a chair, a sofa, etc.

[0175] The second object may be an immovable water fixture such as a sink or washstand.

[0176] The third object is a building structure such as an entrance hall, kitchen, living room, dining room, bedroom, bathroom, elevator, and washroom.

[0177] The installation position is configured, for example, as two-dimensional or three-dimensional coordinate data based on the entrance of the building. The installation position of the third object may be configured as coordinate data indicating the area in which the third object is located. This makes it possible to determine in which space within the building the first object and the second object are to be installed. The installation positions of the first object and the second object may include the names of the spaces in the building in which the first object and the second object are located, in addition to the coordinate data in which the first object and the second object are located.

[0178] The classification information is information indicating which of the first object, second object, and third object an object extracted from the floor plan information 202 corresponds to.

[0179] The floor plan feature quantity extraction unit 302 generates a behavior selection table T1 based on the extracted floor plan feature quantities.

[0180] For example, the installation location of the first object indicates the location where the user will perform an action. For example, the installation location of a coffee maker can be associated with the action of making coffee. For example, the installation location of a microwave oven can be associated with the action of starting to cook.

[0181] Furthermore, since the second object is a plumbing fixture, the installation position of the second object indicates the location where an action using water is performed. For example, the installation position of a sink can be associated with the action of washing dishes.

[0182] The installation position of the third object indicates the name of the space, the user's entry and exit into the space, and the range of movement that the user can make. For example, the installation position of a bathroom door can be associated with the bathing action.

[0183] The floor plan feature extraction unit 302 generates an action selection table T1 based on the extracted floor plan feature. The action selection table T1 is a table in which one or more spaces in a building are associated with target actions that the user is likely to perform in each space, as shown in Figure 4.

[0184] Here, the floor plan feature extraction unit 302 determines what kind of devices or facilities are installed in each space from the floor plan features, and generates the behavior selection table T1 by referring to predetermined rules from the determination results and the classification information. As the rules, for example, rules that associate devices or facilities with target behaviors that are likely to be performed with the devices or facilities, such as heating food for a microwave oven as a first object, brewing coffee for a coffee maker as a first object, and washing dishes for a sink as a second object, can be adopted.

[0185] For example, suppose a kitchen is equipped with a microwave oven, a coffee maker, and a sink. In this case, the floor plan feature quantity extraction unit 302 generates an action selection table T1 in which target actions such as starting cooking, making coffee, and washing dishes are associated with each other for the kitchen. The floor plan feature quantity extraction unit 302 may also associate target actions with other spaces in the building in a similar manner.

[0186] Furthermore, the rules may include target actions that are likely to be performed not only on devices or facilities but also on the spaces themselves. For example, as illustrated in Fig. 4, rules such as starting cooking in the kitchen, doing laundry in the bathroom, reading in the living room, dining room, and bedroom, and eating in the living room and dining room can be adopted.

[0187] Furthermore, the floor plan feature extraction unit 302 may set weighting factors listed in weighting table T3 shown in Fig. 6 from the extracted floor plan features. For example, if a kitchen is equipped with a sink, refrigerator, microwave oven, and gas stove, the weighting factor We for the kitchen may be set to 1 for the sink, refrigerator, microwave oven, and gas stove, and 0 for the other appliances.

[0188] The floor plan feature quantity extraction unit 302 may set the weighting factor We for spaces other than the kitchen in the same manner as for the kitchen.

[0189] The first acquisition unit 101 may acquire the behavior selection table T1 generated in this manner as the target behavior information.

[0190] The installation support unit 301 further includes an installation support unit that acquires the name of the space in the building in which the image sensor 2 is to be installed via the display terminal 400 and outputs installation guidance to the display terminal 400 for installing the image sensor 2 so that specific equipment or specific facilities related to the space are included within the field of view of the image sensor 2.

[0191] Fig. 16 is a diagram showing a scene in which the image sensor 2 is installed at the entrance 501. Fig. 17 is a diagram showing an example of interaction between the user and the display terminal 400 in the scene in which the image sensor 2 is installed at the entrance 501.

[0192] In step ST1, the display terminal 400 accepts an operation to open a setting screen from the user and displays a list of spaces in which the image sensor 2 is planned to be installed. Here, the names of the spaces in which the image sensor 2 will be installed are displayed in a list, such as A: entrance, B: kitchen, C: living room, etc.

[0193] In step ST2, the display terminal 400 accepts a user operation to select the name of the space in which the image sensor 2 is to be installed from the names of the spaces displayed in the list. The display terminal 400 transmits the name of the selected space to the behavior recognition device 1B. In this case, since the image sensor 2 is to be installed at the entrance 501, "A: Entrance" is selected.

[0194] In step ST3, the display terminal 400 displays a message such as "Please install it in a position where the door is visible. After installation, please press OK," encouraging the user to install the image sensor 2. Along with this message, an OK button is displayed on the setting screen so that the user can notify the display terminal 400 that the installation work of the image sensor 2 has been completed.

[0195] By this installation work, the image sensor 2 is installed at the entrance 501 as shown in FIG.

[0196] In step ST4, the display terminal 400 accepts the user's operation of pressing the OK button. The display terminal 400 that has accepted the operation of pressing the OK button transmits information indicating that the OK button has been pressed to the behavior recognition apparatus 1B.

[0197] The installation support unit 301, which has acquired the information indicating that the OK button has been pressed, acquires image data captured by the image sensor 2 and executes a process of detecting the door 401 from the acquired image data. At this time, the installation support unit 301 causes the display terminal 400 to display a message saying "Door is being automatically detected" (step ST5). Here, the installation support unit 301 may detect the door 401 using a predetermined recognizer. The door 401 is an example of specific equipment in the entrance 501. The specific equipment is determined in advance depending on the space in which the image sensor 2 is installed. As the predetermined recognizer, for example, a recognizer prepared in advance for detecting the door 401 is adopted.

[0198] Fig. 18 is a diagram showing the interaction following Fig. 17. In step ST6, the display terminal 400 acquires annotation information indicating the detection result of the door 401 from the installation support unit 301, and displays a setting screen on which the annotation image indicated by the acquired annotation information is superimposed on the image captured by the image sensor 2. This setting screen displays a message urging the user to correct the annotation image, such as "Adjust so that it matches the door at the position of the automatically detected red door guide, and press OK." The annotation information is, for example, coordinate data of the annotation image.

[0199] 19 is a diagram showing an example of setting screens G1 and G2 on which annotation image A1 is superimposed. Setting screen G1 shows annotation image A1 before correction, and setting screen G2 shows annotation image A1 after correction. Annotation image A1 is configured with a rectangular bounding box. Annotation image A1 has a predetermined color (here, red). An OK button 1901 is displayed on setting screens G1 and G2 to notify the completion of the correction work on annotation image A1.

[0200] A circle mark P1 is displayed at the top left corner of the annotation image A1, and a circle mark P2 is displayed at the bottom right corner of the annotation image A1.

[0201] When the display terminal 400 receives an operation to move the circle marks P1 and P2, it changes the size of the annotation image A1 in conjunction with the received operation.

[0202] The setting screen G1 displays an annotation image A1, which is the result of the recognizer detecting the door 401. This annotation image A1 is shifted diagonally upward and to the left from the door 401, and it can be seen that the door 401 has not been correctly recognized.

[0203] The user inputs an operation to move the circles P1 and P2 so that the shape of the annotation image A1 becomes the circumscribing rectangle of the door 401. This results in a setting screen G2. In the setting screen G2, the annotation image A1 is positioned to the circumscribing rectangle of the door 401.

[0204] As a result, the annotation image A1, which was displayed at a position offset from the door 401 on the setting screen G1, is corrected to surround the entire area of ​​the door 401 as shown on the setting screen G2. As a result, even if the recognizer is unable to accurately recognize the position of the door 401 on the image, the recognizer can obtain the correct position of the door 401 on the image.

[0205] In step ST7, the display terminal 400 receives an operation of pressing the OK button 1901 from the user who has completed the adjustment of the annotation image A1. Upon receiving this operation, the display terminal 400 transmits the coordinate data of the corrected annotation image A1 to the installation support unit 301. Upon receiving this coordinate data, the installation support unit 301 stores the coordinate data of the corrected annotation image A1 in the memory 200 in association with the position information 203 of the image sensor 2 installed at the entrance 501.

[0206] Thereafter, the recognizer 111 for recognizing the target behavior corresponding to the entrance 501 recognizes the user's behavior based on the annotation image A1 indicated by the coordinate data for the image data captured by the image sensor 2 installed at the entrance 501. This allows the recognizer 111 to efficiently recognize the user's behavior.

[0207] In step ST8, the display terminal 400 receives information from the installation support unit 301 indicating that the coordinate data has been saved, and therefore displays a message saying "Settings saved."

[0208] In step ST9, the display terminal 400 accepts an operation to close the setting screen from the user, and thereby the display terminal 400 closes the setting screen.

[0209] Fig. 20 is a diagram showing a scene in which the image sensor 2 is installed in the kitchen 502. Fig. 21 is a diagram showing an example of interaction between the user and the display terminal 400 in the scene in which the image sensor 2 is installed in the kitchen 502.

[0210] Step ST1 is the same as step ST1 in Fig. 17. In step ST2, the display terminal 400 receives an operation from the user to select "B: Kitchen" as the installation location of the image sensor 2.

[0211] Step ST3 is the same as step ST3 in Fig. 17. In step ST4, the display terminal 400 accepts the user's operation of pressing the OK button. The display terminal 400 that has accepted the operation of pressing the OK button transmits information indicating that the OK button has been pressed to the behavior recognition apparatus 1B.

[0212] Having acquired the information indicating that the OK button has been pressed, the installation support unit 301 acquires the image data captured by the image sensor 2 and executes a process to detect the refrigerator 402 from the acquired image data. As a result, the installation support unit 301 causes the display terminal 400 to display a message saying "Refrigerator is being automatically detected" (step ST5). Here, the installation support unit 301 may detect the refrigerator 402 using a predetermined recognizer. The refrigerator 402 is an example of a specific appliance in the kitchen 502. The specific appliance is determined in advance depending on the space in which the image sensor 2 is installed. As the predetermined recognizer, for example, a recognizer prepared in advance to detect the refrigerator 402 is used.

[0213] FIG. 22 is a diagram showing the interaction following FIG. 21. Steps ST6 to ST9 are the same as those in FIG. 18. FIG. 23 is a diagram showing an example of setting screens G3 and G4 on which annotation image A1 is superimposed. Setting screen G3 displays annotation image A1 before correction, and setting screen G3 displays annotation image A1 after correction. Setting screen G3 displays annotation image A1, which is the detection result of refrigerator 402 by the recognizer. This annotation image A1 is shifted upward from refrigerator 402, and it can be seen that refrigerator 402 has not been correctly recognized. The user inputs an operation to move circles P1 and P2 so that the shape of annotation image A1 becomes the circumscribed rectangle of refrigerator 402. This results in setting screen G4. In setting screen G4, annotation image A1 is positioned to the circumscribed rectangle of refrigerator 402.

[0214] As a result, annotation image A1, which was displayed at a position offset from refrigerator 402 on setting screen G3, is corrected to surround the entire refrigerator 402, as shown on setting screen G4. As a result, even if the recognizer is unable to accurately recognize the position of refrigerator 402 on the image, the correct position of refrigerator 402 on the image can be obtained.

[0215] Thereafter, the recognizer 111 for recognizing the target behavior corresponding to the refrigerator 402 recognizes the user's behavior based on the annotation image A1 indicated by the coordinate data for the image data captured by the image sensor 2 installed in the refrigerator 402. This allows the recognizer 111 to efficiently recognize the user's behavior.

[0216] 24 is a diagram showing an example of interaction between the user and the display terminal 400 when correcting the annotation image A1 after installation of the image sensor 2. After installation, some external force may be applied to the image sensor 2, causing the angle to change. In this case, the coordinate data indicated by the annotation image A1 set at the time of installation will be shifted from the position on the image of the specific device or specific facility. To correct this shift, the following correction work is performed.

[0217] In step ST1, the display terminal 400 accepts an operation from the user to open the setting screen, and displays a message prompting the user to check for misalignment of the image sensor 2. Here, the message displayed reads, "Please check that the camera position is correct. The red guide is the current setting. The light blue guide is the automatically detected position."

[0218] 25 shows examples of setting screens G5 and G6 with annotation image A1 superimposed. Setting screen G5 shows annotation image A1 before correction, and setting screen G6 shows annotation image A1 after correction. Annotation image A1 corresponds to the red guide. Annotation image A2, which is a dotted line, shows the annotation image that is the detection result of refrigerator 402 by the recognizer. Annotation image A2 corresponds to the light blue guide.

[0219] On the setting screen G5, the angle of the image sensor 2 has shifted from the angle at the time of installation, so the annotation image A1 is displayed shifted upward relative to the annotation image A2.

[0220] In step ST2, the user checks the deviation of the angle of the image sensor 2 on the setting screen G5 and performs the adjustment of the angle of the image sensor 2.

[0221] The display terminal 400 determines by image processing whether the position of the annotation image A1 matches the position of the annotation image A2. If the display terminal 400 determines that they match, it displays a message indicating that the angle of the image sensor 2 has returned to the angle at the time of installation.

[0222] In step ST4, the display terminal 400 accepts an operation to close the setting screen from the user, and thereby the display terminal 400 closes the setting screen.

[0223] By having the user perform such an adjustment operation, even if the angle of the image sensor 2 changes after installation, the angle of the image sensor 2 can be returned to the angle at the time of installation, thereby enabling the recognizer 111 to accurately recognize the user's behavior. [Industrial Applicability]

[0224] The behavior recognition device of the present invention is useful for recognizing human behavior in a building.

Claims

1. An activity recognition device that recognizes human activity in a building, a first acquisition unit that associates a plurality of predetermined spaces with one or more target actions to be recognized for each of the spaces, and acquires target action information, floor plan information of the building, and position information of an image sensor installed in the building; a behavior selection unit that selects a candidate behavior to be recognized from the one or more target behaviors included in the target behavior information based on the position information of the image sensor and the floor plan information; a second acquisition unit that acquires image data detected by the image sensor; a first behavior recognition unit that determines one or more recognizers according to the candidate behaviors and calculates feature quantities of the image data using the one or more recognizers; a second behavior recognition unit that recognizes the candidate behavior based on the feature amount; an output unit that outputs a recognition result by the second behavior recognition unit, the first behavior recognition unit determines a plurality of recognizers when the candidate behavior is a predetermined behavior; the second behavior recognition unit combines the feature amounts calculated by the plurality of recognizers and recognizes the candidate behavior based on the combined feature amounts. Behavior recognition device.

2. The predetermined activity is cleaning, brushing teeth, cooking, doing laundry, using a computer, reading, or eating. The behavior recognition device according to claim 1 .

3. A behavior recognition device that recognizes human behavior in a building, a first acquisition unit that associates a plurality of predetermined spaces with one or more target actions to be recognized for each of the spaces, and acquires target action information, floor plan information of the building, and position information of an image sensor installed in the building; a behavior selection unit that selects a candidate behavior to be recognized from the one or more target behaviors included in the target behavior information based on the position information of the image sensor and the floor plan information; a second acquisition unit that acquires image data detected by the image sensor; a first behavior recognition unit that determines one or more recognizers according to the candidate behaviors and calculates feature quantities of the image data using the one or more recognizers; a second behavior recognition unit that recognizes the candidate behavior based on the feature amount; an output unit that outputs a recognition result by the second behavior recognition unit, Each recognizer consists of a convolutional neural network, the second behavior recognition unit recognizes the candidate behavior using a classifier using any one of a logistic regression, a support vector machine, a decision tree, a random forest, a k-nearest neighbor algorithm, a Gaussian naive Bayes algorithm, a perceptron, and a stochastic descent algorithm; Behavior recognition device.

4. A behavior recognition device that recognizes human behavior in a building, a first acquisition unit that associates a plurality of predetermined spaces with one or more target actions to be recognized for each of the spaces, and acquires target action information, floor plan information of the building, and position information of an image sensor installed in the building; a behavior selection unit that selects a candidate behavior to be recognized from the one or more target behaviors included in the target behavior information based on the position information of the image sensor and the floor plan information; a second acquisition unit that acquires image data detected by the image sensor; a first behavior recognition unit that determines one or more recognizers according to the candidate behaviors and calculates feature quantities of the image data using the one or more recognizers; a second behavior recognition unit that recognizes the candidate behavior based on the feature amount; an output unit that outputs a recognition result by the second behavior recognition unit, the second behavior recognition unit recognizes the candidate behavior using a classifier that has been machine-learned using the feature amounts as explanatory variables and the target behavior as a target variable. Behavior recognition device.

5. the second behavior recognition unit weights each feature amount using a weighting coefficient that is predetermined according to the candidate behavior, and recognizes the candidate behavior based on each weighted feature amount. The behavior recognition device according to claim 4.

6. A behavior recognition device that recognizes human behavior in a building, a first acquisition unit that associates a plurality of predetermined spaces with one or more target actions to be recognized for each of the spaces, and acquires target action information, floor plan information of the building, and position information of an image sensor installed in the building; a behavior selection unit that selects a candidate behavior to be recognized from the one or more target behaviors included in the target behavior information based on the position information of the image sensor and the floor plan information; a second acquisition unit that acquires image data detected by the image sensor; a first behavior recognition unit that determines one or more recognizers according to the candidate behaviors and calculates feature quantities of the image data using the one or more recognizers; a second behavior recognition unit that recognizes the candidate behavior based on the feature amount; an output unit that outputs a recognition result by the second behavior recognition unit, extracting one or more objects installed in the building from the floor plan information, classifying the one or more objects into one of a first object that is a movable object, a second object that is a plumbing facility, and a third object that is a structure of the building, extracting floor plan features that associate classification information indicating the classification result with an installation position for each of the one or more objects, and generating a behavior selection table based on the floor plan features; the behavior selection table is a table in which one or more spaces of the building are associated with the target behaviors corresponding to each space, the first acquisition unit acquires the behavior selection table as the target behavior information; Behavior recognition device.

7. the behavior recognition device is communicably connected to a display terminal, an installation support unit that acquires, via the display terminal, a name of a space in the building in which the image sensor is to be installed, and outputs to the display terminal installation guidance for installing the image sensor so that a specific device or specific facility related to the space is included in the field of view of the image sensor; The behavior recognition device according to claim 6.

8. the installation support unit acquires image data captured by the image sensor, detects the specific device or the specific facility included in the image data, and superimposes an annotation image indicating a detection result of the specific device or the specific facility on an image indicated by the image data. The behavior recognition device according to claim 7.

9. the installation support unit acquires an instruction to correct the annotation image via the display terminal, and stores annotation information indicated by the corrected annotation image in a memory; The behavior recognition device according to claim 8.

10. An activity recognition method for recognizing user activity in a building, comprising: The computer Acquire target behavior information by associating each of a plurality of predetermined spaces with one or more target behaviors to be recognized for each space; acquiring floor plan information of the building; acquiring location information of an image sensor installed in the building; selecting a candidate behavior to be recognized from the one or more target behaviors included in the target behavior information based on the position information of the image sensor and the floor plan information; acquiring image data detected by the image sensor; a first behavior recognition unit determining one or more recognizers according to the candidate behaviors, and calculating feature amounts of the image data using the one or more recognizers; a second behavior recognition unit that recognizes the candidate behavior based on the feature amount; outputting a recognition result by the second behavior recognition unit; Furthermore, the first behavior recognition unit determines a plurality of recognizers when the candidate behavior is a predetermined behavior; the second behavior recognition unit combines the feature amounts calculated by the plurality of recognizers and recognizes the candidate behavior based on the combined feature amounts. Behavior recognition method.

11. A program for recognizing user behavior in a building, comprising: a first acquisition unit that acquires target behavior information in association with one or more target behaviors to be recognized for each of a plurality of predetermined spaces, floor plan information of the building, and location information of an image sensor installed in the building; a behavior selection unit that selects a candidate behavior to be recognized from the one or more target behaviors included in the target behavior information based on the position information of the image sensor and the floor plan information; a second acquisition unit that acquires image data detected by the image sensor; a first behavior recognition unit that determines one or more recognizers according to the candidate behaviors and calculates feature quantities of the image data using the determined one or more recognizers; a second behavior recognition unit that recognizes the candidate behavior based on the feature amount; causing the computer to function as an output unit that outputs a recognition result by the second behavior recognition unit; and the first behavior recognition unit determines a plurality of recognizers when the candidate behavior is a predetermined behavior; causing the computer to function so that the second behavior recognition unit combines the feature amounts calculated by the plurality of recognizers and recognizes the candidate behavior based on the combined feature amounts; program.

Images