State identification system, state identification method, and program

Abstract

To provide a state identification system that can improve the accuracy of identifying the state of a subject.SOLUTION: In a state identification system 1, a relation model creation unit 38 creates a relation model in which the correspondence relationship between distance and index information indicating the value of the distance is defined. A depth image creation unit 39 creates depth image information of a predetermined area on the basis of distance information and the relation model. The distance information is acquired from a distance information acquisition unit. The relation model is created in the relation model creation unit 38. An identification processing unit 44 identifies the state of a subject in the predetermined area on the basis of the depth image information. The depth image information is created by the depth image creation unit 39. The relation model creation unit 38 creates the relation model on the basis of the distance information corresponding to the position of the subject in the predetermined area, of the positions of a plurality of divided areas.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a condition identification system, a condition identification method, and a program. [Background technology]

[0002] There are known techniques for detecting the behavior of a subject, such as a person, an animal, or a machine (for example, Patent Document 1). For example, Patent Document 1 discloses a technique for detecting information related to the behavior of a subject and identifying the behavior of the subject based on the detection result. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-29671 A Summary of the Invention [Problem to be solved by the invention]

[0004] Patent Document 1 shows that temperature distribution or radio wave fluctuations are detected, features are extracted from the detection results, and behaviors such as walking and falling are identified. With such a technique, for example, an abnormal situation of a subject can be quickly discovered. There is an increasing demand for such a technique, and it is required to identify the condition of a subject in more detail while suppressing the computational load. Examples of detailed identification of the condition of a subject include identification of the behavior of a subject, identification of unstable movement of a subject, and individual identification of a subject. For example, if the accuracy of identification of unstable movement is improved, serious abnormal situations of a subject can be predicted more reliably.

[0005] It is also conceivable to obtain depth image information of a subject by LiDAR (Light Detection and Ranging) or the like and detect the state of the subject based on the obtained depth image information. However, in the depth image information, the information of the subject is buried in environmental information other than the subject, making it difficult to detect the state of the subject.

[0006] An object of one aspect of the present invention is to provide a condition identification system that can improve the accuracy of identifying a condition of a subject.Another aspect of the present invention is to provide a condition identification method that can improve the accuracy of identifying a condition of a subject.Another aspect of the present invention is to provide a program that can improve the accuracy of identifying a condition of a subject. [Means for solving the problem]

[0007] A state identification system according to one aspect of the present invention includes a distance information acquisition unit, a relationship model creation unit, a depth image creation unit, and an identification processing unit. The distance information acquisition unit acquires distance information indicating a distance from a reference position in real space to each position of a plurality of partitioned regions. The plurality of partitioned regions are partitions of a predetermined region in real space. The relationship model creation unit creates a relationship model in which a correspondence relationship between a distance and index information indicating the distance is defined. The depth image creation unit creates depth image information of the predetermined region based on the distance information and the relationship model. The distance information is acquired from the distance information acquisition unit. The relationship model is created in the relationship model creation unit. The identification processing unit identifies a state of the subject in the predetermined region based on the depth image information. The depth image information is created by the depth image creation unit. The relationship model creation unit creates the relationship model based on distance information corresponding to the position of the subject in the predetermined region among the positions of the plurality of partitioned regions.

[0008] In another aspect of this embodiment, the state identification method may include acquiring distance information indicating a distance from a reference position in a real space to a position of each of a plurality of partitioned regions into which a predetermined region in the real space is partitioned, creating a relationship model in which a correspondence relationship between the distance and index information indicating the distance is defined, creating depth image information of the predetermined region based on the acquired distance information and the created relationship model, and identifying a state of the subject in the predetermined region based on the created depth image information. In creating the relationship model, the relationship model may be created based on distance information corresponding to a position of the subject in the predetermined region among the positions of the plurality of partitioned regions.

[0009] In yet another aspect of the present embodiment, the program may cause a computer to execute the following steps: acquire distance information indicating a distance from a reference position in real space to each position of a plurality of partitioned regions into which a predetermined region in real space is partitioned; create a relationship model in which a correspondence relationship between the distance and index information indicating the distance is defined; create depth image information of the predetermined region based on the acquired distance information and the created relationship model; and identify a state of the subject in the predetermined region based on the created depth image information. In creating the relationship model, the program creates the relationship model based on distance information corresponding to the position of the subject in the predetermined region among the positions of the plurality of partitioned regions. Effect of the Invention

[0010] One aspect of the present invention provides a condition identification system that can improve the accuracy of identifying a condition of a subject. Another aspect of the present invention provides a condition identification method that can improve the accuracy of identifying a condition of a subject. Yet another aspect of the present invention provides a program that can improve the accuracy of identifying a condition of a subject. [Brief description of the drawings]

[0011] [Figure 1] FIG. 2 is a block diagram of a state identification system according to the present embodiment. [Diagram 2]FIG. 2 is a schematic diagram for explaining a sensor. [Diagram 3] FIG. 13 is a diagram showing skeletal information in an RGB image. [Figure 4] FIG. 13 is a diagram illustrating an example of a depth image. [Diagram 5] FIG. 13 is a diagram showing skeletal information in a depth image. [Figure 6] 13 is a diagram showing a relational model corresponding to depth image information acquired by a distance information acquisition unit. FIG. [Figure 7] FIG. 13 is a diagram showing a relational model updated by a relational model creating unit. [Figure 8] 1 shows a box region in a depth image. [Figure 9] 1 is a graph showing a relational model. [Figure 10] FIG. 2 is a diagram illustrating an example of a hardware configuration of the situation identification system. [Figure 11] 1 is a flowchart illustrating an example of a method for creating a discrimination model. [Figure 12] 13 is a flowchart illustrating an example of a process for creating depth image information. [Figure 13] 13 is a flowchart illustrating an example of a method for identifying a situation in a designation phase. [Figure 14] (a) and (b) show depth images of the subject as it rolls over. [Figure 15] 11 is a graph showing a comparison of feature information extracted from depth image information before and after updating by a relational model creation unit. [Figure 16] FIG. 13 is a diagram showing the performance of detection results of state identification. [Figure 17] 13 is a schematic diagram for explaining a state identification system in a modified example of the present embodiment. FIG. [Figure 18] FIG. 13 is a diagram showing the performance of detection results of state identification. [Figure 19] FIG. 13 is a diagram showing the performance of detection results of state identification. [Figure 20] FIG. 13 is a diagram showing the performance of detection results of state identification. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] [Description of the embodiments of the present disclosure]

[0013] First, embodiments of the present disclosure will be listed and described.

[0014] [1] A state identification system according to an embodiment of the present disclosure includes a distance information acquisition unit, a relationship model creation unit, a depth image creation unit, and an identification processing unit. The distance information acquisition unit acquires distance information indicating a distance from a reference position in real space to each of a plurality of partitioned regions. The plurality of partitioned regions are partitions of a predetermined region in real space. The relationship model creation unit creates a relationship model in which a correspondence relationship between a distance and index information indicating the distance is defined. The depth image creation unit creates depth image information of the predetermined region based on the distance information and the relationship model. The distance information is acquired from the distance information acquisition unit. The relationship model is created in the relationship model creation unit. The identification processing unit identifies a state of the subject in the predetermined region based on the depth image information. The depth image information is created by the depth image creation unit. The relationship model creation unit creates the relationship model based on distance information corresponding to the position of the subject in the predetermined region among the positions of the plurality of partitioned regions.

[0015] In the condition identification system in [1] above, the depth image creation unit creates depth image information of the predetermined region based on the relationship model created by the relationship model creation unit and distance information. The relationship model creation unit creates the relationship model based on distance information corresponding to the position of the subject in the predetermined region among the positions of the multiple divided regions. The depth image information created in this manner can improve the identification accuracy of the subject's condition.

[0016] [2] In the condition identification system of [1] above, the relation model creation unit may create a relation model such that a rate of change in the index information relative to a change in the distance value in a first range corresponding to the position of the subject within the predetermined region among the ranges of distance values ​​indicated by the index information is greater than a rate of change in the index information relative to a change in the distance value in a second range different from the first range. In this case, the accuracy of identifying the subject's condition can be further improved.

[0017] [3] In the state identification system of [2] above, the relational model creation unit may create the relational model such that, in the second range, the rate of change in the index information relative to the change in the distance value increases as the second range approaches the first range. In this case, the robustness of detection of the subject in the depth image information can be improved.

[0018] [4] In the condition identification system of [3] above, the index information may include color information that indicates different colors depending on distance. In this case, the condition of the subject can be more easily identified.

[0019] [5] The condition identification system according to any one of [1] to [4] above may further include a position identification unit. The position identification unit may identify a position of the subject within the predetermined region based on feature information indicating the position of the subject. The feature information may include information indicating a position of an RGB pixel related to the subject among a plurality of RGB pixels included in an RGB image obtained by capturing an image of the subject. The relation model creation unit creates a relation model based on the position of the subject identified by the position identification unit and distance information corresponding to the position of the subject. In this case, the relation model is created based on the information of the RGB image and the distance information. As a result, the relation model creation unit may create a relation model that more accurately corresponds to the position of the subject.

[0020] [6] In any one of the condition identification systems [1] to [5] above, the identification processing unit may obtain an identification model that associates information related to the depth image information created by the depth image creation unit with the condition of the subject. The identification processing unit may identify the condition of the subject in the predetermined region based on the information related to the depth image information created by the depth image creation unit and the identification model. In this case, the identification model associates information related to the depth image information created based on the relationship model created by the relationship model creation unit with the condition of the subject. Therefore, the condition of the subject can be identified more easily and reliably.

[0021] [7] Any one of the above condition identification systems [1] to [6] may further include an identification model creation unit that creates an identification model. The identification model creation unit may create the identification model based on information on the learning depth image information created by the depth image creation unit and the condition of the subject for learning. In this case, the identification model is created using the learning depth image information created by the depth image creation unit. Therefore, the identification model can further improve the accuracy of identifying the subject's condition.

[0022] [8] In another aspect of the present disclosure, a state identification method may include acquiring distance information indicating a distance from a reference position in a real space to a position of each of a plurality of partitioned regions into which a predetermined region in the real space is partitioned, creating a relationship model in which a correspondence relationship between the distance and index information indicating the distance is defined, creating depth image information of the predetermined region based on the acquired distance information and the created relationship model, and identifying a state of a subject in the predetermined region based on the created depth image information. In creating the relationship model, the relationship model may be created based on distance information corresponding to a position of the subject in the predetermined region among the positions of the plurality of partitioned regions.

[0023] [9] In yet another aspect of the present disclosure, a program may cause a computer to execute the following steps: acquire distance information indicating a distance from a reference position in real space to each position of a plurality of partitioned regions into which a predetermined region in real space is partitioned; create a relationship model in which a correspondence relationship between the distance and index information indicating the distance is defined; create depth image information of the predetermined region based on the acquired distance information and the created relationship model; and identify a state of a subject in the predetermined region based on the created depth image information. In creating the relationship model, the program creates the relationship model based on distance information corresponding to a position of the subject in the predetermined region among the positions of the plurality of partitioned regions. [Details of the embodiment of the present disclosure]

[0024] Hereinafter, an embodiment of the system according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same or corresponding parts are given the same reference numerals, and duplicated description will be omitted. First, a schematic configuration of a condition identification system 1 according to an embodiment of the present invention will be described with reference to Figs. 1 to 9. Fig. 1 is a block diagram of the condition identification system 1. Fig. 2 is a schematic diagram for explaining a sensor.

[0025] The condition identification system 1 identifies the condition of the subject 3 to be identified based on the detection result of the sensor 2. The subject 3 includes, for example, a living body to be detected and an object to be detected. The living body to be detected includes a person to be detected and an animal other than a human. The person to be detected is a person. The object to be detected includes a machine such as a robot. As an example in this embodiment, the subject 3 is a person. The condition of the subject 3 identified by the condition identification system 1 includes, for example, the behavior of the subject 3, as well as individual information of the subject 3 and the stability of the movement of the subject 3. The "behavior of the subject" is, for example, moving, rolling over, sitting, standing upright, etc. The "individual information of the subject" is information for identifying the individual of the subject, for example, the name of a person. The "movement of the subject" is, for example, the walking of a person. The "rolling over of the subject" is, for example, a fall of a person. The identification of the "stability of the movement of the subject" is, for example, identification of whether the walking of the subject 3 is unstable or not.

[0026] As shown in FIG. 2, for example, the sensor 2 is located in real space. The sensor 2 is located at an arbitrary position in real space. FIG. 2 shows a state in which a subject 3 is located in a predetermined area in real space. The sensor 2, for example, emits light and detects reflected light of the emitted light. The sensor 2 measures the time from emitting light to detecting reflected light, and detects the distance to an object that reflects the emitted light. The light is, for example, infrared light. The sensor 2 is, for example, a 2D LiDAR (Light Detection and Ranging).

[0027] The sensor 2 sequentially detects a relative position with respect to a reference position in the real space. The sensor 2 detects the relative position for each scanning period. For example, the reference position is the position of the sensor 2. The detection result of the sensor 2 includes, for example, distances from a reference position in the real space to the multiple relative positions, and orientations of the multiple relative positions with respect to the reference position. The multiple relative positions correspond to the positions of multiple partitioned areas into which a predetermined area in the real space is partitioned. The multiple relative positions are positions corresponding to the multiple partitioned areas in the real space. Each partitioned area corresponds to, for example, the detection resolution of the sensor 2. For example, the sensor 2 detects distance information Pi indicating a distance to each of the relative positions in the predetermined area of ​​the real space as a data set V1. The data set V1 is, for example, depth image information indicating, for each of a plurality of pixels, distance information Pi of the relative positions corresponding to each pixel. The "desired area" corresponds, for example, to a field of view identified by the state identification system 1. The "distance information" may be, for example, a distance value, or may be a signal, symbol, color, or light and shade indicating a distance.

[0028] In this embodiment, the condition identification system 1 executes a learning phase and an estimation phase. The condition identification system 1 executes the estimation phase based on the identification model learned in the learning phase. In the estimation phase, the condition identification system 1 identifies the condition of the subject 3 based on the detection result of the sensor 2. The condition identification system 1 may further execute a learning phase based on information acquired in the estimation phase.

[0029] For example, the state identification system 1 uses machine learning in creating an identification model in the learning phase. Machine learning is a method of autonomously finding laws or rules by iteratively learning based on given information. For example, in the machine learning performed in the state identification system 1, model parameters such as activation functions and weighting values ​​are optimized by learning using a learning data set. This creates an identification model.

[0030] The machine learning performed in the state identification system 1 is, for example, deep learning. This machine learning is supervised learning configured by a multilayer perceptron (MLP). The state identification system 1 uses machine learning configured to include a neural network. The machine learning performed in the state identification system 1 is not limited to supervised learning.

[0031] For example, the neural network used in the state identification system 1 is a long short-term memory network (LSTM). The LSTM is one of recurrent neural networks. The type of neural network used in the state identification system 1 is not limited to this.

[0032] The state identification system 1 includes an estimation unit 10 that operates in an estimation phase and a learning unit 20 that operates in a learning phase. The state identification system 1 includes, as the estimation unit 10, an RGB image acquisition unit 31, a sensor unit 32, a distance information acquisition unit 33, a calibration unit 34, a first skeleton extraction unit 35, a feature information acquisition unit 36, a position identification unit 37, a relational model creation unit 38, a depth image creation unit 39, a second skeleton extraction unit 41, a storage unit 42, an identification processing unit 44, and an output unit 45. The state identification system 1 includes, as the learning unit 20, the RGB image acquisition unit 31, the sensor unit 32, the distance information acquisition unit 33, the calibration unit 34, the first skeleton extraction unit 35, the feature information acquisition unit 36, a position identification unit 37, a relational model creation unit 38, a depth image creation unit 39, a second skeleton extraction unit 41, a storage unit 42, and an identification model creation unit 47. In this embodiment, the RGB image acquisition unit 31, the sensor unit 32, the distance information acquisition unit 33, the calibration unit 34, the first skeleton extraction unit 35, the feature information acquisition unit 36, the position identification unit 37, the relationship model creation unit 38, the depth image creation unit 39, the second skeleton extraction unit 41, and the storage unit 42 are included in both the estimation unit 10 and the learning unit 20.

[0033] As a modification of this embodiment, the estimation unit 10 and the learning unit 20 may be separated from each other. In this case, the estimation unit 10 and the learning unit 20 may each include a different RGB image acquisition unit 31, a sensor unit 32, a distance information acquisition unit 33, a calibration unit 34, a first skeleton extraction unit 35, a feature information acquisition unit 36, a position identification unit 37, a relational model creation unit 38, a depth image creation unit 39, a second skeleton extraction unit 41, and a storage unit 42. As a modification of this embodiment, the state identification system 1 may not include the identification model creation unit 47 and may be composed of only the estimation unit 10. The state identification system 1 may be a learning system composed of only the learning unit 20. The state identification system 1 may not include the sensor unit 32 and may acquire detection information from a sensor unit arranged outside the state identification system 1. The estimation unit 10 may not include the RGB image acquisition unit 31, the calibration unit 34, the first skeleton extraction unit 35, and the feature information acquisition unit 36.

[0034] Next, each functional unit of the condition identification system 1 in the estimation phase will be described in more detail. In the estimation phase, the estimation unit 10 operates. The estimation unit 10 estimates and outputs the state of the specimen 3 to be identified. The estimation unit 10 estimates the state of the specimen 3 to be identified based on the detection result of the sensor 2.

[0035] The RGB image acquisition unit 31 acquires information on an RGB image of the subject 3. For example, the RGB image acquisition unit 31 includes a color camera (not shown) as an imaging device (not shown). The RGB image acquisition unit 31 acquires information on an RGB image of the subject 3, for example, by a color camera. The information on the RGB image corresponds to distance information detected by the sensor 2. The RGB image includes a plurality of RGB pixels. The plurality of RGB pixels correspond to a plurality of partitioned regions into which a predetermined region in the real space is partitioned, and each RGB pixel corresponds to distance information Pi of each partitioned region detected by the sensor 2. As a modified example of this embodiment, the RGB image acquisition unit 31 may acquire information on the RGB image from outside the condition identification system 1. The "desired region" is, for example, a region corresponding to the field of view of the sensor 2 or the color camera, that is, a region corresponding to the angle of view of the sensor 2 or the color camera.

[0036] The sensor unit 32 includes a sensor 2. The sensor unit 32 sequentially outputs detection information of the sensor 2 to the distance information acquisition unit 33. For example, the sensor unit 32 sequentially outputs the detection information of the sensor 2 to the distance information acquisition unit 33 as depth image information.

[0037] The distance information acquisition unit 33 acquires distance information indicating the distance from a reference position to the position of each of a plurality of partitioned regions into which a predetermined area in real space is partitioned. The position of each of the plurality of partitioned regions is a relative position corresponding to the reference position. For example, the distance information acquisition unit 33 acquires detection information of the sensor 2 from the sensor unit 32 as distance information. In this case, each partitioned region corresponds to, for example, the detection resolution of the sensor 2. For example, the distance information acquisition unit 33 acquires depth image information from the sensor unit 32 indicating distance information from each pixel to the position of the partitioned region to which the pixel corresponds.

[0038] For example, each pixel of the depth image acquired by the distance information acquisition unit 33 includes index information indicating the distance from the reference position to the position of each divided area in a predetermined correspondence relationship. For example, the depth image acquired by the distance information acquisition unit 33 is shown based on a relationship model in which the correspondence relationship between the distance from the reference position to the position of each divided area and the index information indicating the distance is defined in advance. In other words, the relationship model corresponds to a table or map indicating the correspondence relationship between the distance from the reference position to the position of each divided area and the index information indicating the distance. For example, the index information includes color information indicating a different color depending on the distance. The index information is not limited to color information, and may be brightness or a numerical value. The index information may be information associated with a numerical value of the distance, or may be information associated with the degree of the distance.

[0039] The calibration unit 34 calibrates the distance information acquired by the distance information acquisition unit 33 and the RGB image acquired by the RGB image acquisition unit 31. The calibration unit 34 calibrates the positions of the multiple RGB pixels of the RGB image and the multiple partitioned regions. For example, the calibration unit 34 calibrates the multiple RGB pixels of the RGB image and the multiple pixels of the depth image acquired from the sensor unit 32.

[0040] The first skeleton extraction unit 35 extracts skeleton information DA1 of the subject 3 from the RGB image acquired by the RGB image acquisition unit 31. FIG. 3 shows skeleton information DA1 in an RGB image V2. For example, the first skeleton extraction unit 35 extracts skeleton information DA1 of the subject 3 shown in the RGB image V2 based on information of the RGB image V2 acquired by the RGB image acquisition unit 31 and an existing model. The existing model is, for example, a learning model that has already been learned. The existing model is, for example, "Open Pose".

[0041] The feature information acquisition unit 36 ​​acquires feature information DA2 of the subject 3 from the RGB image V2 acquired by the RGB image acquisition unit 31. FIG. 3 shows feature information DA2 in the RGB image V2. The feature information DA2 is information indicating the position of the subject 3. The feature information DA2 of the subject 3 is, for example, position information indicating the positions of a plurality of feature points of the subject 3. The feature information DA2 includes information indicating the positions of RGB pixels related to the subject 3 among a plurality of RGB pixels included in the RGB image V2 obtained by capturing the subject 3 in the RGB image V2. The feature information acquisition unit 36 ​​inputs information of the RGB image V2 acquired by the RGB image acquisition unit 31 to an existing model, and acquires position information indicating the positions of a plurality of feature points output from the existing model. The existing model is, for example, a learning model that has already been learned, such as "Open Pose". The first skeleton extraction unit 35 and the feature information acquisition unit 36 ​​may be the same functional unit, and extraction of the skeleton information DA1 of the subject 3 and extraction of the feature information DA2 of the subject 3 may be executed in parallel.

[0042] For example, the feature information acquisition unit 36 ​​detects the positions of a plurality of feature points indicating the nose, left eye, right eye, left ear, and right ear in the head, and stores them together with labeling information indicating the type of each part in the storage unit 42. For example, the feature information acquisition unit 36 ​​detects the positions of a plurality of feature points indicating the left shoulder, right shoulder, left elbow, right elbow, left hand, and right hand in the upper body, and stores them together with labeling information indicating the type of each part in the storage unit 42. For example, the feature information acquisition unit 36 ​​detects the positions of a plurality of feature points indicating the left buttock, right buttock, left knee, right knee, left ankle, and right ankle in the lower body, and stores them together with labeling information indicating the type of each part in the storage unit 42.

[0043] The position identifying unit 37 identifies the position of the subject 3 within the above-mentioned predetermined region based on the feature information acquired by the feature information acquiring unit 36. The position identifying unit 37 identifies distance information corresponding to the position of the subject 3 from the distance information acquired by the distance information acquiring unit 33 based on the feature information of the subject 3 acquired by the feature information acquiring unit 36. The position identifying unit 37 identifies, for example, a box region corresponding to the center of the subject 3 and identifies distance information in the box region. For example, the position identifying unit 37 performs transfer learning based on the feature information of the subject 3 acquired by the feature information acquiring unit 36. For example, the position identifying unit 37 identifies the positions of multiple feature points of the subject 3 in the depth image information acquired by the distance information acquiring unit 33 by transfer learning.

[0044] Fig. 4 shows a depth image V3. Fig. 5 shows a state in which the position of the subject 3 is specified in the depth image V3. Fig. 5 shows skeletal information of the subject 3 in the depth image V3, as well as feature information DA2 of the subject 3 and link information DA3 connecting feature points of the subject 3.

[0045] The relational model creation unit 38 creates a relational model in which a correspondence relationship between the distance from the reference position to the position of each divided area and index information indicating the distance is defined. For example, the index information includes color information indicating a different color according to the distance. The index information is not limited to color information, and may be brightness or a numerical value.

[0046] The relation model creation unit 38 creates a relation model based on the position of the subject 3 identified by the position identification unit 37 and distance information corresponding to the position of the subject 3. The relation model creation unit 38 creates a relation model based on the center of the subject 3 identified by the position identification unit 37 and distance information corresponding to the box region. The relation model creation unit 38 creates a relation model based on distance information corresponding to the position of the subject 3 among the positions of the multiple divided regions. For example, the relation model creation unit 38 updates the relation model corresponding to the depth image information acquired by the distance information acquisition unit 33.

[0047] FIG. 6 shows an example of a relation model M1 corresponding to depth image information acquired by the distance information acquisition unit 33. FIG. 7 shows an example of a relation model M2 created by the relation model creation unit. The relation models M1 and M2 include index information DA6 and distance information DA7, and the index information DA6 and the distance information DA7 are associated with each other. The distance information DA7 includes a plurality of distance values. In the relation model M1, the rate of change of the index information relative to the change of the distance value is constant. In other words, in the relation model M1, the change of the index information relative to the change of the distance value is linear. In contrast, in the relation model M2, the change of the index information relative to the change of the distance value is nonlinear.

[0048] 9 shows a box area RA10 corresponding to the position of the subject 3 in a depth image V9. The box area RA10 is determined based on, for example, the position of the subject 3. The box area RA10 is, for example, an area that is placed with a position that is estimated to be the center of gravity of the subject 3 as its center of gravity, and has a predetermined shape and size. The box area RA10 may be set based on the contour or feature points of the subject 3.

[0049] For example, the relational model creation unit 38 specifies a first range R1 from the range R based on distance values ​​in a box region RA10 corresponding to the position of the subject 3. The range R is the maximum range of the index information DA6 in the relational model M2. In other words, the relational model creation unit 38 specifies a range of distance values ​​corresponding to the position of the subject 3 among the positions of the multiple partitioned regions as the first range R1. In the example shown in this embodiment, the relational model creation unit 38 acquires a predetermined number of distance value samples in the box region RA10, and specifies a predetermined numerical range centered on the average value of the samples as the first range R1. As a modified example of this embodiment, the relational model creation unit 38 may specify a predetermined numerical range centered on the average value of all distance values ​​in the box region RA10 as the first range R1.

[0050] The relational model creation unit 38 creates a relational model based on the first range R1. The relational model creation unit 38 specifies a range closer than the first range R1 as a second range R2, and specifies a range farther than the first range R1 as a third range R3.

[0051] FIG. 8 is a graph plotting distance information DA7 in the relation model created by the relation model creation unit 38, with the vertical axis representing the index value and the horizontal axis representing the distance value. The relation model creation unit 38 creates the relation model such that the rate of change in the index information relative to the change in the distance value in a first range R1 corresponding to the position of the subject 3 in the above-mentioned predetermined region among the ranges of distance values ​​indicated by the index information is greater than the rate of change in the index information relative to the change in the distance value in a second range R2 different from the first range R1. The relation model creation unit 38 creates the relation model such that in the second range R2, the rate of change in the index information relative to the change in the distance value becomes higher as the second range R2 approaches the first range R1. The relation model creation unit 38 creates the relation model such that in the third range R3, the rate of change in the index information relative to the change in the distance value becomes higher as the second range R2 approaches the first range R1. As a modified example of this embodiment, the relation model may be created such that the rate of change in the index information relative to the change in the distance value is constant in at least one of the first range R1, the second range R2, and the third range R3.

[0052] The depth image creation unit 39 creates depth image information of the above-mentioned predetermined region based on the distance information acquired from the distance information acquisition unit 33 and the relationship model created by the relationship model creation unit 38. For example, the depth image creation unit 39 creates depth image information indicating, for each pixel, index information corresponding to the distance information of each partitioned region in the created relationship model based on the distance information acquired from the distance information acquisition unit 33 and the relationship model created by the relationship model creation unit 38. For example, the depth image creation unit 39 updates the depth image information acquired from the distance information acquisition unit 33 based on the relationship model created by the relationship model creation unit 38. For example, the depth image creation unit 39 creates second depth image information indicating, for each pixel, index information corresponding to the distance information of each partitioned region in the updated relationship model based on the first depth image information acquired from the distance information acquisition unit 33 and the relationship model updated by the relationship model creation unit 38 from the relationship model in the first depth image information.

[0053] The second skeleton extraction unit 41 extracts skeleton information of the subject 3 from the depth image information created by the depth image creation unit 39. The second skeleton extraction unit 41 stores the extracted skeleton information in the storage unit 42. The second skeleton extraction unit 41 may output the extracted skeleton information to the discrimination processing unit 44 or the discrimination model creation unit 47.

[0054] The storage unit 42 stores information used by each functional unit in advance. The storage unit 42 stores outputs from each functional unit. The storage unit 42 stores, for example, information acquired by the RGB image acquisition unit 31, the sensor unit 32, the distance information acquisition unit 33, and the feature information acquisition unit 36. The storage unit 42 stores, for example, a result of calibration by the calibration unit 34. The storage unit 42 stores, for example, skeleton information extracted by each of the first skeleton extraction unit 35 and the second skeleton extraction unit 41. The storage unit 42 stores, for example, position information identified by the position identification unit 37. The storage unit 42 stores, for example, a relationship model created by the relationship model creation unit 38. The storage unit 42 stores, for example, a depth image created by the depth image creation unit 39. The storage unit 42 stores, for example, an identification result by the identification processing unit 44. The storage unit 42 stores, for example, information output from the output unit 45.

[0055] The identification processing unit 44 identifies the state of the subject 3 in the above-mentioned predetermined region based on the depth image information created by the depth image creation unit 39. For example, the identification processing unit 44 acquires an identification model from the storage unit 42. The identification model associates information on the depth image information created by the depth image creation unit 39 with the state of the subject 3. The identification processing unit 44 identifies the state of the subject 3 in the above-mentioned predetermined region based on the information on the depth image information created by the depth image creation unit 39 and the identification model. The identification model outputs the state of the subject 3 in response to the input of information on the depth image information created by the depth image creation unit 39. The information on the depth image information created by the depth image creation unit 39 is, for example, skeleton information extracted by the second skeleton extraction unit 41. The information on the depth image information created by the depth image creation unit 39 may be, for example, the depth image information itself created by the depth image creation unit 39. The state of the subject 3 output from the identification model is information on the subject 3 included in the depth image information created by the depth image creation unit 39.

[0056] The output unit 45 outputs the condition of the subject 3 identified by the identification processing unit 44. The output unit 45 may output the condition of the subject 3 identified by the identification processing unit 44 to the outside of the condition identification system 1. The output unit 45 may output the condition of the subject 3 identified by the identification processing unit 44 to a server terminal or another terminal via a communication network or the like.

[0057] Next, each functional unit of the state identification system 1 in the learning phase will be described in more detail. Some of the descriptions overlapping with those in the estimation phase will be omitted. In the learning phase, the learning unit 20 operates. The learning unit 20 creates an identification model using an RGB image acquisition unit 31, a sensor unit 32, a distance information acquisition unit 33, a calibration unit 34, a first skeleton extraction unit 35, a feature information acquisition unit 36, a position identification unit 37, a relational model creation unit 38, a depth image creation unit 39, a second skeleton extraction unit 41, a storage unit 42, and an identification model creation unit 47. The learning unit 20 creates an identification model trained using a learning data set including various information. The learning unit 20 acquires a learning data set for learning the discrimination model by using an RGB image acquisition unit 31, a sensor unit 32, a distance information acquisition unit 33, a calibration unit 34, a first skeleton extraction unit 35, a feature information acquisition unit 36, a position identification unit 37, a relational model creation unit 38, a depth image creation unit 39, and a second skeleton extraction unit 41. The learning unit 20 may acquire a learning data set that has been stored in advance in a storage unit 42, or may acquire a learning data set from outside the state discrimination system 1 via a communication network or the like.

[0058] When the learning unit 20 operates, in the example shown in this embodiment, the RGB image acquiring unit 31 acquires information on a learning RGB image obtained by capturing an image of the learning subject 3. The RGB image acquiring unit 31 acquires information on the RGB image of the learning subject 3 by, for example, a color camera. As a modified example of this embodiment, the RGB image acquiring unit 31 may acquire a learning RGB image stored in advance in the storage unit 42, or may acquire information on the RGB image from outside the condition identification system 1.

[0059] The RGB image acquisition unit 31 acquires, for example, information on RGB images captured in different scenes as learning RGB images. The RGB images captured in different scenes refer to RGB images captured under conditions in which at least one of the posture, position, and number of learning subjects 3 shown in the RGB images is different.

[0060] The sensor unit 32 sequentially outputs the detection information of the sensor 2 for learning to the distance information acquisition unit 33. For example, the sensor unit 32 sequentially outputs the detection information of the sensor 2 to the distance information acquisition unit 33 as depth image information for learning.

[0061] The distance information acquisition unit 33 acquires learning distance information from a reference position to each position of the plurality of partitioned regions. The distance information acquisition unit 33 acquires detection information of the learning sensor 2 from the sensor unit 32 as learning distance information. For example, the distance information acquisition unit 33 acquires learning depth image information from the sensor unit 32, which indicates distance information for each pixel regarding the position of the partitioned region corresponding to the pixel.

[0062] The calibration unit 34 calibrates the learning distance information acquired by the distance information acquisition unit 33 and the learning RGB image acquired by the RGB image acquisition unit 31. The calibration unit 34 calibrates the positions of the multiple RGB pixels of the learning RGB image and the multiple partitioned regions. For example, the calibration unit 34 calibrates the multiple RGB pixels of the learning RGB image and the multiple pixels of the learning depth image acquired from the sensor unit 32.

[0063] The first skeleton extraction unit 35 acquires skeleton information of the learning subject 3 from the learning RGB image acquired by the RGB image acquisition unit 31. For example, the first skeleton extraction unit 35 acquires skeleton information of the learning subject 3 shown in the learning RGB image based on information of the RGB image acquired by the RGB image acquisition unit 31 and an existing model. The existing model is, for example, a learning model that has already been learned. The existing model is, for example, "Open Pose".

[0064] The feature information acquiring unit 36 ​​acquires feature information of the learning subject 3 from the learning RGB image acquired by the RGB image acquiring unit 31. The feature information of the learning subject 3 is, for example, position information indicating the positions of a plurality of feature points of the learning subject 3. The feature information acquiring unit 36 ​​inputs information of the learning RGB image to an existing model, and acquires position information indicating the positions of a plurality of feature points output from the existing model. The existing model is, for example, a learning model that has already been trained, such as "Open Pose".

[0065] The position identifying unit 37 identifies the position of the learning subject 3 within the above-mentioned predetermined region based on the learning feature information acquired by the feature information acquiring unit 36. The position identifying unit 37 identifies distance information corresponding to the position of the learning subject 3 from the distance information acquired by the distance information acquiring unit 33 based on the learning feature information of the learning subject 3 acquired by the feature information acquiring unit 36. For example, the position identifying unit 37 performs transfer learning based on the feature information of the learning subject 3 acquired by the feature information acquiring unit 36. For example, the position identifying unit 37 identifies the positions of a plurality of feature points of the learning subject 3 in the learning depth image information acquired by the distance information acquiring unit 33 by transfer learning.

[0066] The relation model creation unit 38 creates a relation model based on the position of the learning subject 3 identified by the position identification unit 37 and distance information corresponding to the position of the learning subject 3. The relation model creation unit 38 creates a relation model based on distance information corresponding to the position of the learning subject 3 in the above-mentioned predetermined area among the positions of the multiple divided areas. For example, the relation model creation unit 38 updates the relation model corresponding to the learning depth image information acquired by the distance information acquisition unit 33.

[0067] The depth image creation unit 39 creates a plurality of pieces of learning depth image information used for learning the discrimination model. The depth image creation unit 39 creates a plurality of pieces of learning depth image information based on the learning distance information acquired from the distance information acquisition unit 33 and the relation model created by the relation model creation unit 38. The plurality of pieces of learning depth image information include information on depth images captured in different scenes. In the example shown in this embodiment, each piece of learning depth image information includes information on a plurality of feature pixels in which a plurality of feature points of the subject 3 corresponding to the learning depth image information are located.

[0068] The second skeleton extraction unit 41 extracts skeleton information of the learning subject 3 from the learning depth image information created by the depth image creation unit 39. The second skeleton extraction unit 41 stores the extracted learning skeleton information in the storage unit .

[0069] The storage unit 42 stores a training data set used for training the discrimination model. The storage unit 42 stores, for example, information to be output from the output unit 45. The storage unit 42 may store the training data set in advance. The storage unit 42 stores, for example, a discrimination model created by the discrimination model creation unit 47.

[0070] The learning data set used for learning the discrimination model includes information on a plurality of learning depth image information created by the depth image creation unit 39 and state information on the learning subject 3. The information on the plurality of learning depth image information created by the depth image creation unit 39 is, for example, a plurality of learning skeletal information extracted by the second skeletal extraction unit 41. The information on the plurality of learning depth image information created by the depth image creation unit 39 may be the plurality of depth image information itself. The information on the plurality of learning depth images includes information on learning depth images captured in different scenes. The learning depth images captured in different scenes refer to learning depth images captured under conditions in which at least one of the posture, position, and number of the learning living organism shown in the learning depth images is different. The plurality of learning skeletal information extracted by the second skeletal extraction unit 41 is, for example, skeletal information extracted from the above-mentioned learning depth image information.

[0071] The discrimination model creation unit 47 creates a discrimination model by using the above-mentioned learning data set. The discrimination model creation unit 47 creates a model that outputs a state of the subject 3 corresponding to information on the input depth image information in response to input of information on the depth image information for learning, by learning using information on the plurality of learning depth image information created by the depth image creation unit 39 and the state of the subject 3 for learning, as a discrimination model. The discrimination model creation unit 47 may create a model that outputs a state of the subject 3 corresponding to the input skeletal information in response to input of the plurality of learning skeletal information, by learning using, for example, the plurality of learning skeletal information extracted by the second skeletal extraction unit 41 and the state of the subject 3 for learning, as a discrimination model.

[0072] The discrimination model creation unit 47 may recreate or update the discrimination model based on the information acquired in the estimation phase. At least one of the RGB image acquisition unit 31, the calibration unit 34, the first skeleton extraction unit 35, the feature information acquisition unit 36, and the discrimination model creation unit 47 may be removed from the state discrimination system 1 after the learning phase is completed.

[0073] Next, a hardware configuration of the condition identification system 1 will be described with reference to Fig. 10. Fig. 10 is a diagram showing an example of the hardware configuration of the condition identification system 1.

[0074] The condition identification system 1 includes a processor 101, a memory 102, a storage 103, a communication device 104, an output device 105, an input device 106, and a sensor 2. The condition identification system 1 includes one or more computers configured with these hardware devices and software such as programs. The condition identification system 1 may be configured with one computer, or may be configured with multiple computers. The condition identification system 1 is realized in cooperation with the hardware.

[0075] When the condition identification system 1 is composed of multiple computers, these computers may be connected locally or via a communication network such as the Internet or an intranet. This connection logically constructs one condition identification system 1.

[0076] The processor 101 executes an operating system, application programs, etc. The memory 102 is composed of a ROM (Read Only Memory) and a RAM (Random Access Memory). For example, at least some of the various functional units of the RGB image acquisition unit 31, the sensor unit 32, the distance information acquisition unit 33, the calibration unit 34, the first skeleton extraction unit 35, the feature information acquisition unit 36, the position identification unit 37, the relational model creation unit 38, the depth image creation unit 39, the second skeleton extraction unit 41, the storage unit 42, the discrimination processing unit 44, the output unit 45, and the discrimination model creation unit 47 can be realized by the processor 101 and the memory 102.

[0077] The storage 103 is a storage medium configured by a hard disk, a flash memory, etc. The storage 103 generally stores a larger amount of data than the memory 102. For example, at least a part of the storage unit 42 may be realized by the storage 103.

[0078] The communication device 104 is configured with a network card or a wireless communication module. For example, at least a part of the RGB image acquisition unit 31, the distance information acquisition unit 33, the feature information acquisition unit 36, and the output unit 45 may be realized by the communication device 104. The output device 105 is configured with a printer, a display, and the like. For example, at least a part of the output unit 45 may be realized by the output device 105.

[0079] The input device 106 includes a keyboard, a mouse, a touch panel, etc. For example, at least a part of the RGB image acquisition unit 31, the distance information acquisition unit 33, and the feature information acquisition unit 36 ​​can be realized by the input device 106. The input device 106 may further include an imaging device of the RGB image acquisition unit 31.

[0080] The storage 103 stores in advance a program and data necessary for processing. This program causes a computer to execute each functional element of the condition identification system 1. This program causes, for example, each process in a condition identification method described below to be executed in the computer. This program may be provided in a state recorded on a tangible recording medium such as a CD-ROM, a DVD-ROM, or a semiconductor memory. This program may be provided as a data signal via a communication network.

[0081] Next, an example of a state identification method in this embodiment will be described with reference to Fig. 11 to Fig. 13. For example, the state identification method includes a relational model creation method and a discrimination model creation method. The state identification method executes a learning phase in which an identification model is created, and an estimation phase in which the state of the subject 3 is identified using the created discrimination model. First, an example of a discrimination model creation method in the learning phase will be described with reference to Fig. 11. Fig. 11 is a flowchart showing an example of a discrimination model creation method.

[0082] First, information on a learning RGB image is acquired (process S1). In process S1, for example, the RGB image acquisition unit 31 acquires information on a plurality of learning RGB images. In the example shown in this embodiment, the information on the learning RGB images is an RGB image obtained by capturing an image of the subject 3 by an imaging device.

[0083] Next, learning distance information is acquired (process S2). In process S2, the distance information acquisition unit 33 acquires, for example, a plurality of pieces of learning distance information Pi corresponding to the plurality of learning RGB images acquired in process S1. For example, the plurality of pieces of learning distance information Pi corresponding to the plurality of learning RGB images is learning distance information acquired in the same scene as each of the learning RGB images. The distance information acquisition unit 33 acquires, for example, distance information Pi from a reference position to each position of a plurality of divided regions into which the above-mentioned predetermined region is divided. The distance information acquisition unit 33 acquires, for example, distance information of the learning subject 3 detected by the sensor unit 32.

[0084] The distance information acquisition unit 33 acquires, for example, a plurality of pieces of learning distance information Pi for the same scene as the information of a plurality of learning RGB images acquired by the RGB image acquisition unit 31. At this time, a region in real space from which the distance information acquisition unit 33 acquires the distance information Pi overlaps with a region in real space from which the RGB image acquisition unit 31 acquires the RGB image. For example, the distance information acquisition unit 33 acquires the learning distance information Pi as depth image information. For example, the process S1 and the process S2 are performed in parallel.

[0085] Next, calibration of the learning distance information and the learning RGB image is performed (process S3). In process S3, for example, the calibration unit 34 calibrates the positions of the multiple RGB pixels of the learning RGB image and the multiple partitioned regions. For example, the calibration unit 34 calibrates the multiple RGB pixels of the RGB image and the multiple pixels of the depth image acquired in process S2.

[0086] Next, skeleton information is extracted from the learning RGB image (process S4). In process S4, for example, the first skeleton extraction unit 35 extracts skeleton information from the learning RGB image acquired in process S1.

[0087] Next, feature information for learning is acquired (process S5). For example, the feature information acquiring unit 36 ​​acquires feature information of the subject 3 from the learning RGB image acquired in process S1, and stores the acquired feature information in the storage unit 42. Process S4 and process S5 may be performed in parallel.

[0088] Next, the position of the learning subject 3 is identified (process S6). For example, the position identification unit 37 identifies distance information corresponding to the position of the subject 3 from the distance information acquired in process S2 based on the characteristic information of the subject 3 acquired in process S5.

[0089] Next, a process of creating depth image information is executed (process S7). For example, the relational model creation unit 38 and the depth image creation unit 39 execute a process of creating learning depth image information based on the position of the learning subject 3 identified in process S6. For example, in process S7, the relational model creation unit 38 and the depth image creation unit 39 update the depth image information acquired in process S2 based on distance information of the position of the learning subject 3, and create new learning depth image information.

[0090] Next, skeletal information is extracted from the learning depth image information (process S8). For example, the second skeletal extraction unit 41 extracts skeletal information of the learning subject 3 from the learning depth image information created in process S7. The second skeletal extraction unit 41 stores the extracted skeletal information in the storage unit .

[0091] Next, a discrimination model is created (process S9). For example, the discrimination model creation unit 47 creates, as a discrimination model, a model that outputs a state of the subject 3 corresponding to information on the input depth image information in response to input of information on the plurality of learning depth image information by learning using information on the plurality of learning depth image information created in process S7 and the state of the learning subject 3. For example, the discrimination model creation unit 47 creates, as a discrimination model, a model that outputs a state of the subject 3 corresponding to the input skeletal information in response to input of the plurality of learning skeletal information by learning using, for example, the plurality of learning skeletal information extracted in process S8 and the state of the learning subject 3.

[0092] The feature detection model created in process S9 is stored in the storage unit 42. When process S9 is completed, the series of processes in the method for creating an identification model is completed. Although an example of the method for creating an identification model has been described above, the order of each process is not limited to this. For example, either process S1 or process S2 may be performed first, or they may be performed in parallel. Instead of processes S1 to S6, distance information corresponding to the position of the subject 3 may be obtained from outside the condition identification system 1, or may be stored in advance in the storage unit 42.

[0093] Next, the depth image information creation process in step S7 will be described in detail with reference to Fig. 12. Fig. 12 is a flowchart showing an example of the depth image information creation process.

[0094] First, a range of distance values ​​corresponding to the position of the object 3 is identified (step S11). For example, the relational model creation unit 38 identifies a range of distance values ​​corresponding to the position of the learning object 3 among the positions of a plurality of divided regions as a first range. The relational model creation unit 38 identifies a range of distances closer than the first range as a second range, and identifies a range of distances farther than the first range as a third range.

[0095] Next, a relational model is created (process S12). For example, the relational model creation unit 38 creates a relational model in which a correspondence relationship between distance and index information indicating the distance is defined. The relational model creation unit 38 creates the relational model based on distance information corresponding to the position of the learning subject 3 among the positions of the multiple divided regions. For example, the relational model creation unit 38 updates the relational model corresponding to the depth image information acquired in process S2.

[0096] For example, the relationship model creation unit 38 creates a relationship model such that a rate of change in the index information relative to a change in the distance value in a first range identified in step S11 among the ranges of distance values ​​indicated by the index information is greater than a rate of change in the index information relative to a change in the distance value in a second range different from the first range. For example, the index information includes color information that indicates different colors according to the distance value. The index information is not limited to color information, and may be brightness or a numerical value.

[0097] Next, depth image information is created (process S13). For example, the depth image creation unit 39 creates a plurality of pieces of learning depth image information used for learning the discrimination model. The depth image creation unit 39 creates the plurality of pieces of learning depth image information based on the learning distance information acquired in process S2 and the relational model created in process S12. The plurality of pieces of learning depth image information include information on depth images captured in different scenes.

[0098] Information on the multiple depth images created in the process S13 is stored in the storage unit 42. When the process S13 ends, the process of creating the depth image information ends.

[0099] Next, an example of a state identification method in the estimation phase will be described with reference to Fig. 13. Fig. 13 is a flowchart showing an example of the state identification method.

[0100] First, information on an RGB image is acquired (step S21). In step S21, for example, the RGB image acquisition unit 31 acquires information on the RGB image. In the example shown in this embodiment, the information on the RGB image is an RGB image of the subject 3 captured by an imaging device.

[0101] Next, distance information is acquired (process S22). In process S22, the distance information acquisition unit 33 acquires, for example, distance information Pi corresponding to the RGB image acquired in process S21. For example, the distance information Pi corresponding to each RGB image is distance information acquired in the same scene as the RGB image. The distance information acquisition unit 33 acquires, for example, distance information of the subject 3 detected by the sensor unit 32. The distance information acquisition unit 33 acquires, for example, distance information Pi in the same scene as the information of the RGB image acquired by the RGB image acquisition unit 31. At this time, the area of ​​the real space from which the distance information acquisition unit 33 acquires the distance information Pi overlaps with the area of ​​the real space from which the RGB image acquisition unit 31 acquires the RGB image. For example, the distance information acquisition unit 33 acquires the distance information Pi as depth image information. For example, the process S21 and the process S22 are performed in parallel.

[0102] Next, the distance information and the RGB image are calibrated (process S23). In process S23, for example, the calibration unit 34 calibrates a plurality of RGB pixels of the RGB image and a plurality of pixels of the depth image acquired in process S22.

[0103] Next, skeleton information is extracted from the RGB image (process S24). In process S24, for example, the first skeleton extraction unit 35 extracts skeleton information from the RGB image acquired in process S21.

[0104] Next, characteristic information is acquired (process S25). In process S25, for example, the characteristic information acquisition unit 36 ​​acquires the characteristic information of the subject 3 from the RGB image acquired in process S21, and stores the acquired characteristic information in the storage unit 42. For example, process S24 and process S25 may be performed in parallel.

[0105] Next, the position of the subject 3 is identified (process S26). In process S26, for example, the position identifying unit 37 identifies distance information corresponding to the position of the subject 3 from the distance information acquired in process S22, based on the characteristic information of the subject 3 acquired in process S25.

[0106] Next, a process for creating depth image information is executed (process S27). In process S27, for example, the relation model creation unit 38 and the depth image creation unit 39 execute a process for creating depth image information based on the position of the subject 3 specified in process S26, similar to process S7. For example, in process S27, the relation model creation unit 38 and the depth image creation unit 39 update the depth image information acquired in process S22 based on distance information of the position of the subject 3, and create new depth image information.

[0107] Next, skeletal information is extracted from the depth image information (process S28). For example, the second skeletal extraction unit 41 extracts skeletal information of the subject 3 from the depth image information created in process S27. The second skeletal extraction unit 41 stores the extracted skeletal information in the storage unit 42.

[0108] Next, the state of the subject 3 is identified (process S29). For example, the identification processing unit 44 identifies the state of the subject 3 in the above-mentioned predetermined region based on the depth image information created in process S27. The identification processing unit 44 identifies the state of the subject 3 in the above-mentioned predetermined region based on the information on the depth image information created in process S27 and the identification model. For example, the identification model is trained by the above-mentioned learning data set, and outputs the state of the subject 3 in response to the input of the information on the depth image information. The information on the depth image information created in process S27 is, for example, the skeletal information extracted in process S28. The information on the depth image information created in process S28 may be, for example, the depth image information itself created in process S27.

[0109] The identification result identified in process S29 is stored in the storage unit 42 and output from the output unit 45. When process S29 is completed, the series of processes of the condition identification method is completed. Although an example of the condition identification method has been described above, the order of each process is not limited to this. For example, either process S21 or process S22 may be performed first. Instead of processes S21 to S26, distance information corresponding to the position of the subject 3 may be obtained from outside the condition identification system 1, or may be stored in advance in the storage unit 42.

[0110] As a modification of this embodiment, the steps S21, S23, S24, and S25 may not be performed. In this case, for example, in step S26, the position identifying unit 37 may identify distance information corresponding to the position of the subject 3 from the distance information acquired in step S22 by a predetermined algorithm. The predetermined algorithm may be, for example, a model learned in the learning phase.

[0111] Next, the effects of the condition identification system 1, the condition identification method, and the program in the above-described embodiment will be described.

[0112] In the above-mentioned condition identification system 1, condition identification method, and program, the depth image creation unit 39 creates depth image information of the predetermined region based on the relationship model created by the relationship model creation unit 38 and distance information. The relationship model creation unit 38 creates the relationship model based on distance information corresponding to the position of the subject 3 in the predetermined region among the positions of the multiple divided regions. The depth image information created in this manner can improve the identification accuracy of the condition of the subject 3.

[0113] For example, the relational model creation section 38 creates the relational model M2 so that the rate of change in the index information relative to the change in the distance value in a first range R1 corresponding to the position of the subject 3 within the above-mentioned predetermined region within the range of distance values ​​indicated by the index information is greater than the rate of change in the index information relative to the change in the distance value in a second range R2 different from the first range R1. In this case, the accuracy of identifying the state of the subject 3 can be further improved.

[0114] For example, the relational model creation unit 38 creates the relational model M2 such that the rate of change in the index information relative to the change in the distance value increases as the second range R2 approaches the first range R1. In this case, the robustness of the detection of the object in the depth image information can be improved.

[0115] For example, the index information includes color information that indicates different colors according to distance values, so that the state of the subject can be more easily identified.

[0116] For example, the position identifying unit 37 identifies the position of the subject 3 within the predetermined region based on feature information indicating the position of the subject 3. The feature information includes information indicating the position of an RGB pixel related to the subject among a plurality of RGB pixels included in an RGB image obtained by capturing an image of the subject 3. The relation model creating unit 38 creates a relation model based on the position of the subject 3 identified by the position identifying unit 37 and distance information corresponding to the position of the subject 3. Since the relation model creating unit 38 creates a relation model based on the information of the RGB image and the distance information, the relation model creating unit 38 can create a relation model that more accurately corresponds to the position of the subject 3.

[0117] For example, the identification processing unit 44 acquires an identification model that associates information related to the depth image information created by the depth image creation unit 39 with the state of the subject. The identification processing unit 44 identifies the state of the subject in the above-mentioned predetermined region based on the information related to the depth image information created by the depth image creation unit 39 and the identification model. In this case, the identification model associates information related to the depth image information created based on the relation model created by the relation model creation unit 38 with the state of the subject. Therefore, the state of the subject 3 can be identified more easily and reliably.

[0118] For example, the discrimination model creation unit 47 creates a discrimination model based on information on the learning depth image information created by the depth image creation unit 39 and the state of the learning subject 3. In this case, the discrimination model is created using the learning depth image information created by the depth image creation unit 39. Therefore, according to the discrimination model, the discrimination accuracy of the subject's state can be further improved.

[0119] Next, an example of verification of the above-mentioned condition identification system 1 and the condition identification method by steps S1 to S29 will be described with reference to Fig. 14 to Fig. 16. Hereinafter, the condition identification method using the above-mentioned condition identification system 1 and the condition identification method by steps S1 to S29 will be referred to as "the proposed method".

[0120] In this verification, the presence or absence of the subject 3 rolling over was determined using data of the subject 3 rolling over and data of the subject 3 not rolling over. The data of the subject 3 not rolling over includes 465 pieces of data of the subject 3 walking, 110 pieces of data of the subject 3 squatting, 64 pieces of data of the subject 3 lying down, and 7 pieces of data of the subject 3 doing push-ups. The data of the subject rolling over includes 106 pieces of data. A VGG19 model was used to extract the skeletal information of the subject 3, and the learning rate was 4e -5 The batch size is 10, and the epoch is 200. The LSTM model is used to extract the detected skeletal information of the state of the subject 3, and the learning rate is 5e -3 where BatchSize is 64 and Epoch is 70000.

[0121] 14(a) and 14(b) show depth images in the process of the subject rolling over. FIG. 14(a) shows group G1 of a series of depth images acquired by distance information acquisition unit 33. FIG. 14(b) shows group G2 of a series of depth images created using a relational model updated by relational model creation unit 38. In group G1, the subject 3 is buried in the background, whereas in group G2, the boundary between the subject 3 and the background can be clearly seen. As a result, more skeletal information of the subject 3 is extracted in group G2 than in group G1.

[0122] FIG. 15 is a graph showing a comparison of feature information extracted from depth image information with and without updating by the relational model creation unit 38. In FIG. 15, data DA11 is data of depth image information before the relational model M2 created by the relational model creation unit 38 is applied. The data DA11 corresponds to data of depth image information acquired by the distance information acquisition unit 33. The data DA12 is data of depth image information after the relational model M2 created by the relational model creation unit 38 is applied. As shown in FIG. 15, when the relational model M2 created by the relational model creation unit 38 is applied, the number of connections of the extracted skeletal information of the subject 3 is increased. It has been confirmed that the skeletal information of the subject 3 is easily detected by applying the relational model M2 created by the relational model creation unit 38.

[0123] 16 is a diagram showing data DA20 of the performance of the detection result of state classification. In the data DA20, a "True label" indicating whether the subject 3 has actually rolled over and a "Predicted label" indicating whether the subject 3 has rolled over in a prediction are shown. The data DA20 shows the comparison result between the "True label" and the "Predicted label" in terms of depth image information before and after the relational model M2 created by the relational model creating unit 38 is applied.

[0124] According to the depth image information before the relational model M2 created by the relational model creation unit 38 is applied, the PCK (Percentage of Correct Keypoints) of "rollover", "squat", "walking", and "push-ups" are 53.3%, 68.1%, 65.0%, and 66.6%, respectively. According to the depth image information after the relational model M2 created by the relational model creation unit 38 is applied, the PCK of "rollover", "squat", "walking", and "push-ups" are 69.5%, 83.6%, 91.5%, and 72.0%, respectively.

[0125] According to the depth image information before the relational model M2 created by the relational model creation unit 38 is applied, the connectivity rates of the extracted skeletal information in "rollover", "squat", "walking", and "push-ups" were 46.3%, 58.9%, 43.4%, and 48.1%, respectively. According to the depth image information after the relational model M2 created by the relational model creation unit 38 is applied, the connectivity rates of the extracted skeletal information in "rollover", "squat", "walking", and "push-ups" were 60.0%, 81.2%, 92.2%, and 67.5%, respectively. As described above, it was confirmed that the connectivity of the skeletal information was improved in all of "rollover", "squat", "walking", and "push-ups", and that the PCK value was also improved.

[0126] According to the depth image information before the relational model M2 created by the relational model creation unit 38 is applied, the "Accuracy", "Precision", "Recall", and "F1 Score" are 85.2%, 94.2%, 75.0%, and 83.5%, respectively. According to the depth image information after the relational model M2 created by the relational model creation unit 38 is applied, the "Accuracy", "Precision", "Recall", and "F1 Score" are 98.5%, 97.0%, 100.0%, and 98.5%, respectively. From the above, it was confirmed that the accuracy of the prediction result is improved when the relational model M2 created by the relational model creation unit 38 is applied.

[0127] Next, a state identification system in a modified example of this embodiment will be described with reference to Figs. 17 to 20. Fig. 17 is a schematic diagram for explaining a state identification system in a modified example of this embodiment. Figs. 18 to 20 are diagrams showing the performance of the detection results of state identification by the state identification system in the modified example of this embodiment. This modified example is generally similar or the same as the above-mentioned embodiment. This modified example differs from the above-mentioned embodiment in terms of the creation of a relational model. Below, differences between the above-mentioned embodiment and the modified example will be mainly described. In this modified example, the state identification system 1A has the same functional units as the state identification system 1.

[0128] First, each functional unit of the condition identification system 1A in the estimation phase will be described in more detail. The condition identification system 1A identifies the position of the subject to be identified by dividing it into a plurality of regions, and creates a relational model for each region. The condition identification system 1A creates depth image information according to the relational model corresponding to each region, and identifies the state of the subject 3 to be identified based on the plurality of depth image information created corresponding to each region.

[0129] For example, in the condition identification system 1A, the position identifying unit 37 identifies the position of the subject 3 by dividing it into a plurality of regions based on the feature information acquired by the feature information acquiring unit 36. The plurality of regions are, for example, the positions of a plurality of parts of the subject 3. The positions of the plurality of parts are, for example, the centers of each part. The position identifying unit 37 is, for example, YOLOv5. For example, the position identifying unit 37 identifies the center of the entire subject 3, the center of the upper body of the subject 3, and the center of the lower body of the subject 3. The center of the upper body of the subject 3 is, for example, the neck joint. The position identifying unit 37 identifies a box region corresponding to the center of each part of the subject 3, and identifies distance information in the box region.

[0130] The relational model creating unit 38 creates a plurality of relational models based on the position of each part of the subject 3 identified by the position identifying unit 37 and distance information corresponding to the position of the part. The relational model creating unit 38 creates a plurality of relational models based on the center of each part of the subject 3 identified by the position identifying unit 37 and distance information corresponding to the box region. The created plurality of relational models correspond to different parts of the subject 3. The relational model creating unit 38 creates a relational model based on distance information corresponding to the position of the subject 3 among the positions of the plurality of divided regions, and updates the relational model corresponding to the depth image information acquired by the distance information acquiring unit 33. The relational model creating unit 38 identifies a first range R1 corresponding to each of the plurality of parts based on distance values ​​in a plurality of box regions RA10 corresponding to the positions of the plurality of parts of the subject 3, respectively.

[0131] The depth image creation unit 39 creates depth image information of the above-mentioned multiple regions based on the distance information acquired from the distance information acquisition unit 33 and the multiple relationship models created by the relationship model creation unit 38. For example, the depth image creation unit 39 creates multiple pieces of depth image information indicating, for each pixel, index information corresponding to the distance information in each of the multiple relationship models created based on the distance information acquired from the distance information acquisition unit 33 and the multiple relationship models created by the relationship model creation unit 38. For example, the depth image creation unit 39 creates multiple pieces of second depth image information indicating, for each pixel, index information corresponding to the multiple updated relationship models based on the first depth image information acquired from the distance information acquisition unit 33 and the multiple relationship models updated by the relationship model creation unit 38 from the relationship model in the first depth image information. The depth image creation unit 39 creates multiple second depth images indicating, for each pixel, index information corresponding to the multiple updated relationship models based on the first depth image acquired from the distance information acquisition unit 33 and the multiple updated relationship models.

[0132] The second skeleton extraction unit 41 extracts skeleton information of the subject 3 from each of the plurality of pieces of depth image information created by the depth image creation unit 39. The second skeleton extraction unit 41 extracts skeleton information of the subject 3, for example, from each of the plurality of depth images respectively corresponding to a plurality of parts of the subject 3. The second skeleton extraction unit 41 stores the extracted skeleton information in the storage unit 42. The second skeleton extraction unit 41 may output the extracted skeleton information to the discrimination processing unit 44 or the discrimination model creation unit 47.

[0133] The discrimination processing unit 44 discriminates the state of the subject 3 in the above-mentioned predetermined region based on, for example, information on the multiple depth images created by the depth image creation unit 39 and a discrimination model. The discrimination model associates information on the multiple depth images created by the depth image creation unit 39 with the state of the subject 3. The discrimination model outputs the state of the subject 3 in response to input of information on the multiple depth images created by the depth image creation unit 39.

[0134] Next, each functional unit of the condition identification system 1A in the learning phase will be described in more detail. Some explanations that overlap with those in the estimation phase will be omitted. The condition identification system 1A identifies the positions of the learning objects to be identified by dividing them into a plurality of regions, and creates a relational model for each region.

[0135] For example, in the condition identification system 1A, the position identifying unit 37 identifies the position of the learning subject 3 by dividing it into a plurality of regions based on the learning feature information acquired by the feature information acquiring unit 36. The position identifying unit 37 identifies distance information corresponding to the position of the learning subject 3 among the distance information acquired by the distance information acquiring unit 33 based on the feature information of the learning subject 3 acquired by the feature information acquiring unit 36. The plurality of regions are, for example, the positions of a plurality of parts of the learning subject 3. For example, the position identifying unit 37 identifies the center of the learning subject 3, the center of the upper body of the learning subject 3, and the center of the lower body of the learning subject 3. The center of the upper body of the subject 3 is, for example, the neck joint. The position identifying unit 37 identifies a box region corresponding to the center of each part of the learning subject 3, and identifies distance information in the box region. For example, the position identifying unit 37 performs transfer learning based on the feature information of each part of the learning subject 3 acquired by the feature information acquiring unit 36. For example, the position specifying unit 37 specifies the positions of a plurality of feature points of each part of the learning subject 3 in the learning depth image information acquired by the distance information acquiring unit 33 through transfer learning.

[0136] The relational model creating unit 38 creates a relational model based on the position of each part of the learning subject 3 identified by the position identifying unit 37 and distance information corresponding to the position of the part. The relational model creating unit 38 creates a plurality of relational models based on the center of each part of the learning subject 3 identified by the position identifying unit 37 and distance information corresponding to the box region. The created plurality of relational models correspond to different parts of the learning subject 3. The relational model creating unit 38 creates a relational model based on distance information corresponding to the position of the learning subject 3 among the positions of the plurality of partition regions, and updates the relational model corresponding to the depth image information acquired by the distance information acquiring unit 33. The relational model creating unit 38 identifies a first range R1 corresponding to each of the plurality of parts based on distance values ​​in a plurality of box regions corresponding to the positions of the plurality of parts of the learning subject 3, respectively.

[0137] The depth image creation unit 39 creates learning depth image information of the above-mentioned multiple regions based on the learning distance information acquired from the distance information acquisition unit 33 and the multiple relation models created by the relation model creation unit 38. For example, the depth image creation unit 39 creates learning depth image information indicating, for each pixel, index information corresponding to the distance information in each of the multiple relation models created based on the learning distance information acquired from the distance information acquisition unit 33 and the multiple relation models created by the relation model creation unit 38. In this modification, each learning depth image information includes information on multiple feature pixels where multiple feature points of each part of the subject 3 corresponding to the learning depth image information are located.

[0138] The second skeleton extraction unit 41 extracts skeleton information of the learning subject 3 from a plurality of pieces of learning depth image information created by the depth image creation unit 39. The second skeleton extraction unit 41 extracts skeleton information of the learning subject 3 from, for example, each of a plurality of learning depth images corresponding to a plurality of parts of the learning subject 3, respectively.

[0139] The learning data set used for training the discrimination model includes information on a plurality of pieces of learning depth image information created by the depth image creation unit 39, and state information on the learning subject 3. The discrimination model creation unit 47 creates a discrimination model using the above-mentioned learning data set.

[0140] The condition identification system 1A, the condition identification method corresponding to the condition identification system 1A, and the program have the same effects as the condition identification system 1, the condition identification method, and the program in the above-mentioned embodiment. According to the condition identification system 1A, the condition identification method corresponding to the condition identification system 1A, and the program, the accuracy of identifying the subject's condition can be further improved.

[0141] Next, an example of verification of the above-described condition identification system 1A and a condition identification method using the condition identification system 1A will be described with reference to Fig. 18 to Fig. 20. Hereinafter, the condition identification method using the above-described condition identification system 1A will be referred to as "the proposed method".

[0142] In FIG. 18, skeleton extraction results from depth images corresponding to various states of the subject 3 are shown. The "Open Pose" row in FIG. 18 shows PCK using the original depth images without using a relational model. The "SDNLM (Single Dynamic Nonlinear Mapping)" row shows PCK when a relational model updated based on the subject's global center is applied. Hereinafter, the case where a relational model updated based on the subject's global center is applied is simply referred to as "SDNLM". According to the "SDNLM" row, the average value of all actions is improved by 16.4%.

[0143] The row "MDNLM (Multiple Dynamic Nonlinear Mapping)" shows the PCK when the relational model updated based on the center of multiple parts of the subject is applied. Hereinafter, the case where the relational model updated based on the center of multiple parts of the subject is applied is simply referred to as "MDNLM". According to the row "MDNLM", the average value of all actions is further improved by 30.0%. PCK is improved across all actions, regardless of the difference between activities with few obstacles to the skeleton and activities with many body overlaps or obstacles. Activities with few obstacles to the skeleton are, for example, walking and wiping a desk. Activities with many body overlaps or obstacles are, for example, squatting, lying down, and falling.

[0144] In Figure 19, the performance (precision, precision, recall, F1 score) of three classification tasks, 2-class, 7-class, and 10-class, is shown. For the 2-class classification task (fall detection), MDNLM achieves perfect accuracy of 100%. For the 2-class classification task (fall detection), MDNLM shows 4.1% and 0.8% improvement in accuracy compared to the original depth images without relational model and SDNLM, respectively, despite the inclusion of confusing activities such as lying in the non-fall dataset.

[0145] For the 7-class classification task, MDNLM achieves an accuracy of 99.0%, which is an improvement of 32.0% over the original depth images without the relational model and 14.0% over SDNLM. For the 10-class classification task, MDNLM achieves an improvement of 39.0% and 19.5% over the original depth images without the relational model and SDNLM, respectively.

[0146] Figure 20 shows the confusion matrix of SDNLM and MDNLM. In the fall detection task, the results of MDNLM were completely accurate. In the 10-class classification task, moving furniture causes confusion with very similar actions such as walking. In this regard, MDNLM captures a sufficient number of skeletal joints from the depth images, which is believed to overcome the difficult problem. Furthermore, we believe that the classification network effectively captures temporal features and identifies subtle differences between similar activities, which enables accurate classification.

[0147] The above describes the embodiments and modifications of the present invention, but the present invention is not necessarily limited to the above-described embodiments and modifications, and various modifications are possible without departing from the spirit of the present invention.

[0148] For example, the identification by the state identification system 1 is not limited to the rollover of the subject. For example, the individual subject may be identified according to the movement of the subject. For example, the individual subject may be identified based on the depth image information and the identification model. [Explanation of symbols]

[0149] 1...state identification system, 3...subject, 33...distance information acquisition unit, 37...position identification unit, 38...relationship model creation unit, 39...depth image creation unit, 44...identification processing unit, 47...identification model creation unit, DA2...feature information, DA6...index information, DA7,Pi...distance information, M2...relationship model, R1...first range, R2...second range.

Claims

1. a distance information acquisition unit that acquires distance information indicating a distance from a reference position in a real space to each position of a plurality of partitioned regions into which a predetermined area in the real space is partitioned; a relational model creating unit that creates a relational model in which a correspondence relationship between the distance and index information indicating the distance is defined; a depth image creation unit that creates depth image information of the predetermined area based on the distance information acquired from the distance information acquisition unit and the relation model created by the relation model creation unit; an identification processing unit that identifies a state of the subject in the predetermined region based on the depth image information created by the depth image creation unit; A state identification system, wherein the relational model creation unit creates the relational model based on the distance information corresponding to a position of the subject within the specified area among the positions of the multiple divided areas.

2. 2. The condition identification system according to claim 1, wherein the relational model creation unit creates the relational model so that a ratio of a change in the index information to a change in the distance value in a first range corresponding to the position of the subject within the specified region, within the range of the distance values ​​indicated by the index information, is greater than a ratio of a change in the index information to a change in the distance value in a second range different from the first range.

3. The condition identification system according to claim 2 , wherein the relational model creation unit creates the relational model such that, in the second range, the closer the range is to the first range, the higher the rate of change in the index information relative to the change in the distance value becomes.

4. The condition identification system according to claim 1 , wherein the index information includes color information that indicates a different color depending on the distance.

5. a position specifying unit that specifies a position of the subject within the predetermined region based on feature information indicating the position of the subject, the feature information includes information indicating positions of RGB pixels related to the subject among a plurality of RGB pixels included in an RGB image obtained by capturing an image of the subject, The condition identification system according to claim 1 , wherein the relational model creation unit creates the relational model based on the position of the subject identified by the position identification unit and the distance information corresponding to the position of the subject.

6. The condition identification system of claim 1 or 2, wherein the identification processing unit acquires an identification model that associates information regarding the depth image information created by the depth image creation unit with the condition of the subject, and identifies the condition of the subject in the specified area based on the information regarding the depth image information created by the depth image creation unit and the identification model.

7. A discrimination model creation unit that creates the discrimination model, The condition identification system of claim 6, wherein the identification model creation unit creates the identification model based on information regarding the learning depth image information created by the depth image creation unit and the condition of the learning subject.

8. acquiring distance information from a reference position in a real space to each position of a plurality of partitioned regions into which a predetermined area in the real space is partitioned; creating a relation model in which a correspondence relationship between the distance and index information indicating the distance is defined; creating depth image information of the predetermined area based on the acquired distance information and the created relational model; and identifying a state of the subject in the predetermined region based on the created depth image information; A state identification method, in which, in creating the relational model, the relational model is created based on the distance information corresponding to a position of the subject within the specified region among the positions of the multiple divided regions.

9. acquiring distance information from a reference position in a real space to each position of a plurality of partitioned regions into which a predetermined area in the real space is partitioned; creating a relation model in which a correspondence relationship between the distance and index information indicating the distance is defined; creating depth image information of the predetermined area based on the acquired distance information and the created relational model; and identifying a state of the subject in the predetermined region based on the created depth image information. A program for creating the relational model, the program creating the relational model based on the distance information corresponding to the position of the subject within the specified region among the positions of the plurality of divided regions.

Images