Sign detection device, sign detection method, and recording medium

Abstract

To appropriately detect signs for upcoming behavior of a subject from image data in which the subject is photographed.SOLUTION: A sign detection device acquires image data in which a subject is photographed and acquires posture information by estimating a posture of the subject from the image data. The sign detection device acquires an emotion value by estimating an emotion of the subject from the posture information. The sign detection device stores the posture information and the emotion value in a storage unit in chronological order. By referring to the storage unit, the sign detection device detects signs of some upcoming behavior by the subject from fluctuations in the posture information and fluctuations in the emotion value.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a device for detecting a sign related to a subject's behavior. [Background technology]

[0002] Elderly people and patients with dementia (hereinafter simply referred to as "patients") sometimes exhibit abnormal behaviors that may appear sudden when observed objectively, such as wandering or sudden violent outbursts, leaving their families and caregivers struggling to deal with the situation. For example, even if patients are given a device capable of measuring their location, such as a Global Positioning System (GPS) (hereinafter also referred to as a "positioning device"), as a way to prevent wandering, the patient will often remove it. Therefore, if the patient is forced to wear a positioning device that cannot be removed, there is a concern that the patient may injure themselves by trying to tear it off or by scratching themselves, or that the device may get caught on some structure, leading to an accident. For this reason, it is difficult to have patients wear positioning devices.

[0003] For this reason, a system has been proposed that uses a camera to monitor a patient and detect abnormal behavior by the patient. For example, Patent Document 1 describes a prediction device that can predict the occurrence of symptoms such as wandering based on information about the patient's posture. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2023 / 058143 Summary of the Invention [Problem to be solved by the invention]

[0005] However, a problem with systems that predict behavior from posture information is that they can only detect signs of behavior from the continuity of movements. Checking each patient's individual movement habits over time requires enormous computational resources, and even if machine learning is used, it requires a huge amount of training data. Furthermore, in either case, if the patient performs a movement that does not exist in the combination, there is a high possibility that signs of behavior will not be detected.

[0006] One of the objects of the present invention is to appropriately detect signs of a target person's behavior from image data of the target person. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, in one aspect of the present invention, a sign detection device includes: image acquisition means for acquiring image data of a subject; a posture estimation means for estimating a posture of the subject from the image data to acquire posture information; an emotion estimation means for estimating an emotion of the subject from the posture information to acquire an emotion value; a time-series storage means for storing the posture information and the emotion values ​​in time series; a sign detection means for detecting a sign of some kind of behavior by the subject based on the change in the posture information and the change in the emotion value; Equipped with.

[0008] In another aspect of the present invention, a method for detecting a sign includes: is executed by a computer, Acquire image data of the subject, acquiring posture information by estimating a posture of the subject from the image data; acquiring an emotion value by estimating an emotion of the subject from the posture information; storing the posture information and the emotion value in a time series in a storage unit; A sign of some kind of behavior by the subject is detected from the change in the posture information and the change in the emotion value.

[0009] In another aspect of the invention, a program includes: Acquire image data of the subject, acquiring posture information by estimating a posture of the subject from the image data; acquiring an emotion value by estimating an emotion of the subject from the posture information; storing the posture information and the emotion value in a time series in a storage unit; The computer is caused to execute a process of detecting a sign of some kind of behavior by the subject from the change in the posture information and the change in the emotion value. [Effects of the Invention]

[0010] According to the present invention, it is possible to appropriately detect signs of a target person's behavior from image data obtained by capturing an image of the target person. [Brief explanation of the drawings]

[0011] [Figure 1] The configuration of the warning sign detection system is shown below. [Figure 2] 1 shows the hardware configuration of a warning sign detection device. [Figure 3] 1 shows the functional configuration of a warning sign detection device. [Figure 4] 10 is an example of notification of a sign detection result. [Figure 5] 10 is an example of image data of a subject. [Figure 6] 10 is an example of a change in posture information and an accompanying change in emotion value. [Figure 7] 10 is a flowchart of a behavior warning process. [Figure 8] 10 is a flowchart of a position warning process. [Figure 9] 10 is a flowchart of emotion correction value update processing. [Figure 10] 10 is a flowchart of a caution correction value update process. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. [Hardware configuration] FIG. 1 shows the configuration of a sign detection system incorporating the sign detection device of the present invention. The sign detection system 100 monitors and predicts emotional fluctuations based on the posture of a subject estimated from image data of the subject, and detects signs of the subject's actions. Specifically, in this embodiment, the subject is an elderly person or a dementia patient with cognitive problems. The sign detection system 100 can detect signs of unexpected behavior (hereinafter also referred to as "abnormal behavior"), such as the subject's wandering or sudden violent outbursts. In other words, the sign detection system 100 can assist supervisors of individuals with cognitive problems, such as doctors and other medical professionals or caregivers, in making decisions to prevent the subject from engaging in abnormal behavior. Note that, although this embodiment detects signs of abnormal behavior, the present disclosure is not limited to this. The sign detection system 100 can detect signs of any behavior, including unexpected and expected behavior, using any settings.

[0013] In the sign detection system 100, a photographing device 10 and a sign detection device 20 are communicatively connected via a network 5 such as the Internet. The photographing device 10 is configured with a depth sensor (3D sensor) such as a Lidar or a ToF (Time of Flight) camera, is installed in a specific location such as the subject's room, and constantly photographs the subject. The photographing device 10 is also capable of transmitting and receiving data via the network 5, and image data D1 captured by the photographing device 10 is sequentially transmitted to the sign detection device 20. In this embodiment, the image data D1 is a video captured of the subject, but it may also be a series of still images in a time series. The sign detection device 20 is an information processing device that processes, stores, and transmits and receives various data, and analyzes the image data D1 to detect signs of abnormal behavior in the subject.

[0014] In this embodiment, the image capturing device 10 is configured, for example, with a depth sensor, and image data D1 captured by the image capturing device 10 is used for processing by the sign detection device 20. However, the present disclosure is not limited to this, and the image capturing device 10 may be an RGB camera such as a general digital camera or a surveillance camera. In this case, the sign detection device 20 acquires image data equivalent to the image data D1 by performing estimation using AI based on the image data acquired from the image capturing device 10. Alternatively, image data equivalent to that of a depth sensor may be acquired by beam sensing, ultrasonic sensing, or the like, and the acquisition method can be set as desired.

[0015] 2 is a block diagram showing an example of the hardware configuration of the sign detection device 20. As shown in the figure, the sign detection device 20 includes an interface 21, a processor 22, a memory 23, a recording medium 24, a display unit 25, and an input unit 26. These components, an emotion estimation DB 31, a time-series DB 32, an emotion correction value DB 33, and an attention correction value DB 34 are interconnected via a bus.

[0016] The interface 21 exchanges data with the image capturing device 10. The interface 21 is used when receiving image data D1 from the image capturing device 10. The interface 21 is also used when the sign detection device 20 exchanges data with a predetermined device connected by wire or wirelessly.

[0017] The processor 22 is a computer such as a CPU (Central Processing Unit), and executes a prepared program to control the entire sign detection device 20. Note that the processor 22 may be a CPU, a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating Point number Processing Unit), a PPU (Physics Processing Unit), a TPU (Tensor Processing Unit), a quantum processor, a microcontroller, or a combination thereof.

[0018] The memory 23 is composed of a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The memory 23 stores programs executed by the processor 22. The memory 23 is also used as a working memory while the processor 22 is executing various processes.

[0019] The recording medium 24 is a non-volatile, non-transitory recording medium such as a disk-shaped recording medium or semiconductor memory, and is configured to be detachable from the sign detection device 20. The recording medium 24 records various programs to be executed by the processor 22. When the sign detection device 20 executes the sign detection process, the programs recorded on the recording medium 24 are loaded into the memory 23 and executed by the processor 22.

[0020] The display unit 25 displays a predetermined image on, for example, an LCD (Liquid Crystal Display), etc. The input unit 26 includes a keyboard, mouse, touch panel, etc., and is used by an operator who manages the sign detection device 20, such as a supervisor.

[0021] The emotion estimation DB 31 stores a group of learning data used to estimate an emotion value that represents the emotion of a person from posture information that represents the posture of the person.

[0022] The time series DB32 stores posture information representing the posture of the subject estimated from the image data D1, an emotion value estimated based on the posture, and position information of the subject in space in a time series, in association with the subject's identification information.

[0023] The emotion correction value DB 33 stores emotion correction values, which are combinations of posture information and emotion values ​​specific to a subject, in association with the identification information of the subject.

[0024] If there is a change in the emotional value of a subject again during the time period when the subject exhibited abnormal behavior, the attention correction value DB34 stores an attention correction value, which is a combination of the change in the emotional value and the change in posture information when the emotional value changed, in association with the subject's identification information.

[0025] For ease of explanation, the sign detection device 20 has an emotion estimation DB 31, a time series DB 32, an emotion correction value DB 33, and an attention correction value DB 34, but the present invention is not limited to this, and the type of DB and the data structure of the DB are arbitrary as long as the necessary data can be obtained. [Function Configuration]

[0026] 3 is a block diagram showing an example of the functional configuration of the sign detection device 20. Functionally, the sign detection device 20 includes an image acquisition unit 41, a posture estimation unit 42, a position detection unit 43, a feeling estimation unit 44, a sign detection unit 45, an alarm unit 46, an input / output unit 47, and a correction unit 48. The image acquisition unit 41, the posture estimation unit 42, the position detection unit 43, the feeling estimation unit 44, the sign detection unit 45, the alarm unit 46, the input / output unit 47, and the correction unit 48 are realized by the processor 22 executing a program.

[0027] The image acquisition unit 41 acquires image data D1 of the subject from the photographing device 10. The acquired image data D1 is passed to the posture estimation unit 42. Note that the method of identifying the subject from the image data D1 can be a method using any predetermined information, such as location information indicating that the subject is in a specific position or biometric information regarding physical characteristics.

[0028] The posture estimation unit 42 acquires posture information by calculating the posture estimation result of the subject from the image data D1. Specifically, the posture estimation unit 42 acquires posture information by analyzing three-dimensional point cloud data based on the image data D1. The posture information is a coordinate group obtained by converting position information of specific body parts and joints constituting the subject into relative coordinates with the coordinate of the waist as the origin, and the posture of the subject can be estimated from the posture information. Note that using the coordinate of the waist as the origin is just one example, and the coordinate of the chest or the coordinate of the abdomen may also be used as the origin. Alternatively, using the coordinate of the head as the origin can be used as long as it can correct the point where the relative position coordinates of the entire body rotate due to head movement. The same applies to using the coordinates of the hands and feet on the left and right as the origin.

[0029] The position detection unit 43 acquires position information of the subject in space. Because the image capture device 10 is configured with a depth sensor, the position detection unit 43 can measure the distance from the image capture device 10 to an object or the distance between two objects based on the image data D1. Here, the position coordinates of the subject are different from posture information and are information indicating where the subject is located in the capture space, and can be acquired from the image data D1 by providing the installation position of the image capture device 10 and map information of the room in advance. The position information makes it possible to identify where the subject is located in the capture space, for example, whether the subject is near a bed or near a doorway.

[0030] The emotion estimation unit 44 estimates the emotion value of the subject at that time from the posture information of the subject, the learning data group accumulated in the emotion estimation DB 31, and the emotion correction value of the subject stored in the emotion correction value DB 33. In this embodiment, the emotion value is a value that represents the emotion of a person, and is a combination of a type item that is the type of emotion and a numerical value that indicates the degree of the emotion as a percentage.

[0031] Examples of category items include sadness, gladness, madness, nervous, impatience, anger, hungry, thirsty, hurt, sleepy, loneliness, nervous, anger, impatience, anger, guilt, embarrassment, exhaustion, anxiety, disappointment, surprise, fear, confidence, and contempt.

[0032] The type items can be set arbitrarily to include not only joy, anger, sadness, and happiness, but also various other emotions during design. Furthermore, the emotion value is not limited to a combination of the type item and a numerical value, and can be set arbitrarily as long as it can represent a person's emotion numerically.

[0033] Specifically, the emotion estimation unit 44 creates training data based on a group of learning data accumulated in the emotion estimation DB 31, with posture information as input data and the corresponding emotion values ​​as correct data, and generates an emotion estimation model by training a machine learning model with the training data. The emotion estimation unit 44 then uses the emotion estimation model to input the posture information of the subject and acquires, as the emotion estimation result, an emotion value that is output by the input. Furthermore, if an emotion correction value of the subject is stored in the emotion correction value DB 33, the emotion estimation unit 44 corrects the emotion value based on the emotion correction value and acquires the corrected emotion value as the emotion estimation result. The emotion values ​​acquired as the emotion estimation result are stored in chronological order in the time series DB 32 together with the posture information and position information of the subject.

[0034] The sign detection unit 45 detects signs of abnormal behavior by determining whether there is an increasing likelihood that the subject will attempt to perform abnormal behavior from continuous fluctuations in the subject's posture information and continuous fluctuations in the emotion value with reference to the time series DB 32. At this time, if an attention correction value for the subject is stored in the attention correction value DB 34, the sign detection unit 45 determines whether there is an increasing likelihood that the subject will attempt to perform abnormal behavior taking into account the attention correction value, and detects signs by prioritizing this determination.

[0035] Specifically, the sign detection unit 45 generates a sign detection model by having a machine learning model learn training data in which the input data is a change in posture information and a change in emotion value, and the corresponding possibility of abnormal behavior is used as correct answer data. In this embodiment, the possibility of abnormal behavior is expressed in five levels, from 0 to 4, with level 0 being the lowest possibility of abnormal behavior and level 4 being the highest possibility of abnormal behavior.

[0036] Then, the sign detection unit 45 uses the sign detection model to input the variation in posture information of the subject and the variation in emotion value, and acquires the possibility of abnormal behavior as a sign detection result. The possibility of abnormal behavior acquired as a sign detection result is notified to the alarm unit 46, which will be described later. In this embodiment, if a warning level of 1 to 4 is acquired as the sign detection result, the sign detection unit 45 determines that a sign of abnormal behavior has been detected. On the other hand, if a warning level of 0 is acquired as the sign detection result, the sign detection unit 45 determines that a sign of abnormal behavior has not been detected.

[0037] Figure 4 shows an example of a notification of the sign detection result. As shown in Figure 4, the attention level, which indicates the possibility of abnormal behavior, is expressed as a numerical value and the corresponding number of black stars. In this case, as shown in Figure 4, the notification may also include the subject's latest emotional value.

[0038] Figure 5 shows an example of image data captured by a camera 10 installed above a room. It shows a series of actions taken by the subject, from sitting on the bed to walking to the doorway and going outside. The attention level increases only if the subject's emotions are judged to fluctuate significantly in the negative direction during the action. Specifically, Figure 5 shows five scenes extracted from image data D1, with pattern A consisting of scenes 50, 51, and 52, and pattern B consisting of scenes 50, 53, and 54, and shows the subject's emotional value and attention level in each scene.

[0039] In pattern A, the subject goes from "sitting with head up" (scene 50) to maintaining the same posture for a while (scene 51) and then "standing up and facing the entrance to open the door" (scene 52). From scene 50 to scene 51, there is no change in posture information, so there is also little change in emotional value, and the attention level drops from 1 to 0. Also, from scene 51 to scene 52, there is a change in posture information as the subject goes from "sitting" to "standing up and facing the entrance to open the door," and as a result there is some change in emotional value, so the attention level rises from 0 to 1.

[0040] On the other hand, in pattern B, the subject changes from "sitting with head held high" (scene 50) to "holding head" (scene 53), and then to "standing up and facing the entrance to open the door" (scene 54). From scene 50 to scene 53, the subject's posture information fluctuates significantly as they change from "sitting with head held high" to "holding head," and the emotional values ​​also fluctuate significantly, with nervousness increasing by 70% and anger increasing by 60%. Due to these changes in posture information and emotional values, the attention level rises significantly from 0 to 3. Furthermore, from scene 53 to scene 54, the subject changes from "holding head" to "standing up and facing the entrance to open the door," and the emotional values ​​also fluctuate significantly, with madness increasing by 20% and impatience increasing by 24%. Due to these changes in posture information and emotional values, the attention level rises from 3 to 4, which indicates the highest possibility of abnormal behavior.

[0041] Note that Fig. 5 is just one example, and other possible variations in posture information and accompanying variations in emotional values ​​are as shown in Fig. 6. In the example of Fig. 5, the attention level increases when it is determined that the emotion is fluctuating significantly in the negative direction, but the present disclosure is not limited to this, and even in the case of positive or neutral emotions, signs of abnormal behavior can be detected from variations in posture information and accompanying variations in emotional values.

[0042] When the warning unit 46 determines that there is a high possibility of abnormal behavior based on the notification from the sign detection unit 45, in other words, when it detects a sign of abnormal behavior, it outputs a behavior warning instruction to the supervisor via the input / output unit 47. Specifically, when it detects a sign of abnormal behavior, the warning unit 46 outputs a behavior warning instruction. On the other hand, when it does not detect a sign of abnormal behavior, the warning unit 46 does not output a behavior warning instruction.

[0043] Furthermore, by operating the sign detection device 20, the supervisor can pre-register location information indicating the location of caution areas where the subject should be careful not to enter, such as entrances and restricted areas. This allows the alarm unit 46 to output a location alarm instruction to the supervisor via the input / output unit 47 when the value of the subject's location information is the value of location information for a caution area. Such a location alarm instruction is used as an aid for storing attention correction values ​​in the attention correction value DB 34 in the early stages of introducing the sign detection system 100.

[0044] It should be noted that outputting a location alarm is not essential, and the supervisor can set in advance whether or not to output a location alarm in response to the movement of the subject.

[0045] The input / output unit 47 is used to visualize information and issue alarms to the supervisor, as well as to issue commands to the sign detection device 20 by command input, GUI (Graphical User Interface) operation, etc. In this embodiment, the input / output unit 47 is provided in the sign detection device 20, but the present disclosure is not limited to this, and the input / output unit 47 may be made available from one or more other terminal devices connected to the sign detection device 20 via the network 5.

[0046] Correction unit 48 references time series DB 32 and detects a combination of posture and emotion specific to the subject, thereby creating an emotion correction value and associating it with the subject's identification information, and updates emotion correction value DB 33. Specifically, if there are two fluctuations in the emotion value that are greater than the threshold value within a certain period of time, and the emotion value due to the second fluctuation has returned to a value close to the emotion value before the first fluctuation, correction unit 48 sets the combination of the posture information at the time of the first fluctuation and the emotion value before the fluctuation as the emotion correction value.

[0047] Furthermore, if there is a change in the emotion value of the subject again during a time period in which the subject exhibited abnormal behavior, the correction unit 48 references the time-series DB 32, creates an attention correction value that is a combination of the change in the emotion value and the change in the posture information at the time the emotion value changed, and associates it with the subject's identification information to update the attention correction value DB 34. Specifically, if the subject exhibits abnormal behavior despite not detecting a sign of abnormal behavior, the correction unit 48 specifies the time period in which the abnormal behavior occurred. Then, the correction unit 48 reads the posture information and emotion value of the subject for the specified time period from the time-series DB 32, and if there is a change in the read emotion value, sets the combination of the change in the emotion value and the posture information at the time of the change as the attention correction value. In this way, even if the correction unit 48 fails to detect a sign of abnormal behavior, it improves the accuracy of sign detection from the next time onwards by setting the attention correction value to a combination of posture and emotion that require attention specific to the subject. In addition, if the fluctuation in posture information and the fluctuation in emotion value correspond to the attention correction value specific to the subject, the warning unit 46 may output a behavior warning regardless of the detection result of the sign by the sign detection unit 45.

[0048] In this way, by using the emotion correction value and the attention correction value, correction specific to the subject can be performed, and emotion value estimation and sign detection can be personalized.

[0049] In the present embodiment, the sign detection result is set as a caution level, and whether or not a sign of abnormal behavior has been detected is determined based on the caution level, but the present disclosure is not limited to this, and the sign detection result can be set as any numerical value that represents the possibility of abnormal behavior. Also, in the present embodiment, whether or not to output a behavior alert is determined based on whether or not a sign has been detected, but the present disclosure is not limited to this, and whether or not to output a behavior alert may be determined based on whether or not the numerical value that represents the possibility of abnormal behavior, which is the sign detection result, is equal to or greater than a threshold.

[0050] In addition, the emotion correction value and the attention correction value may be used by performing transfer learning or the like as soon as they are updated by the correction unit 48.

[0051] In the above configuration, the image acquisition unit 41, posture estimation unit 42, position detection unit 43, emotion estimation unit 44, sign detection unit 45, and warning unit 46 of the sign detection device 20 are examples of the image acquisition means, posture estimation means, position detection means, emotion estimation means, sign detection means, and warning means of the sign detection device of the present disclosure, respectively. The correction unit 48 is an example of the correction means and time period designation means. The time series DB 32, emotion correction value DB 33, and attention correction value DB 34 are examples of the time series storage means, emotion correction value storage means, and attention correction value storage means.

[0052] [Behavior Alert Processing] Next, a description will be given of the behavior warning process performed by the sign detection device 20. Fig. 7 is a flowchart of the behavior warning process performed by the sign detection device 20. This process is realized by the processor 22 shown in Fig. 2 executing a program prepared in advance.

[0053] First, the sign detection device 20 acquires image data D1 of a subject captured by the image capture device 10 (step S101). Then, the sign detection device 20 calculates a posture estimation result from the image data D1 and acquires posture information (step S102). Using an emotion estimation model, the sign detection device 20 acquires an emotion value output by inputting the posture information of the subject as the emotion estimation result (step S103). At this time, if an emotion correction value of the subject is stored in the emotion correction value DB 33, the sign detection device 20 assumes that there is a change in posture and emotion unique to the subject, corrects the emotion value based on the emotion correction value, and sets the corrected emotion value as the emotion estimation result (step S104). Then, the sign detection device 20 stores the posture information and emotion values ​​in the time series DB 32 in chronological order (step S105). The time series DB 32 is subsequently referenced in the repetitive behavior warning process.

[0054] The sign detection device 20 refers to the time-series DB 32 and determines whether or not there is a high possibility that the subject will behave abnormally, based on the continuous fluctuations in the subject's posture information and the continuous fluctuations in the emotion value (step S106). Specifically, the sign detection device 20 uses a sign detection model to input the fluctuations in the subject's posture information and the fluctuations in the emotion value, and acquires the possibility of abnormal behavior as the sign detection result. If the sign detection result acquires a caution level of 0, the sign detection unit 45 determines that there is almost no possibility of abnormal behavior occurring, since no sign of abnormal behavior was detected (step S106; No), and returns to the processing of step S101.

[0055] On the other hand, if the warning level 1 to 4 is acquired as the sign detection result, the sign detection unit 45 determines that a sign of abnormal behavior has been detected and therefore there is a possibility that abnormal behavior will occur (step S106; Yes). In this case, the sign detection device 20 outputs a behavior warning instruction (step S107). Specifically, the sign detection device 20 displays or outputs audio to the supervisor to inform them that there is a possibility that the subject will be behaving abnormally. Then, the sign detection device 20 returns to the processing of step S101.

[0056] [Location alarm processing] Next, a description will be given of the position warning process performed by the sign detection device 20. Fig. 8 is a flowchart of the position warning process performed by the sign detection device 20. This process is realized by the processor 22 shown in Fig. 2 executing a program prepared in advance.

[0057] The position warning process is a process that should assist in relatively safely executing the attention correction value update process in the early stages of introducing the sign detection system 100, and it can be set as desired whether or not to execute the position warning process. Also, the processes of steps S201 and S202 shown in Fig. 8 are assumed to be executed simultaneously with the processes of steps S101 and S102 shown in Fig. 7.

[0058] First, the sign detection device 20 acquires image data D1 of an image of a target from the imaging device 10 (step S201). Then, the sign detection device 20 acquires position information of the target in space from the image data D1 (step S202). Then, the sign detection device 20 stores the position information in time series in the time series DB 32. Furthermore, before executing the position warning process, the sign detection device 20 determines whether or not a position warning is set to be issued (step S203). If a position warning is not set to be issued (step S203; No), the sign detection device 20 returns to the process of step S201. On the other hand, if a position warning is set to be issued (step S203; Yes) and position information of a caution area is registered, the sign detection device 20 compares the position information of the caution area with the position information of the target to determine whether or not the target has approached the caution area (step S204).

[0059] If it is determined that the target person is not approaching the caution zone (step S204; No), the sign detection device 20 returns to the processing of step S201. On the other hand, if it is determined that the target person is approaching the caution zone (step S204; Yes), the sign detection device 20 outputs a position warning instruction (step S205). Specifically, the sign detection device 20 displays or outputs audio to the supervisor to inform them that the target person is approaching the caution zone. Then, the sign detection device 20 returns to the processing of step S201.

[0060] For example, when a subject approaches a room entrance, the sign detection device 20 outputs a location alert through the location alert process, allowing a supervisor to receive the alert and address the subject's behavior. In this case, the sign detection device 20 will output a location alert even if the subject's legitimate outing, such as to the bathroom, does not lead to behavior such as wandering. However, in the early stages of introducing the sign detection system 100, it may be impossible to detect signs of abnormal behavior from the subject's unique posture and emotional transitions. However, the sign detection device 20 can assist in addressing the subject's behavior by outputting a location alert. Furthermore, establishing an emotion correction value and an attention correction value contributes to smooth operation.

[0061] [Emotion correction value update processing] Next, we will explain the emotion correction value update processing performed by the sign detection device 20. Figure 9 is a flowchart of the emotion correction value update processing performed by the sign detection device 20. This processing is realized by the processor 22 shown in Figure 2 executing a program prepared in advance. Note that the emotion correction value update processing is executed in parallel with the behavior alert output processing and the position alert output processing.

[0062] If data exists in the time-series DB 32, the sign detection device 20 begins monitoring the fluctuations in emotion values ​​(step S301). The sign detection device 20 then compares the latest emotion value stored in the time-series DB 32 with the immediately preceding emotion value, and determines whether or not an abrupt fluctuation has been detected (step S302). If no abrupt fluctuation is detected (step S302; No), the sign detection device 20 resumes monitoring the time-series DB 32. On the other hand, if an abrupt fluctuation is detected (step S302; Yes), the sign detection device 20 records the fact that a fluctuation has occurred (step S303). What is preferably recorded at this time is the latest emotion value, the immediately preceding emotion value, and time information about the difference between them. Furthermore, even if there is an abrupt fluctuation, the sign detection device 20 does not update the emotion correction value unless two or more fluctuations are detected.

[0063] The sign detection device 20 determines whether there has been another change within a certain period of time. That is, it determines whether there have been two or more changes within the certain period of time (step S304). If there have not been two or more changes within the certain period of time (step S304; No), the sign detection device 20 deletes the first change record (step S305) and returns to the processing of step S302. In this way, if there have not been two or more changes within the certain period of time, although there has been a sudden change in the emotion value, it is determined that this change occurred within the subject and is not a sign of abnormal behavior, and the sign detection device 20 deletes the first change record.

[0064] On the other hand, if there are two or more fluctuations within a certain period of time (Step S304: Yes), the sign detection device 20 records the fact that a second fluctuation has occurred and reads out the posture information and emotion value from the time-series DB 32 (Step S306). The sign detection device 20 then determines whether the emotion value resulting from the second fluctuation has returned to a value close to the emotion value prior to the first fluctuation (Step S307). If it is determined that the emotion value has not returned to a close value (Step S307: No), the sign detection device 20 deletes the record of the first fluctuation (Step S305) and returns to the processing of Step S302. On the other hand, if it is determined that the emotion value has returned to a close value (Step S307: Yes), the sign detection device 20 creates an emotion correction value that is a combination of the posture information at the time of the first fluctuation and the emotion value prior to the fluctuation, and stores and updates the emotion correction value DB 33 (Step S308). This is because the emotion value to be estimated from the posture information at the time of the first fluctuation is specific to the subject and is considered to correspond to the emotion value prior to the fluctuation or after the second fluctuation. Then, the sign detection device 20 deletes the fluctuation record (step S309) and returns to the processing of step S302.

[0065] [Caution correction value update process] Next, the attention correction value update process performed by the sign detection device 20 will be described. The attention correction value update process is a process for storing and updating an attention correction value in the attention correction value DB 34 to prevent recurrence when detection of a sign of abnormal behavior by a subject has failed. Fig. 10 is a flowchart of the attention correction value update process performed by the sign detection device 20. This process is realized by the processor 22 shown in Fig. 2 executing a program prepared in advance. Note that the attention correction value update process is executed in parallel with the behavior alert output process, position alert output process, and emotion correction value update process.

[0066] If the detection of a sign of abnormal behavior by a subject fails and a behavior alert is not output, and the subject actually does behave abnormally, the supervisor subsequently specifies to the sign detection device 20 the time period in which the subject's abnormal behavior occurred (step S401). The sign detection device 20 then reads out posture information, emotion value, and position information from the time-series DB 32 (step S402). The sign detection device 20 determines whether there has been a change in the emotion value that should be updated as an attention correction value, for example, if there have been multiple sudden changes in the emotion value within the same time period (step S403).

[0067] If it is determined that there has been no change in the emotion value that should be updated as the attention correction value (step S403; No), the sign detection device 20 returns to the processing of step S401. On the other hand, if it is determined that there has been a change in the emotion value that should be updated as the attention correction value (step S403; Yes), the sign detection device 20 creates a combination of the changed emotion value and the posture information at the time of the change as an attention correction value, and stores and updates the attention correction value DB 34 (step S404). This ends the attention correction value update processing.

[0068] According to the sign detection device 20 of the present disclosure, posture information of the subject estimated from image data D1 using a depth sensor is continuously accumulated, and by further identifying the distribution of emotions obtained from the posture information, it is possible to simultaneously monitor changes in the subject's emotions and behavior. In this way, the sign detection device 20 can detect signs that the subject is about to engage in abnormal behavior and can be used to assist supervisors and others in suppressing the abnormal behavior.

[0069] In this way, the sign detection device 20 not only performs behavior analysis using image data and point cloud data from a depth sensor, but also performs emotion analysis estimated from posture information, and as a result of combining these, it is possible to detect signs of abnormal behavior by the subject.In addition, by automatically or manually storing correction data such as emotion correction values ​​and attention correction values ​​in a DB, the sign detection device 20 can avoid fluctuations and singularities in the posture, behavior, and emotions unique to the subject in the sign detection process.

[0070] This allows for monitoring the subject's internal state, such as their emotions, without requiring the subject to carry any specific equipment or personal belongings, making it possible to detect signs of abnormal behavior with higher accuracy than behavior monitoring alone. Furthermore, the longer the sign detection system 100 is in operation, the more it can learn the subject's unique tendency to move on to abnormal behavior, making it possible to detect signs with higher accuracy. By detecting signs of abnormal behavior by a subject and outputting an alert to a supervisor, it is possible to support supervisors who are struggling to deal with abnormal behavior.

[0071] For example, suppose there is a subject who exhibits abnormal behavior when experiencing negative emotions. In the case of monitoring this subject, if there are two patterns, one where abnormal behavior occurs after a prolonged "head bowed" posture, and the other where abnormal behavior occurs after a "head-holding" posture, then with conventional methods that only analyze behavior, unless the two consecutive behavioral patterns of abnormal behavior after the "head bowed" posture and the "head-holding" posture are obtained in advance as known information, it will be impossible to detect signs of abnormal behavior after a posture for which no information is available.

[0072] Here, if the subject's emotions can be analyzed, as with the sign detection device 20 of the present disclosure, it can be inferred that both the "head bowed" and "head in hands" postures are expressions of negative emotions. By using emotional fluctuations to detect signs of abnormal behavior, the sign detection device 20 can detect both the "head bowed" posture and the "head in hands" posture as signs of abnormal behavior, provided that information on when the subject begins abnormal behavior from some negative posture is known.

[0073] Note that having the subject carry or wear a device that reads emotional fluctuations should be avoided as it may lead to injury or accident. The sign detection device 20 of the present disclosure estimates an emotional value based on posture information obtained from image capture by the imaging device 10 and detects signs of abnormal behavior from fluctuations in posture information and fluctuations in the emotional value, so another advantage is that there is no need for the subject to wear a specific device.

[0074] In addition, while discrepancies in predicted behavior and emotions due to differences in the gender of the subject have been a concern, other differences between subjects are also of concern.Unlike conventional methods, the sign detection device 20 of the present disclosure improves the accuracy of detecting signs of abnormal behavior by using an emotion correction value and an attention correction value.

[0075] [Variations] (First Modification) The above embodiment applies an indicator management system that targets elderly people with cognitive problems and detects signs of abnormal behavior, such as wandering or sudden violence. However, the present disclosure is not limited to this. In a work environment such as a factory, construction site, or warehouse, the indicator may be a worker, and indicators of poor health or mental anxiety may be detected based on fluctuations in posture information and fluctuations in emotional values, such as fatigue, tiredness, and distress. This allows workers to be given appropriate breaks at appropriate times, and the indicator management system can be applied as a work environment improvement system. Similarly, the indicator management system can be applied as a safety management system by detecting signs of distraction, such as emotional instability.

[0076] Furthermore, it is also possible to detect signs of school refusal from changes in students' posture information and emotional values. This allows schools to check for students who are not fitting in well, which could lead to school refusal, while respecting privacy, and the sign management system can be used as a check system.

[0077] In this way, the symptom detection system of the present disclosure can be applied to a variety of environments, such as hospitals, nursing homes, retirement homes, schools, workplaces, inside vehicles while driving, construction sites, factories, and warehouses.

[0078] (Second Modification) If the environment allows, the image capturing device 10 may be a surveillance camera or the like.

[0079] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0080] (Appendix 1) image acquisition means for acquiring image data of a subject; a posture estimation means for estimating a posture of the subject from the image data to acquire posture information; an emotion estimation means for estimating an emotion of the subject from the posture information to acquire an emotion value; a time-series storage means for storing the posture information and the emotion values ​​in time series; a sign detection means for detecting a sign of some kind of behavior by the subject based on the change in the posture information and the change in the emotion value; A sign detection device comprising:

[0081] (Appendix 2) 2. The sign detection device according to claim 1, further comprising: an alarm means for outputting a behavior alarm when the sign detection means detects the sign.

[0082] (Appendix 3) The posture information is information indicating the positions of the parts and joints that make up the subject, and is a group of coordinates converted into relative coordinates with the coordinates of the waist as the origin.

[0083] (Appendix 4) An emotion correction value storage means for storing an emotion correction value, which is a combination of posture information and emotion value specific to a subject, in association with subject identification information for identifying the subject, 2. The sign detection device according to claim 1, wherein the emotion estimation means corrects the emotion value of the subject based on the emotion correction value of the subject.

[0084] (Appendix 5) The sign detection device according to appendix 4, wherein the emotion correction value storage means, when there are two fluctuations in the emotion value that are equal to or greater than the threshold value within a certain period of time and the emotion value due to the second fluctuation has returned to a value close to the emotion value before the first fluctuation, sets as an emotion correction value a combination of the posture information at the time of the first fluctuation and the emotion value before the fluctuation.

[0085] (Appendix 6) a registration means for registering location information of a warning area where entry of the target person should be warned; a position detection means for detecting position information of the subject from the image data; The warning means is a sign detection device described in Appendix 2 that outputs a location warning when it detects that the target person is approaching the attention area based on the location information of the target person and the location information of the attention area.

[0086] (Appendix 7) a time period designation means for designating a time period for a certain behavior when the subject person has performed some behavior despite the fact that the sign has not been detected; and an attention correction value storage means for reading out the posture information and emotion value of the subject for the time period from the time series storage means, and when there is a change in the read emotion value, storing an attention correction value which is a combination of the change in the emotion value and the change in the posture information at the time of the change in association with subject identification information which identifies the subject, The warning means is an indicator detection device as described in Appendix 2, which outputs the behavioral warning when the change in the posture information and the change in the emotion value correspond to the attention correction value of the subject, regardless of the detection result by the indicator detection means.

[0087] (Appendix 8) The sign detection device according to claim 1, wherein the emotion estimation means acquires the emotion value of the subject using a machine learning model trained to output an optimized emotion value in response to the input of the posture information.

[0088] (Appendix 9) is executed by a computer, Acquire image data of the subject, acquiring posture information by estimating a posture of the subject from the image data; acquiring an emotion value by estimating an emotion of the subject from the posture information; storing the posture information and the emotion value in a time series in a storage unit; A sign detection method for detecting a sign of some kind of behavior by the subject based on the change in the posture information and the change in the emotion value.

[0089] (Appendix 10) Acquire image data of the subject, acquiring posture information by estimating a posture of the subject from the image data; acquiring an emotion value by estimating an emotion of the subject from the posture information; storing the posture information and the emotion value in a time series in a storage unit; A program that causes a computer to execute a process of detecting signs of some kind of behavior by the subject from changes in the posture information and changes in the emotion value.

[0090] Although the present invention has been described above with reference to the embodiments and examples, the present invention is not limited to the above-described embodiments and examples. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. [Explanation of symbols]

[0091] 5. Network 10 Imaging equipment 20 Premonition detection device 21 Interface 22 processors 23 Memory 24 Recording media 25 Display section 26 Input section 31 Emotion estimation DB 32 Time Series DB 33 Emotion Correction Value DB 34 Caution Correction Value DB

Claims

1. image acquisition means for acquiring image data of a subject; a posture estimation means for estimating a posture of the subject from the image data to acquire posture information; an emotion estimation means for estimating an emotion of the subject from the posture information to acquire an emotion value; a time-series storage means for storing the posture information and the emotion values ​​in time series; a sign detection means for detecting a sign of some kind of behavior by the subject based on the change in the posture information and the change in the emotion value; A sign detection device comprising:

2. The sign detection device according to claim 1 , further comprising: a warning unit that outputs a behavior warning when the sign detection unit detects the sign.

3. The sign detection device according to claim 1 , wherein the posture information is information indicating the positions of the parts and joints that make up the subject, and is a group of coordinates converted into relative coordinates with the coordinate of the waist as the origin.

4. An emotion correction value storage means for storing an emotion correction value, which is a combination of posture information and emotion value specific to a subject, in association with subject identification information for identifying the subject, The sign detection device according to claim 1 , wherein the emotion estimation means corrects the emotion value of the subject based on the emotion correction value of the subject.

5. 5. The sign detection device according to claim 4, wherein when there are two fluctuations in the emotion value that are greater than or equal to the threshold value within a certain period of time and the emotion value due to the second fluctuation has returned to a value close to the emotion value prior to the first fluctuation, the emotion correction value storage means uses the combined posture information at the time of the first fluctuation and the emotion value prior to the fluctuation as the emotion correction value.

6. a registration means for registering location information of a warning area where entry of the target person should be warned; a position detection means for detecting position information of the subject from the image data; 3. The sign detection device according to claim 2, wherein the warning means outputs a position warning when it detects that the target person has approached the attention area based on the position information of the target person and the position information of the attention area.

7. a time period designation means for designating a time period for a certain behavior when the subject person has performed some behavior despite the fact that the sign has not been detected; and an attention correction value storage means for reading out the posture information and emotion value of the subject for the time period from the time series storage means, and when there is a change in the read emotion value, storing an attention correction value which is a combination of the change in the emotion value and the change in the posture information at the time of the change in association with subject identification information which identifies the subject, The precursor detection device according to claim 2, wherein the warning means outputs the behavioral warning when the change in the posture information and the change in the emotion value correspond to the attention correction value of the subject, regardless of the detection result by the precursor detection means.

8. The sign detection device according to claim 1 , wherein the emotion estimation means acquires the emotion value of the subject using a machine learning model that has been trained to output an optimized emotion value in response to the input of the posture information.

9. is executed by a computer, Acquire image data of the subject, acquiring posture information by estimating a posture of the subject from the image data; acquiring an emotion value by estimating an emotion of the subject from the posture information; storing the posture information and the emotion value in a time series in a storage unit; A sign detection method for detecting a sign of some kind of behavior by the subject based on the change in the posture information and the change in the emotion value.

10. Acquire image data of the subject, acquiring posture information by estimating a posture of the subject from the image data; acquiring an emotion value by estimating an emotion of the subject from the posture information; storing the posture information and the emotion value in a time series in a storage unit; A program that causes a computer to execute a process of detecting signs of some kind of behavior by the subject from changes in the posture information and changes in the emotion value.

Images