Data processing device and data processing program

Abstract

A data processing device according to an embodiment includes a memory circuit and a processor. The memory circuit can store video data and sound data. The processor performs a monitoring process. The processor: estimates, in the monitoring process, the direction and number of at least one visitor staying in a monitored area on the basis of the video data (S11); estimates the direction of sound generated in the monitored area, on the basis of the sound data (S12); and determines whether or not a soliloquy has been uttered on the basis of the direction and number of the at least one visitor and the direction of the generated sound (S18).

Description

[Technical field]

[0001] The embodiments relate to a data processing device, a data processing method, and a data processing program. [Background technology]

[0002] Hallucinations such as visual hallucinations and auditory hallucinations are known as one type of behavioral and psychological symptoms of dementia (BPSD). In order for a caregiver to provide proper care to a care recipient with BPSD symptoms in a care setting, it is preferable for the caregiver to be able to properly grasp whether the care recipient is experiencing hallucinations. The caregiver must be able to grasp whether the care recipient has experienced hallucinations by observing the care recipient's state. In other words, the caregiver cannot grasp whether the care recipient has experienced hallucinations in a place where the caregiver is not looking at the care recipient.

[0003] A possible system for monitoring people is to use a nursing management system that records whether or not a person is being properly cared for by a doctor, nurse, caregiver, etc., as in Patent Document 1. This allows caregivers, etc. to refer to the nursing process, etc., as necessary, without direct human monitoring. In addition, when a caregiver experiences visual or auditory hallucinations, he or she may talk to himself or herself. In order to detect such talking to oneself, as in Patent Document 2, a voice input device that can estimate the position of the operator based on the voice and distinguish the operator's voice from environmental sounds, etc., can be used. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent No. 4069061 [Patent Document 2] Japanese Patent Application Publication No. 2020-034597 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the nursing management system of Patent Document 1 does not support detection of the occurrence of hallucinations in the care recipient. In addition, the voice input device of Patent Document 2 can detect human speech, but cannot distinguish between conversation and monologue, and cannot determine the occurrence of hallucinations based on monologue.

[0006] Therefore, an object of the present invention is to provide a data processing device, a data processing method, and a data processing program that can detect when a cared-for person is experiencing hallucinations indoors, even in an environment out of the caregiver's reach. [Means for solving the problem]

[0007] A data processing device according to an embodiment includes a memory circuit and a processor. The memory circuit is capable of storing video data and audio data. The processor executes a monitoring process. In the monitoring process, the processor estimates the direction and number of at least one visitor in a monitoring area based on the video data, estimates the direction of a sound generated in the monitoring area based on the audio data, and determines whether or not a monologue has occurred based on the direction and number of the at least one visitor and the direction of the sound. Effect of the Invention

[0008] The data processing device of the embodiment can detect when a care recipient is experiencing hallucinations in a room. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram showing an example of use of a data processing system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of a hardware configuration of a data processing device according to an embodiment. [Diagram 3] FIG. 3 is a block diagram illustrating an example of a functional configuration of the data processing device according to the embodiment. [Figure 4]FIG. 4 is a flowchart showing an example of the monitoring process of the data processing device according to the embodiment. [Diagram 5] FIG. 5 is a schematic diagram showing a specific example of the monitoring process by the data processing device according to the embodiment. [Figure 6] FIG. 6 is a schematic diagram showing a specific example of the monitoring process by the data processing device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments will be described with reference to the drawings. The embodiments illustrate devices and methods for embodying the technical ideas of the invention. The drawings are schematic or conceptual. In the following description, components having substantially the same functions and configurations are denoted by the same reference numerals.

[0011] <Embodiment> The data processing system 1 according to the embodiment will be described below.

[0012] <1> composition Fig. 1 is a schematic diagram showing an example of use of a data processing system 1 according to an embodiment. As shown in Fig. 1, the data processing system 1 monitors the state of a monitoring target MT, which is a person (visitor) in a monitoring area MA. For example, the monitoring target MT is a care recipient, and the monitoring area MA is the care recipient's room. The data processing system 1 includes, for example, a data processing device 10, a camera 20, and a microphone array 30.

[0013] The data processing device 10 is a computer capable of analyzing the state of the monitored object MT in the monitored area MA using input video (image) data and audio data. The data processing device 10 may be installed inside a room corresponding to the monitored area MA, or may be installed outside the room. The data processing device 10 receives input of image data from the camera 20 and audio data from the microphone array 30 via the network NW.

[0014] The camera 20 is a video capture device that uses, for example, an RGB+D sensor, and is installed in a room corresponding to the monitoring area MA. The camera 20 is configured to be able to acquire an image within the monitoring area MA and depth information of the image. The data processing device 10 can estimate whether or not a person is present within the monitoring area MA based on the video data acquired by the camera 20.

[0015] The microphone array 30 is a sound collecting device equipped with a plurality of microphones, and is installed in a room corresponding to the monitoring area MA. The microphone array 30 is configured to be able to acquire spatial information of sounds generated in the monitoring area MA. The data processing device 10 can estimate the generation position, directivity, etc. of the detected sound based on the sound data acquired by the microphone array 30.

[0016] The camera 20 and the microphone array 30 may be connected to the data processing device 10 without going through the network NW. The data processing device 10 may be connected to the camera 20 and the microphone array 30 wirelessly or by wire. A plurality of cameras 20 may be connected to the data processing device 10, and a plurality of microphone arrays 30 may be connected to the data processing device 10. The number and arrangement of the cameras 20 and the microphone arrays 30 connected to the data processing device 10 may be changed as appropriate as long as the monitoring process described below can be executed.

[0017] <1-1> Hardware configuration of data processing device 10 2 is a block diagram showing an example of a hardware configuration of a data processing device 10 according to an embodiment. As shown in FIG. 2, the data processing device 10 includes, for example, a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a communication device 14, and a storage device 15.

[0018] The CPU 11 is an integrated circuit capable of executing various programs. The CPU 11 executes programs expanded in, for example, the RAM 13 to realize various functions of the data processing device 10. The ROM 12 is a non-volatile semiconductor memory. The ROM 12 stores programs and control data for controlling the data processing device 10. The RAM 13 is, for example, a volatile semiconductor memory. The RAM 13 is used as a working area for the CPU 11. The communication device 14 is a communication circuit configured to be connectable to the network NW. The data processing device 10 transfers the video data and audio data received via the communication device 14 to the RAM 13 or the storage device 15. The storage device 15 is a non-volatile storage device. The storage device 15 stores, for example, system software of the data processing device 10, analysis results of the video data and audio data, and the like.

[0019] The data processing device 10 may have other hardware configurations as long as it is capable of executing the monitoring process described below. The data processing device 10 may include a connection interface to which the camera 20 and / or the microphone array 30 can be connected. In this case, the data processing device 10 receives video data and audio data from the camera 20 and the microphone array 30 via the connection interface. In addition, a display, an input interface, a removable storage device, etc. may be connected to the data processing device 10.

[0020] <1-2> Functional configuration of data processing device 10 Fig. 3 is a block diagram showing an example of a functional configuration of the data processing device 10 according to the embodiment. As shown in Fig. 3, the data processing device 10 includes, for example, a person position estimation unit 100, a visitor number estimation unit 110, a voice direction estimation unit 120, an utterance detection unit 130, a monologue detection unit 140, a hallucination determination unit 150, and a recording unit 160.

[0021] The person position estimation unit 100 detects people in the monitoring area MA based on the video data input from the camera 20. LocationFor example, the person position estimation unit 100 recognizes people from video data by skeletal estimation or the like, and estimates the direction in which the people are located. Then, the person position estimation unit 100 transfers the estimation result regarding the position of the recognized person, etc. to the visitor number estimation unit 110.

[0022] The visitor number estimation unit 110 estimates the number of visitors (the number of monitoring targets MT) in the monitored area MA based on the data transferred from the person position estimation unit 100. Then, the visitor number estimation unit 110 transfers the estimation result of the number of visitors in the monitored area MA to the monologue detection unit 140.

[0023] The voice direction estimation unit 120 estimates the sound pressure and the sound generation direction of the sound generated in the monitoring area MA based on the sound data input from the microphone array 30. Then, the voice direction estimation unit 120 transfers the estimation result of the sound generated in the monitoring area MA to the speech detection unit 130.

[0024] The speech detection unit 130 distinguishes between human speech and environmental noise and detects human speech based on the estimation results of the person position estimation unit 100 and the voice direction estimation unit 120. For example, the speech detection unit 130 determines that a speech is a human speech when the direction of sound pressure coincides with the direction of a person. The speech detection unit 130 then transfers the detection result to the monologue detection unit 140.

[0025] The monologue detection unit 140 detects monologue within the monitoring area MA based on the estimation result of the visitor number estimation unit 110 and the detection result of the speech detection unit 130. For example, when the number of visitors within the monitoring area MA is one and a human speech occurs, the monologue detection unit 140 recognizes the detected human voice as monologue. The monologue detection unit 140 then transfers information related to the recognized monologue to the hallucination determination unit 150.

[0026] The hallucination determination unit 150 estimates whether or not a hallucination has occurred in the monitoring target MT within the monitoring area MA based on the information transferred from the monologue detection unit 140. For example, the hallucination determination unit 150 estimates the occurrence of a visual hallucination or an auditory hallucination from the frequency of monologue occurrence. The hallucination determination unit 150 then determines that a hallucination has occurred in the monitoring target MT when the continuous occurrence time of monologue or the number of occurrences of monologue per unit time exceeds a predetermined threshold. The hallucination determination unit 150 may also determine that monologue has occurred continuously when the time interval between occurrences of monologue is equal to or less than a predetermined threshold. The hallucination determination unit 150 transfers the hallucination determination result based on the detected monologue to the recording unit 160.

[0027] The recording unit 160 stores the hallucination determination result input from the hallucination determination unit 150. The hallucination determination result may be recorded as time-series data, or the number of occurrences may be counted for each predetermined item. For example, the number of occurrences of monologue and the continuous occurrence of monologue may be recorded separately. The hallucination determination result may be recorded in the recording unit 160 with settings and items according to the desires of the user (caregiver).

[0028] In the data processing device 10, the processing of each of the person position estimation unit 100, visitor number estimation unit 110, voice direction estimation unit 120, speech detection unit 130, monologue detection unit 140, hallucination determination unit 150, and recording unit 160 is realized by, for example, the CPU 11 and the RAM 13. The processing of the recording unit 160 is realized by, for example, the storage device 15. This is not limiting, and the functional configuration of the data processing device 10 may be classified in other ways.

[0029] <2> operation In the monitoring process, the data processing device 10 according to the embodiment estimates the position of a person in a room based on video data, estimates the direction of a sound source based on acquired sound data, and estimates whether a person in the room is talking to himself / herself. When the data processing device 10 detects talking to himself / herself, it judges whether the care recipient is hallucinating based on the duration and frequency of the talking to himself / herself.

[0030] <2-1> Monitoring process flow 4 is a flowchart showing an example of the monitoring process of the data processing device 10 according to the embodiment. The flow of the monitoring process of the data processing device 10 according to the embodiment will be described below with reference to FIG.

[0031] The CPU 11 starts (starts) the monitoring process in response to, for example, a program for the monitoring process being executed by a user operation.

[0032] First, the CPU 11 acquires video data and audio data of the monitoring area MA (step S10). In other words, the video data acquired by the camera 20 is input to the person position estimation unit 100, and the audio data acquired by the microphone array 30 is input to the voice direction estimation unit 120.

[0033] Next, the CPU 11 estimates the positions of people in the monitoring area MA based on the acquired video data (step S11). In other words, the person position estimation unit 100 estimates the number and positions of people in the monitoring area MA based on the image and depth information of the video data.

[0034] Next, the CPU 11 estimates the direction of the sound generated in the monitoring area MA based on the acquired sound data (step S12). In other words, the sound direction estimation unit 120 estimates the sound pressure of the sound generated in the monitoring area MA and the direction of the sound based on the sound data.

[0035] Next, CPU 11 checks whether or not the sound is coming from the direction where the person is located (step S13). Specifically, speech detection unit 130 checks whether or not the location of the visitor estimated by the process of step S11 matches the location of the sound estimated by the process of step S12.

[0036] In the process of step S13, if it is confirmed that no sound is coming from the direction where people are present (step S13, NO), the speech detection unit 130 determines that an environmental sound has been detected (step S14). In this specification, "environmental sound (environmental noise)" refers to a sound generated by something other than a person's speech. When the process of step S14 is completed, the CPU 11 proceeds to the process of step S21.

[0037] In the process of step S13, if it is confirmed that the sound is coming from the direction in which a person is present (step S13, YES), the speech detection unit 130 determines that a person's speech has been detected (step S15).

[0038] After step S15, the CPU 11 checks whether or not the number of visitors in the monitoring area MA is one (step S16). In other words, the monologue detection unit 140 checks whether or not the number of visitors estimated by the process of step S11 is one.

[0039] In the process of step S16, if it is not confirmed that there is only one visitor in the monitoring area MA (step S16, NO), the monologue detection unit 140 determines that a conversation has been detected (step S17). In this specification, a "conversation" refers to an exchange of words between at least two people. When the process of step S17 is completed, the CPU 11 proceeds to the process of step S21.

[0040] In the process of step S16, if it is confirmed that there is only one visitor in the monitoring area MA (step S16, YES), the monologue detection unit 140 determines that monologue has been detected (step S18). In other words, when the only visitor in the room speaks, the monologue detection unit 140 determines that the monitored object MT is talking to itself.

[0041] After step S18, the CPU 11 executes a hallucination determination process (step S19). Specifically, the hallucination determination unit 150 determines whether or not a hallucination is occurring in the monitoring target MT based on the occurrence time and occurrence interval of the monologue transferred from the monologue detection unit 140. Then, the hallucination determination unit 150 transfers the determination result of the hallucination determination process to the recording unit 160.

[0042] After step S19, the CPU 11 executes a recording process (step S20). In other words, the recording unit 160 records the determination result transferred from the hallucination determination unit 150. After completing the process of step S19, the CPU 11 proceeds to the process of step S21.

[0043] In the process of step S21, the CPU 21 checks whether or not to end the monitoring (step S21). Specifically, the CPU 21 checks whether or not, for example, the user has performed an operation to end the monitoring process.

[0044] In the process of step S16, if the monitoring is not to be ended (step S21, NO), for example, if the user has not performed an operation to end the monitoring process, the CPU 11 proceeds to the process of step S10. As a result, the CPU 11 appropriately executes the processes of steps S10 to S20 described above, and monitors whether or not a hallucination has occurred in the monitoring target MT within the monitoring area MA.

[0045] In the process of step S16, when monitoring is to be ended (step S21, YES), for example, when an operation to end the monitoring process is performed by the user, the CPU 11 ends the series of processes in FIG. 4 (end).

[0046] The monitoring process described above may be performed based on a preset schedule, or may be performed in association with the use of a specific device in the room (for example, an indoor light installed in the monitoring area MA). After the process of step S17, the CPU 21 may proceed to the process of step S20. That is, the CPU 21 may cause the recording unit 160 to store the fact that a conversation has been detected in the monitoring area MA.

[0047] <2-2> Specific examples of monitoring processing A specific example of the monitoring process of the data processing device 10 according to the embodiment will be described below.

[0048] (If conversation is detected) FIG. 5 is a schematic diagram showing a specific example of the monitoring process by the data processing device 10 according to the embodiment, and corresponds to a case where a conversation is detected in the monitoring process. As shown in FIG. 5, in this example, two monitoring targets MT1 and MT2 are present in the monitoring area MA. The monitoring target MT1 is speaking toward the monitoring target MT2. In this case, the data processing device 10 first confirms that a voice VO is emitted from the monitoring target MT1. Then, the data processing device 10 determines that the monitoring target MT1 is talking (conversation state) based on the fact that the two monitoring targets MT1 and MT2 are present in the monitoring area MA. Note that the respective estimation results of the position of the monitoring target MT and the generation position of the voice VO do not need to completely match. A predetermined threshold value or margin is provided for determining whether or not the voice VO is emitted by the monitoring target MT, and a certain degree of error can be tolerated.

[0049] (If talking to yourself is detected) Fig. 6 is a schematic diagram showing a specific example of the monitoring process by the data processing device 10 according to the embodiment, and corresponds to a case where monologue is detected in the monitoring process. As shown in Fig. 6, in this example, one monitored subject MT1 is present in the monitoring area MA. The monitored subject MT1 is speaking into an empty space. In this case, the data processing device 10 first confirms that the voice VO is coming from the monitored subject MT1. Then, based on the fact that there is no one other than the monitored subject MT1 present in the monitoring area MA, the data processing device 10 determines that the monitored subject MT1 is speaking to itself (monolithic state).

[0050] <3> effect The data processing device 10 according to the embodiment can continuously monitor and record the condition of the care recipient even when the care recipient is not directly looking at the care recipient. The data processing device 10 can also detect when the care recipient has hallucinations and record the frequency of hallucinations and the severity of hallucinations in the care recipient. As a result, the care recipient can easily grasp the condition of the care recipient by referring to the data of the care recipient accumulated by the data processing device 10.

[0051] <4> others In the embodiment, the case where the camera 20 generates video data has been described, but the present invention is not limited to this. Only an RGB-D sensor unit may be installed in a room corresponding to the monitoring area MA. In this case, the data processing device 10 generates video data including depth information based on a signal detected by the RGB-D sensor unit. The data processing device 10 can also realize the operation described in the embodiment by combining a camera that cannot measure distance with a distance measuring device such as LiDAR (Light Detection and Ranging) that can measure distance. In the room corresponding to the monitoring area MA, multiple microphones may be installed instead of the microphone array 30. In this case, the data processing device 10 generates spatial sound data based on signals detected by the multiple microphones.

[0052] In the flowchart shown in FIG. 4, the order of processing may be changed as far as possible, or other processing may be added, as long as the same results as those of the embodiment are obtained. In the embodiment, the case of detecting monologue in real time is illustrated, but the present invention is not limited to this. The data processing device 10 may detect the occurrence of monologue by the monitored object MT based on the video data and audio data recorded in the storage device 15. In this specification, the data processing device 10 may be called a "server" or a "processing server". The CPU 11 may be called a "processor". Each of the ROM 12, the RAM 13, and the storage device 15 may be called a "storage circuit".

[0053] The hardware configuration of the data processing device 10 described in the embodiment is merely an example. The CPU 11 included in the data processing device 10 may be another circuit. For example, the data processing device 10 may use an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (field-programmable gate array), or the like instead of the CPU 11. The analysis process described in the embodiment may be realized by dedicated hardware. The analysis process of the data processing device 10 may be a mixture of processes executed by software and processes executed by hardware, or may be only one of them.

[0054] The present invention is not limited to the above-mentioned embodiment, and can be modified in various ways without departing from the gist of the present invention. The embodiments may be combined as appropriate, and in that case, the combined effect can be obtained. Furthermore, the above-mentioned embodiment includes various inventions, and various inventions can be extracted by combinations selected from the multiple components disclosed. For example, if the problem can be solved and the effect can be obtained even if some components are deleted from all the components shown in the embodiment, the configuration from which the components are deleted can be extracted as an invention. [Explanation of symbols]

[0055] 1. Data processing system 10...Data processing device 11...CPU 12...ROM 13…RAM 14...Communication equipment 15...Storage device 20…Camera 21...CPU 30…Microphone array 100...Person position estimation section 110…Resident Number Estimation Department 120...Speech direction estimation unit 130…Speech detection unit 140…Muttering to oneself detection unit 150…Hallucination Judgment Department 160…Recording section

Claims

1. A memory circuit capable of storing video data and audio data; A processor for executing a monitoring process, In the monitoring process, the processor Estimating the number of people in a monitored area based on the video data; Estimating a direction of a sound generated within the monitoring area based on the sound data; When the visitor is alone and a sound generated from the direction of the visitor is detected, it is determined that the visitor is talking to himself / herself, When the continuous duration of monologue exceeds a first threshold, or when the number of times monologue occurs per unit time exceeds a second threshold, it is determined that the visitor has experienced a hallucination. Data processing device.

2. The processor determines that the monologue is occurring continuously when a time interval between occurrences of the monologue is equal to or less than a third threshold value.

2. A data processing apparatus according to claim 1.

3. On the computer, Estimating the direction and number of at least one person in a monitoring area based on the video data; estimating a direction of a sound generated within the monitoring area based on the sound data; determining that the visitor is talking to himself / herself when the visitor is alone and a sound generated from a direction in which the visitor is present is detected; determining that the visitor has experienced a hallucination when a continuous duration of monologue exceeds a first threshold or when a number of times the visitor has monologue per unit time exceeds a second threshold; A data processing program that executes the above.

4. A computer comprising: determining that the monologue is occurring continuously when the time interval between occurrences of the monologue is equal to or less than a third threshold value; 4. The data processing program according to claim 3, further comprising:

Images