Data Processing System

Abstract

A data processing system is provided that can determine medically important information from a conversation and perform document creation and data input. [Solution] The data processing system includes a necklace-type terminal including a sensor that detects biometric data of the wearer, who is a medical professional, a microphone that picks up the conversation between the wearer and a patient and converts it into audio data, a collection unit that collects the output of each of the sensor and microphone, and a communication unit that transmits the collected output of each of the sensor and microphone to an external device, and a data processing device.The data processing device includes an input unit that accepts the biometric data collected by the sensor and the audio data picked up by the microphone, a processing unit that obtains medically important information from the audio data based on changes in the wearer's emotions estimated from the biometric data, creates a summary that summarizes the content of the conversation, and records it in a recording unit, and an output unit that outputs the summary to a pre-set output destination.

Description

[Technical Field]

[0001] The techniques of this disclosure relate to data processing systems. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] It is important for medical professionals and other employees of medical institutions to spend more time on patient care, such as medical examinations and rehabilitation. However, medical institution employees spend a lot of time on document preparation, such as preparing medical certificates, and data entry, such as entering electronic medical records, which reduces the time they can spend on patient care. While the conventional technology described above can generate chatbot utterances corresponding to user utterances, it cannot determine important information from dialogues such as medical interviews and perform document preparation or data entry. [Means for solving the problem]

[0005] A first aspect of the technology of the present disclosure is a necklace-type terminal including: a sensor that detects biometric data of a wearer who is a medical professional; a microphone that picks up a conversation between the wearer and a patient and converts the conversation into audio data; a collection unit that collects the outputs of the sensor and the microphone; and a communication unit that transmits the outputs of the sensor and the microphone collected by the collection unit to an external device; a data processing device; and a data processing system comprising: The data processing device is a data processing system that includes: an input unit that accepts biometric data collected by the sensor and audio data picked up by the microphone; a processing unit that acquires medically important information from the audio data based on changes in the wearer's emotions estimated from the biometric data, creates a summary that summarizes the content of the dialogue by preferentially including the important information, and records the summary in a recording unit; and an output unit that outputs the summary to a predetermined output destination.

[0006] A second aspect of the technology of the present disclosure is the data processing system of the first aspect, wherein the processing unit creates the summary in a manner that makes the important information identifiable.

[0007] A third aspect of the technology disclosed herein is a data processing system according to the first aspect, wherein the processing unit inputs the wearer's emotional change pattern and a prompt including the voice data into a data generation model, and obtains the important information based on the output of the data generation model.

[0008] A fourth aspect of the technology disclosed herein is a data processing system according to the third aspect, wherein when a pattern of change in the wearer's emotions satisfies a predetermined trigger condition, the processing unit inputs a prompt including the pattern of change in the wearer's emotions into the data generation model, and obtains the important information based on the output of the data generation model.

[0009] A fifth aspect of the technology disclosed herein is a data processing system according to the third or fourth aspect, wherein when the voice data during the dialogue satisfies a predetermined trigger condition, the processing unit inputs a prompt including the voice data into the data generation model, and obtains the important information based on the output of the data generation model.

[0010] A sixth aspect of the technology of the present disclosure is that, in the data processing system of the first aspect, the processing unit detects the content of the voice data at that time from a pattern of changes in the wearer's emotions estimated from the biometric data, and determines the importance of the detected voice data in the voice data during the conversation.

[0011] A seventh aspect of the technique of the present disclosure is the data processing system of the sixth aspect, wherein the processing unit creates the summary based on the importance. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a necklace-type terminal according to a first embodiment. [Figure 3] 1 is a side view showing the configuration of a necklace-type terminal according to a first embodiment. [Figure 4] 1 is a top view showing the configuration of a necklace-type terminal according to a first embodiment. FIG. [Figure 5] 2 is a schematic diagram showing the functional configuration of a control unit of the necklace-type terminal according to the first embodiment. [Figure 6] 2 shows a schematic functional configuration of a specific processing unit of the data processing device according to the first embodiment. [Figure 7] 3 shows an example of an operational flow of a specific process performed by the data processing device according to the first embodiment. [Figure 8] 10 is a diagram illustrating an example of an operational flow of a specific process performed by a data processing device according to a second embodiment. [Figure 9] 10 shows an example of an operation flow when updating a data generation model according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, an example of an embodiment of a data processing device, a data processing method, and a program according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), or an APU (Accelerated Processing Unit).

[0016] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0017] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0018] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0020] [First embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to a first embodiment of the present disclosure.

[0021] 1, the data processing system 10 includes a data processing device 12 and a necklace-type terminal 14. An example of the data processing device 12 is a server. In this embodiment, the data processing device 12 is an example of a "data processing device" according to the technology of the present disclosure, and the necklace-type terminal 14 is an example of a "necklace-type terminal" according to the technology of the present disclosure.

[0022] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0023] The necklace type terminal 14 includes a computer 36, a microphone 38, a sensor 39, a speaker 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 38, the speaker 40, and the camera 42 are also connected to the bus 52.

[0024] The user 20 who wears the necklace-type terminal 14 may be, for example, a patient whose health condition is to be diagnosed, or may be a normal user.

[0025] The microphone 38 picks up the voice uttered by the user 20 who is wearing the necklace-type terminal 14, as well as sounds around the user 20. The microphone 38 also receives instructions and the like from the user 20 by receiving the voice uttered by the user 20. The microphone 38 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 40 outputs audio in accordance with instructions from the processor 46. The speaker 40 is, for example, a directional speaker, and outputs audio toward the ears of the user 20.

[0026] The sensor 39 is a sensor that detects biological data of the user 20 who is wearing the necklace-type terminal. For example, the sensor 39 is a heart rate sensor or a blood oxygen sensor.

[0027] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the necklace-type terminal 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58. The data generation model 58 is used by the specific processing unit 290. The storage 32 also includes a data accumulation unit 57.

[0032] In the necklace-type terminal 14, the data collection process is performed by the processor 46. A data collection program 60 is stored in the storage 50. The processor 46 reads the data collection program 60 from the storage 50 and executes the read data collection program 60 on the RAM 48. The data collection process is realized by the processor 46 operating as the control unit 46A in accordance with the data collection program 60 executed on the RAM 48.

[0033] As shown in FIGS. 3 and 4, the necklace type terminal 14 includes multiple microphones 38, multiple sensors 39, multiple speakers 40, and multiple cameras 42. FIGS. 3 and 4 show an example in which two microphones 38 are arranged so as to be located in front of the user 20 when the user 20 wears the necklace type terminal 14. FIGS. 3 and 4 also show an example in which two sensors 39 are arranged so as to be located on the right and left sides of the user 20 when the user 20 wears the necklace type terminal 14. FIGS. 3 and 4 also show an example in which two speakers 40 are arranged so as to be located on the right and left rear sides of the user 20 when the user 20 wears the necklace type terminal 14. FIGS. 3 and 4 also show an example in which two cameras 42 are arranged so as to be located on the right and left front sides of the user 20 when the user 20 wears the necklace type terminal 14. FIGS. 3 and 4 also show an example in which two sensors 39 are arranged inside the necklace type terminal 14 so as to come into contact with the neck of the user 20 when the user 20 wears the necklace type terminal 14.

[0034] Next, the processing of the control unit 46A when the necklace-type terminal 14 performs a data collection process for collecting data will be described.

[0035] In the data collection process of this embodiment, biometric data of the user is collected in real time. Furthermore, not only biometric data but also all situational data surrounding the user is collected. This makes it possible to detect early signs of, for example, Alzheimer's disease and dementia. It also makes it possible to monitor the user's health condition (for example, heart disease).

[0036] As shown in FIG. 5, the control unit 46A includes a data collection unit 100 and a communication unit 102.

[0037] The data collection unit 100 collects the output of each of the microphone 38, the sensor 39, and the camera 42.

[0038] The communication unit 102 transmits the outputs of the microphone 38 , the sensor 39 , and the camera 42 collected by the data collection unit 100 to the data processing device 12 .

[0039] Next, a description will be given of the processing of the specific processing unit 290 when the data processing device 12 performs the specific processing to acquire a response corresponding to a user utterance.

[0040] In the identification process of this embodiment, a response corresponding to a user utterance picked up by the microphone 38 of the necklace-type terminal 14 is acquired using the data generation model 58.

[0041] As shown in FIG. 6, the specific processing unit 290 includes an input unit 292, a processing unit 294, and an output unit 296.

[0042] The input unit 292 stores the outputs of the microphone 38 , the sensor 39 , and the camera 42 received from the necklace-type terminal 14 in the data storage unit 57 .

[0043] The input unit 292 acquires the user's utterance received by the necklace type terminal 14. Specifically, the input unit 292 acquires the user's utterance picked up by the microphone 38 of the necklace type terminal 14.

[0044] The processing unit 294 performs a specific process using the data generation model 58. Specifically, a prompt including a user utterance is input to the data generation model 58 to obtain a generation result. At this time, the prompt may further include outputs from the sensor 39 and the camera 42 collected by the data collection unit 100.

[0045] The output unit 296 transmits the result of the identification process to the necklace type terminal 14. In the necklace type terminal 14, the control unit 46A causes the speaker 40 to output the result of the identification process. In this way, a response corresponding to the user utterance picked up by the microphone 38 is output to the user 20 by the speaker 40. The microphone 38 further acquires the user utterance in response to the result of the identification process. The control unit 46A transmits voice data indicating the user utterance acquired by the microphone 38 to the data processing device 12. In the data processing device 12, the identification processing unit 290 acquires the user utterance.

[0046] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generative AI models. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0047] The outputs of the microphone 38, the sensor 39, and the camera 42 stored in the data storage unit 57 are used, for example, to diagnose the health condition of the user 20. In this case, the outputs of the microphone 38, the sensor 39, and the camera 42 stored in the data storage unit 57 may be transmitted to a terminal on the medical institution side. Alternatively, the data processing device 12 may analyze the outputs of the microphone 38, the sensor 39, and the camera 42 stored in the data storage unit 57 to diagnose the health condition of the user 20.

[0048] Next, the operation of the data processing system 10 will be described.

[0049] First, an example of the flow of the data collection process will be described.

[0050] When the user 20 is wearing the necklace-type terminal 14, the data collection unit 100 sequentially collects the outputs of the microphone 38, the sensor 39, and the camera 42. The communication unit 102 sequentially transmits the outputs of the microphone 38, the sensor 39, and the camera 42 collected by the data collection unit 100 to the data processing device 12.

[0051] Next, an example of the flow of the identification process will be described with reference to Fig. 7. Here, it is assumed that the input unit 292 of the data processing device 12 sequentially acquires the outputs of the microphone 38, the sensor 39, and the camera 42 received from the necklace-type terminal 14 and stores them in the data accumulation unit 57.

[0052] In step S300, the processing unit 294 determines whether a predetermined trigger condition is satisfied. Specifically, the trigger condition may be that a specific word (for example, the name of an agent installed in the necklace type terminal 14) or phrase (for example, "Hi! XXX" (XXX is the name of the agent)) is included in the user utterance picked up by the microphone 38.

[0053] If the trigger condition is met in step S300 (step S300; Yes), the data processing system 10 proceeds to step S301. On the other hand, if the trigger condition is not met in step S300 (step S300; No), the data processing system 10 ends the identification process.

[0054] In step S301, the processing unit 294 generates a prompt by adding an instruction sentence for obtaining a result of a specific process to text representing a user utterance picked up by the microphone 38.

[0055] For example, a prompt such as "The user is saying the following: XXX. Please respond as an agent." (XXX is the user's utterance) may be generated. Alternatively, the outputs of the sensor 39 and the camera 42 may be added to the prompt to generate a prompt such as "This is biometric data representing the user's heart rate and video data representing the user's surroundings. The user is also saying the following: XXX. Please respond as an agent." (XXX is the user's utterance).

[0056] In step S303, the processing unit 294 inputs the generated prompt to the data generation model 58, and obtains the result of the specific process based on the output of the data generation model 58.

[0057] In step S304, the output unit 296 outputs the result of the identification process to the necklace-type terminal 14, and the identification process ends.

[0058] [Second embodiment] Next, a second embodiment will be described. In the second embodiment, as the identification process described in the first embodiment, for example, a conversation between a medical professional and a patient during a medical interview is recorded in real time in a medical setting. Then, medically important information for appropriate diagnosis and treatment is obtained from biometric data of the medical professional during the medical interview and audio data including the content of the recorded conversation, and a summary of the medical interview is created. Here, "medical professional" refers to a person engaged in medical care, such as a doctor or nurse. Furthermore, "medically important information" refers to information indicating symptoms such as a physical condition or change that a patient consciously or unconsciously feels, and is information linked to a specific disease. Note that this embodiment will also be described using the configuration and symbols of the data processing system 10 in the first embodiment described above.

[0059] In this embodiment, the user 20, who is a medical professional, wears the necklace-type terminal 14. As described above, the necklace-type terminal 14 includes a sensor 39 that detects biometric data of the user 20, who is a medical professional and wears the necklace-type terminal 14, in real time, a microphone 38 that picks up the conversation between the user 20 and a patient and converts it into audio data, a data collection unit 100 that collects the outputs of the sensor 39 and the microphone 38, and a communication unit 102 that transmits the outputs of the sensor 39 and the microphone 38 collected by the data collection unit 100 to the data processing device 12, which is an external device.

[0060] The data processing device 12 in this embodiment includes an input unit 292 that receives biometric data collected by the sensor 39 and voice data picked up by the microphone 38, a processing unit 294 that acquires medically important information from the voice data during the dialogue based on the pattern of emotional changes estimated from the biometric data of the user 20, creates a summary that summarizes the content of the dialogue so as to prioritize the medically important information, and stores (records) it in the data accumulation unit 57 as a recording unit, and an output unit 296 that outputs the summary including the medically important information to a predetermined output destination.

[0061] Here, the "preset output destination" may be, for example, a terminal of a medical professional or a terminal of a medical institution. The medical professional can access their own terminal or the terminal of the medical institution to check the summary including medically important information.

[0062] The sensor 39 estimates the emotional change pattern of the user 20 by measuring in real time the heart rate, pulse rate, respiratory rate, blood pressure, sweating, and other biological data of the user 20 wearing the necklace-type device 14. Here, the emotional change pattern may be an absolute emotional value that indicates the intensity of the emotion, an index value that indicates any of joy, anger, sadness, and pleasure, or the amount of change in the emotional value that changes within a predetermined period of time. The emotional change pattern serves as an index for estimating the emotion of the user 20, such as whether the user 20 is tense, relaxed, agitated, tired, excited, relieved, anxious, angry, sad, etc.

[0063] Here, when conversing with a patient, not only the tone of voice of the medical professional but also the information conveyed by the medical professional's emotions is important. For this reason, the processing unit 294 detects patterns of emotional changes of the user 20, who is a medical professional, using voice data collected by the microphone 38 and biometric data of the user 20 collected by the sensor 39. As a result, the processing unit 294 acquires medically important information in the conversation with the patient and creates a summary that summarizes the content of the conversation by prioritizing the acquired medically important information. At this time, the processing unit 294 captures subtle changes in the user 20's emotions using the biometric data and, based on the pattern of changes in the user 20's emotions during the conversation, evaluates what is important and what is not important in the conversation with the patient and determines the level of importance.

[0064] The processing unit 294 automatically converts the voice data collected by the microphone 38 into a text format to generate text. The processing unit 294 also recognizes the voice of the user 20 from the voice data collected by the microphone 38 and automatically converts it into text.

[0065] The processing unit 294 inputs prompts including emotion change patterns estimated from the biometric data of the user 20 and voice data during dialogue with the patient into the data generation model 58, and obtains a generation result. That is, the prompts include the outputs of the sensor 39 and the microphone 38 collected by the data collection unit 100.

[0066] Furthermore, when the pattern of changes in the emotions of the user 20 satisfies a predetermined trigger condition, the processing unit 294 inputs a prompt including the pattern of changes in the emotions of the user 20 to the data generation model 58, and acquires medically important information based on the output of the data generation model 58. Furthermore, when voice data during a dialogue with the patient satisfies a predetermined trigger condition, the processing unit 294 inputs a prompt including the voice data during a dialogue with the patient to the data generation model 58, and acquires medically important information based on the output of the data generation model 58. Note that the trigger condition may not be required.

[0067] The processing unit 294 then inputs the emotional change pattern of the user 20, who is a medical professional, and a prompt including voice data during the conversation with the patient to the data generation model 58, and acquires medically important information based on the output of the data generation model 58. The data generation model 58 infers the input emotional change pattern of the user 20 and the voice data during the conversation with the patient in accordance with the instructions given by the prompt, and outputs the inference result, medically important information during the conversation, in the form of data such as voice data and / or text data.

[0068] Furthermore, the processing unit 294 detects the content of the voice data at this time from the pattern of changes in emotion estimated from the biometric data of the user 20, and assigns a label indicating the type of emotion to the detected voice data to label it.

[0069] Then, the processing unit 294 determines the medical importance of the voice data converted into text based on the assigned label. That is, the processing unit 294 acquires medically important information from the voice data during the conversation with the patient based on the assigned label.

[0070] The processing unit 294 then creates a summary of the dialogue with the patient based on the determined medical importance. At this time, the processing unit 294 may create the summary in such a way that medically important information can be distinguished from other information. Here, "distinguishable" means that the medically important information is displayed in a manner that makes it distinguishable from other information, for example, by underlining the medically important information, adding a color to it, or making it bolder than other information.

[0071] Furthermore, the processing unit 294 may estimate a change in the emotion of the user 20 based on the speaking speed, intonation, tone, and other aspects of the user's speaking style in the audio data collected by the microphone 38. Specifically, for example, during a medical interview with a patient, if the user 20, who is a medical professional and wears the necklace-type terminal 14, utters "Do you have pain in your lower abdomen?" and the user 20 speaks with a stronger tone, the processing unit 294 acquires the patient's response to the user's utterance of "Do you have pain in your lower abdomen?", that is, "I don't have pain in my lower abdomen," as medically important information.

[0072] That is, the processing unit 294 may determine the medical importance of the dialogue with the patient based on the change pattern of the user's tone and emotions during the dialogue with the patient, and create a summary that prioritizes information of high medical importance and medical significance. For example, each time collected voice data is obtained, an emotion engine that estimates emotions based on the collected voice data may be used to estimate the emotion of the user 20. Then, the data generation model 58 may be used to determine the medical importance of the voice data based on the time-series changes in the estimated emotion of the user 20. For example, the outputs of the sensor 39 and the microphone 38 may be added to a prompt to generate a prompt such as, "This is voice data including biometric data representing the user's heart rate and the dialogue with the patient. Please summarize the contents of the medical interview in a way that identifies medically significant information."

[0073] Furthermore, the processing unit 294 may acquire, from the voice data during a conversation with a patient, utterances made when there is a change in the emotions of the user 20, who is a medical professional and wears the necklace-type terminal 14, as medically important information. The processing unit 294 may also determine the medical importance of the medically important information using the frequency of utterances associated with physical changes during a conversation with a patient. The medical importance may also be determined according to the severity of the patient's illness during the conversation. Furthermore, when creating a summary, the patient may be asked to wear the necklace-type terminal 14, and biometric data of the patient who is wearing the necklace-type terminal 14 may be detected in real time to acquire, as medically important information, utterances made when there is a change in the patient's emotions.

[0074] Next, an example of the operation of the specific processing unit 290 in this embodiment will be described with reference to FIG.

[0075] Here, when the user 20, who is a medical professional, wears the necklace-type terminal 14, the data collection unit 100 sequentially collects the outputs of the microphone 38 and the sensor 39. The communication unit 102 sequentially transmits the outputs of the microphone 38 and the sensor 39 collected by the data collection unit 100 to the data processing device 12.

[0076] In step S401, the processing unit 294 receives, via the input unit 292, the biometric data collected by the sensor 39 and the voice data including the content of the conversation picked up by the microphone .

[0077] In step S402, the processing unit 294 automatically converts the voice data into text format.

[0078] In step S403, the processing unit 294 determines the medical importance of the voice data converted into text. Specifically, the processing unit 294 detects the content of the voice data converted into text at this time from a pattern of changes in emotion estimated from the biometric data of the user 20, and assigns a label indicating the type of emotion to the detected voice data. Then, the processing unit 294 determines the medical importance of the voice data based on the assigned label.

[0079] The emotion change pattern may be, for example, an absolute value of an emotion value that indicates the intensity of an emotion, an index value that indicates any of "joy, anger, sadness, or pleasure," or the amount of change in the emotion value for each preset period. Furthermore, the processing unit 294 may detect the content of the voice data converted to text at that time from the tone of voice of the user 20 estimated from the voice data of the user 20, and may label the detected voice data by assigning a label indicating the type of emotion.

[0080] Specifically, for example, if the absolute value of the emotion value suddenly increases, if the amount of change in the emotion value over a certain period of time is large, or if the user 20 speaks forcefully, the processing unit 294 labels the utterance, which is the content of the voice data at that time, as "tension." Furthermore, if the absolute value of the emotion value gradually decreases, if the amount of change in the emotion value over a certain period of time is small, or if the user 20 speaks calmly, the processing unit 294 labels the utterance, which is the content of the voice data at that time, as "relief." Furthermore, the processing unit 294 labels the utterance, which is the content of the voice data, as "sad" if the index value indicating "joy, anger, sadness, and happiness" is a value of "sad." At this time, an index value estimated based on biometric data is used as the label indicating the type of emotion.

[0081] Then, the processing unit 294 determines that an utterance labeled with "tension" is more important than an utterance labeled with "relaxation," for example. Also, the processing unit 294 determines that an utterance labeled with "anxiety" is more important than an utterance labeled with "relief."

[0082] In step S404, the processing unit 294 creates a summary including medically important information based on the medical importance of the converted audio data, and stores (records) it in the data storage unit 57. For example, the processing unit 294 creates a summary including utterances labeled with "tension," "anxiety," etc. that have been determined to have a high level of medical importance, and stores it in the data storage unit 57.

[0083] Specifically, for example, when user 20, who is a medical professional and wears necklace-type terminal 14, utters "Are you getting enough sleep?" during a medical interview with a patient, if there is a large change in the emotional value of user 20, processing unit 294 labels the patient's utterance "I'm not getting enough sleep" in response to user 20's utterance "Are you getting enough sleep?" as "tension" and creates a summary sentence including the information "I'm not getting enough sleep" as having high medical importance.

[0084] Also, for example, when the patient utters "I have no appetite" during a medical interview, and the user 20, who is a medical professional and wears the necklace-type terminal 14, utters "It's okay even if you have no appetite as long as you stay hydrated," if the change in the emotional value of the user 20 is small, the processing unit 294 labels the patient's utterance "I have no appetite" as "reassuring" and creates a summary sentence as having low medical importance.

[0085] In step S405, the processing unit 294 outputs the created summary to a preset output destination via the output unit 296, and the identification process ends. Specifically, the processing unit 294 outputs the summary including the medically important information to a preset terminal of a medical professional or a terminal of a medical institution, and the identification process ends. At this time, the summary may be output so that the medically important information can be distinguished from other information.

[0086] In other words, it reduces the time medical professionals spend on preparing documents such as medical certificates and entering data such as electronic medical records. As a result, medical professionals can spend more time caring for patients, such as through medical examinations and rehabilitation, which also reduces their stress. Furthermore, by automatically converting information obtained from conversations with patients into text and judging its medical importance, medical professionals can reduce the time they spend laboriously inputting and searching. Furthermore, by creating summaries through conversations with patients, more appropriate diagnoses and treatment methods can be selected. It also reduces the burden on patients of having to fill out and input information into medical questionnaires. It also enables more accurate medical interviews even when patients are unable to clearly communicate their physical condition.

[0087] When a summary containing medically important information is output to a predetermined output destination by the process described above, a medical professional, i.e., a doctor, will check the output content and select a diagnosis and treatment method for the user 20. However, if the determination accuracy of the data generation model 58 is not high, even if a summary containing medically important information is output, the part determined to be medically important information may not actually be important. Therefore, it is expected that the determination accuracy will improve if a medical professional actually provides feedback to the data generation model 58 regarding the effectiveness of the summary, thereby updating the data generation model 58.

[0088] The operation when updating the data generation model 58 will be described with reference to the flowchart of FIG.

[0089] When the processing unit 294 outputs a summary containing medically important information to a predetermined output destination as a result of a dialogue with a patient, the processing unit 294 receives a feedback result regarding the effectiveness of the summary containing the medically important information (step S501). Then, the processing unit 294 evaluates the determination accuracy of the data generation model 58 based on the received feedback result (step S502). Finally, the processing unit 294 performs deep learning based on the evaluation result to update the data generation model 58 (step S503).

[0090] By performing the processing described above, the data generation model 58 undergoes machine learning again based on the feedback results each time it outputs a summary sentence containing medically important information through a dialogue with a patient, thereby improving the accuracy of the judgment.

[0091] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[0092] In the above embodiment, an example was given in which a specific process is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may be performed by multiple computers including computer 22.

[0093] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0094] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0095] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0096] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[0097] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[0098] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0099] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[0100] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0101] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0102] 10 Data Processing System 12 Data Processing Device 14 Necklace-type device 38 Microphone 39 Sensors 40 speakers 42 Camera 46A Control Unit 100 Data Collection Department 102 Communications Department 290 Special Processing Department 292 Input section 294 Processing Section 296 Output Section< / url:>

Claims

1. a sensor for detecting biometric data of a wearer who is a medical professional; a microphone that picks up the conversation between the wearer and the patient and converts it into voice data; a collection unit that collects outputs from the sensors and the microphones; a communication unit that transmits the outputs of the sensor and the microphone collected by the collection unit to an external device; and a data processing device; and a data processing system comprising: The data processing device includes: an input unit that receives the biological data collected by the sensor and the audio data collected by the microphone; a processing unit that acquires medically important information from the voice data based on changes in the wearer's emotions estimated from the biometric data, creates a summary that summarizes the content of the dialogue so as to preferentially include the important information, and records the summary in a recording unit; an output unit that outputs the summary to a predetermined output destination, Data processing system.

2. the processing unit creates the summary in such a way that the important information can be identified; 10. The data processing system of claim 1.

3. the processing unit inputs the emotion change pattern of the wearer and the prompt including the voice data into a data generation model, and acquires the important information based on an output of the data generation model.

10. The data processing system of claim 1.

4. 4. The data processing system according to claim 3, wherein, when a pattern of change in the wearer's emotions satisfies a predetermined trigger condition, the processing unit inputs a prompt including the pattern of change in the wearer's emotions into the data generation model, and acquires the important information based on an output of the data generation model.

5. 5. The data processing system according to claim 3, wherein when the voice data during the dialogue satisfies a predetermined trigger condition, the processing unit inputs a prompt including the voice data into the data generation model, and acquires the important information based on an output of the data generation model.

6. 2. The data processing system according to claim 1, wherein the processing unit detects the content of the voice data at that time from a pattern of changes in the wearer's emotions estimated from the biometric data, and determines the importance of the detected voice data in the voice data during the conversation.

7. 7. The data processing system according to claim 6, wherein the processing unit creates the summary based on the importance.

Images