Terminal equipment and data processing system

Abstract

A terminal and a data processing system for collecting biometric data of a user are provided. [Solution] The terminal device includes a camera that captures the wearer's surroundings, a sensor that detects the wearer's biometric data, a microphone, and a control unit that sets the storage mode of each output data collected over a certain period of time in a specific memory according to a specific index value calculated based on the output data of each of the camera, sensor, and microphone, and performs control to record each output data in memory according to the set storage mode as the wearer's life log.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a terminal device and a data processing system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional techniques leave room for improvement in collecting biometric data of users. [Means for solving the problem]

[0005] A first aspect of the technology of the present disclosure includes a terminal device including a camera that captures an image of a wearer's surroundings, a sensor that detects biometric data of the wearer, a microphone, and a control unit that sets a storage mode of each piece of output data collected over a certain period of time in a specific memory in accordance with a specific index value calculated based on output data from the camera, the sensor, and the microphone, and executes control to record each piece of output data in accordance with the set storage mode in the memory as a life log of the wearer.

[0006] In a first aspect of the technology of the present disclosure, the control unit further analyzes at least one of the wearer's emotions, the content of the wearer's voice, the wearer's biometric information, and the content of the medical professional's remarks collected by the microphone based on each of the output data, and calculates the value of the index related to the content of the voice related to the wearer's symptoms based on the analyzed information.

[0007] A second aspect of the technology of the present disclosure is that the storage manner includes at least part of the compression rate of each of the output data, the storage format of the output data, deletion of all or part of each of the output data, the storage period of each of the output data, and the storage destination of each of the output data.

[0008] In a third aspect of the technique of the present disclosure, the control unit of the first aspect associates each of the output data with the index and records the output data in the memory.

[0009] In a fourth aspect of the technique of the present disclosure, the terminal device of the first aspect is a necklace-type terminal that is worn around the neck of the wearer.

[0010] A fifth aspect of the technique of the present disclosure includes a data processing system including the terminal device of the first aspect and a data processing device, wherein the data processing device includes an input unit that receives an utterance from the wearer picked up by the microphone, a processing unit that inputs a prompt including the utterance into a data generation model and acquires a response to the utterance using an output of the data generation model, and an output unit that reproduces the acquired response from a speaker of the terminal device. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a conceptual diagram illustrating an example of a configuration of a data processing system. [Figure 2] FIG. 2 is a conceptual diagram showing an example of main functions of a data processing device and a necklace-type terminal. [Figure 3] FIG. 2 is a side view showing the configuration of the necklace-type terminal. [Figure 4] FIG. 2 is a top view showing the configuration of the necklace-type terminal. [Figure 5] 10 is a schematic diagram showing the functional configuration of a control unit of the necklace-type terminal. [Figure 6] 2 shows a schematic functional configuration of a specific processing unit of the data processing device. [Figure 7] 10 is a diagram illustrating an example of an operational flow of specific processing by a data processing device. [Figure 8A] FIG. 10 is a diagram illustrating an example of an index used in the processing of the second and third embodiments. [Figure 8B] FIG. 10 is a diagram illustrating an example of an index used in the processing of the second and third embodiments. [Figure 8C] FIG. 10 is a diagram illustrating an example of an index used in the processing of the second and third embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an example of an embodiment of a data processing device, a data processing method, and a program according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0013] First, the terms used in the following description will be explained.

[0014] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), or an APU (Accelerated Processing Unit).

[0015] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0016] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0017] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0018] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0019] [First embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the embodiment.

[0020] As shown in FIG. 1, a data processing system 10 includes a data processing device 12 and a necklace-type terminal 14. An example of the data processing device 12 is a server. In this embodiment, the data processing device 12 is an example of a "data processing device" according to the technology of the present disclosure, and the necklace-type terminal 14 is an example of a "terminal device" according to the technology of the present disclosure. Note that the terminal device of the present disclosure is not limited to the necklace-type terminal 14. The terminal device of the present disclosure may include a robot, a doll, a stuffed animal, a wearable device (pendant, smart watch, smart glasses), a smartphone, a smart speaker, earphones, a personal computer, and the like.

[0021] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 53. Examples of the network 53 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0022] The necklace type terminal 14 includes a computer 36, a microphone 38, a sensor 39, a speaker 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 38, the speaker 40, and the camera 42 are also connected to the bus 52.

[0023] The user 20 who wears the necklace-type terminal 14 may be, for example, a patient whose health condition is to be diagnosed, or may be a normal user.

[0024] The microphone 38 picks up the voice uttered by the user 20 who is wearing the necklace-type terminal 14, as well as sounds around the user 20. The microphone 38 also receives instructions and the like from the user 20 by receiving the voice uttered by the user 20. The microphone 38 captures the voice uttered by the user 20 and / or people near the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 40 outputs audio in accordance with instructions from the processor 46. The speaker 40 is, for example, a directional speaker, and outputs audio toward the ears of the user 20.

[0025] The sensor 39 is a sensor that detects biological data of the user 20 who is wearing the necklace-type terminal. For example, the sensor 39 is a heart rate sensor or a blood oxygen sensor.

[0026] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures an image of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the width of the field of vision of a typical healthy person). Camera 42 may be capable of capturing an image of the body of a person near user 20, for example.

[0027] The communication I / F 44 is connected to a network 53. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 53.

[0028] FIG. 2 shows an example of the main functions of the data processing device 12 and the necklace-type terminal 14.

[0029] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores a data generation model 58. The data generation model 58 is used by the specific processing unit 290. The storage 32 also includes a data accumulation unit .

[0031] In the necklace-type terminal 14, the data collection process is performed by the processor 46. A data collection program 60 is stored in the storage 50. The processor 46 reads the data collection program 60 from the storage 50 and executes the read data collection program 60 on the RAM 48. The data collection process is realized by the processor 46 operating as the control unit 46A in accordance with the data collection program 60 executed on the RAM 48.

[0032] As shown in FIGS. 3 and 4, the necklace type terminal 14 includes multiple microphones 38, multiple sensors 39, multiple speakers 40, and multiple cameras 42. FIGS. 3 and 4 show an example in which two microphones 38 are arranged so as to be located in front of the user 20 when the user 20 wears the necklace type terminal 14. FIGS. 3 and 4 also show an example in which two sensors 39 are arranged so as to be located on the right and left sides of the user 20 when the user 20 wears the necklace type terminal 14. FIGS. 3 and 4 also show an example in which two speakers 40 are arranged so as to be located on the right and left rear sides of the user 20 when the user 20 wears the necklace type terminal 14. FIGS. 3 and 4 also show an example in which two cameras 42 are arranged so as to be located on the right and left front sides of the user 20 when the user 20 wears the necklace type terminal 14. FIGS. 3 and 4 also show an example in which two sensors 39 are arranged inside the necklace type terminal 14 so as to come into contact with the neck of the user 20 when the user 20 wears the necklace type terminal 14.

[0033] Next, the processing of the control unit 46A when the necklace-type terminal 14 performs a data collection process for collecting data will be described.

[0034] In the data collection process of this embodiment, biometric data of the user is collected in real time. Furthermore, not only biometric data but also all situational data surrounding the user is collected. This makes it possible to detect early signs of, for example, Alzheimer's disease and dementia. It also makes it possible to monitor the user's health condition (for example, heart disease).

[0035] As shown in FIG. 5, the control unit 46A includes a data collection unit 100 and a communication unit 102.

[0036] The data collection unit 100 collects the output of each of the microphone 38, the sensor 39, and the camera 42.

[0037] The communication unit 102 transmits the outputs of the microphone 38 , the sensor 39 , and the camera 42 collected by the data collection unit 100 to the data processing device 12 .

[0038] Next, a description will be given of the processing of the specific processing unit 290 when the data processing device 12 performs the specific processing to acquire a response corresponding to a user utterance.

[0039] In the identification process of this embodiment, a response corresponding to a user utterance picked up by the microphone 38 of the necklace-type terminal 14 is acquired using the data generation model 58.

[0040] As shown in FIG. 6, the specific processing unit 290 includes an input unit 292, a processing unit 294, and an output unit 296.

[0041] The input unit 292 stores the outputs of the microphone 38 , the sensor 39 , and the camera 42 received from the necklace-type terminal 14 in the data storage unit 54 .

[0042] The input unit 292 acquires the user's utterance received by the necklace type terminal 14. Specifically, the input unit 292 acquires the user's utterance picked up by the microphone 38 of the necklace type terminal 14.

[0043] The processing unit 294 performs a specific process using the data generation model 58. Specifically, a prompt including a user utterance is input to the data generation model 58 to obtain a generation result. At this time, the prompt may further include outputs from the sensor 39 and the camera 42 collected by the data collection unit 100.

[0044] The output unit 296 transmits the result of the identification process to the necklace type terminal 14. In the necklace type terminal 14, the control unit 46A causes the speaker 40 to output the result of the identification process. In this way, a response corresponding to the user utterance picked up by the microphone 38 is output to the user 20 by the speaker 40. The microphone 38 further acquires the user utterance in response to the result of the identification process. The control unit 46A transmits voice data indicating the user utterance acquired by the microphone 38 to the data processing device 12. In the data processing device 12, the identification processing unit 290 acquires the user utterance.

[0045] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generative AI models. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0046] The outputs of the microphone 38, the sensor 39, and the camera 42 stored in the data storage unit 54 are used, for example, to diagnose the health condition of the user 20. In this case, the outputs of the microphone 38, the sensor 39, and the camera 42 stored in the data storage unit 54 may be transmitted to a terminal on the medical institution side. Alternatively, the data processing device 12 may analyze the outputs of the microphone 38, the sensor 39, and the camera 42 stored in the data storage unit 54 to diagnose the health condition of the user 20.

[0047] Next, the operation of the data processing system 10 will be described.

[0048] First, an example of the flow of the data collection process will be described.

[0049] When the user 20 is wearing the necklace-type terminal 14, the data collection unit 100 sequentially collects the outputs of the microphone 38, the sensor 39, and the camera 42. The communication unit 102 sequentially transmits the outputs of the microphone 38, the sensor 39, and the camera 42 collected by the data collection unit 100 to the data processing device 12.

[0050] Next, an example of the flow of the identification process will be described with reference to Fig. 7. Here, it is assumed that the input unit 292 of the data processing device 12 sequentially acquires the outputs of the microphone 38, the sensor 39, and the camera 42 received from the necklace-type terminal 14 and stores them in the data accumulation unit 54.

[0051] In step S300, the processing unit 294 determines whether a predetermined trigger condition is satisfied. Specifically, the trigger condition may be that a specific word (for example, the name of an agent installed in the necklace type terminal 14) or phrase (for example, "Hi! XXX" (XXX is the name of the agent)) is included in the user utterance picked up by the microphone 38.

[0052] If the trigger condition is met in step S300 (step S300; Yes), the data processing system 10 proceeds to step S301. On the other hand, if the trigger condition is not met in step S300 (step S300; No), the data processing system 10 ends the identification process.

[0053] In step S301, the processing unit 294 generates a prompt by adding an instruction sentence for obtaining a result of a specific process to text representing a user utterance picked up by the microphone 38.

[0054] For example, a prompt such as "The user is saying the following: XXX. Please respond as an agent." (XXX is the user's utterance) may be generated. Alternatively, the outputs of the sensor 39 and the camera 42 may be added to the prompt to generate a prompt such as "This is biometric data representing the user's heart rate and video data representing the user's surroundings. The user is also saying the following: XXX. Please respond as an agent." (XXX is the user's utterance).

[0055] In step S303, the processing unit 294 inputs the generated prompt to the data generation model 58, and obtains the result of the specific process based on the output of the data generation model 58.

[0056] In step S304, the output unit 296 outputs the result of the identification process to the necklace-type terminal 14, and the identification process ends.

[0057] [Second embodiment]

[0058] The control unit 46A of the necklace type terminal 14 may set a storage mode in a specific memory (such as the storage 50 or the database 24) for each piece of output data collected by the collection unit (data collection unit 100) over a certain period of time, according to a specific index value calculated based on the output data of the camera 42, the sensor 39, and the microphone 38. The control unit 46A may also execute control to record each piece of output data according to the set storage mode in a memory (such as the storage 50 or the database 24) as a life log of the wearer.

[0059] The control unit 46A of the necklace-type terminal 14 may analyze at least one of the wearer's emotions, the content of the wearer's voice, the wearer's biometric information, the content of the voice of a person near the wearer, and the body movements of a person near the wearer based on each output data, and calculate an index value based on the analyzed information.

[0060] The storage manner may include at least some of the compression rate of each of the output data, the storage format (data format, file format) of each of the output data, deletion of all or part of each of the output data, the storage period of each of the output data, and the storage destination of each of the output data.

[0061] The output data may include images, sounds, biometric information, etc. collected by the necklace-type device 14. The images may include either still images or moving images. The biometric information may include electrocardiogram data, pulse rate, body temperature, oxygen concentration, etc. The certain period of time may be interpreted as, for example, 1 second, 1 minute, 10 minutes, 1 hour, 2 hours, etc.

[0062] The index may include, for example, an importance index as shown in FIG. 8A, an excitement index as shown in FIG. 8B, an emotion index as shown in FIG. 8C, and the like.

[0063] The importance index may be interpreted as an index representing the degree of importance in the activity of the user 20 who is wearing the necklace-type device 14. The importance index may include values ​​such as "1," "2," and "3." A higher importance index may be interpreted as a higher importance.

[0064] For example, when the necklace type terminal 14 or the data processing device 12 determines that the user 20 in a meeting is participating in an important meeting such as a board meeting by analyzing output data collected by the camera 42, the sensor 39, the microphone 38, etc., the necklace type terminal 14 or the data processing device 12 may set the importance index to "3." When the necklace type terminal 14 or the data processing device 12 determines that the user 20 is participating in a regular group meeting, the necklace type terminal 14 or the data processing device 12 may set the importance index to "2." When the necklace type terminal 14 or the data processing device 12 determines that the user 20 is having a casual conversation at work (everyday conversation), the necklace type terminal 14 or the data processing device 12 may set the importance index to "1."

[0065] The excitement index may be interpreted as an index representing the degree of excitement of the user 20 who is wearing the necklace-type terminal 14. The excitement index may include values ​​such as "1," "2," and "3." A higher excitement index may be interpreted as a higher level of excitement.

[0066] For example, if the necklace-type terminal 14 or the data processing device 12 determines, by analyzing the output data collected by the camera 42, the sensor 39, the microphone 38, etc., that the excitement level of the user 20 at a concert venue or the like is very high, the excitement index may be set to "3." If the necklace-type terminal 14 or the data processing device 12 determines, by analyzing the output data collected by the camera 42, the sensor 39, the microphone 38, etc., that the excitement level of the user 20 riding a bus, train, etc. is relatively low, the excitement index may be set to "2." If the necklace-type terminal 14 or the data processing device 12 determines, by analyzing the output data collected by the camera 42, the sensor 39, the microphone 38, etc., that the excitement level of the user 20 meditating or walking is very low, the excitement index may be set to "1."

[0067] The emotion index may be interpreted as an index representing the degree of emotion of the user 20 who is wearing the necklace-type terminal 14, for example. The emotion index may include values ​​such as "1," "2," and "3." A higher emotion index may be interpreted as a higher degree of emotion.

[0068] For example, when the necklace-type terminal 14 or the data processing device 12 determines, by analyzing the output data collected by the camera 42, the sensor 39, the microphone 38, etc., that the user 20 is in an excited mood while eating their favorite food, the necklace-type terminal 14 or the data processing device 12 may set the emotion index to "3." When the necklace-type terminal 14 or the data processing device 12 determines, by analyzing the output data collected by the camera 42, the sensor 39, the microphone 38, etc., that the user 20 is reading and is feeling close to being calm, the necklace-type terminal 14 or the data processing device 12 may set the emotion index to "2." When the necklace-type terminal 14 or the data processing device 12 determines, by analyzing the output data collected by the camera 42, the sensor 39, the microphone 38, etc., that the user 20 who is not good at exercise is feeling close to being depressed while exercising, the necklace-type terminal 14 or the data processing device 12 may set the emotion index to "1."

[0069] These indexes may be associated with data storage modes. Storage modes may include storing the output data of each of the camera 42, the sensor 39, and the microphone 38 at a high compression rate, storing the output data at a low compression rate, storing the output data in a large-capacity storage format (data format, file format) without compression, storing the output data in a small-capacity storage format without compression, or deleting specific data from the output data as unnecessary data. Storage modes may include a storage period (e.g., several months, one year, several years, ten years, etc.) for all or part of the output data of each of the camera 42, the sensor 39, and the microphone 38. Storage modes may include deleting all or part of the output data of each of the camera 42, the sensor 39, and the microphone 38 collected over a certain period of time. Storage modes may include storing the output data of each of the camera 42, the sensor 39, and the microphone 38 collected over a certain period of time in a location other than a specific memory. Specifically, the storage mode may include storing the output data on a specific server instead of the storage 50 of the necklace-type terminal 14. More specifically, the storage mode may include changing the storage destination of the output data to an important cloud (such as a cloud for which a separate contract has been made) that is different from the usual storage destination. The storage modes can be classified into, for example, "3," "2," and "1." Furthermore, for example, storage mode "3" may include a mode in which data is stored at a high compression rate, a mode in which data is stored in a storage format with a larger capacity, and a mode in which data is stored for a longer period of time. Furthermore, for example, storage mode "1" may include a mode in which data is stored at a lower compression rate than storage mode "3," a mode in which data is stored in a storage format with a smaller capacity than storage mode "3," a mode in which data is stored for a shorter period of time than storage mode "3," and a mode in which data is deleted as unnecessary data. Furthermore, storage mode "2" may include a mode in which data is stored at a compression rate lower than storage mode "3" and higher than "1", a mode in which data is stored in a storage format with a capacity smaller than storage mode "3" and larger than "1", and a mode in which the storage period is shorter than storage mode "3" and longer than "1".

[0070] If the user 20 is attending an important meeting and it is desirable to store the output data for a long period of time, the necklace-type terminal 14 or the data processing device 12 may set the storage mode "3" for the excitement index "3" to increase the compression rate of each output data, thereby enabling effective use of memory capacity.

[0071] If the user 20 who is not good at exercise is feeling depressed, the necklace-type terminal 14 or the data processing device 12 may regard the output data collected at this time as unnecessary data with low usefulness and set the excitement index to "1" and the storage mode to "1". This can save memory capacity.

[0072] The life log may be interpreted as a history of actions taken by the user 20 in daily life, and may include sounds and images associated with the user 20, specifically, sounds collected by the microphone 38 in daily life and images taken by the camera 42. The life log may record sounds and images associated with the user 20 in association with the date, time, and location at which they were acquired.

[0073] The sounds collected by the microphone 38 may include the voices of the user 20 and / or those near the user 20 in conversation, sounds occurring around the user 20 while walking or cycling (voices in a meeting, cars driving by, birds chirping, the sound of a river flowing, trees rustling in the wind), and the like.

[0074] For example, the camera 42 may capture an image of scenery within an angle of view that captures what is in front of the user 20, or may capture an image of scenery within an angle of view that captures what is not in front of the user 20, such as what is to the side, behind, below, or above the user 20. The image captured by the camera 42 may include the image of someone the user 20 is talking to, the image of the scenery around the user 20 when taking a walk or cycling, the image of a pet walking with the user 20, etc.

[0075] While the necklace type terminal 14 is attached to the user 20, all or part of the collected sounds and images may be recorded as a life log in the storage 50, the database 24, etc. Specifically, when the necklace type terminal 14 is attached to the user 20, recording of sounds and images may start, and when the necklace type terminal 14 is removed from the user 20, recording of sounds and images may end.

[0076] The control unit 46A of the necklace-type terminal 14 may associate each piece of output data with an index and record it in memory. For example, the output data collected when the user 20 is attending an important meeting may be associated with an excitement index of "3."

[0077] The specific processing unit 290 of the data processing device 12 may include an input unit 292 that receives the wearer's speech picked up by the microphone 38, a processing unit 294 that inputs a prompt including the speech into a data generation model and obtains a response to the speech using the output of the data generation model, and an output unit 296 that plays the obtained response from a speaker of the terminal device.

[0078] Specifically, when the processing unit 294 receives an utterance from the user 20 relating to the memory or behavior of the user 20 as user data, the processing unit 294 may execute a process of suggesting information corresponding to the content of the utterance to the user 20, for example, by referring to the database 24 in which the life log is recorded.

[0079] (First example of information corresponding to the content of the utterance) When a user wearing the necklace-type terminal 14 requests a message that will trigger a specific memory as the content of the utterance, the specific processing unit 290 may suggest one or more messages selected based on the life log to the user who requested the message as information corresponding to the content (request) of the utterance.

[0080] For example, if the user 20 wearing the necklace-type terminal 14 tries to recall his / her memory and utters, "What did you say at the meeting I attended on XX date at around XX time?", the identification processing unit 290 inputs the message as a prompt into the data generation model 58 as an identification process. The identification processing unit 290 may refer to the life log in the database 24 and generate a message such as "I think I said something about negotiating with Company A" based on the output obtained by the data generation model 58. The message may be interpreted as an example of information corresponding to the content of the user 20's utterance.

[0081] For example, if the user 20 wearing the necklace-type terminal 14 tries to recall his or her memory and utters, "I wonder who I was talking to at around XX time on XX month," the identification processing unit 290 inputs the message as a prompt into the data generation model 58 as an identification process. The identification processing unit 290 may refer to the life log in the database 24 and, based on the output obtained by the data generation model 58, generate a message such as, "It seems that at that time, two friends, probably Mr. B and Mr. C, were talking." This message may be interpreted as an example of information corresponding to the content of the user 20's utterance.

[0082] For example, if a user 20 wearing the necklace-type terminal 14 tries to recall his or her own feelings and utters, "How did I feel when I was talking to A-san around XX time on XX month?", the identification processing unit 290 inputs the message as a prompt into the data generation model 58 as an identification process. The identification processing unit 290 may refer to the life log in the database 24 and, based on the output obtained by the data generation model 58, generate a message such as, "You were laughing a lot at that time, so you seemed to like your friend and be very happy." This message may be interpreted as an example of information corresponding to the content of the user 20's utterance.

[0083] (Second example of information corresponding to the content of the utterance) When a user 20 wearing the necklace-type terminal 14 tweets a specific matter as the content of the speech, the specific processing unit 290 may suggest to the user 20 who requested the message, as information corresponding to the content of the speech (tweet), the behavior of the user 20 that is recommended for the matter based on the life log.

[0084] For example, if the user 20 wearing the necklace-type terminal 14 utters, "What should I buy?" while shopping at a particular retail store, the identification processing unit 290, as an identification process, inputs the message as a prompt into the data generation model 58. The identification processing unit 290 may generate a message such as, "A few months ago, after purchasing product A at this store, you commented that it wasn't very tasty, so how about purchasing product B or product C, which were recently released, this time," based on the output obtained by the data generation model 58 by referring to the life log in the database 24. The message may be interpreted as an example of information corresponding to the content of the user 20's utterance.

[0085] (Third example of information corresponding to the content of the utterance) For example, if the user 20 wearing the necklace-type terminal 14 utters, "I wonder what the name of product A I searched for the day before yesterday" while operating a personal computer, the identification processing unit 290 inputs the message as a prompt to the data generation model 58 as an identification process. The data generation model 58 generates a specific output by referencing the life log in the database 24 and analyzing images of the personal computer screen when the user 20 was operating the computer in the past. The identification processing unit 290 may generate a message such as "Product A is XXX" based on the output obtained by the data generation model 58. The message may be interpreted as an example of information corresponding to the content of the user 20's utterance.

[0086] (Fourth example of information corresponding to the content of the utterance) For example, if a user 20 wearing the necklace-type device 14 utters, while cycling, "There's a place nearby with a spectacular view, but I wonder where it is," the identification processing unit 290 inputs the message as a prompt to the data generation model 58 as an identification process. The data generation model 58 generates a specific output by referencing the life log in the database 24 and analyzing places the user 20 has previously visited and the route to those places. Based on the output obtained by the data generation model 58, the identification processing unit 290 may generate a message such as, "I think Cape XX is 500 meters from here." This message may be interpreted as an example of information corresponding to the content of the user 20's utterance.

[0087] (Fifth example of information corresponding to the content of an utterance) For example, if the user 20 wearing the necklace-type device 14 utters "Tell me his name" when meeting Mr. X from Company A while visiting, the identification processing unit 290 inputs the message as a prompt into the data generation model 58 as an identification process. The data generation model 58 references the life log in the database 24 and generates a specific output based on the history of people the user 20 met while visiting Company A. The identification processing unit 290 may generate a message such as "I think his name is XX" based on the output obtained by the data generation model 58. The message may be interpreted as an example of information corresponding to the content of the user 20's utterance.

[0088] [Third embodiment]

[0089] In the third embodiment, the necklace-type terminal 14 or the data processing device 12 sets an importance index for the user 20 by analyzing output data of a person other than the user 20 collected by the camera 42, the sensor 39, the microphone 38, etc. Specifically, the person other than the patient user 20 may be a medical professional, such as a doctor, who is involved with the patient. In the following, a doctor is used as an example of a medical professional. When the necklace-type terminal 14 or the data processing device 12 determines that the patient user 20 is being examined by a doctor, the necklace-type terminal 14 or the data processing device 12 may set the importance index to "3" if the doctor's comments, the physical movements of the doctor and the patient, and the patient's biological information satisfy predetermined conditions. That is, for example, if the doctor makes a statement such as "explaining the patient's symptoms" or "listening to the patient's symptoms," the necklace-type terminal 14 or the data processing device 12 may set the importance index to "3." For example, if the patient blinks frequently or moves their body frequently, the necklace-type terminal 14 or the data processing device 12 may set the importance index to "3."

[0090] "Explaining about the patient's symptoms" includes, for example, explaining information about the results of the patient's tests, information about the causes, symptoms, and prognosis of the patient's illness, future treatment plans, information about medications that have been or will be prescribed to the patient, and points that the patient should be aware of in their daily lives.

[0091] "Asking about the patient's symptoms" includes, for example, asking about symptoms present at the time of the examination, information about the patient's daily physical and mental condition, information about the patient's medical history, information about medications the patient takes on a daily basis, information about the patient's appetite, information about the patient's drinking or smoking, information about the type and frequency of the patient's usual exercise, and information about the patient's sleep.

[0092] "When the patient blinks a lot" refers to, for example, when the number of blinks within a predetermined time period is equal to or exceeds a predetermined threshold number. When the patient blinks a lot, it is considered that there is a high possibility that the patient is nervous. In other words, when the patient blinks a lot, there is a high possibility that the patient is making important statements.

[0093] "When the patient is moving a lot" includes, for example, when the patient uses gestures and hand movements while speaking. In such cases, the patient is likely desperately trying to communicate his or her thoughts to the doctor.

[0094] When the biological information satisfies a predetermined condition, it means, for example, a state in which electrocardiogram data, pulse rate, body temperature, oxygen concentration, etc. are outside the range of a predetermined threshold.

[0095] On the other hand, if the doctor determines that he is exchanging seasonal greetings or making small talk that is unrelated to the state of "explaining the patient's symptoms" or "listening to the patient's symptoms," the necklace-type terminal 14 or the data processing device 12 may set the importance index to "1."

[0096] "Seasonal greetings and small talk" includes, for example, talk about today's or recent weather, talk about social topics that are being covered prominently on television, talk about the hobbies of the doctor and patient, and talk about the transportation the patient used to arrive at the hospital.

[0097] As described above, the importance index for the states in which the doctor is "explaining the patient's symptoms" and "listening to the patient's symptoms" is set to "3." In these cases, the save state "3" is set to the importance index "3," and the output data for the importance index "3" is saved in a format with a large capacity. For example, the image data and audio data included in the output data in these cases are saved as video data.

[0098] On the other hand, when the doctor is making seasonal greetings or making small talk, the importance index is set to "1." In this case, the save state is set to "1" for the importance index "1," and the output data for the importance index "1" is saved in a format with a small capacity. For example, the image data included in the output data in these cases may be saved as photograph (still image) data, or only the audio data may be saved without saving the image data, or it may be saved as text data.

[0099] In this way, by setting an importance index and changing the storage format of output data based on the doctor's remarks and body movements, memory can be used efficiently.

[0100] The processing unit 294 of the third embodiment may also, when receiving an utterance from the user 20 relating to the memories or behavior of the user 20 as user data, perform a process of suggesting information corresponding to the content of the utterance to the user 20, for example, by referring to the database 24 in which the life log is recorded.

[0101] (Example of information corresponding to the content of the utterance) For example, if a user 20 wearing the necklace-type terminal 14 tries to recall his or her memory and utters, "I wonder what kind of conversation I had with Dr. X, who examined me at XX Clinic on XX date at around XX time," the identification processing unit 290 inputs the message as a prompt into the data generation model 58 as an identification process. The identification processing unit 290 may refer to the life log in the database 24 and, based on the output obtained from the data generation model 58, generate a message such as, "At that time, it seems that you and Dr. X discussed your symptoms, physical and mental condition, medical history, appetite, drinking, smoking, sleep, test results, causes of illness, symptoms, prognosis, future treatment plans, information about medications, and points to note in daily life, along the lines of XX..." This message may be interpreted as an example of information corresponding to the content of the user 20's utterance.

[0102] According to the present disclosure, by combining the necklace-type terminal 14, the emotion engine (data generation model 58), and an electrocardiogram using a probe (such as a sensor 39), it is possible to grasp the excitement state of the user 20, trigger video recording or audio capture, and compress and store the life log content to an appropriate size (such as 1 / 100,000 of the collected data) based on the level of excitement or the content of an important meeting, for example, or record it in an appropriate storage format, and individually learn from it to infer it to be useful for providing advice.

[0103] According to the present disclosure, it is possible to reduce battery consumption in the necklace-type device 14. It is also possible to prevent the memory capacity of the necklace-type device 14 from being depleted. Furthermore, since most of the data collected in a day may be useless, for example, it is possible to compress and store such data. For example, phrases such as "good morning" and "good night" or views from roads that are regularly used may all be compressed. Furthermore, the human brain also forgets or compresses huge amounts of data from one minute, one hour, one day, one week, one month, one year, ten years ago, etc. According to the present disclosure, it is possible to record a life log by determining a compression coefficient based on such concepts of time and an index of importance.

[0104] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[0105] In the above embodiment, an example was given in which a specific process is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may be performed by multiple computers including computer 22.

[0106] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0107] In addition, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 53, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0108] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 53, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0109] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[0110] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[0111] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0112] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[0113] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0114] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0115] 10 Data Processing System 12 Data Processing Device 14 Necklace-type device 38 Microphone 39 Sensors 40 speakers 42 Camera 46A Control Unit 100 Data Collection Department 102 Communications Department 290 Special Processing Department 292 Input section 294 Processing Section 296 Output Section< / url:>

Claims

1. A camera that captures the wearer's surroundings; a sensor for detecting biological data of the wearer; A microphone and a control unit that sets a storage mode in a specific memory for each of the output data collected over a certain period of time according to a value of a specific index calculated based on the output data of each of the camera, the sensor, and the microphone, and executes control to record each of the output data according to the set storage mode in the memory as a life log of the wearer; Including, The control unit analyzes at least one of the wearer's emotions, the content of the wearer's voice, the wearer's biometric information, and the content of the medical professional's remarks collected by the microphone based on each of the output data, and calculates the value of the index related to the content of the voice related to the wearer's symptoms based on the analyzed information.

2. 2. The terminal device according to claim 1, wherein the storage mode includes at least part of a compression rate of each of the output data, a storage format of the output data, deletion of all or part of each of the output data, a storage period of each of the output data, and a storage destination of each of the output data.

3. The terminal device according to claim 1 , wherein the control unit associates each of the output data with the index and records the associated data in the memory.

4. The terminal device according to claim 1 , which is a necklace-type terminal worn around the neck of the wearer.

5. A data processing system including the terminal device according to claim 1 and a data processing device, The data processing device includes: an input unit that receives the wearer's speech picked up by the microphone; a processing unit that inputs a prompt including the utterance into a data generation model and obtains a response to the utterance using an output of the data generation model; an output unit that reproduces the acquired response from a speaker of the terminal device; 2. The data processing system of claim 1, comprising:

Images