Data processing system

Abstract

To provide a data processing system for properly recording data according to the intensity of feeling.SOLUTION: The necklace-type terminal 14 includes at least a camera 42 for photographing the periphery of the wearer, a sensor 39 for detecting biological data of the wearer, and a microphone 38. The data processing device 12 functions as a processing unit. The processing unit estimates the level of the wearer's emotion based on the output data from each of the camera 42, the sensor 39, and the microphone 38, generates data in a first format from the output data collected during a period in which the value indicating the estimated level of the emotion is less than a predetermined value, and stores the generated data in the first format in a specific memory.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The techniques of this disclosure relate to data processing systems. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional techniques have the problem that data may not be recorded appropriately depending on the intensity of emotions. [Means for solving the problem]

[0005] A data processing system according to a first aspect includes a terminal device including at least a camera that photographs the wearer's surroundings, a sensor that detects the wearer's biometric data, and a microphone; and a processing unit that estimates the strength of the wearer's emotion based on at least a portion of the output data of the camera, the sensor, and the microphone, generates data in a first format from each of the output data collected during a period in which a value indicating the estimated strength of the emotion is less than a predetermined value and stores the data in a specific memory, and generates data in a second format having more information than the first format from each of the output data collected during a period in which a value indicating the estimated strength of the emotion is equal to or greater than a predetermined value and stores the data in the specific memory.

[0006] In a second aspect, in the first aspect, the processing unit estimates a negative emotion of the wearer as the emotion of the wearer.

[0007] A third aspect is the second aspect, wherein the negative emotion is at least one of "anger," "sadness," and "anxiety."

[0008] In a fourth aspect, in the first aspect, the processing unit estimates the emotion using a neural network based on at least a part of each of the output data.

[0009] In a fifth aspect, in the fourth aspect, the plurality of artificial neurons constituting the neural network include emotional artificial neurons that are artificial neurons in which a current emotion is defined, and the processing unit estimates the intensity of the emotion based on an internal state of the emotional artificial neurons.

[0010] A sixth aspect is the first aspect, wherein the terminal device is a necklace-type terminal worn around the neck of the wearer.

[0011] The seventh aspect is the first aspect, further including an input unit that receives the wearer's speech picked up by the microphone, an acquisition unit that inputs a prompt including the speech into a data generation model and acquires a response to the speech using the output of the data generation model, and an output unit that plays the acquired response from a speaker of the terminal device. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a conceptual diagram illustrating an example of a configuration of a data processing system. [Figure 2] FIG. 2 is a conceptual diagram showing an example of main functions of a data processing device and a necklace-type terminal. [Figure 3] FIG. 2 is a side view showing the configuration of the necklace-type terminal. [Figure 4] FIG. 2 is a top view showing the configuration of the necklace-type terminal. [Figure 5] 10 is a schematic diagram showing the functional configuration of a control unit of the necklace-type terminal. [Figure 6] 2 shows a schematic functional configuration of a specific processing unit of the data processing device. [Figure 7] 10 is a diagram illustrating an example of an operational flow of specific processing by a data processing device. [Figure 8A] FIG. 10 is a diagram illustrating an example of an index used in the processing of the second embodiment. [Figure 8B] FIG. 10 is a diagram illustrating an example of an index used in the processing of the second embodiment. [Figure 8C] FIG. 10 is a diagram illustrating an example of an index used in the processing of the second embodiment. [Figure 9] FIG. 10 is a diagram schematically illustrating a block configuration of a data processing device according to a fourth embodiment. [Figure 10] FIG. 1 is a diagram illustrating a neural network. [Figure 11] FIG. 1 is a diagram showing a schematic diagram of neural network parameters in table form. [Figure 12]FIG. 10 is a diagram illustrating an outline of an operation flow when the data processing device is started or reset. [Figure 13] FIG. 10 is a diagram for explaining the calculation of the coupling coefficient of an artificial synapse. [Figure 14] FIG. 10 is a diagram schematically illustrating the time evolution of a coupling coefficient when a function ht ij is defined as a parameter for increasing or decreasing the coupling coefficient. [Figure 15] FIG. 10 is a diagram schematically showing the time evolution of the coupling coefficient when there is further simultaneous firing at time t2. [Figure 16] FIG. 2 is a diagram showing an outline of influence definition information that defines a chemical influence given to a parameter. [Figure 17] 1 shows a flowchart for calculating internal state and status. [Figure 18] FIG. 10 is a diagram for explaining an example of calculation of the internal state when the artificial neuron does not fire. [Figure 19] FIG. 10 is a diagram for explaining an example of calculation of an output when an artificial neuron fires. [Figure 20] FIG. 10 is a diagram illustrating the time evolution of a coupling coefficient when a function is defined as an increase / decrease parameter of an artificial neuron. [Figure 21] FIG. 10 is a diagram showing an example of rules stored in a switching rule in a table format. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, an example of an embodiment of a data processing device, a data processing method, and a program according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), or an APU (Accelerated Processing Unit).

[0016] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0017] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0018] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0020] [First embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the embodiment.

[0021] As shown in FIG. 1, a data processing system 10 includes a data processing device 12 and a necklace-type terminal 14. An example of the data processing device 12 is a server. In this embodiment, the data processing device 12 is an example of a "data processing device" according to the technology of the present disclosure, and the necklace-type terminal 14 is an example of a "terminal device" according to the technology of the present disclosure. Note that the terminal device of the present disclosure is not limited to the necklace-type terminal 14. The terminal device of the present disclosure may include a robot, a doll, a stuffed animal, a wearable device (pendant, smart watch, smart glasses), a smartphone, a smart speaker, earphones, a personal computer, and the like.

[0022] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 53. Examples of the network 53 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0023] The necklace type terminal 14 includes a computer 36, a microphone 38, a sensor 39, a speaker 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 38, the speaker 40, and the camera 42 are also connected to the bus 52.

[0024] The user 20 who wears the necklace-type terminal 14 may be, for example, a patient whose health condition is to be diagnosed, or may be a normal user.

[0025] The microphone 38 picks up the voice uttered by the user 20 who is wearing the necklace-type terminal 14, as well as sounds around the user 20. The microphone 38 also receives instructions and the like from the user 20 by receiving the voice uttered by the user 20. The microphone 38 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 40 outputs audio in accordance with instructions from the processor 46. The speaker 40 is, for example, a directional speaker, and outputs audio toward the ears of the user 20.

[0026] The sensor 39 is a sensor that detects biological data of the user 20 who is wearing the necklace-type terminal. For example, the sensor 39 is a heart rate sensor or a blood oxygen sensor.

[0027] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0028] The communication I / F 44 is connected to a network 53. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 53.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the necklace-type terminal 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58. The data generation model 58 is used by the specific processing unit 290. The storage 32 also includes a data accumulation unit .

[0032] In the necklace-type terminal 14, the data collection process is performed by the processor 46. A data collection program 60 is stored in the storage 50. The processor 46 reads the data collection program 60 from the storage 50 and executes the read data collection program 60 on the RAM 48. The data collection process is realized by the processor 46 operating as the control unit 46A in accordance with the data collection program 60 executed on the RAM 48.

[0033] As shown in FIGS. 3 and 4, the necklace type terminal 14 includes multiple microphones 38, multiple sensors 39, multiple speakers 40, and multiple cameras 42. FIGS. 3 and 4 show an example in which two microphones 38 are arranged so as to be located in front of the user 20 when the user 20 wears the necklace type terminal 14. FIGS. 3 and 4 also show an example in which two sensors 39 are arranged so as to be located on the right and left sides of the user 20 when the user 20 wears the necklace type terminal 14. FIGS. 3 and 4 also show an example in which two speakers 40 are arranged so as to be located on the right and left rear sides of the user 20 when the user 20 wears the necklace type terminal 14. FIGS. 3 and 4 also show an example in which two cameras 42 are arranged so as to be located on the right and left front sides of the user 20 when the user 20 wears the necklace type terminal 14. FIGS. 3 and 4 also show an example in which two sensors 39 are arranged inside the necklace type terminal 14 so as to come into contact with the neck of the user 20 when the user 20 wears the necklace type terminal 14.

[0034] Next, the processing of the control unit 46A when the necklace-type terminal 14 performs a data collection process for collecting data will be described.

[0035] In the data collection process of this embodiment, biometric data of the user is collected in real time. Furthermore, not only biometric data but also all situational data surrounding the user is collected. This makes it possible to detect early signs of, for example, Alzheimer's disease and dementia. It also makes it possible to monitor the user's health condition (for example, heart disease).

[0036] As shown in FIG. 5, the control unit 46A includes a data collection unit 100 and a communication unit 102.

[0037] The data collection unit 100 collects the output of each of the microphone 38, the sensor 39, and the camera 42.

[0038] The communication unit 102 transmits the outputs of the microphone 38 , the sensor 39 , and the camera 42 collected by the data collection unit 100 to the data processing device 12 .

[0039] Next, a description will be given of the processing of the specific processing unit 290 when the data processing device 12 performs the specific processing to acquire a response corresponding to a user utterance.

[0040] In the identification process of this embodiment, a response corresponding to a user utterance picked up by the microphone 38 of the necklace-type terminal 14 is acquired using the data generation model 58.

[0041] As shown in FIG. 6, the specific processing unit 290 includes an input unit 292, a processing unit 294, and an output unit 296.

[0042] The input unit 292 stores the outputs of the microphone 38 , the sensor 39 , and the camera 42 received from the necklace-type terminal 14 in the data storage unit 54 .

[0043] The input unit 292 acquires the user's utterance received by the necklace type terminal 14. Specifically, the input unit 292 acquires the user's utterance picked up by the microphone 38 of the necklace type terminal 14.

[0044] The processing unit 294 performs a specific process using the data generation model 58. Specifically, a prompt including a user utterance is input to the data generation model 58 to obtain a generation result. At this time, the prompt may further include outputs from the sensor 39 and the camera 42 collected by the data collection unit 100.

[0045] The output unit 296 transmits the result of the identification process to the necklace type terminal 14. In the necklace type terminal 14, the control unit 46A causes the speaker 40 to output the result of the identification process. In this way, a response corresponding to the user utterance picked up by the microphone 38 is output to the user 20 by the speaker 40. The microphone 38 further acquires the user utterance in response to the result of the identification process. The control unit 46A transmits voice data indicating the user utterance acquired by the microphone 38 to the data processing device 12. In the data processing device 12, the identification processing unit 290 acquires the user utterance.

[0046] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generative AI models. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0047] The outputs of the microphone 38, the sensor 39, and the camera 42 stored in the data storage unit 54 are used, for example, to diagnose the health condition of the user 20. In this case, the outputs of the microphone 38, the sensor 39, and the camera 42 stored in the data storage unit 54 may be transmitted to a terminal on the medical institution side. Alternatively, the data processing device 12 may analyze the outputs of the microphone 38, the sensor 39, and the camera 42 stored in the data storage unit 54 to diagnose the health condition of the user 20.

[0048] Next, the operation of the data processing system 10 will be described.

[0049] First, an example of the flow of the data collection process will be described.

[0050] When the user 20 is wearing the necklace-type terminal 14, the data collection unit 100 sequentially collects the outputs of the microphone 38, the sensor 39, and the camera 42. The communication unit 102 sequentially transmits the outputs of the microphone 38, the sensor 39, and the camera 42 collected by the data collection unit 100 to the data processing device 12.

[0051] Next, an example of the flow of the identification process will be described with reference to Fig. 7. Here, it is assumed that the input unit 292 of the data processing device 12 sequentially acquires the outputs of the microphone 38, the sensor 39, and the camera 42 received from the necklace-type terminal 14 and stores them in the data accumulation unit 54.

[0052] In step S300, the processing unit 294 determines whether a predetermined trigger condition is satisfied. Specifically, the trigger condition may be that a specific word (for example, the name of an agent installed in the necklace type terminal 14) or phrase (for example, "Hi! XXX" (XXX is the name of the agent)) is included in the user utterance picked up by the microphone 38.

[0053] If the trigger condition is met in step S300 (step S300; Yes), the data processing system 10 proceeds to step S301. On the other hand, if the trigger condition is not met in step S300 (step S300; No), the data processing system 10 ends the identification process.

[0054] In step S301, the processing unit 294 generates a prompt by adding an instruction sentence for obtaining a result of a specific process to text representing a user utterance picked up by the microphone 38.

[0055] For example, a prompt such as "The user is saying the following: XXX. Please respond as an agent." (XXX is the user's utterance) may be generated. Alternatively, the outputs of the sensor 39 and the camera 42 may be added to the prompt to generate a prompt such as "This is biometric data representing the user's heart rate and video data representing the user's surroundings. The user is also saying the following: XXX. Please respond as an agent." (XXX is the user's utterance).

[0056] In step S303, the processing unit 294 inputs the generated prompt to the data generation model 58, and obtains the result of the specific process based on the output of the data generation model 58.

[0057] In step S304, the output unit 296 outputs the result of the identification process to the necklace-type terminal 14, and the identification process ends.

[0058] [Second embodiment]

[0059] The control unit 46A of the necklace type terminal 14 may set a storage mode in a specific memory (such as the storage 50 or the database 24) for each piece of output data collected by the collection unit (data collection unit 100) over a certain period of time, according to a specific index value calculated based on the output data of the camera 42, the sensor 39, and the microphone 38. The control unit 46A may also execute control to record each piece of output data according to the set storage mode in a memory (such as the storage 50 or the database 24) as a life log of the wearer.

[0060] The control unit 46A of the necklace-type terminal 14 may analyze at least one of the wearer's emotions, the content of the wearer's voice, and the wearer's biometric information based on each output data, and calculate an index value based on the analyzed information.

[0061] The storage manner may include at least some of the compression rate of each of the output data, deletion of all or part of each of the output data, storage period of each of the output data, and storage destination of each of the output data.

[0062] The output data may include images, sounds, biometric information, etc. collected by the necklace-type device 14. The images may include either still images or moving images. The biometric information may include electrocardiogram data, pulse rate, body temperature, oxygen concentration, etc. The certain period of time may be interpreted as, for example, 1 second, 1 minute, 10 minutes, 1 hour, 2 hours, etc.

[0063] The index may include, for example, an importance index as shown in FIG. 8A, an excitement index as shown in FIG. 8B, an emotion index as shown in FIG. 8C, and the like.

[0064] The importance index may be interpreted as an index representing the degree of importance in the activity of the user 20 who is wearing the necklace-type device 14. The importance index may include values ​​such as "1," "2," and "3." A higher importance index may be interpreted as a higher importance.

[0065] For example, when the necklace type terminal 14 or the data processing device 12 determines that the user 20 in a meeting is participating in an important meeting such as a board meeting by analyzing output data collected by the camera 42, the sensor 39, the microphone 38, etc., the necklace type terminal 14 or the data processing device 12 may set the importance index to "3." When the necklace type terminal 14 or the data processing device 12 determines that the user 20 is participating in a regular group meeting, the necklace type terminal 14 or the data processing device 12 may set the importance index to "2." When the necklace type terminal 14 or the data processing device 12 determines that the user 20 is having a casual conversation at work (everyday conversation), the necklace type terminal 14 or the data processing device 12 may set the importance index to "1."

[0066] The excitement index may be interpreted as an index representing the degree of excitement of the user 20 who is wearing the necklace-type terminal 14. The excitement index may include values ​​such as "1," "2," and "3." A higher excitement index may be interpreted as a higher level of excitement.

[0067] For example, if the necklace-type terminal 14 or the data processing device 12 determines, by analyzing the output data collected by the camera 42, the sensor 39, the microphone 38, etc., that the excitement level of the user 20 at a concert venue or the like is very high, the excitement index may be set to "3." If the necklace-type terminal 14 or the data processing device 12 determines, by analyzing the output data collected by the camera 42, the sensor 39, the microphone 38, etc., that the excitement level of the user 20 riding a bus, train, etc. is relatively low, the excitement index may be set to "2." If the necklace-type terminal 14 or the data processing device 12 determines, by analyzing the output data collected by the camera 42, the sensor 39, the microphone 38, etc., that the excitement level of the user 20 meditating or walking is very low, the excitement index may be set to "1."

[0068] The emotion index may be interpreted as an index representing, for example, the degree of emotion (intensity of emotion) of the user 20 who is wearing the necklace-type terminal 14. The emotion index may include values ​​such as "1," "2," and "3." A higher emotion index may be interpreted as a higher degree of emotion.

[0069] For example, when the necklace type terminal 14 or the data processing device 12 determines, by analyzing the output data collected by the camera 42, the sensor 39, the microphone 38, etc., that the user 20 is in an excited mood while eating their favorite food, the necklace type terminal 14 or the data processing device 12 may set the emotion index to "3." When the necklace type terminal 14 or the data processing device 12 determines, by analyzing the output data collected by the camera 42, the sensor 39, the microphone 38, etc., that the user 20 is in a neutral mood while reading, the necklace type terminal 14 or the data processing device 12 may set the emotion index to "2." When the necklace type terminal 14 or the data processing device 12 determines, by analyzing the output data collected by the camera 42, the sensor 39, the microphone 38, etc., that the user 20 who is not good at exercise is in a depressed mood while exercising, the necklace type terminal 14 or the data processing device 12 may set the emotion index to "1."

[0070] These indexes may be associated with data storage modes. The storage modes may include storing the output data of the camera 42, the sensor 39, and the microphone 38 at a high compression rate, storing the output data at a low compression rate, deleting specific data from the output data as unnecessary data, and the like. The storage modes may include a storage period (e.g., several months, one year, several years, ten years, etc.) for all or part of the output data of the camera 42, the sensor 39, and the microphone 38. The storage modes may include deleting all or part of the output data of the camera 42, the sensor 39, and the microphone 38 collected over a certain period of time. The storage modes may include storing the output data of the camera 42, the sensor 39, and the microphone 38 collected over a certain period of time in a location other than a specific memory, instead of in the specific memory. Specifically, the storage modes may include storing the output data in a specific server instead of in the storage 50 of the necklace-type terminal 14. More specifically, the saving mode may include changing the saving destination of the output data to an important cloud (such as a cloud for which a separate contract has been made) that is different from the usual saving destination.

[0071] If the user 20 is attending an important meeting and it is desirable to store the output data for a long period of time, the necklace-type terminal 14 or the data processing device 12 may set the storage mode "3" for the excitement index "3" to increase the compression rate of each output data, thereby enabling effective use of memory capacity.

[0072] If the user 20 who is not good at exercise is feeling depressed, the necklace-type terminal 14 or the data processing device 12 may regard the output data collected at this time as unnecessary data with low usefulness and set the excitement index to "1" and the storage mode to "1". This can save memory capacity.

[0073] The life log may be interpreted as a history of actions taken by the user 20 in daily life, and may include sounds and images associated with the user 20, specifically, sounds collected by the microphone 38 in daily life and images taken by the camera 42. The life log may record sounds and images associated with the user 20 in association with the date, time, and location at which they were acquired.

[0074] The sounds collected by the microphone 38 may include the voice of the person with whom the user 20 is talking, sounds occurring around the user 20 while walking or cycling (voices in a meeting, cars driving by, birds chirping, the sound of a river flowing, trees rustling in the wind), and the like.

[0075] For example, the camera 42 may capture an image of scenery within an angle of view that captures what is in front of the user 20, or may capture an image of scenery within an angle of view that captures what is not in front of the user 20, such as what is to the side, behind, below, or above the user 20. The image captured by the camera 42 may include the image of someone the user 20 is talking to, the image of the scenery around the user 20 when taking a walk or cycling, the image of a pet walking with the user 20, etc.

[0076] While the necklace type terminal 14 is worn by the user 20, all or part of the collected sounds and images may be recorded as a life log in the storage 50, the database 24, etc. Specifically, when the necklace type terminal 14 is worn by the user 20, recording of sounds and images may start, and when the necklace type terminal 14 is removed from the user 20, recording of sounds and images may end.

[0077] The control unit 46A of the necklace-type terminal 14 may associate each piece of output data with an index and record it in memory. For example, the output data collected when the user 20 is attending an important meeting may be associated with an excitement index of "3."

[0078] The specific processing unit 290 of the data processing device 12 may include an input unit 292 that receives the wearer's speech picked up by the microphone 38, a processing unit 294 that inputs a prompt including the speech into a data generation model and obtains a response to the speech using the output of the data generation model, and an output unit 296 that plays the obtained response from a speaker of the terminal device.

[0079] Specifically, when the processing unit 294 receives an utterance from the user 20 relating to the memory or behavior of the user 20 as user data, the processing unit 294 may execute a process of suggesting information corresponding to the content of the utterance to the user 20, for example, by referring to the database 24 in which the life log is recorded.

[0080] (First example of information corresponding to the content of the utterance) When a user wearing the necklace-type terminal 14 requests a message that will trigger a specific memory as the content of the utterance, the specific processing unit 290 may suggest one or more messages selected based on the life log to the user who requested the message as information corresponding to the content (request) of the utterance.

[0081] For example, if the user 20 wearing the necklace-type terminal 14 tries to recall his / her memory and utters, "What did you say at the meeting I attended on XX date at around XX time?", the identification processing unit 290 inputs the message as a prompt into the data generation model 58 as an identification process. The identification processing unit 290 may refer to the life log in the database 24 and generate a message such as "I think I said something about negotiating with Company A" based on the output obtained by the data generation model 58. The message may be interpreted as an example of information corresponding to the content of the user 20's utterance.

[0082] For example, if the user 20 wearing the necklace-type terminal 14 tries to recall his or her memory and utters, "I wonder who I was talking to at around XX time on XX month," the identification processing unit 290 inputs the message as a prompt into the data generation model 58 as an identification process. The identification processing unit 290 may refer to the life log in the database 24 and, based on the output obtained by the data generation model 58, generate a message such as, "It seems that at that time, two friends, probably Mr. B and Mr. C, were talking." This message may be interpreted as an example of information corresponding to the content of the user 20's utterance.

[0083] For example, if a user 20 wearing the necklace-type terminal 14 tries to recall his or her own feelings and utters, "How did I feel when I was talking to A-san around XX time on XX month?", the identification processing unit 290 inputs the message as a prompt into the data generation model 58 as an identification process. The identification processing unit 290 may refer to the life log in the database 24 and, based on the output obtained by the data generation model 58, generate a message such as, "You were laughing a lot at that time, so you seemed to like your friend and be very happy." This message may be interpreted as an example of information corresponding to the content of the user 20's utterance.

[0084] (Second example of information corresponding to the content of the utterance) When a user 20 wearing the necklace-type terminal 14 tweets a specific matter as the content of the speech, the specific processing unit 290 may suggest to the user 20 who requested the message, as information corresponding to the content of the speech (tweet), the behavior of the user 20 that is recommended for the matter based on the life log.

[0085] For example, if the user 20 wearing the necklace-type terminal 14 utters, "What should I buy?" while shopping at a particular retail store, the identification processing unit 290, as an identification process, inputs the message as a prompt into the data generation model 58. The identification processing unit 290 may generate a message such as, "A few months ago, after purchasing product A at this store, you commented that it wasn't very tasty, so how about purchasing product B or product C, which were recently released, this time," based on the output obtained by the data generation model 58 by referring to the life log in the database 24. The message may be interpreted as an example of information corresponding to the content of the user 20's utterance.

[0086] (Third example of information corresponding to the content of the utterance) For example, if the user 20 wearing the necklace-type terminal 14 utters, "I wonder what the name of product A I searched for the day before yesterday" while operating a personal computer, the identification processing unit 290 inputs the message as a prompt to the data generation model 58 as an identification process. The data generation model 58 generates a specific output by referencing the life log in the database 24 and analyzing images of the personal computer screen when the user 20 was operating the computer in the past. The identification processing unit 290 may generate a message such as "Product A is XXX" based on the output obtained by the data generation model 58. The message may be interpreted as an example of information corresponding to the content of the user 20's utterance.

[0087] (Fourth example of information corresponding to the content of the utterance) For example, if a user 20 wearing the necklace-type device 14 utters, while cycling, "There's a place nearby with a spectacular view, but I wonder where it is," the identification processing unit 290 inputs the message as a prompt to the data generation model 58 as an identification process. The data generation model 58 generates a specific output by referencing the life log in the database 24 and analyzing places the user 20 has previously visited and the route to those places. Based on the output obtained by the data generation model 58, the identification processing unit 290 may generate a message such as, "I think Cape XX is 500 meters from here." This message may be interpreted as an example of information corresponding to the content of the user 20's utterance.

[0088] (Fifth example of information corresponding to the content of an utterance) For example, if the user 20 wearing the necklace-type device 14 utters "Tell me his name" when meeting Mr. X from Company A while visiting, the identification processing unit 290 inputs the message as a prompt into the data generation model 58 as an identification process. The data generation model 58 references the life log in the database 24 and generates a specific output based on the history of people the user 20 met while visiting Company A. The identification processing unit 290 may generate a message such as "I think his name is XX" based on the output obtained by the data generation model 58. The message may be interpreted as an example of information corresponding to the content of the user 20's utterance.

[0089] According to the present disclosure, by combining the necklace-type terminal 14, the emotion engine (data generation model 58), and an electrocardiogram using a probe (such as a sensor 39), it is possible to grasp the excitement state of the user 20, trigger video recording and audio capture, and compress and store the life log content to an appropriate size (such as 1 / 100,000 of the collected data) based on the level of excitement or the content of an important meeting, for example, and then learn from this individually and use it to infer useful information for providing advice.

[0090] According to the present disclosure, it is possible to reduce battery consumption in the necklace-type device 14. It is also possible to prevent the memory capacity of the necklace-type device 14 from being depleted. Furthermore, since most of the data collected in a day may be useless, for example, it is possible to compress and store such data. For example, phrases such as "good morning" and "good night" or views from roads that are regularly used may all be compressed. Furthermore, the human brain also forgets or compresses huge amounts of data from one minute, one hour, one day, one week, one month, one year, ten years ago, etc. According to the present disclosure, it is possible to record a life log by determining a compression coefficient based on such concepts of time and an index of importance.

[0091] [Third embodiment] In the second embodiment described above, an aspect was described in which an emotion index representing the intensity of an emotion of the user 20 is set (estimated) by analyzing output data collected by the camera 42, the sensor 39, the microphone 38, etc. In the third embodiment, an aspect is described in which the emotion index is subdivided into a plurality of specific emotion indexes corresponding to any of a plurality of different emotions of the user 20, and the intensity of the plurality of emotions of the user 20 (values ​​of the plurality of specific emotion indexes) is estimated by analyzing the output data. The plurality of emotions of the user 20 estimated using the specific emotion index can include, for example, negative emotions of the user 20 (as an example, at least one of "anger," "sadness," and "anxiety").

[0092] As an example, when a user 20 wearing the necklace-type terminal 14 is conversing with a doctor as a patient and a first event is detected in which the user 20 is talking about a symptom that the user 20 is anxious about, or when a second event is detected in which the doctor is expressing the user 20's feelings of anxiety, the necklace-type terminal 14 or the data processing device 12 will estimate that the user 20's "anxiety" has become stronger by setting the value of the specific emotion index corresponding to "anxiety" among multiple emotions higher than when the first event or the second event is not detected.

[0093] As another example, when a third event is detected in which a pet owned by a user 20 wearing the necklace-type terminal 14 has died, the necklace-type terminal 14 or the data processing device 12 estimates that the "sadness" of the user 20 has become stronger by increasing the value of the specific emotion index corresponding to "sadness" among multiple emotions compared to when the third event is not detected.

[0094] In the third embodiment, when the values ​​of all of the multiple specific emotion indices are less than a predetermined value, the necklace-type terminal 14 or the data processing device 12 generates first-format data from output data collected during a period when the value of the specific emotion index was less than the predetermined value and stores the data in a specific memory. When the value of at least one of the multiple specific emotion indices is equal to or greater than a predetermined value, the necklace-type terminal 14 or the data processing device 12 generates second-format data, which contains more information than the first format, from output data collected during a period when the value of at least one of the multiple specific emotion indices was equal to or greater than the predetermined value and stores the second-format data in a specific memory. The first-format data is, for example, highly compressed data, such as text data obtained by speech recognition of audio data or the audio data itself. The second-format data is, for example, low-compression data, such as video data. This allows, for example, a life log of when the user 20 felt strong negative emotions to be clearly recorded, while a life log of when the user did not feel strong negative emotions can be roughly recorded.

[0095] [Fourth embodiment] In the second and third embodiments, an aspect has been described in which an emotion index is set (estimated) by analyzing output data collected by the camera 42, the sensor 39, the microphone 38, etc. In the fourth embodiment, an aspect will be described in which a neural network including an emotion artificial neuron, which is an artificial neuron that estimates the current emotion of the user 20, is used to estimate the emotion of the user 20 based on the internal state of the emotion artificial neuron.

[0096] 9, the specific processing unit 290 of the data processing device 12 according to the fourth embodiment includes an initial value setting unit 210, an external input data generating unit 230, a parameter processing unit 240, and a recording control unit 270. The parameter processing unit 240 and the recording control unit 270 are examples of processing units in the present disclosure. The storage 32 of the data processing device 12 stores definition information 284, parameter initial values ​​286, the latest parameters 288, a switching rule 291, and recording data 293.

[0097] The control unit 46A of the necklace type terminal 14 transmits output data collected by the camera 42, the sensor 39, the microphone 38, etc. to the data processing device 12 via the communication I / F 44. In the data processing device 12, the communication I / F 26 outputs the output data received from the necklace type terminal 14 to the specific processing unit 290.

[0098] The initial value setting unit 210 stores the initial values ​​of the parameters that indicate the initial state of the neural network in the parameter initial value 286 in the storage 32. The initial values ​​of the parameters of the neural network may be predetermined in the data processing device 12, or may be changeable by the user via the network 53.

[0099] The external input data generation unit 230 processes at least a portion of the output data received by the communication I / F 26 to generate input information from outside the neural network and outputs it to the parameter processing unit 240. The parameter processing unit 240 performs calculations on the neural network based on the input information and the current parameters 288 and definition information 284 of the neural network stored in the storage 32.

[0100] The artificial neurons of the neural network include a plurality of artificial neurons in which the situation of the user 20 is defined, a plurality of emotion artificial neurons in which a plurality of emotions of the user 20 are defined, and a plurality of endocrine artificial neurons in which the production state of the user's endocrine substances is defined. Note that endocrine substances refer to substances that are secreted in the body and transmit signals, such as neurotransmitters and hormones. Also, endocrine refers to the secretion of endocrine substances in the body.

[0101] The parameter processing unit 240 calculates parameters representing the internal states of multiple artificial neurons in the neural network based on the input information generated by the external input data generation unit 230. For example, the parameter processing unit 240 updates the current internal state parameters of multiple artificial neurons, etc., in which the user's situation is defined, based on the input information generated by the external input data generation unit 230. The parameter processing unit 240 also calculates the internal state parameters of other artificial neurons in the neural network. This calculates the internal state parameters of an emotional artificial neuron in which, for example, the emotion "happy" is defined. The internal state parameters of this emotional artificial neuron are an example of an index representing the intensity of the emotion "happy." Therefore, the parameter processing unit 240 can estimate the intensity of the user's emotion based on the internal states of the emotional artificial neuron. In this way, the parameter processing unit 240 functions as an emotion estimation unit that estimates the intensity of an emotion using a neural network based on at least a portion of the output data collected by the camera 42, the sensor 39, the microphone 38, etc.

[0102] The neural network parameters calculated by the parameter processing unit 240 are supplied to the recording control unit 270. The recording control unit 270 processes at least a portion of the output data received from the necklace type terminal 14 to generate record data in the first format or record data in a second format that contains more information than the first format, and records the generated record data in the storage 32 as record data 293. The recording control unit 270 also switches between generating record data in the first format and generating record data in the second format based on the parameters supplied from the parameter processing unit 240.

[0103] For example, when the intensity of the emotion estimated by the parameter processing unit 240 increases while record data in the first format is being generated, the recording control unit 270 switches the format of the generated record data from the first format to the second format, which contains more information. This allows the record data for the period when the emotion of the user 20 is increasing to be kept in detail as record data 293.

[0104] Furthermore, for example, when the intensity of the emotion estimated by the parameter processing unit 240 decreases while record data in the second format is being generated, the recording control unit 270 switches the format of the generated record data from the second format to the first format. This makes it possible to compress the volume of record data during periods when the intensity of the emotion of the user 20 decreases.

[0105] 10 schematically illustrates a neural network 310. The neural network 310 is an example neural network for explaining the operation of the parameter processing unit 240. The neural network 310 includes a plurality of artificial neurons, including artificial neuron 1, artificial neuron 2, artificial neuron 3, artificial neuron 4, artificial neuron 5, artificial neuron 6, artificial neuron 7, artificial neuron 8, artificial neuron 9, artificial neuron a, artificial neuron b, and artificial neuron c. Neural network 310 includes a plurality of artificial synapses, including artificial synapse 311, artificial synapse 312, artificial synapse 313, artificial synapse 314, artificial synapse 315, artificial synapse 316, artificial synapse 317, artificial synapse 318, artificial synapse 319, artificial synapse 320, artificial synapse 321, artificial synapse 322, artificial synapse 323, artificial synapse 324, artificial synapse 325, artificial synapse 326, artificial synapse 327, artificial synapse 328, and artificial synapse 329. The artificial neurons correspond to neurons in a living organism. The artificial synapses correspond to synapses in a living organism.

[0106] Artificial synapse 311 connects artificial neuron 4 and artificial neuron 1. Artificial synapse 311 is a unidirectional artificial synapse, as indicated by the arrow of artificial synapse 311. Artificial neuron 4 is an artificial neuron connected to the input of artificial neuron 1. Artificial synapse 312 connects artificial neuron 1 and artificial neuron 2. Artificial synapse 312 is a bidirectional artificial synapse, as indicated by the arrows at both ends of artificial synapse 312. Artificial neuron 1 is an artificial neuron connected to the input of artificial neuron 2. Artificial neuron 2 is an artificial neuron connected to the input of artificial neuron 1.

[0107] In this embodiment, an artificial neuron may be represented by N, and an artificial synapse may be represented by S. When identifying each artificial neuron, a superscripted reference symbol may be used as an identification letter. When identifying an arbitrary artificial neuron, i or j may be used as an identification letter. For example, N i represents an arbitrary artificial neuron.

[0108] An artificial synapse may also be identified by the identification numbers i and j of the two artificial neurons connected to it. For example, S 41 is N 1 and N 4 It represents an artificial synapse that connects ij is N i The output of N j represents an artificial synapse input to S ji is N j The output of N i represents an artificial synapse input to

[0109] In FIG. 10, A to J indicate that the state of the user 20 is defined. The state of the user 20 includes the emotions of the user 20, the state of production of endocrine substances, the situation of the user 20, etc. As an example, N 4 , N 6 , and N 7 is a conceptual artificial neuron in which a concept representing the situation of the user 20 is defined.

[0110] N 1 , N 3 , N b and N c is an emotional artificial neuron in which the emotion of the user 20 is defined. 1 is an emotional artificial neuron to which the emotion "happy" is assigned. 3 is an emotional artificial neuron to which the emotion "anger" is assigned. b is an emotional artificial neuron to which the emotion "sadness" is assigned. c is an emotional artificial neuron to which the emotion "anxiety" is assigned.

[0111] N 2 , N 5 and N a is an endocrine artificial neuron with the endocrine state of the user 20 defined. 5 is an endocrine artificial neuron that is assigned the developmental state of dopamine. Dopamine is an example of an endocrine substance involved in the reward system. 5 is an example of an endocrine artificial neuron involved in the reward system. N2 is an endocrine artificial neuron assigned the generation state of serotonin. Serotonin is an example of an endocrine substance involved in the sleep system. That is, N 2 is an example of an endocrine artificial neuron involved in the sleep system. a is an endocrine artificial neuron assigned the generation state of noradrenaline. Noradrenaline is an example of an endocrine substance involved in the sympathetic nervous system. a is an endocrine artificial neuron involved in the sympathetic nervous system.

[0112] The definition information 284 in the storage 32 stores information defining the state of the user 20 as described above for each of the multiple artificial neurons that make up the neural network. Thus, the neural network 310 includes conceptual artificial neurons, emotional artificial neurons, and endocrine artificial neurons. The conceptual artificial neurons, emotional artificial neurons, and endocrine artificial neurons are artificial neurons in which the meanings of concepts, emotions, endocrine functions, etc. are explicitly defined. In contrast, N 8 and N 9 is an artificial neuron with no user-defined state. 8 and N 9 is an artificial neuron whose meanings, such as concepts, emotions, and endocrine secretions, are not explicitly defined.

[0113] The parameters of the neural network 310 are i I is the input to t i and N from outside the neural network i E, which is the input to t i And, N i parameters and S i and the parameters of

[0114] N i The parameters of N i S indicates the status of t i And, N i represents the internal state of the artificial neuron, V i m t And, N i T represents the firing threshold of i t And, N i represents the last firing time, t f and the artificial neuron N at the final firing time i V represents the internal state of i m tf and the output increase / decrease parameter a t i , b t i , h t i The output increase / decrease parameter is an example of a parameter that determines the time evolution of the output when the artificial neuron fires. In this embodiment, the subscript t indicates that the parameter can be updated as time progresses. Also, V i m t is information corresponding to the membrane potential of the artificial neuron, and is an example of a parameter representing the internal state or output of the artificial neuron.

[0115] S ij The parameters of S ij BS represents the coupling coefficient of the artificial synapse t ij And, S ij N connected i and N j The time when the two last fired simultaneously is t cf and BS, which represents the coupling coefficient at the time of the last simultaneous firing. ij tcf and a, which is the coupling coefficient increase / decrease parameter t ij , b t ij , h t ij The coupling coefficient increase / decrease parameter is an example of a parameter that determines the time evolution of the coupling coefficient after the two artificial neurons connected by the artificial synapse last fired simultaneously.

[0116] The parameter processing unit 240 updates the above-mentioned parameters based on the input from the external input data generation unit 230 and the neural network to determine the activation state of each artificial neuron. The recording control unit 270 determines whether to generate record data in the first format or the second format based on the internal state or activation state of at least some of the artificial neurons in the neural network, which is determined by the parameter values ​​of at least some of the artificial neurons, and the state defined for at least some of the artificial neurons by the definition information 284. Note that the activation state can be an activated state or an inactivated state. In this embodiment, activation is sometimes referred to as "firing," and inactivation is sometimes referred to as "not firing." Note that, as will be described later, the "firing" state is divided into an "upward phase" and a "downward phase" depending on whether the internal state is rising or not. "Not firing," "upward phase," and "downward phase" are referred to as status S. t i is expressed by

[0117] Figure 11 shows the neural network parameters in a table format. Each neuron N is assigned a threshold T t and the increase / decrease parameter h t , a t and b t Each artificial synapse has a coupling coefficient BS t and the increase / decrease parameter h t , a t and b t and are included as parameters. i Every time, N i Each parameter of all the artificial neurons directly connected to the corresponding artificial synapse is shown in a single line.

[0118] 12 shows a schematic flow of the operation of the data processing device 12 when the data processing device 12 is started or reset. When the data processing device 12 is started or reset, the parameter processing unit 240 performs initial setting of the parameters of the neural network. For example, the parameter processing unit 240 obtains initial values ​​of the parameters from the storage 32 and generates parameter data of the neural network in a predetermined data structure (S502). The parameter processing unit 240 also sets the values ​​of the parameters of the neural network at time t0. When the initial setting is complete, a loop for time t is started in S504.

[0119] In S510, the parameter processing unit 240 calculates the time step t n+1 We calculate the parameters corresponding to the changes due to the electrical influence of the artificial synapse in S. ij BS t ij Calculate.

[0120] In S520, the parameter processing unit 240 calculates the time step t n+1 Calculate the parameters corresponding to the changes due to the chemical influence of endocrine substances in N i and S ij More specifically, we calculate the change in the parameters of n+1 In the artificial neuron N, the endocrine artificial neuron influences i The internal state of the S ij Calculate the coupling coefficient increase / decrease parameters and coupling coefficients.

[0121] In S530, the parameter processing unit 240 acquires an input from outside the neural network. Specifically, the parameter processing unit 240 acquires the output of the external input data generation unit 230.

[0122] In S540, the parameter processing unit 240 calculates the time step t n+1 In N i Calculate the internal state of V i m tn+1 and Status S tt i Then, in S550, at time t n+1 The values ​​of the parameters at time t are stored in the storage 32 as parameters 288. n+1 The values ​​of the parameters in the above are output to the recording control unit 270.

[0123] In S560, the recording control unit 270 performs the recording at time step t n+1 N in i It is determined whether the parameters satisfy the conditions for switching the format of the record data to be recorded as the record data 293. n+1 N in i If the parameters satisfy the conditions for switching the format of the recorded data, the recording control unit 270 switches the format of the recorded data (S570) and proceeds to S506. n+1 N in i If the parameters do not satisfy the conditions for switching the format of the recorded data, the process proceeds to S506.

[0124] In S506, the parameter processing unit 240 determines whether to end the loop. For example, it determines to end the loop when the time represented by the time step reaches a predetermined time or when output data from the necklace-type terminal 14 has not been received for a predetermined time. If the loop is not to be ended, the process returns to S510 and the next time step is calculated. If the loop is to be ended, this flow ends.

[0125] FIG. 13 is a diagram for explaining the calculation of the coupling coefficient of the artificial synapse. Here, the constant a is used as the initial value of the increase / decrease parameter. ij and b ij This section explains the case where is defined.

[0126] time t n At the time step S ij N on both ends of i and N j are both firing, the parameter processing unit 240 n+1 in B.S. tn+1 ij ,BS tn+1 ij =BS tn ij +a tn ij ×(t n+1 -t n ) is calculated. On the other hand, at time t n At the time step S i and S j If none of the above fires, then at time t n+1 Coupling coefficient BS tn+1 ij ,BS tn+1 ij =BS tn ij +b tn ij ×(t n+1 -t n ) is calculated. Also, BS tn+1 ij If is negative, BS tn+1 ij is set to 0. Note that BS ij is a positive value of S ij So, a t ij is positive and b t ij is a negative value. ij is a negative value of S ij So, a t ij is a positive value, and b t ij is a negative value.

[0127] As shown in Figure 13, at time t0, the artificial neurons on both ends fire simultaneously. t ij is a per unit time t0 ij Also, since there are no simultaneous firings at time t1, BS t ij is the number of |b t1 ij Also, due to simultaneous firing at time t4, BS t ij is a per unit time t4 ij increases with.

[0128] Figure 14 shows the function h t ij The time evolution of the coupling coefficient when is defined is shown below. t ij is t cf The elapsed time from Δt (= tt cf ) ≧ 0. t ij is a function of at least Δt and takes real values.

[0129] The function 700 shown in FIG. t ij The function 700 is an example of cf Coupling coefficient BS tcf ij and Δt. The function 700 monotonically increases when Δt is less than a predetermined value, and monotonically decreases and gradually tapers off to 0 when Δt is greater than the predetermined value. The function 700 has a value BS tcf ij Take.

[0130] FIG. 14 shows a case where a function 700 is defined as a parameter for increasing or decreasing the coupling coefficient, and N i and N j The parameter processing unit 240 calculates the coupling coefficients of the BSs at each time t1 to t6 based on the function 700 and Δt. t ij In the time range from time t1 to time t6, N i and N j do not fire simultaneously. Therefore, for example, after time t2, the coupling coefficient decreases monotonically.

[0131] Figure 15 shows the N i and N j The graph shows the time evolution of the coupling coefficient when N fires simultaneously. The coupling coefficient is calculated in the same way as in Figure 14 from time t0 to time t2. At time t2, N i and N j When these two fire simultaneously, the parameter processing unit 240 t ij (t-t2,BS t2 ij ) and calculate the coupling coefficient at each time from t3 to t6. In this way, the coupling coefficient increases each time simultaneous firing occurs. This has the effect of strengthening the artificial synaptic connection, similar to Hebb's law in living organisms. On the other hand, as shown in Figures 13 and 14, if the time without simultaneous firing becomes long, the effect of weakening the artificial synaptic connection is obtained.

[0132] 16 shows a schematic diagram of effect definition information that defines the chemical effect given to a parameter. This effect definition information is used to calculate the change in the parameter in S520 of FIG. 12. The definition information includes conditions related to the internal state of the endocrine artificial neuron, information specifying the influencing artificial neuron or artificial synapse, and an equation that defines the effect content.

[0133] In the example of Figure 16, the endocrine artificial neuron N 2 is an endocrine artificial neuron to which the endocrine substance responsible for sleepiness is assigned. 2 For definition information about tn 2 >T tn 2 "Conditions for Endocrine Artificial Neuron N 2 Emotional Artificial Neuron N 1 and N 3 ", and "T" is the formula that determines the impact content. tn+1 i =T tn i Parameter processing unit 240 determines Vm tn 2 T tn 2 If it exceeds t n+1 Emotional Artificial Neuron N 1 and N 3 The threshold is increased by 10%. This makes it harder for the emotional artificial neuron to fire when, for example, drowsiness occurs.

[0134] In addition, the endocrine artificial neuron N 5 is an endocrine artificial neuron to which dopamine is assigned. Endocrine artificial neuron N 5 The first definition information about "Vm tn 5 >T tn 5 and Vm tn 4 >T tn 4 "Conditions for Endocrine Artificial Neuron N 5 As an artificial synapse that is influenced by 49 and S 95 ", and "a" is the formula that determines the impact content. tn+1 ij =a tn ij The parameter processing unit 240 calculates Vm tn 5 T tn 5 Exceeds Vm tn 4 T tn 4 If it exceeds t n+1 Artificial Synapse S 49 and S 95 Increases the increase / decrease parameters by 10%.

[0135] In addition, the endocrine artificial neuron N 5 The second definition information about "Vm tn 5 >T tn 5 "Conditions for Endocrine Artificial Neuron N 5 "N" is an artificial neuron that is influenced by 1 ", and "T" is the formula that determines the impact content. tn+1 i =T tn i The parameter processing unit 240 calculates Vm tn 5 T tn 5 If it exceeds t n+1 Artificial neuron N 1 This reduces the increase / decrease parameter of the reward system endocrine artificial neuron N 5 When this emotion is ignited, it becomes easier for the emotion of happiness to be ignited.

[0136] The influence definition information is not limited to the example shown in FIG. 16. For example, a condition may be defined that the internal state of the artificial neuron is equal to or less than a threshold. A condition related to the status of the artificial neuron, such as an ascending phase, a descending phase, or non-firing, may also be defined. The influence range may be defined by directly specifying an artificial neuron or an artificial synapse, or by defining "all artificial synapses connected to a specific artificial neuron." Regarding the influence formula, when the target is an artificial neuron, in addition to multiplying the threshold by a constant, a formula may be defined that adds a constant to the threshold or multiplies an increase / decrease parameter of the internal state by a constant. When the target is an artificial synapse, in addition to multiplying an increase / decrease parameter by a constant, a formula may be defined that multiplies a coupling coefficient by a constant.

[0137] The influence definition information is stored in the definition information 284 of the storage 32. Thus, the storage 32 stores influence definition information that defines the influence of at least one of the internal state and firing state of an endocrine artificial neuron on the parameters of at least one of other artificial neurons and artificial synapses that are not directly connected to the endocrine artificial neuron via an artificial synapse. The parameter processing unit 240 then updates the parameters of at least one of other artificial neurons and artificial synapses that are not directly connected to the endocrine artificial neuron via an artificial synapse, based on at least one of the internal state and firing state of the endocrine artificial neuron and the influence definition information. Furthermore, the parameters of other artificial neurons that are influenced by at least one of the internal state and firing state of an endocrine artificial neuron can include at least one of the parameters that determine the threshold, firing state, and time evolution of the output at the time of firing of the other artificial neuron. Furthermore, the parameters of the artificial synapse that are influenced by at least one of the internal state and firing state of an endocrine artificial neuron can include at least one of the parameters that determine the coupling coefficient of the artificial synapse and the time evolution of the coupling coefficient after the two artificial neurons connected by the artificial synapse last simultaneously fired. The influence definition information also includes information that defines the influence that the firing state of the endocrine artificial neuron associated with the reward system has on the threshold of the emotional artificial neuron, and when the endocrine artificial neuron fires, the parameter processing unit 240 updates the threshold of the emotional artificial neuron in accordance with the influence definition information.

[0138] Figure 17 shows the V tn+1 i and S tn+1 i The process of this flowchart can be applied to part of the process in S540 in FIG. 12. In S1100, the parameter processing unit 240 calculates S tn i indicates non-firing.

[0139] S tn i If N indicates non-firing, the parameter processing unit 240 i Input to I tn+1 i Specifically, when an input from outside the neural network is N i If not connected to tn+1 i =Σ j BS tn+1 ji ×VM tn j ×f(S tn j ) is calculated by the external input of the neural network. i If connected to tn+1 i =Σ j BS tn+1 ji ×VM tn j ×f(S tn j )+E tn+1 i Here, E tn i is the time t from outside the neural network. n This is the input in

[0140] Furthermore, f(S) returns 0 if S is a value that indicates no firing, and returns 1 if S is a value that indicates an ascending or descending phase. This model corresponds to a model in which a synapse transmits an action potential only when the neuron fires. Note that f(S) = 1 may also be returned. This corresponds to a model in which a membrane potential is transmitted regardless of the firing state of the neuron.

[0141] In S1112, the parameter processing unit 240 tn+1 i T tn+1 i Determine whether it exceeds I tn+1 i T tn+1 i If it exceeds Vm tn+1 i is calculated based on the increase / decrease parameters, and Vm tn+1 i Depending on the S tn+1 i is set to a value indicating the rising phase or the falling phase (S1114), and this flow ends.

[0142] In the S1100, S tn i is in the rising phase or the falling phase, the parameter processing unit 240 tn+1 i Then, the parameter processing unit 240 calculates t n+1 By Vm t i When Vmin is reached, S tn+1 i Set to the unfired value and t n+1 By Vm t i If Vmin is not reached, S tn+1 i The parameter processing unit 240 sets t to the value of the rising phase or the falling phase, and then ends this flow. n+1 By Vm t i When Vmax is reached, S tn+1 i Set the value of the falling phase to t n+1 By Vm t i If Vmax is not reached, S tn+1 i Set the value of the rising phase to

[0143] In this way, N i If is firing, even if the output falls below the threshold, N i The output of is independent of the input. Such a period corresponds to the absolute refractory period in biological neurons.

[0144] Figure 18 shows the N i V when does not fire t i FIG. 10 is a diagram for explaining an example of calculation of

[0145] At time step t0, N i is not fired. I at time t1 t1 i T t1 i If V is equal to or less than V at time t1, the parameter processing unit 240 t1 i ,V t1 i =I t1 i V for the period from time t0 to t1 t i ,V t i =I t0 i Similarly, the parameter processing unit 240 calculates the time step t n V calculated using tn The value of is maintained until the next time step, and V tn+1 In tn+1 Change it to.

[0146] Figure 19 shows the N i V when ignition occurs t i 19 is a diagram for explaining an example of calculation of the constant a i and b i This is a calculation example when is defined.

[0147] At time step t0, N i is not fired. I at time t1 t1 i T t1 i If V exceeds V at time t1, the parameter processing unit 240 t1 i ,V t1 i =I t1 i V for the period from time t0 to t1 t i ,V t i =I t0 i Here, the calculation is performed using I at time t1. t1 i is less than Vmax. I at time t1 t1 i If exceeds Vmax, I t1 i =Vmax.

[0148] As shown in FIG. 19, after time t1, the parameter processing unit 240 t i Until V reaches Vmax, V t i per unit time a ti j The parameter processing unit 240 also increases the N i Status S t i is determined to be the ascending phase.

[0149] Also, V t i When V reaches Vmax, V t i Until V reaches Vmin, V t i per unit time |b t i In addition, the parameter processing unit 240 decreases N i The status of V is determined to be in the descending phase. t i When Vmin is reached, V at the next time t6 i ,V t6 i =I t6 i It is calculated by V t i After reaching Vmin, the status is determined as not fired.

[0150] In addition, N i If the status of is in a declining phase, the calculated Vm t i T t i Even if it falls below Vm t i I t i The parameter processing unit 240 does not depend on Vm t i T t i Even if it falls below Vm t i Vm according to the increase / decrease parameters until Vmin is reached t i Calculate.

[0151] Figure 20 shows the N i As an increasing or decreasing parameter of the function h t i The time evolution of the coupling coefficient when is defined is shown below. In general, h t i is the firing time t f The elapsed time from Δt (= tt f ) ≧ 0. t i is a function of at least Δt. t i takes real values, and h t i The value range is Vmin or more and Vmax or less.

[0152] The function 1300 shown in FIG. t i The function 1300 is an example of f Vm in tf i and Δt. The function 1300 monotonically increases when Δt is less than a predetermined value, and monotonically decreases when Δt is greater than a predetermined value. The function 1300 has a value Vm tf i Take.

[0153] FIG. 20 shows a function 1300 defined as an increase / decrease parameter of the internal state, and N i The parameter processing unit 240 calculates the output when the function 1300, Δt, and Vm f i Based on this, Vm at each time from time t1 to time t5 t i Calculate Vm t i reaches Vmin at time t5, so at time t6, Vm t i =I t6 i This becomes:

[0154] 21 shows an example of a rule 1400 stored in the switching rule 291 in table format. 1 , N 3 , N b and N c Any VM t i is greater than the threshold value, the operation of "switching the format of the recorded data to the low-compression second format" is defined. As a result, when the recorded data is recorded in the high-compression first format, the recording control unit 270 determines to switch the format of the recorded data to the low-compression second format when the state where the first condition is not satisfied changes to the state where the first condition is satisfied. Note that as the threshold value, j The example shows the value obtained by multiplying Vmax by a constant 0.9. i t Higher is better.

[0155] Also, rule 1400 states that N 5 and N a VM t i When at least the second condition that the sum of the values ​​exceeds the threshold is satisfied, the operation of "switching the data recording format to the low-compression second format" is defined. As a result, when the recording data is recorded in the high-compression first format, the recording control unit 270 determines to switch the recording data format to the low-compression second format when the second condition is satisfied after not being satisfied. Note that as the threshold, each N j The value obtained by multiplying the total value of Vmax of each N by a constant 0.9 is shown as an example. j T i t It may be higher than the total value of

[0156] N 1 , N 3 , N b and N c are emotion artificial neurons in which the emotions of "happiness," "anger," "sadness," and "anxiety" are defined, respectively. Therefore, the parameter processing unit 240 estimates the strength of each of the emotions of "happiness," "anger," "sadness," and "anxiety" of the user 20 based on the internal state of the emotion artificial neuron, and can switch the format of the recorded data to a low-compression second format when the strength of at least one of the estimated emotions of "happiness," "anger," "sadness," and "anxiety" exceeds a predetermined threshold.

[0157] N 5 and N a are endocrine artificial neurons in which the endocrine substances "dopamine" and "noradrenaline" are defined, respectively. The sum of the parameters of the internal states of these endocrine artificial neurons is an example of an index representing the strength of the emotion "excitement." Therefore, the parameter processing unit 240 estimates the strength of the emotion "excitement" of the user 20 based on the internal states of the endocrine artificial neurons, and can switch the format of the recorded data to the low-compression second format when the estimated strength of the emotion "excitement" exceeds a predetermined threshold.

[0158] Also, rule 1400 states that N 1 , N 3 , N b and N c Any VM t i is also less than or equal to the first threshold, and N 5 and N a VM t i The operation of "switching the format of the recorded data to the highly compressed first format" is defined when a third condition is satisfied, that is, when the total value of the above three conditions is equal to or less than a second threshold. Therefore, when the recorded data is recorded in the low-compression second format, the recording control unit 270 determines to switch the format of the recorded data to the highly compressed first format when the third condition is satisfied after the state where the third condition is not satisfied. In this way, the format of the recorded data can be switched to the highly compressed first format in response to the estimated intensity of the emotion of the user 20 becoming equal to or less than a predetermined threshold.

[0159] The first threshold of the third condition is the j The second threshold value of the third condition is the value obtained by multiplying Vmax of each N j The value is obtained by multiplying the total value of Vmax of the three conditions by a constant 0.8. In this way, the case where the first threshold value of the third condition is smaller than the threshold value of the first condition, and the second threshold value of the third condition is smaller than the threshold value of the second condition is exemplified. However, the first threshold value may be the same as the threshold value of the first condition, and the second threshold value may be the same as the threshold value of the second condition. In addition, the first threshold value of the third condition is set to a value obtained by multiplying the total value of Vmax of the three conditions by a constant 0.8. j T i t The second threshold of the third condition can be higher than the respective N j T i t The threshold values ​​for each condition are not limited to these examples, and various values ​​can be applied.

[0160] According to the data processing system 10, the data processing device 12 generates recording data in a highly compressed format, such as text data or audio data, during a period when the emotion of the user 20 is not high, and continuously records the data as recording data 293. Furthermore, when the emotion of the user 20 increases, the data processing device 12 generates recording data in a low compressed format, such as video data, and continuously records the data as recording data 293 during a period when the emotion continues to be at or above a certain level.

[0161] In this way, the data processing system 10 can accumulate and record video data of scenes in which the user 20 felt strong emotions as the recording data 293. On the other hand, when the user 20 did not feel strong emotions, it can accumulate and record summary information such as text data or audio data as the recording data 293. Therefore, the data processing system 10 can clearly preserve memories (records) of when the user 20 felt strong emotions, while briefly preserving memories (records) of when the user 20 did not feel strong emotions.

[0162] In this embodiment, "happiness," "anger," "sadness," and "anxiety" have been described as emotions, but the emotions handled by the data processing system 10 are not limited to these. In addition, in this embodiment, "dopamine," "serotonin," and "noradrenaline" have been described as endocrine substances, but the endocrine substances handled by the data processing system 10 are not limited to these.

[0163] The functions of the data processing device 12 may be implemented by one or more computers. At least some of the functions of the data processing device 12 may be implemented by a virtual machine. At least some of the functions of the data processing device 12 may be implemented in the cloud. Of the functions of the data processing device 12, the functions of the components excluding the storage 32 can be realized by a CPU operating based on a program. For example, at least some of the processes described as the operation of the data processing device 12 can be realized by a processor controlling each piece of hardware (e.g., a hard disk, memory, etc.) of the computer in accordance with a program. In this way, at least some of the processes of the data processing device 12 can be realized by the processor operating based on a program to control each piece of hardware, thereby causing each piece of hardware, including the processor, hard disk, memory, etc., to operate in cooperation with the program. In other words, the program can cause the computer to function as each component of the data processing device 12. Similarly, the functions of the control unit 46A, one of the components of the necklace-type terminal 14, can be realized by a CPU operating based on a program. In other words, the program can cause the computer to function as the control unit 46A of the necklace-type terminal 14. The computer may load a program that controls the execution of the above-mentioned processes and operate in accordance with the loaded program to execute the processes. The computer can read the program from a computer-readable recording medium that stores the program. Alternatively, the program may be supplied to the computer via a communication line, and the computer may read the program supplied via the communication line.

[0164] In the embodiment described above, a data processing device 12 different from the necklace-type terminal 14 is responsible for neural network processing. Also, a data processing device 12 different from the necklace-type terminal 14 stores information such as video data. However, the necklace-type terminal 14 may also perform the functions of the data processing device 12, such as neural network processing. Also, the necklace-type terminal 14 may store recorded data, etc.

[0165] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[0166] In the above embodiment, an example was given in which a specific process is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may be performed by multiple computers including computer 22.

[0167] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0168] In addition, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 53, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0169] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 53, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0170] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[0171] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[0172] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0173] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[0174] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0175] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0176] 10 Data Processing System 12 Data Processing Device 14 Necklace-type device 38 Microphone 39 Sensors 40 speakers 42 Camera 46A Control Unit 100 Data Collection Department 102 Communications Department 240 Parameter Processing Section 270 Recording control section 290 Special Processing Department 292 Input section 294 Processing Section 296 Output Section< / url:>

Claims

1. a terminal device including at least a camera that captures images of the wearer's surroundings, a sensor that detects biometric data of the wearer, and a microphone; a processing unit that estimates the intensity of emotion of the wearer based on at least a portion of the output data of the camera, the sensor, and the microphone, generates data in a first format from each of the output data collected during a period in which a value indicating the estimated intensity of emotion is less than a predetermined value, and stores the data in a specific memory, and generates data in a second format having a larger amount of information than the first format from each of the output data collected during a period in which a value indicating the estimated intensity of emotion is equal to or greater than a predetermined value, and stores the data in the specific memory; A data processing system comprising:

2. The data processing system according to claim 1 , wherein the processing unit estimates a negative emotion of the wearer as the emotion of the wearer.

3. 3. The data processing system according to claim 2, wherein the negative emotion is at least one of "anger," "sadness," and "anxiety."

4. The data processing system according to claim 1 , wherein the processing unit estimates the emotion based on at least a part of each of the output data using a neural network.

5. The plurality of artificial neurons constituting the neural network include emotion artificial neurons that are artificial neurons in which a current emotion is defined, 5. The data processing system according to claim 4, wherein the processing unit estimates the intensity of the emotion based on an internal state of the emotion artificial neuron.

6. 2. The data processing system according to claim 1, wherein said terminal device is a necklace-type terminal worn around the neck of said wearer.

7. an input unit that receives the wearer's speech picked up by the microphone; an acquisition unit that inputs a prompt including the utterance into a data generation model and acquires a response to the utterance using an output of the data generation model; an output unit that reproduces the acquired response from a speaker of the terminal device; 10. The data processing system of claim 1, further comprising:

Images