Necklace type terminal and data processing system

Abstract

To provide a necklace type terminal and a data processing system for collecting biological data of a user.SOLUTION: A necklace-type terminal includes a camera that captures an image of surroundings of a wearer, a sensor that detects biological data of the wearer, a microphone, and a collection unit that collects outputs of the camera, the sensor, and the microphone.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a necklace-type terminal and a data processing system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional techniques leave room for improvement in collecting biometric data of users. [Means for solving the problem]

[0005] A first aspect of the technology of the present disclosure is a necklace-type terminal that includes a camera that captures images of the wearer's surroundings, a sensor that detects the wearer's biometric data, a microphone, and a collection unit that collects the outputs of the camera, the sensor, and the microphone.

[0006] A second aspect of the technology disclosed herein is a necklace-type terminal of the first aspect, further including a communication unit that transmits the outputs of the camera, the sensor, and the microphone collected by the collection unit to a data processing device.

[0007] A third aspect of the technique of the present disclosure is the necklace type terminal of the second aspect, further including a speaker that outputs a response corresponding to a user utterance picked up by the microphone.

[0008] A fourth aspect of the technology of the present disclosure is a data processing system including the necklace-type terminal of the third aspect and a data processing device, wherein the data processing device includes an input unit that accepts user utterances picked up by the microphone, a processing unit that inputs prompts including the user utterances into a data generation model and obtains a response to the user utterances using the output of the data generation model, and an output unit that outputs the obtained response to the necklace-type terminal.

[0009] A fifth aspect of the technology disclosed herein is a data processing system of the fourth aspect, wherein when the user utterance satisfies a predetermined trigger condition, the processing unit inputs a prompt including the user utterance into the data generation model, and obtains a response to the user utterance using the output of the data generation model. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a conceptual diagram illustrating an example of a configuration of a data processing system. [Figure 2] FIG. 2 is a conceptual diagram showing an example of main functions of a data processing device and a necklace-type terminal. [Figure 3] FIG. 2 is a side view showing the configuration of the necklace-type terminal. [Figure 4] FIG. 2 is a top view showing the configuration of the necklace-type terminal. [Figure 5] 10 is a schematic diagram showing the functional configuration of a control unit of the necklace-type terminal. [Figure 6]2 shows a schematic functional configuration of a specific processing unit of the data processing device. [Figure 7] 10 is a diagram illustrating an example of an operational flow of specific processing by a data processing device. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an example of an embodiment of a data processing device, a data processing method, and a program according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0012] First, the terms used in the following description will be explained.

[0013] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), or an APU (Accelerated Processing Unit).

[0014] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0015] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0016] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0017] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0018] FIG. 1 shows an example of the configuration of a data processing system 10 according to the embodiment.

[0019] 1, the data processing system 10 includes a data processing device 12 and a necklace-type terminal 14. An example of the data processing device 12 is a server. In this embodiment, the data processing device 12 is an example of a "data processing device" according to the technology of the present disclosure, and the necklace-type terminal 14 is an example of a "necklace-type terminal" according to the technology of the present disclosure.

[0020] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0021] The necklace type terminal 14 includes a computer 36, a microphone 38, a sensor 39, a speaker 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 38, the speaker 40, and the camera 42 are also connected to the bus 52.

[0022] The user 20 who wears the necklace-type terminal 14 may be, for example, a patient whose health condition is to be diagnosed, or may be a normal user.

[0023] The microphone 38 picks up the voice uttered by the user 20 who is wearing the necklace-type terminal 14, as well as sounds around the user 20. The microphone 38 also receives instructions and the like from the user 20 by receiving the voice uttered by the user 20. The microphone 38 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 40 outputs audio in accordance with instructions from the processor 46. The speaker 40 is, for example, a directional speaker, and outputs audio toward the ears of the user 20.

[0024] The sensor 39 is a sensor that detects biological data of the user 20 who is wearing the necklace-type terminal. For example, the sensor 39 is a heart rate sensor or a blood oxygen sensor.

[0025] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0026] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0027] FIG. 2 shows an example of the main functions of the data processing device 12 and the necklace-type terminal 14.

[0028] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0029] The storage 32 stores a data generation model 58. The data generation model 58 is used by the specific processing unit 290. The storage 32 also includes a data accumulation unit .

[0030] In the necklace-type terminal 14, the data collection process is performed by the processor 46. A data collection program 60 is stored in the storage 50. The processor 46 reads the data collection program 60 from the storage 50 and executes the read data collection program 60 on the RAM 48. The data collection process is realized by the processor 46 operating as the control unit 46A in accordance with the data collection program 60 executed on the RAM 48.

[0031] As shown in FIGS. 3 and 4, the necklace type terminal 14 includes multiple microphones 38, multiple sensors 39, multiple speakers 40, and multiple cameras 42. FIGS. 3 and 4 show an example in which two microphones 38 are arranged so as to be located in front of the user 20 when the user 20 wears the necklace type terminal 14. FIGS. 3 and 4 also show an example in which two sensors 39 are arranged so as to be located on the right and left sides of the user 20 when the user 20 wears the necklace type terminal 14. FIGS. 3 and 4 also show an example in which two speakers 40 are arranged so as to be located on the right and left rear sides of the user 20 when the user 20 wears the necklace type terminal 14. FIGS. 3 and 4 also show an example in which two cameras 42 are arranged so as to be located on the right and left front sides of the user 20 when the user 20 wears the necklace type terminal 14. FIGS. 3 and 4 also show an example in which two sensors 39 are arranged inside the necklace type terminal 14 so as to come into contact with the neck of the user 20 when the user 20 wears the necklace type terminal 14.

[0032] Next, the processing of the control unit 46A when the necklace-type terminal 14 performs a data collection process for collecting data will be described.

[0033] In the data collection process of this embodiment, biometric data of the user is collected in real time. Furthermore, not only biometric data but also all situational data surrounding the user is collected. This makes it possible to detect early signs of, for example, Alzheimer's disease and dementia. It also makes it possible to monitor the user's health condition (for example, heart disease).

[0034] As shown in FIG. 5, the control unit 46A includes a data collection unit 100 and a communication unit 102.

[0035] The data collection unit 100 collects the output of each of the microphone 38, the sensor 39, and the camera 42.

[0036] The communication unit 102 transmits the outputs of the microphone 38 , the sensor 39 , and the camera 42 collected by the data collection unit 100 to the data processing device 12 .

[0037] Next, a description will be given of the processing of the specific processing unit 290 when the data processing device 12 performs the specific processing to acquire a response corresponding to a user utterance.

[0038] In the identification process of this embodiment, a response corresponding to a user utterance picked up by the microphone 38 of the necklace-type terminal 14 is acquired using the data generation model 58.

[0039] As shown in FIG. 6, the specific processing unit 290 includes an input unit 292, a processing unit 294, and an output unit 296.

[0040] The input unit 292 stores the outputs of the microphone 38 , the sensor 39 , and the camera 42 received from the necklace-type terminal 14 in the data storage unit 54 .

[0041] The input unit 292 acquires the user's utterance received by the necklace type terminal 14. Specifically, the input unit 292 acquires the user's utterance picked up by the microphone 38 of the necklace type terminal 14.

[0042] The processing unit 294 performs a specific process using the data generation model 58. Specifically, a prompt including a user utterance is input to the data generation model 58 to obtain a generation result. At this time, the prompt may further include outputs from the sensor 39 and the camera 42 collected by the data collection unit 100.

[0043] The output unit 296 transmits the result of the identification processing to the necklace type terminal 14. In the necklace type terminal 14, the control unit 46A causes the speaker 40 to output the result of the identification processing. In this way, a response corresponding to the user utterance picked up by the microphone 38 is output to the user 20 by the speaker 40. The microphone 38 further acquires the user utterance in response to the result of the identification processing. The control unit 46A transmits voice data indicating the user utterance acquired by the microphone 38 to the data processing device 12. In the data processing device 12, the identification processing unit 290 acquires the user utterance.

[0044] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generative AI models. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0045] The outputs of the microphone 38, the sensor 39, and the camera 42 stored in the data storage unit 54 are used, for example, to diagnose the health condition of the user 20. In this case, the outputs of the microphone 38, the sensor 39, and the camera 42 stored in the data storage unit 54 may be transmitted to a terminal on the medical institution side. Alternatively, the data processing device 12 may analyze the outputs of the microphone 38, the sensor 39, and the camera 42 stored in the data storage unit 54 to diagnose the health condition of the user 20.

[0046] Next, the operation of the data processing system 10 will be described.

[0047] First, an example of the flow of the data collection process will be described.

[0048] When the user 20 is wearing the necklace-type terminal 14, the data collection unit 100 sequentially collects the outputs of the microphone 38, the sensor 39, and the camera 42. The communication unit 102 sequentially transmits the outputs of the microphone 38, the sensor 39, and the camera 42 collected by the data collection unit 100 to the data processing device 12.

[0049] Next, an example of the flow of the identification process will be described with reference to Fig. 7. Here, it is assumed that the input unit 292 of the data processing device 12 sequentially acquires the outputs of the microphone 38, the sensor 39, and the camera 42 received from the necklace-type terminal 14 and stores them in the data accumulation unit 54.

[0050] In step S300, the processing unit 294 determines whether a predetermined trigger condition is satisfied. Specifically, the trigger condition may be that a specific word (for example, the name of an agent installed in the necklace type terminal 14) or phrase (for example, "Hi! XXX" (XXX is the name of the agent)) is included in the user utterance picked up by the microphone 38.

[0051] If the trigger condition is met in step S300 (step S300; Yes), the data processing system 10 proceeds to step S301. On the other hand, if the trigger condition is not met in step S300 (step S300; No), the data processing system 10 ends the identification process.

[0052] In step S301, the processing unit 294 generates a prompt by adding an instruction sentence for obtaining a result of a specific process to text representing a user utterance picked up by the microphone 38.

[0053] For example, a prompt such as "The user is saying the following: XXX. Please respond as an agent." (XXX is the user's utterance) may be generated. Alternatively, the outputs of the sensor 39 and the camera 42 may be added to the prompt to generate a prompt such as "This is biometric data representing the user's heart rate and video data representing the user's surroundings. The user is also saying the following: XXX. Please respond as an agent." (XXX is the user's utterance).

[0054] In step S303, the processing unit 294 inputs the generated prompt to the data generation model 58, and obtains the result of the specific process based on the output of the data generation model 58.

[0055] In step S304, the output unit 296 outputs the result of the identification process to the necklace-type terminal 14, and the identification process ends.

[0056] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[0057] In the above embodiment, an example was given in which a specific process is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may be performed by multiple computers including computer 22.

[0058] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0059] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0060] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0061] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[0062] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[0063] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0064] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[0065] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0066] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0067] 10 Data Processing System 12 Data Processing Device 14 Necklace-type device 38 Microphone 39 Sensors 40 speakers 42 Camera 46A Control Unit 100 Data Collection Department 102 Communications Department 290 Special Processing Department 292 Input section 294 Processing Section 296 Output Section< / url:>

Claims

1. A camera that captures the wearer's surroundings; a sensor for detecting biological data of the wearer; A microphone and a collection unit that collects outputs from the camera, the sensor, and the microphone; A necklace-type terminal including:

2. The necklace-type terminal according to claim 1 , further comprising a communication unit that transmits the outputs of the camera, the sensor, and the microphone collected by the collection unit to a data processing device.

3. The necklace-type terminal according to claim 2, further comprising a speaker that outputs a response corresponding to a user utterance picked up by the microphone.

4. The necklace-type terminal according to claim 3, a data processing device; and a data processing system comprising: The data processing device includes: an input unit that receives a user utterance picked up by the microphone; a processing unit that inputs a prompt including the user utterance into a data generation model and obtains a response to the user utterance using an output of the data generation model; an output unit that outputs the acquired response to the necklace type terminal; A data processing system comprising:

5. 5. The data processing system of claim 4, wherein when the user utterance satisfies a predetermined trigger condition, the processing unit inputs a prompt including the user utterance into the data generation model and obtains a response to the user utterance using an output of the data generation model.

Images