System

Abstract

A system is provided.SOLUTION: A system comprising: means for a user to input text data, voice data, and image data; means for tokenizing the text data and performing grammatical analysis; means for noise removing and processing the voice data; means for pre-processing the image data; means for generating a diagnosis result based on analysis results of the text data, voice data, and image data; and means for displaying the diagnosis result to a user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional diagnostic methods require patients to visit a medical institution in person and receive a consultation from a doctor, which is time-consuming and labor-intensive. Furthermore, in emergencies or when travel is difficult, it can be difficult to receive appropriate diagnosis and treatment. Given these circumstances, there is a demand for a system that can quickly and accurately diagnose health conditions online. [Means for solving the problem]

[0005] This invention provides a system that includes a means for a user to input text data, voice data, and image data, a means for tokenizing the text data and performing grammatical analysis, a means for removing noise from and processing the voice data, a means for preprocessing the image data, a means for generating a diagnosis based on the results of the analysis of the text data, voice data, and image data, and a means for displaying the diagnosis result to the user, thereby enabling a prompt and accurate online diagnosis of health conditions. This allows a user to receive an appropriate diagnosis and receive the necessary treatment or response early, even if it is difficult for the user to immediately visit a medical institution.

[0006] A "user" is an entity that utilizes the system to input symptoms and receive diagnostic results.

[0007] "Text data" refers to character information that is input by a user using a language.

[0008] "Audio data" refers to audio information recorded and input by the user.

[0009] "Image data" is visual information captured and input by a user.

[0010] "Means" is a broad term that refers to a method or device for achieving a particular purpose.

[0011] "Tokenization" is the process of dividing text data into semantic units.

[0012] "Grammar analysis" is the process of analyzing the grammatical structure of text data and understanding its meaning.

[0013] "Noise reduction" is the process of removing unwanted background or interfering sounds from audio data.

[0014] "Preprocessing" is a preliminary process that converts data into a format that is easy to analyze.

[0015] "Analysis results" are diagnostic information and features obtained through data analysis.

[0016] The "diagnosis result" is an evaluation or advice regarding the user's health condition that is generated based on the analysis result.

[0017] A "system" is a collection of processes and devices that function in an integrated manner.

[0018] Inflammation is an immune response that body tissues display in response to irritation, and is often accompanied by symptoms such as redness, swelling, and pain.

[0019] Swelling is a condition in which bodily tissues swell larger than normal, and is often associated with inflammation or infection. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] This system is designed to allow users to easily diagnose their own health condition from home or on the go. The system analyzes the text, voice, and image data entered by the user and provides a diagnosis result. Below, the program processing of this system is explained in natural language and shown in detail with concrete examples.

[0042] System configuration

[0043] User terminal

[0044] 1. Data input interface:

[0045] The user launches a dedicated application and inputs text data about the symptoms.

[0046] The user uses a microphone to record voice data such as coughing.

[0047] The user uses a camera to capture image data of the affected area.

[0048] 2. Data transmission function:

[0049] In order to transmit the input data to the server, the terminal packetizes the data and transmits it to the server using a secure communication protocol (for example, HTTPS).

[0050] server

[0051] 3. Data reception and preprocessing:

[0052] The server receives the data sent from the terminal and identifies the data type of each data.

[0053] The text data is first tokenized and then grammatically analyzed.

[0054] The audio data is soundproofed and analyzed based on specific frequencies.

[0055] The image data is filtered and preprocessed to convert it into a format that is easy to analyze.

[0056] 4. Data Analysis:

[0057] The server passes the preprocessed data to the AI ​​model for further analysis.

[0058] The main symptoms and related symptoms are extracted from the text data.

[0059] The audio data is analyzed for specific acoustic features, and the symptoms are characterized based on the results.

[0060] The image data recognizes specific patterns and identifies the presence or absence of inflammation or swelling.

[0061] 5. Generating diagnostic results:

[0062] The server integrates the analysis results and generates a diagnosis based on the user's symptoms.

[0063] Diagnostic findings may include recommended treatments and further medical consultations.

[0064] 6. Sending and viewing diagnostic results:

[0065] The server transmits the generated diagnosis results to the terminal.

[0066] The terminal displays the received diagnostic results to the user.

[0067] Specific examples

[0068] Diagnosing sore throat and cough

[0069] 1. Data Entry:

[0070] The user enters the text "I've had a persistent cough for the past three days and my throat hurts," then records and uploads an audio recording of themselves coughing.

[0071] 2. Data transmission:

[0072] The terminal converts text and audio files into packets and sends them to the server.

[0073] 3. Data reception and preprocessing:

[0074] The server receives the data, passes the text to an NLP model for analysis, and passes the audio data to an audio processing module for noise reduction.

[0075] 4. Data Analysis:

[0076] The server extracts key symptoms from the text and analyzes cough characteristics from the audio.

[0077] 5. Generating diagnostic results:

[0078] The server aggregates the results and generates a report with a diagnosis of "possible mild bronchitis" and recommended treatment.

[0079] 6. Sending and viewing diagnostic results:

[0080] The server sends the diagnostic report to the device.

[0081] The device displays the message, "You may have mild bronchitis. We recommend you drink plenty of fluids and get plenty of rest. If necessary, seek medical advice."

[0082] The above is a detailed embodiment of this system, which allows users to quickly and effectively check their own health status.

[0083] The processing flow will be explained below.

[0084] Step 1:

[0085] The user launches a dedicated application to describe their symptoms. In the text box, they enter, "I've had a persistent cough for the past three days and my throat hurts." They then use the recording function to record the sound of their cough and, if necessary, take an image of the affected area.

[0086] Step 2:

[0087] The device detects the input text data, voice data, and image data, and packetizes them for transmission to the server using the secure HTTPS communication protocol.

[0088] Step 3:

[0089] The server receives data packets sent from the terminal and separates them by data type: text data is sent to the text processing module, voice data to the voice processing module, and image data to the image processing module.

[0090] Step 4:

[0091] The server uses a Natural Language Processing (NLP) model to analyze the text data. The text data is first tokenized and then grammatically analyzed. The analysis results identify the primary symptom and related secondary symptoms.

[0092] Step 5:

[0093] The server pre-processes the audio data, applying noise reduction filters to bring the audio data into a clean format for analysis. Then, the audio analysis model extracts audio features and analyzes cough patterns.

[0094] Step 6:

[0095] The server preprocesses the image data, filtering and normalizing it, and then passes it to the image analysis model, which detects patterns of inflammation and swelling from the image data.

[0096] Step 7:

[0097] The server combines the results of the text, audio, and image analysis to generate a comprehensive diagnosis of the user's health, including recommended treatments and further medical consultations.

[0098] Step 8:

[0099] The server sends the generated diagnostic results to the device, where they are transmitted in a secure format and read by the device application.

[0100] Step 9:

[0101] The device displays the diagnosis results it received to the user. For example, it might say, "You may have mild bronchitis. We recommend you drink plenty of fluids and get plenty of rest. If necessary, consult a doctor."

[0102] Example 1

[0103] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0104] In modern society, users need to be able to easily diagnose their own health status from home or on the go. However, there is no system that can integrate and effectively analyze multiple data types (e.g., text data, voice data, and image data) to provide diagnostic results. Secure transmission of this data and accurate preprocessing are also challenges.

[0105] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0106] In this invention, the server includes means for a user to input text data, voice data, and image data, means for tokenizing the text data and performing grammatical analysis, means for denoising and processing the voice data, means for preprocessing the image data, means for passing the text data, voice data, and image data to an AI model for analysis, means for generating a diagnosis result based on the analysis result, means for displaying the diagnosis result to the user, means for transmitting the input data to the server using a secure communication protocol, and means for identifying the received data and preprocessing each data type, thereby enabling a user to quickly and effectively diagnose their health condition.

[0107] A "user" is an individual who utilizes the system to diagnose their own health condition.

[0108] "Text data" is data that represents character information entered by a user.

[0109] "Audio data" is data representing acoustic information recorded by a user.

[0110] "Image data" is data that represents visual information captured by a user.

[0111] "Tokenization" is the process of dividing text data into words and phrases.

[0112] "Grammar analysis" is a process of analyzing the grammatical structure of tokenized text data.

[0113] "Noise reduction" is a process of removing unnecessary background sounds from audio data.

[0114] "Preprocessing" is a series of steps to prepare data in a format that is easy to analyze.

[0115] An "AI model" is a system that uses artificial intelligence techniques to analyze data and recognize patterns.

[0116] "Analysis" is the process of examining data in detail and extracting its meaning and characteristics.

[0117] A "diagnostic result" is an assessment of health status or identification of symptoms generated based on the results of the analysis.

[0118] A "secure communication protocol" is a communication protocol for securely transmitting data.

[0119] A "server" is a computer that receives, preprocesses, and analyzes data, and generates and transmits diagnostic results.

[0120] A "user terminal" is a device through which a user inputs and transmits data.

[0121] The term "said" refers to something that has already been mentioned in the text.

[0122] A "means" is a device or process for achieving a particular function.

[0123] This invention is a system that allows users to easily diagnose their own health condition from home or on the go. In this system, users input text data, voice data, and image data, and generate diagnostic results by analyzing the data. Detailed embodiments of this system are described below.

[0124] System configuration

[0125] User terminal

[0126] 1. Data input interface:

[0127] The user launches a dedicated application and inputs text data about the symptoms.

[0128] The user uses a microphone to record voice data such as coughing.

[0129] The user uses a camera to capture image data of the affected area.

[0130] 2. Data transmission function:

[0131] The terminal packetizes the input data and transmits it to the server using a secure communication protocol (for example, HTTPS).

[0132] server

[0133] 3. Data reception and preprocessing:

[0134] The server receives the data sent from the terminal and identifies the data type of each data.

[0135] The text data is first tokenized and then grammatically analyzed.

[0136] The audio data is subjected to noise reduction and analyzed based on specific frequencies.

[0137] The image data is filtered and preprocessed to convert it into a format that is easy to analyze.

[0138] 4. Data Analysis:

[0139] The server passes the pre-processed data to a generative AI model for further analysis.

[0140] The main symptoms and related symptoms are extracted from the text data.

[0141] The audio data is analyzed for specific acoustic features, and the symptoms are characterized based on the results.

[0142] The image data recognizes specific patterns and identifies the presence or absence of inflammation or swelling.

[0143] 5. Generating diagnostic results:

[0144] The server integrates the analysis results and generates a diagnosis based on the user's symptoms.

[0145] Diagnostic findings may include recommended treatments and further medical consultations.

[0146] 6. Sending and viewing diagnostic results:

[0147] The server transmits the generated diagnosis results to the terminal.

[0148] The terminal displays the received diagnostic results to the user.

[0149] Specific examples

[0150] Diagnosing sore throat and cough

[0151] 1. Data Entry:

[0152] The user enters the text "I've had a persistent cough for the past three days and my throat hurts," then records and uploads an audio recording of themselves coughing.

[0153] 2. Data transmission:

[0154] The terminal converts text and audio files into packets and sends them to the server.

[0155] 3. Data reception and preprocessing:

[0156] The server receives the data, passes the text to a natural language processing (NLP) model for analysis, and passes the audio data to an audio processing module for noise reduction.

[0157] 4. Data Analysis:

[0158] The server extracts key symptoms from the text and analyzes cough characteristics from the audio.

[0159] 5. Generating diagnostic results:

[0160] The server aggregates the results and generates a report with a diagnosis of "possible mild bronchitis" and recommended treatment.

[0161] 6. Sending and viewing diagnostic results:

[0162] The server sends the diagnostic report to the device.

[0163] The device displays the message, "You may have mild bronchitis. We recommend you drink plenty of fluids and get plenty of rest. If necessary, seek medical advice."

[0164] The above is a detailed embodiment of this system, which allows users to quickly and effectively check their own health status.

[0165] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0166] Step 1:

[0167] Use of a data entry interface

[0168] User:

[0169] Start the dedicated application.

[0170] In the application's text entry field, enter your symptoms: "I've had a persistent cough and sore throat for the past three days."

[0171] Use the application's voice recording feature to record the sound of your cough using your microphone.

[0172] Use the application's camera function to take a picture of your throat, for example if you have a sore throat.

[0173] Input: Text data, audio data, and image data about the user's symptoms

[0174] Output: Symptom data stored within the application

[0175] Step 2:

[0176] Data transmission

[0177] Device:

[0178] Text data entered by the user, recorded voice data, and captured image data are packetized.

[0179] This data is sent to the server using a secure communication protocol such as HTTPS.

[0180] Input: Text data, voice data, and image data entered by the user

[0181] Output: Packetized data sent to the server

[0182] Step 3:

[0183] Data reception and preprocessing

[0184] server:

[0185] Receives data sent from the device and identifies the data type (text, audio, image).

[0186] The text data is first tokenized and then grammatically analyzed using natural language processing (NLP) techniques.

[0187] For example, extract key keywords such as "cough" and "sore throat."

[0188] The audio data is subjected to noise removal processing and specific frequencies are analyzed to extract cough patterns and characteristics.

[0189] For example, the intensity and frequency of coughing sounds are analyzed.

[0190] The image data is subjected to filtering and pre-processing to improve the image quality.

[0191] A Gaussian filter is used for filtering, and image resolution is adjusted for preprocessing.

[0192] Input: Text data, audio data, and image data sent from the device

[0193] Output: Preprocessed text data, audio data, and image data

[0194] Step 4:

[0195] Data analysis

[0196] server:

[0197] Preprocessed text data, audio data, and image data are passed to a generative AI model.

[0198] A generative AI model analyzes each piece of data in detail.

[0199] From the text data, major symptoms such as "last three days," "cough," and "sore throat" are detected.

[0200] Cough characteristics (e.g., frequency, intensity) are analyzed from the audio data.

[0201] The image data is used to evaluate the condition of the throat and determine whether inflammation is present.

[0202] Input: Preprocessed text data, audio data, and image data

[0203] Output: Analysis results for each data

[0204] Step 5:

[0205] Generating diagnostic results

[0206] server:

[0207] The results of each data analysis are integrated to generate a comprehensive diagnostic result.

[0208] For example, it generates a diagnosis such as "You may have mild bronchitis" and compiles a report with recommended treatment methods.

[0209] Input: Analysis results of text data, audio data, and image data

[0210] Output: A report containing the diagnosis and recommended treatment.

[0211] Step 6:

[0212] Sending and displaying diagnostic results

[0213] server:

[0214] The generated diagnostic results are transmitted to the terminal via secure communication.

[0215] Device:

[0216] The received diagnostic results are displayed to the user.

[0217] For example, it might say, "You may have mild bronchitis. We recommend you drink plenty of fluids and get plenty of rest. If necessary, seek medical advice."

[0218] Input: Diagnostic results sent from the server

[0219] Output: Diagnostic results displayed on the user's terminal

[0220] (Application example 1)

[0221] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0222] In recent years, there has been an increasing need for systems that allow individuals to easily diagnose their health conditions from home or on the go. However, there is a lack of systems that can comprehensively analyze text data, voice data, and image data to provide highly reliable diagnoses. There is also a need for systems that can efficiently process user-entered data and quickly display diagnostic results. A technical method is needed to solve these issues.

[0223] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0224] In this invention, the server includes means for a user to input text data, voice data, and image data, means for tokenizing the text data and performing grammatical analysis, means for removing noise and processing the voice data, means for preprocessing the image data, means for generating a diagnosis result based on the analysis results of the text data, voice data, and image data, means for displaying the diagnosis result to the user, means for collecting data from a smartphone terminal and transmitting it to the server, and means for generating a diagnosis result based on a prompt sentence using a generative AI model, thereby enabling a user to quickly and accurately diagnose their own health condition from home or on the go.

[0225] A "user terminal" is an electronic device that allows a user to input, send, and receive text data, voice data, and image data.

[0226] "Text data" refers to text information entered by a user, and is data for performing natural language processing.

[0227] "Audio data" refers to audio information recorded by a user, and is data for performing acoustic analysis and noise removal processes.

[0228] "Image data" is visual information captured by a user using a camera, and is data for preprocessing.

[0229] "Tokenization" is a process of dividing text data into small units, and is a preprocessing step for grammatical analysis.

[0230] "Grammar analysis" is a process of analyzing the grammatical structure of tokenized text data.

[0231] "Noise reduction" is a process that removes unnecessary sounds such as background noise from audio data.

[0232] "Preprocessing" is a process for converting image data into a format that is easy to analyze.

[0233] The "diagnosis result" is estimated information about the health condition that is generated based on the analyzed text data, voice data, and image data.

[0234] The "means for displaying to the user" is a function or device for providing the diagnostic results to the user in a format that is easy to view.

[0235] A "smartphone terminal" is a small, portable electronic device that is used to collect and transmit user data.

[0236] A "server" is a computer system that receives, analyzes, and transmits data over a network.

[0237] A "generative AI model" is an artificial intelligence model that automatically generates diagnostic results based on input data.

[0238] A "prompt sentence" is an instruction sentence used as input to a generative AI model.

[0239] This system is designed to allow users to easily diagnose their own health condition from home or on the go. It analyzes text, voice, and image data entered by the user and provides diagnostic results. The components of the system and specific processing steps are shown below.

[0240] System configuration

[0241] User terminal

[0242] The user terminal includes a portable device such as a smartphone or tablet. The user inputs the following data through the application:

[0243] 1. Text data input: The user inputs information about symptoms and physical condition in text format.

[0244] 2. Recording voice data: The user records coughing, breathing, etc. and inputs it as voice data.

[0245] 3. Capturing image data: The user uses a camera to capture an image of the affected area.

[0246] server

[0247] The server is a central computing device that receives and analyzes data sent from the terminals. The server is configured as follows:

[0248] 1. Data reception: Receives text data, voice data, and image data sent from the user terminal.

[0249] 2. Preprocessing: The received data is processed by various preprocessing modules. Text data is tokenized and grammatically analyzed, audio data is denoised, and image data is filtered and format converted.

[0250] 3. Data analysis: The preprocessed data is then further analyzed by generative AI models: text data is analyzed for key symptoms and related symptoms, audio data is analyzed for specific acoustic features, and image data is analyzed for specific patterns.

[0251] 4. Generating a diagnosis: The analysis results are combined to generate a diagnosis based on the user's symptoms, which may include recommended treatments or further medical consultations.

[0252] 5. Sending the diagnosis result: Send the generated diagnosis result to the user terminal.

[0253] Hardware and software used

[0254] User devices: smartphones, tablets

[0255] Speech processing software: speech_recognition library

[0256] Image processing software: OpenCV library

[0257] Sending data: the requests library

[0258] Generative AI models: various AI diagnostic models, such as those provided by the transformers library

[0259] Specific examples

[0260] The user launches a smartphone application, enters the text "I've had a persistent cough for the past three days and my throat hurts," records the coughing sound, and takes an image of their throat. The user's device sends this data to the server. The server receives the data, performs grammatical analysis on the text data, removes noise from the audio data, and filters the image data. It then analyzes the data based on the generative AI model and generates a diagnosis: "You may have mild bronchitis. We recommend you drink plenty of fluids and rest. If necessary, consult a doctor." This is then sent to the user's device.

[0261] Prompt Sentence Examples

[0262] User input: "I've had a constant cough and sore throat for the past three days."

[0263] Audio Data: [Recorded coughing sound]

[0264] Image Data: [User-taken image of throat]

[0265] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0266] Step 1:

[0267] A user starts the application using a smartphone and inputs text, voice, and image data. Specifically, the user inputs text such as "I've had a persistent cough for the past three days and my throat hurts," records the sound of their cough, and takes an image of their throat.

[0268] Input: Text data, audio data, image data

[0269] Output: Input text, recorded audio files, captured image files

[0270] Step 2:

[0271] The device prepares the input data and sends it to the server using a secure communication protocol (e.g., HTTPS). The data is packetized as text, audio files, or image files.

[0272] Input: text, audio files, image files

[0273] Output: Packetized data

[0274] Step 3:

[0275] The server receives packetized data sent from the terminal, identifies the received data, and initiates appropriate pre-processing for each data type.

[0276] Input: Packetized data

[0277] Output: Received text data, audio data, and image data

[0278] Step 4:

[0279] The server tokenizes the received text data and performs grammatical analysis, extracting the grammatical structure and semantic features of the text data, for example, identifying key symptoms such as "cough" or "sore throat."

[0280] Input: Text data

[0281] Output: Tokenized text, grammar analysis results

[0282] Step 5:

[0283] The server processes the audio data through noise reduction and analyzes its acoustic features to identify coughs and other distinctive acoustic patterns, such as the frequency range and duration of a cough.

[0284] Input: Audio data

[0285] Output: Noise-removed audio data, acoustic analysis results

[0286] Step 6:

[0287] The server performs preprocessing on the image data, such as filtering and format conversion, to convert the image data into a format that is easier to analyze. For example, contrast adjustment and edge detection are performed on the throat image.

[0288] Input: Image data

[0289] Output: Preprocessed image data

[0290] Step 7:

[0291] The server passes the preprocessed text, audio, and image data to the generative AI model for detailed data analysis, such as analyzing the prompt sentences and generating a comprehensive diagnosis.

[0292] Input: Tokenized text, acoustic analysis results, preprocessed image data

[0293] Output: Diagnostic results

[0294] Step 8:

[0295] The server integrates the generated diagnostic results and generates appropriate feedback to the user, including recommended treatments and next steps (e.g., consulting a medical professional).

[0296] Input: Diagnostic results

[0297] Output: Consolidated diagnostic report

[0298] Step 9:

[0299] The server sends the diagnosis results to the user's device, which then displays the received diagnosis results in an easy-to-read format. For example, it might say, "You may have mild bronchitis. We recommend you drink plenty of fluids and get plenty of rest."

[0300] Input: Consolidated diagnostic report

[0301] Output: Diagnostic results displayed to the user

[0302] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0303] This invention combines a system that allows users to diagnose their own health condition using text data, voice data, and image data with an emotion engine that recognizes the user's emotions, thereby providing comprehensive diagnostic results that take the user's emotional state into consideration.

[0304] System configuration

[0305] User terminal

[0306] 1. Data input interface:

[0307] The user launches a dedicated application and inputs text data about the symptoms.

[0308] The user uses a microphone to record audio data such as coughing, and a camera to take an image of the affected area.

[0309] 2. Data transmission function:

[0310] The terminal converts the input text data, voice data, and image data into packets and transmits them to the server using a secure communication protocol (for example, HTTPS).

[0311] server

[0312] 3. Data reception and preprocessing:

[0313] The server receives data packets sent from the terminal and separates them by data type: text data is sent to the text processing module, voice data is sent to the voice processing module, and image data is sent to the image processing module.

[0314] 4. Data Analysis:

[0315] The text data is tokenized and grammatically analyzed using a natural language processing (NLP) model to identify primary symptoms and associated secondary symptoms.

[0316] The audio data is denoised and filtered, and then the cough pattern is analysed through a speech analysis model.

[0317] The image data is preprocessed and an image analysis model is used to detect patterns of inflammation and swelling.

[0318] 5. Emotion analysis:

[0319] The server's emotion engine analyzes text data, voice data, and image data to recognize the user's emotional state.

[0320] From the text data, emotions are identified based on the user's writing style and expression.

[0321] Emotions are analyzed from audio data based on fluctuations in tone and pitch.

[0322] Emotions are recognized from image data based on facial expressions and body movements.

[0323] 6. Generating diagnostic results:

[0324] The server integrates the data analysis results with the emotion analysis results to generate a comprehensive diagnosis, which also reflects the user's emotional state.

[0325] The diagnosis will include recommended treatment and medical consultation if necessary.

[0326] 7. Sending and viewing diagnostic results:

[0327] The server sends the generated diagnostic results to the terminal, where an application displays the diagnostic results to the user.

[0328] Specific examples

[0329] Diagnosis by combining sore throat, cough, and emotion analysis

[0330] 1. Data Entry:

[0331] The user enters the text "I've had a persistent cough for the past three days and my throat hurts," records and sends an audio recording of the cough, and also takes a photo of their face.

[0332] 2. Data transmission:

[0333] The device sends text, audio, and image files to the server.

[0334] 3. Data reception and preprocessing:

[0335] The server receives the data, analyzes the text using NLP models, denoises the audio data, and preprocesses the image data.

[0336] 4. Data Analysis:

[0337] It extracts key symptoms from text, analyzes cough characteristics from audio, and detects patterns of inflammation and swelling from images.

[0338] 5. Emotion analysis:

[0339] The server uses an emotion engine to analyze stress and anxiety from text, tension from audio, and sadness and fatigue from images.

[0340] 6. Generating diagnostic results:

[0341] The server combines the results of the data analysis and the emotion analysis and generates a diagnosis that reads, "You may have mild bronchitis, but you appear to be under stress. We recommend that you drink plenty of fluids and get plenty of rest. Also, if you need psychological support, please consult a specialist."

[0342] 7. Sending and viewing diagnostic results:

[0343] The server sends the diagnostic results to the terminal, which displays them to the user.

[0344] The above is a specific embodiment of the present invention. By using this system, users can receive diagnostic results that take into account their physical and emotional state comprehensively, and obtain appropriate treatment and countermeasures.

[0345] The processing flow will be explained below.

[0346] Step 1:

[0347] The user starts a dedicated symptom recording application and enters text data about the symptoms. For example, the user might write "I've had a persistent cough and a sore throat for the past three days" in the text box.

[0348] Step 2:

[0349] The user uses a microphone to record the sound of their cough, as well as take a photo of their face and the affected area, all of which is uploaded to the application.

[0350] Step 3:

[0351] The device combines the input text, voice, and image data into a single data packet, which is then sent to the server using a secure communication protocol (e.g., HTTPS).

[0352] Step 4:

[0353] The server receives data packets sent from the device and separates them by data type (text, audio, image), and passes each data to the corresponding analysis module.

[0354] Step 5:

[0355] The server passes the text data to a natural language processing (NLP) model for tokenization and grammatical analysis, which identifies the primary symptom and associated secondary symptoms.

[0356] Step 6:

[0357] The server passes the audio data to a speech processing module for noise reduction and filtering, and then a speech analysis model analyzes the audio characteristics (e.g., coughing patterns).

[0358] Step 7:

[0359] The server passes the image data to the image processing module for preprocessing (filtering, normalization), after which an image analysis model detects patterns of inflammation and swelling.

[0360] Step 8:

[0361] The server passes the analyzed text data to an emotion engine to identify the user's emotions. For example, if there are many negative expressions, it can detect stress or anxiety.

[0362] Step 9:

[0363] The server also analyzes emotions from the voice data, identifying emotions such as tension or impatience based on the tone and pitch of the voice.

[0364] Step 10:

[0365] The server uses the image data to analyze the user's facial expressions and physical state to identify emotions, such as sadness or fatigue from facial distortion or puffy eyes.

[0366] Step 11:

[0367] The server integrates the data obtained from each analysis module (text, audio, image analysis results, and emotion analysis results) to generate a comprehensive diagnosis that also reflects the user's emotional state.

[0368] Step 12:

[0369] The server sends the generated diagnostic results to the device, where they are transmitted in a secure format and displayed in the device application.

[0370] Step 13:

[0371] The device will display the diagnosis results it receives to the user, such as "You may have mild bronchitis, but you appear to be under stress. We recommend you drink plenty of fluids and get plenty of rest. Also, if you need psychological support, please consult a specialist."

[0372] Example 2

[0373] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0374] Conventional health checkup systems provide diagnostic results based on the analysis of text data, voice data, and image data collected from users, but do not take the user's emotional state into consideration. As a result, comprehensive health diagnosis is insufficient and advice on the user's emotional health is lacking. The present invention aims to provide more accurate comprehensive diagnostic results by analyzing data including the user's emotional state.

[0375] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0376] In this invention, the server includes a means for tokenizing and syntactically analyzing text data, a means for denoising and processing audio data, a means for preprocessing image data, and a means for analyzing emotional states, thereby enabling the generation and display of comprehensive diagnostic results based on the analysis results of the text data, audio data, and image data, as well as the emotional states.

[0377] "User" refers to a person who uses the system to input data to diagnose their own health condition.

[0378] "Text data" refers to character information such as symptoms and emotions entered by the user.

[0379] "Audio data" refers to acoustic information such as coughs and voices recorded by a user.

[0380] "Image data" refers to image information such as the affected area and facial expression photographed by the user.

[0381] "Tokenization" refers to the process of breaking down text data into smaller units such as words or phrases.

[0382] "Syntax analysis" refers to the process of analyzing the grammatical structure of tokenized text data.

[0383] "Noise reduction" refers to the process of removing unnecessary noise from audio data.

[0384] "Preprocessing" refers to the process of formatting data to make it easier to analyze.

[0385] "Diagnostic results" refer to health assessments and recommended treatments generated based on data and sentiment analysis.

[0386] "Emotional state" refers to a user's mental state, such as stress, anxiety, tension, sadness, etc.

[0387] "Comprehensive diagnostic results" refer to diagnostic information obtained by taking into account both physical symptoms and emotional state.

[0388] "Means for displaying" refers to a method or device for delivering the generated diagnostic results to a user.

[0389] MODE FOR CARRYING OUT THE INVENTION

[0390] The present invention is a system that allows users to diagnose their own health condition using text data, voice data, and image data, and also provides comprehensive diagnostic results that take into account the user's emotional state. This system is composed of a user terminal and a server.

[0391] User terminal

[0392] 1. Data Entry Interface

[0393] The user launches a dedicated application and enters text data about their symptoms. For example, they can enter, "I've had a persistent cough and a sore throat for the past three days."

[0394] The user uses a microphone to record audio data such as coughing, and a camera to take an image of the affected area.

[0395] 2. Data transmission function

[0396] The terminal converts the input text data, voice data, and image data into packets and transmits them to the server using a secure communication protocol (for example, HTTPS).

[0397] server

[0398] 3. Data Reception and Preprocessing

[0399] The server receives data packets sent from the terminal and separates them by data type (passing text data to the text processing module, voice data to the voice processing module, and image data to the image processing module).

[0400] The main technologies used in this case include NLP (Natural Language Processing) modules, voice analysis models, and image analysis models.

[0401] 4. Data Analysis

[0402] Text data analysis: The server uses NLP models to tokenize the text data and perform grammatical analysis, which identifies primary and secondary symptoms, such as "cough" and "sore throat."

[0403] Voice data analysis: The server performs noise reduction and filtering on the voice data, and then analyzes the cough pattern using a voice analysis model.

[0404] Image data analysis: The server preprocesses the image data and uses image analysis models to detect patterns of inflammation and swelling.

[0405] 5. Emotion analysis

[0406] The server's emotion engine analyzes text, voice, and image data to recognize the user's emotional state.

[0407] Sentiment analysis from text: Identify emotions based on the user's writing style and expressions. For example, analyze stress from the emphasis on a period such as "3 days" in the text.

[0408] Emotion analysis from audio: Analyzes tone and pitch fluctuations in audio data to detect tension.

[0409] Emotion analysis from images: Analyzing facial expressions and body movements to recognize feelings of sadness or fatigue.

[0410] 6. Generating diagnostic results

[0411] The server integrates the data analysis results with the emotion analysis results to generate a comprehensive diagnosis, which also reflects the user's emotional state.

[0412] For example, it generates a diagnosis such as, "You may have mild bronchitis, but you appear to be under stress. We recommend you drink plenty of fluids and get plenty of rest. Also, if you need psychological support, consult a specialist."

[0413] 7. Sending and displaying diagnostic results

[0414] The server transmits the generated diagnostic results to the terminal, and an application on the terminal displays the diagnostic results to the user.

[0415] This system allows users to obtain comprehensive health checkup results that take into account not only their physical symptoms but also their emotional state.

[0416] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0417] Program processing flow

[0418] Step 1: User Data Entry

[0419] Using a dedicated application, users input text data about their symptoms. For example, they can describe specific symptoms such as, "I've had a persistent cough for the past three days, and my throat hurts." In addition, the system collects audio and image data by recording the sound of the cough using a microphone and taking images of the affected area using a camera.

[0420] Input: Text data describing symptoms, recorded voice data, photographed image data

[0421] Output: Collected dataset (text data, audio data, image data)

[0422] Step 2: Data collection and transmission by the device

[0423] The device collects text data entered by the user, recorded voice data, and captured image data, then encrypts this data using the HTTPS protocol and securely transmits it to a server.

[0424] Input: Collected datasets (text data, audio data, image data)

[0425] Output: Dataset sent securely to server

[0426] Step 3: Data reception and preprocessing by the server

[0427] The server receives data packets sent from the device. The received data is first classified into text data, voice data, and image data. The text data is then passed to the NLP module, the voice data to the voice analysis module, and the image data to the image analysis module.

[0428] Input: Securely transmitted dataset

[0429] Output: Classified dataset (text data, audio data, image data)

[0430] Step 4: Analyzing the text data

[0431] The server uses an NLP model to tokenize and parse the received text data, extracting key symptoms such as "cough" or "sore throat."

[0432] Input: Text data

[0433] Output: Extracted main and secondary symptoms

[0434] Step 5: Analyzing the audio data

[0435] The server denoises and filters the audio data, then uses a speech analysis model to analyze cough patterns.

[0436] Input: Audio data

[0437] Output: Analyzed cough pattern

[0438] Step 6: Analyzing the image data

[0439] The server preprocesses the image data and uses image analysis models to detect patterns of inflammation and swelling.

[0440] Input: Image data

[0441] Output: Detected inflammation and swelling patterns

[0442] Step 7: Sentiment Analysis

[0443] The server's emotion engine analyzes text data, voice data, and image data to recognize the user's emotional state. Specifically, it identifies emotions from text data based on writing style and expression, analyzes tone and pitch fluctuations from voice data, and analyzes facial expressions and body movements from image data.

[0444] Input: text data, audio data, image data

[0445] Output: User's emotional state

[0446] Step 8: Generate comprehensive diagnostic results

[0447] The server combines the data analysis results with the sentiment analysis results to generate a comprehensive diagnosis. This diagnosis includes recommended treatment and, if necessary, consultation with a medical institution. For example, a diagnosis might read, "You may have mild bronchitis, but you appear to be under stress. We recommend you drink plenty of fluids and rest. We also recommend consulting a specialist if you need psychological support."

[0448] Input: Data analysis results, emotion analysis results

[0449] Output: Comprehensive diagnostic results

[0450] Step 9: Send and view diagnostic results

[0451] The server sends the generated diagnostic results to the terminal, and the terminal application displays the received diagnostic results to the user.

[0452] Input: Comprehensive diagnostic results

[0453] Output: Diagnostic results displayed to the user

[0454] Through the above processing steps, the user can receive a diagnosis result that takes into consideration the user's physical and emotional state comprehensively.

[0455] (Application example 2)

[0456] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0457] Conventional health checkup systems typically analyze symptoms and make diagnoses based on text data, voice data, and image data entered by the user. However, these systems were unable to consider the user's emotional state, making it difficult to provide comprehensive diagnostic results. Furthermore, it was difficult to propose services in physical stores that took into account the customer's health and emotional state, making improving customer satisfaction a challenge. Therefore, there was a need for a system that could analyze not only the user's health condition but also their emotional state, reflect this in the diagnostic results, and then propose appropriate products and services in physical stores.

[0458] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0459] In this invention, the server includes means for a user to input text data, voice data, and image data, means for tokenizing the text data and performing grammatical analysis, means for removing noise and processing the voice data, means for preprocessing the image data, means for generating a diagnostic result based on the analysis results of the text data, voice data, and image data, means for displaying the diagnostic result to the user, means for proposing products and services in a physical store, and means for performing emotion analysis on the text data, voice data, and image data and reflecting the emotion in the diagnostic result. This makes it possible to provide a comprehensive diagnostic result based on the user's health condition and emotional state, and to propose appropriate products and services in a physical store based on the result.

[0460] "User" refers to a person who uses the system to diagnose a health or emotional state.

[0461] "Text data" refers to data that includes character information entered by a user.

[0462] "Voice data" refers to data containing voice information input by a user.

[0463] "Image data" refers to data that includes information about an image captured by a user.

[0464] "Tokenization" refers to the process of dividing text data into smaller units such as words or phrases.

[0465] "Grammar analysis" refers to the process of analyzing the syntax and meaning of text data.

[0466] "Noise reduction" refers to the process of removing unnecessary background sounds and noise from audio data.

[0467] "Preprocessing" refers to the process of shaping or adjusting image data before analysis.

[0468] "Diagnosis results" refer to the evaluation results of the user's health condition and emotional state generated based on the analysis results of text data, audio data, and image data.

[0469] "Display means" refers to a device or method for displaying the diagnostic results to the user or store clerk.

[0470] "Brick and mortar store" refers to a physical location that offers goods and services.

[0471] "Product and service suggestions" refers to suggesting appropriate products and services based on the diagnostic results.

[0472] "Emotion analysis" refers to the process of analyzing a user's emotional state from text data, audio data, and image data.

[0473] This will help the reader to understand the invention more clearly.

[0474] MODE FOR CARRYING OUT THE INVENTION

[0475] The present invention is a system for analyzing the health and emotional state of customers in a physical store and recommending appropriate products and services based on the results. The system includes a user terminal, a server, and a store device that displays the summary results.

[0476] User terminal

[0477] The user terminal has the following functions:

[0478] Data Input Interface:

[0479] The user launches a dedicated application and inputs text data about the symptoms.

[0480] The user uses a microphone to record audio data such as coughing, and a camera to take an image of the affected area.

[0481] Data transmission function:

[0482] The terminal converts the input text data, voice data, and image data into packets and transmits them to the server using a secure communication protocol (for example, HTTPS).

[0483] server

[0484] The server has the following functions:

[0485] Data reception and preprocessing:

[0486] The server receives data packets sent from the terminal and separates them by data type: text data is sent to the text processing module, voice data is sent to the voice processing module, and image data is sent to the image processing module.

[0487] Data Analysis:

[0488] The text data is tokenized and grammatically analyzed using a natural language processing (NLP) model to identify primary symptoms and associated secondary symptoms.

[0489] The audio data is denoised and filtered, and then the cough pattern is analysed through a speech analysis model.

[0490] The image data is preprocessed and an image analysis model is used to detect patterns of inflammation and swelling.

[0491] Emotion analysis:

[0492] The server's emotion engine analyzes text data, voice data, and image data to recognize the user's emotional state.

[0493] From the text data, emotions are identified based on the user's writing style and expression.

[0494] Emotions are analyzed from audio data based on fluctuations in tone and pitch.

[0495] Emotions are recognized from image data based on facial expressions and body movements.

[0496] Generating diagnostic results:

[0497] The server integrates the data analysis results and sentiment analysis results to generate a comprehensive diagnosis.

[0498] The diagnosis will include recommended treatment and medical consultation if necessary.

[0499] Product and service suggestions for physical stores are also generated based on the diagnostic results.

[0500] Sending and viewing diagnostic results:

[0501] The server transmits the generated diagnostic results to the terminal and the store equipment. The application on the terminal displays the diagnostic results to the user, and the store equipment also displays the diagnostic results to the store clerk.

[0502] Specific examples

[0503] For example, a customer visits a physical store and enters text such as, "I've been sleep-deprived these past few days due to work stress. My throat is a little sore," records the sound of a cough, and takes a photo of their face. The device then sends this text, audio, and image data to a server. The server analyzes the data and generates a diagnosis that also takes into account their emotional state. Based on the results, a suggestion is displayed, such as, "We think you may be overworked due to stress. We recommend some relaxing herbal tea or aroma oil to relieve stress."

[0504] Prompt Sentence Examples

[0505] Customer: "I've been sleep deprived these past few days due to work stress. My throat is a bit sore."

[0506] Audio Recording: [Coughing]

[0507] Headshot: [Customer photo]

[0508] Input data:

[0509] Text: "I've been sleep deprived these past few days due to work stress. My throat is a bit sore."

[0510] Audio data: Coughing sound

[0511] Image data: customer photos

[0512] Prompt statement:

[0513] "Analyze the text, voice, and image data entered by the customer and generate a diagnosis that takes into account their health status and emotions. The results should also include recommendations for appropriate products and services."

[0514] As described above, the present invention can comprehensively diagnose a user's health and emotional state, and can also suggest appropriate products and services to customers in physical stores.

[0515] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0516] Processing Steps

[0517] Step 1: Data entry

[0518] The user launches a dedicated application and inputs information about their symptoms and emotional state. The input includes text data, voice data, and image data, and is input using the smartphone's keyboard, microphone, and camera, respectively.

[0519] Input: Text, audio, and image data about symptoms and emotional states

[0520] Output: Input dataset in the device

[0521] Step 2: Send data

[0522] The device then packets the input data and sends it to the server using the HTTPS protocol. The data is then organized into text, audio, and image formats and securely sent to the server.

[0523] Input: Input dataset in the device

[0524] Output: Sending a data packet to the server

[0525] Step 3: Data reception and preprocessing

[0526] The server receives data packets sent from the device and separates them into text data, voice data, and image data. Each data type is passed to a different pre-processing module: text data is tokenized, voice data is denoised, and image data is formatted and adjusted.

[0527] Input: Data packet sent

[0528] Output: Separated text data, audio data, and image data

[0529] Step 4: Text data analysis

[0530] The server uses a Natural Language Processing (NLP) model to tokenize and analyze the text data, identifying key and secondary symptoms.

[0531] Input: Text data

[0532] Output: Symptom identification analysis results

[0533] Step 5: Audio data analysis

[0534] The voice processing module filters the voice data and performs noise reduction, then analyzes the cough pattern through a voice analysis model to analyze the user's symptom characteristics and emotional state.

[0535] Input: Denoised audio data

[0536] Output: Cough pattern analysis results

[0537] Step 6: Image data analysis

[0538] The image processing module preprocesses the image data and uses image analysis models to detect patterns of inflammation and swelling, as well as recognize the user's emotional state from facial expressions and body movements.

[0539] Input: Preprocessed image data

[0540] Output: Analysis of inflammation and swelling patterns and emotional state

[0541] Step 7: Sentiment Analysis

[0542] The server's emotion engine analyzes the user's emotional state from text data, voice data, and image data. From the text data, it analyzes emotions based on the user's expression, from voice data, it analyzes emotions based on tone and pitch fluctuations, and from image data, it analyzes emotions based on facial expressions and body movements.

[0543] Input: Text data, audio data, image data

[0544] Output: Overall emotional state analysis

[0545] Step 8: Generate diagnostic results

[0546] The server integrates the results of data analysis and emotion analysis to generate a comprehensive diagnosis that reflects estimated symptoms and emotional state, and also includes recommendations for appropriate products and services.

[0547] Input: Data analysis results and sentiment analysis results

[0548] Output: Comprehensive diagnostic results

[0549] Step 9: Send and view diagnostic results

[0550] The server sends the generated diagnostic results to the terminal and the store equipment. The terminal application displays the diagnostic results to the user, and the store equipment displays them to the store clerk. This allows the user and the store clerk to confirm the appropriate products and services.

[0551] Input: Comprehensive diagnostic results

[0552] Output: Diagnostic results and recommendations displayed to the user and the store clerk

[0553] The above are the specific processing steps of the system that realizes the application example.

[0554] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0555] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0556] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0557] [Second embodiment]

[0558] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0559] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0560] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0561] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0562] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0563] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0564] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0565] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0566] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0567] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0568] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0569] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0570] This system is designed to allow users to easily diagnose their own health condition from home or on the go. The system analyzes the text, voice, and image data entered by the user and provides a diagnosis result. Below, the program processing of this system is explained in natural language and shown in detail with concrete examples.

[0571] System configuration

[0572] User terminal

[0573] 1. Data input interface:

[0574] The user launches a dedicated application and inputs text data about the symptoms.

[0575] The user uses a microphone to record voice data such as coughing.

[0576] The user uses a camera to capture image data of the affected area.

[0577] 2. Data transmission function:

[0578] In order to transmit the input data to the server, the terminal packetizes the data and transmits it to the server using a secure communication protocol (for example, HTTPS).

[0579] server

[0580] 3. Data reception and preprocessing:

[0581] The server receives the data sent from the terminal and identifies the data type of each data.

[0582] The text data is first tokenized and then grammatically analyzed.

[0583] The audio data is soundproofed and analyzed based on specific frequencies.

[0584] The image data is filtered and preprocessed to convert it into a format that is easy to analyze.

[0585] 4. Data Analysis:

[0586] The server passes the preprocessed data to the AI ​​model for further analysis.

[0587] The main symptoms and related symptoms are extracted from the text data.

[0588] The audio data is analyzed for specific acoustic features, and the symptoms are characterized based on the results.

[0589] The image data recognizes specific patterns and identifies the presence or absence of inflammation or swelling.

[0590] 5. Generating diagnostic results:

[0591] The server integrates the analysis results and generates a diagnosis based on the user's symptoms.

[0592] Diagnostic findings may include recommended treatments and further medical consultations.

[0593] 6. Sending and viewing diagnostic results:

[0594] The server transmits the generated diagnosis results to the terminal.

[0595] The terminal displays the received diagnostic results to the user.

[0596] Specific examples

[0597] Diagnosing sore throat and cough

[0598] 1. Data Entry:

[0599] The user enters the text "I've had a persistent cough for the past three days and my throat hurts," then records and uploads an audio recording of themselves coughing.

[0600] 2. Data transmission:

[0601] The terminal converts text and audio files into packets and sends them to the server.

[0602] 3. Data reception and preprocessing:

[0603] The server receives the data, passes the text to an NLP model for analysis, and passes the audio data to an audio processing module for noise reduction.

[0604] 4. Data Analysis:

[0605] The server extracts key symptoms from the text and analyzes cough characteristics from the audio.

[0606] 5. Generating diagnostic results:

[0607] The server aggregates the results and generates a report with a diagnosis of "possible mild bronchitis" and recommended treatment.

[0608] 6. Sending and viewing diagnostic results:

[0609] The server sends the diagnostic report to the device.

[0610] The device displays the message, "You may have mild bronchitis. We recommend you drink plenty of fluids and get plenty of rest. If necessary, seek medical advice."

[0611] The above is a detailed embodiment of this system, which allows users to quickly and effectively check their own health status.

[0612] The processing flow will be explained below.

[0613] Step 1:

[0614] The user launches a dedicated application to describe their symptoms. In the text box, they enter, "I've had a persistent cough for the past three days and my throat hurts." They then use the recording function to record the sound of their cough and, if necessary, take an image of the affected area.

[0615] Step 2:

[0616] The device detects the input text data, voice data, and image data, and packetizes them for transmission to the server using the secure HTTPS communication protocol.

[0617] Step 3:

[0618] The server receives data packets sent from the terminal and separates them by data type: text data is sent to the text processing module, voice data to the voice processing module, and image data to the image processing module.

[0619] Step 4:

[0620] The server uses a Natural Language Processing (NLP) model to analyze the text data. The text data is first tokenized and then grammatically analyzed. The analysis results identify the primary symptom and related secondary symptoms.

[0621] Step 5:

[0622] The server pre-processes the audio data, applying noise reduction filters to bring the audio data into a clean format for analysis. Then, the audio analysis model extracts audio features and analyzes cough patterns.

[0623] Step 6:

[0624] The server preprocesses the image data, filtering and normalizing it, and then passes it to the image analysis model, which detects patterns of inflammation and swelling from the image data.

[0625] Step 7:

[0626] The server combines the results of the text, audio, and image analysis to generate a comprehensive diagnosis of the user's health, including recommended treatments and further medical consultations.

[0627] Step 8:

[0628] The server sends the generated diagnostic results to the device, where they are transmitted in a secure format and read by the device application.

[0629] Step 9:

[0630] The device displays the diagnosis results it received to the user. For example, it might say, "You may have mild bronchitis. We recommend you drink plenty of fluids and get plenty of rest. If necessary, consult a doctor."

[0631] Example 1

[0632] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0633] In modern society, users need to be able to easily diagnose their own health status from home or on the go. However, there is no system that can integrate and effectively analyze multiple data types (e.g., text data, voice data, and image data) to provide diagnostic results. Secure transmission of this data and accurate preprocessing are also challenges.

[0634] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0635] In this invention, the server includes means for a user to input text data, voice data, and image data, means for tokenizing the text data and performing grammatical analysis, means for denoising and processing the voice data, means for preprocessing the image data, means for passing the text data, voice data, and image data to an AI model for analysis, means for generating a diagnosis result based on the analysis result, means for displaying the diagnosis result to the user, means for transmitting the input data to the server using a secure communication protocol, and means for identifying the received data and preprocessing each data type, thereby enabling a user to quickly and effectively diagnose their health condition.

[0636] A "user" is an individual who utilizes the system to diagnose their own health condition.

[0637] "Text data" is data that represents character information entered by a user.

[0638] "Audio data" is data representing acoustic information recorded by a user.

[0639] "Image data" is data that represents visual information captured by a user.

[0640] "Tokenization" is the process of dividing text data into words and phrases.

[0641] "Grammar analysis" is a process of analyzing the grammatical structure of tokenized text data.

[0642] "Noise reduction" is a process of removing unnecessary background sounds from audio data.

[0643] "Preprocessing" is a series of steps to prepare data in a format that is easy to analyze.

[0644] An "AI model" is a system that uses artificial intelligence techniques to analyze data and recognize patterns.

[0645] "Analysis" is the process of examining data in detail and extracting its meaning and characteristics.

[0646] A "diagnostic result" is an assessment of health status or identification of symptoms generated based on the results of the analysis.

[0647] A "secure communication protocol" is a communication protocol for securely transmitting data.

[0648] A "server" is a computer that receives, preprocesses, and analyzes data, and generates and transmits diagnostic results.

[0649] A "user terminal" is a device through which a user inputs and transmits data.

[0650] The term "said" refers to something that has already been mentioned in the text.

[0651] A "means" is a device or process for achieving a particular function.

[0652] This invention is a system that allows users to easily diagnose their own health condition from home or on the go. In this system, users input text data, voice data, and image data, and generate diagnostic results by analyzing the data. Detailed embodiments of this system are described below.

[0653] System configuration

[0654] User terminal

[0655] 1. Data input interface:

[0656] The user launches a dedicated application and inputs text data about the symptoms.

[0657] The user uses a microphone to record voice data such as coughing.

[0658] The user uses a camera to capture image data of the affected area.

[0659] 2. Data transmission function:

[0660] The terminal packetizes the input data and transmits it to the server using a secure communication protocol (for example, HTTPS).

[0661] server

[0662] 3. Data reception and preprocessing:

[0663] The server receives the data sent from the terminal and identifies the data type of each data.

[0664] The text data is first tokenized and then grammatically analyzed.

[0665] The audio data is subjected to noise reduction and analyzed based on specific frequencies.

[0666] The image data is filtered and preprocessed to convert it into a format that is easy to analyze.

[0667] 4. Data Analysis:

[0668] The server passes the pre-processed data to a generative AI model for further analysis.

[0669] The main symptoms and related symptoms are extracted from the text data.

[0670] The audio data is analyzed for specific acoustic features, and the symptoms are characterized based on the results.

[0671] The image data recognizes specific patterns and identifies the presence or absence of inflammation or swelling.

[0672] 5. Generating diagnostic results:

[0673] The server integrates the analysis results and generates a diagnosis based on the user's symptoms.

[0674] Diagnostic findings may include recommended treatments and further medical consultations.

[0675] 6. Sending and viewing diagnostic results:

[0676] The server transmits the generated diagnosis results to the terminal.

[0677] The terminal displays the received diagnostic results to the user.

[0678] Specific examples

[0679] Diagnosing sore throat and cough

[0680] 1. Data Entry:

[0681] The user enters the text "I've had a persistent cough for the past three days and my throat hurts," then records and uploads an audio recording of themselves coughing.

[0682] 2. Data transmission:

[0683] The terminal converts text and audio files into packets and sends them to the server.

[0684] 3. Data reception and preprocessing:

[0685] The server receives the data, passes the text to a natural language processing (NLP) model for analysis, and passes the audio data to an audio processing module for noise reduction.

[0686] 4. Data Analysis:

[0687] The server extracts key symptoms from the text and analyzes cough characteristics from the audio.

[0688] 5. Generating diagnostic results:

[0689] The server aggregates the results and generates a report with a diagnosis of "possible mild bronchitis" and recommended treatment.

[0690] 6. Sending and viewing diagnostic results:

[0691] The server sends the diagnostic report to the device.

[0692] The device displays the message, "You may have mild bronchitis. We recommend you drink plenty of fluids and get plenty of rest. If necessary, seek medical advice."

[0693] The above is a detailed embodiment of this system, which allows users to quickly and effectively check their own health status.

[0694] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0695] Step 1:

[0696] Use of a data entry interface

[0697] User:

[0698] Start the dedicated application.

[0699] In the application's text entry field, enter your symptoms: "I've had a persistent cough and sore throat for the past three days."

[0700] Use the application's voice recording feature to record the sound of your cough using your microphone.

[0701] Use the application's camera function to take a picture of your throat, for example if you have a sore throat.

[0702] Input: Text data, audio data, and image data about the user's symptoms

[0703] Output: Symptom data stored within the application

[0704] Step 2:

[0705] Data transmission

[0706] Device:

[0707] Text data entered by the user, recorded voice data, and captured image data are packetized.

[0708] This data is sent to the server using a secure communication protocol such as HTTPS.

[0709] Input: Text data, voice data, and image data entered by the user

[0710] Output: Packetized data sent to the server

[0711] Step 3:

[0712] Data reception and preprocessing

[0713] server:

[0714] Receives data sent from the device and identifies the data type (text, audio, image).

[0715] The text data is first tokenized and then grammatically analyzed using natural language processing (NLP) techniques.

[0716] For example, extract key keywords such as "cough" and "sore throat."

[0717] The audio data is subjected to noise removal processing and specific frequencies are analyzed to extract cough patterns and characteristics.

[0718] For example, the intensity and frequency of coughing sounds are analyzed.

[0719] The image data is subjected to filtering and pre-processing to improve the image quality.

[0720] A Gaussian filter is used for filtering, and image resolution is adjusted for preprocessing.

[0721] Input: Text data, audio data, and image data sent from the device

[0722] Output: Preprocessed text data, audio data, and image data

[0723] Step 4:

[0724] Data analysis

[0725] server:

[0726] Preprocessed text data, audio data, and image data are passed to a generative AI model.

[0727] A generative AI model analyzes each piece of data in detail.

[0728] From the text data, major symptoms such as "last three days," "cough," and "sore throat" are detected.

[0729] Cough characteristics (e.g., frequency, intensity) are analyzed from the audio data.

[0730] The image data is used to evaluate the condition of the throat and determine whether inflammation is present.

[0731] Input: Preprocessed text data, audio data, and image data

[0732] Output: Analysis results for each data

[0733] Step 5:

[0734] Generating diagnostic results

[0735] server:

[0736] The results of each data analysis are integrated to generate a comprehensive diagnostic result.

[0737] For example, it generates a diagnosis such as "You may have mild bronchitis" and compiles a report with recommended treatment methods.

[0738] Input: Analysis results of text data, audio data, and image data

[0739] Output: A report containing the diagnosis and recommended treatment.

[0740] Step 6:

[0741] Sending and displaying diagnostic results

[0742] server:

[0743] The generated diagnostic results are transmitted to the terminal via secure communication.

[0744] Device:

[0745] The received diagnostic results are displayed to the user.

[0746] For example, it might say, "You may have mild bronchitis. We recommend you drink plenty of fluids and get plenty of rest. If necessary, seek medical advice."

[0747] Input: Diagnostic results sent from the server

[0748] Output: Diagnostic results displayed on the user's terminal

[0749] (Application example 1)

[0750] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0751] In recent years, there has been an increasing need for systems that allow individuals to easily diagnose their health conditions from home or on the go. However, there is a lack of systems that can comprehensively analyze text data, voice data, and image data to provide highly reliable diagnoses. There is also a need for systems that can efficiently process user-entered data and quickly display diagnostic results. A technical method is needed to solve these issues.

[0752] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0753] In this invention, the server includes means for a user to input text data, voice data, and image data, means for tokenizing the text data and performing grammatical analysis, means for removing noise and processing the voice data, means for preprocessing the image data, means for generating a diagnosis result based on the analysis results of the text data, voice data, and image data, means for displaying the diagnosis result to the user, means for collecting data from a smartphone terminal and transmitting it to the server, and means for generating a diagnosis result based on a prompt sentence using a generative AI model, thereby enabling a user to quickly and accurately diagnose their own health condition from home or on the go.

[0754] A "user terminal" is an electronic device that allows a user to input, send, and receive text data, voice data, and image data.

[0755] "Text data" refers to text information entered by a user, and is data for performing natural language processing.

[0756] "Audio data" refers to audio information recorded by a user, and is data for performing acoustic analysis and noise removal processes.

[0757] "Image data" is visual information captured by a user using a camera, and is data for preprocessing.

[0758] "Tokenization" is a process of dividing text data into small units, and is a preprocessing step for grammatical analysis.

[0759] "Grammar analysis" is a process of analyzing the grammatical structure of tokenized text data.

[0760] "Noise reduction" is a process that removes unnecessary sounds such as background noise from audio data.

[0761] "Preprocessing" is a process for converting image data into a format that is easy to analyze.

[0762] The "diagnosis result" is estimated information about the health condition that is generated based on the analyzed text data, voice data, and image data.

[0763] The "means for displaying to the user" is a function or device for providing the diagnostic results to the user in a format that is easy to view.

[0764] A "smartphone terminal" is a small, portable electronic device that is used to collect and transmit user data.

[0765] A "server" is a computer system that receives, analyzes, and transmits data over a network.

[0766] A "generative AI model" is an artificial intelligence model that automatically generates diagnostic results based on input data.

[0767] A "prompt sentence" is an instruction sentence used as input to a generative AI model.

[0768] This system is designed to allow users to easily diagnose their own health condition from home or on the go. It analyzes text, voice, and image data entered by the user and provides diagnostic results. The components of the system and specific processing steps are shown below.

[0769] System configuration

[0770] User terminal

[0771] The user terminal includes a portable device such as a smartphone or tablet. The user inputs the following data through the application:

[0772] 1. Text data input: The user inputs information about symptoms and physical condition in text format.

[0773] 2. Recording voice data: The user records coughing, breathing, etc. and inputs it as voice data.

[0774] 3. Capturing image data: The user uses a camera to capture an image of the affected area.

[0775] server

[0776] The server is a central computing device that receives and analyzes data sent from the terminals. The server is configured as follows:

[0777] 1. Data reception: Receives text data, voice data, and image data sent from the user terminal.

[0778] 2. Preprocessing: The received data is processed by various preprocessing modules. Text data is tokenized and grammatically analyzed, audio data is denoised, and image data is filtered and format converted.

[0779] 3. Data analysis: The preprocessed data is then further analyzed by generative AI models: text data is analyzed for key symptoms and related symptoms, audio data is analyzed for specific acoustic features, and image data is analyzed for specific patterns.

[0780] 4. Generating a diagnosis: The analysis results are combined to generate a diagnosis based on the user's symptoms, which may include recommended treatments or further medical consultations.

[0781] 5. Sending the diagnosis result: Send the generated diagnosis result to the user terminal.

[0782] Hardware and software used

[0783] User devices: smartphones, tablets

[0784] Speech processing software: speech_recognition library

[0785] Image processing software: OpenCV library

[0786] Sending data: the requests library

[0787] Generative AI models: various AI diagnostic models, such as those provided by the transformers library

[0788] Specific examples

[0789] The user launches a smartphone application, enters the text "I've had a persistent cough for the past three days and my throat hurts," records the coughing sound, and takes an image of their throat. The user's device sends this data to the server. The server receives the data, performs grammatical analysis on the text data, removes noise from the audio data, and filters the image data. It then analyzes the data based on the generative AI model and generates a diagnosis: "You may have mild bronchitis. We recommend you drink plenty of fluids and rest. If necessary, consult a doctor." This is then sent to the user's device.

[0790] Prompt Sentence Examples

[0791] User input: "I've had a constant cough and sore throat for the past three days."

[0792] Audio Data: [Recorded coughing sound]

[0793] Image Data: [User-taken image of throat]

[0794] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0795] Step 1:

[0796] A user starts the application using a smartphone and inputs text, voice, and image data. Specifically, the user inputs text such as "I've had a persistent cough for the past three days and my throat hurts," records the sound of their cough, and takes an image of their throat.

[0797] Input: Text data, audio data, image data

[0798] Output: Input text, recorded audio files, captured image files

[0799] Step 2:

[0800] The device prepares the input data and sends it to the server using a secure communication protocol (e.g., HTTPS). The data is packetized as text, audio files, or image files.

[0801] Input: text, audio files, image files

[0802] Output: Packetized data

[0803] Step 3:

[0804] The server receives packetized data sent from the terminal, identifies the received data, and initiates appropriate pre-processing for each data type.

[0805] Input: Packetized data

[0806] Output: Received text data, audio data, and image data

[0807] Step 4:

[0808] The server tokenizes the received text data and performs grammatical analysis, extracting the grammatical structure and semantic features of the text data, for example, identifying key symptoms such as "cough" or "sore throat."

[0809] Input: Text data

[0810] Output: Tokenized text, grammar analysis results

[0811] Step 5:

[0812] The server processes the audio data through noise reduction and analyzes its acoustic features to identify coughs and other distinctive acoustic patterns, such as the frequency range and duration of a cough.

[0813] Input: Audio data

[0814] Output: Noise-removed audio data, acoustic analysis results

[0815] Step 6:

[0816] The server performs preprocessing on the image data, such as filtering and format conversion, to convert the image data into a format that is easier to analyze. For example, contrast adjustment and edge detection are performed on the throat image.

[0817] Input: Image data

[0818] Output: Preprocessed image data

[0819] Step 7:

[0820] The server passes the preprocessed text, audio, and image data to the generative AI model for detailed data analysis, such as analyzing the prompt sentences and generating a comprehensive diagnosis.

[0821] Input: Tokenized text, acoustic analysis results, preprocessed image data

[0822] Output: Diagnostic results

[0823] Step 8:

[0824] The server integrates the generated diagnostic results and generates appropriate feedback to the user, including recommended treatments and next steps (e.g., consulting a medical professional).

[0825] Input: Diagnostic results

[0826] Output: Consolidated diagnostic report

[0827] Step 9:

[0828] The server sends the diagnosis results to the user's device, which then displays the received diagnosis results in an easy-to-read format. For example, it might say, "You may have mild bronchitis. We recommend you drink plenty of fluids and get plenty of rest."

[0829] Input: Consolidated diagnostic report

[0830] Output: Diagnostic results displayed to the user

[0831] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0832] This invention combines a system that allows users to diagnose their own health condition using text data, voice data, and image data with an emotion engine that recognizes the user's emotions, thereby providing comprehensive diagnostic results that take the user's emotional state into consideration.

[0833] System configuration

[0834] User terminal

[0835] 1. Data input interface:

[0836] The user launches a dedicated application and inputs text data about the symptoms.

[0837] The user uses a microphone to record audio data such as coughing, and a camera to take an image of the affected area.

[0838] 2. Data transmission function:

[0839] The terminal converts the input text data, voice data, and image data into packets and transmits them to the server using a secure communication protocol (for example, HTTPS).

[0840] server

[0841] 3. Data reception and preprocessing:

[0842] The server receives data packets sent from the terminal and separates them by data type: text data is sent to the text processing module, voice data is sent to the voice processing module, and image data is sent to the image processing module.

[0843] 4. Data Analysis:

[0844] The text data is tokenized and grammatically analyzed using a natural language processing (NLP) model to identify primary symptoms and associated secondary symptoms.

[0845] The audio data is denoised and filtered, and then the cough pattern is analysed through a speech analysis model.

[0846] The image data is preprocessed and an image analysis model is used to detect patterns of inflammation and swelling.

[0847] 5. Emotion analysis:

[0848] The server's emotion engine analyzes text data, voice data, and image data to recognize the user's emotional state.

[0849] From the text data, emotions are identified based on the user's writing style and expression.

[0850] Emotions are analyzed from audio data based on fluctuations in tone and pitch.

[0851] Emotions are recognized from image data based on facial expressions and body movements.

[0852] 6. Generating diagnostic results:

[0853] The server integrates the data analysis results with the emotion analysis results to generate a comprehensive diagnosis, which also reflects the user's emotional state.

[0854] The diagnosis will include recommended treatment and medical consultation if necessary.

[0855] 7. Sending and viewing diagnostic results:

[0856] The server sends the generated diagnostic results to the terminal, where an application displays the diagnostic results to the user.

[0857] Specific examples

[0858] Diagnosis by combining sore throat, cough, and emotion analysis

[0859] 1. Data Entry:

[0860] The user enters the text "I've had a persistent cough for the past three days and my throat hurts," records and sends an audio recording of the cough, and also takes a photo of their face.

[0861] 2. Data transmission:

[0862] The device sends text, audio, and image files to the server.

[0863] 3. Data reception and preprocessing:

[0864] The server receives the data, analyzes the text using NLP models, denoises the audio data, and preprocesses the image data.

[0865] 4. Data Analysis:

[0866] It extracts key symptoms from text, analyzes cough characteristics from audio, and detects patterns of inflammation and swelling from images.

[0867] 5. Emotion analysis:

[0868] The server uses an emotion engine to analyze stress and anxiety from text, tension from audio, and sadness and fatigue from images.

[0869] 6. Generating diagnostic results:

[0870] The server combines the results of the data analysis and the emotion analysis and generates a diagnosis that reads, "You may have mild bronchitis, but you appear to be under stress. We recommend that you drink plenty of fluids and get plenty of rest. Also, if you need psychological support, please consult a specialist."

[0871] 7. Sending and viewing diagnostic results:

[0872] The server sends the diagnostic results to the terminal, which displays them to the user.

[0873] The above is a specific embodiment of the present invention. By using this system, users can receive diagnostic results that take into account their physical and emotional state comprehensively, and obtain appropriate treatment and countermeasures.

[0874] The processing flow will be explained below.

[0875] Step 1:

[0876] The user starts a dedicated symptom recording application and enters text data about the symptoms. For example, the user might write "I've had a persistent cough and a sore throat for the past three days" in the text box.

[0877] Step 2:

[0878] The user uses a microphone to record the sound of their cough, as well as take a photo of their face and the affected area, all of which is uploaded to the application.

[0879] Step 3:

[0880] The device combines the input text, voice, and image data into a single data packet, which is then sent to the server using a secure communication protocol (e.g., HTTPS).

[0881] Step 4:

[0882] The server receives data packets sent from the device and separates them by data type (text, audio, image), and passes each data to the corresponding analysis module.

[0883] Step 5:

[0884] The server passes the text data to a natural language processing (NLP) model for tokenization and grammatical analysis, which identifies the primary symptom and associated secondary symptoms.

[0885] Step 6:

[0886] The server passes the audio data to a speech processing module for noise reduction and filtering, and then a speech analysis model analyzes the audio characteristics (e.g., coughing patterns).

[0887] Step 7:

[0888] The server passes the image data to the image processing module for preprocessing (filtering, normalization), after which an image analysis model detects patterns of inflammation and swelling.

[0889] Step 8:

[0890] The server passes the analyzed text data to an emotion engine to identify the user's emotions. For example, if there are many negative expressions, it can detect stress or anxiety.

[0891] Step 9:

[0892] The server also analyzes emotions from the voice data, identifying emotions such as tension or impatience based on the tone and pitch of the voice.

[0893] Step 10:

[0894] The server uses the image data to analyze the user's facial expressions and physical state to identify emotions, such as sadness or fatigue from facial distortion or puffy eyes.

[0895] Step 11:

[0896] The server integrates the data obtained from each analysis module (text, audio, image analysis results, and emotion analysis results) to generate a comprehensive diagnosis that also reflects the user's emotional state.

[0897] Step 12:

[0898] The server sends the generated diagnostic results to the device, where they are transmitted in a secure format and displayed in the device application.

[0899] Step 13:

[0900] The device will display the diagnosis results it receives to the user, such as "You may have mild bronchitis, but you appear to be under stress. We recommend you drink plenty of fluids and get plenty of rest. Also, if you need psychological support, please consult a specialist."

[0901] Example 2

[0902] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0903] Conventional health checkup systems provide diagnostic results based on the analysis of text data, voice data, and image data collected from users, but do not take the user's emotional state into consideration. As a result, comprehensive health diagnosis is insufficient and advice on the user's emotional health is lacking. The present invention aims to provide more accurate comprehensive diagnostic results by analyzing data including the user's emotional state.

[0904] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0905] In this invention, the server includes a means for tokenizing and syntactically analyzing text data, a means for denoising and processing audio data, a means for preprocessing image data, and a means for analyzing emotional states, thereby enabling the generation and display of comprehensive diagnostic results based on the analysis results of the text data, audio data, and image data, as well as the emotional states.

[0906] "User" refers to a person who uses the system to input data to diagnose their own health condition.

[0907] "Text data" refers to character information such as symptoms and emotions entered by the user.

[0908] "Audio data" refers to acoustic information such as coughs and voices recorded by a user.

[0909] "Image data" refers to image information such as the affected area and facial expression photographed by the user.

[0910] "Tokenization" refers to the process of breaking down text data into smaller units such as words or phrases.

[0911] "Syntax analysis" refers to the process of analyzing the grammatical structure of tokenized text data.

[0912] "Noise reduction" refers to the process of removing unnecessary noise from audio data.

[0913] "Preprocessing" refers to the process of formatting data to make it easier to analyze.

[0914] "Diagnostic results" refer to health assessments and recommended treatments generated based on data and sentiment analysis.

[0915] "Emotional state" refers to a user's mental state, such as stress, anxiety, tension, sadness, etc.

[0916] "Comprehensive diagnostic results" refer to diagnostic information obtained by taking into account both physical symptoms and emotional state.

[0917] "Means for displaying" refers to a method or device for delivering the generated diagnostic results to a user.

[0918] MODE FOR CARRYING OUT THE INVENTION

[0919] The present invention is a system that allows users to diagnose their own health condition using text data, voice data, and image data, and also provides comprehensive diagnostic results that take into account the user's emotional state. This system is composed of a user terminal and a server.

[0920] User terminal

[0921] 1. Data Entry Interface

[0922] The user launches a dedicated application and enters text data about their symptoms. For example, they can enter, "I've had a persistent cough and a sore throat for the past three days."

[0923] The user uses a microphone to record audio data such as coughing, and a camera to take an image of the affected area.

[0924] 2. Data transmission function

[0925] The terminal converts the input text data, voice data, and image data into packets and transmits them to the server using a secure communication protocol (for example, HTTPS).

[0926] server

[0927] 3. Data Reception and Preprocessing

[0928] The server receives data packets sent from the terminal and separates them by data type (passing text data to the text processing module, voice data to the voice processing module, and image data to the image processing module).

[0929] The main technologies used in this case include NLP (Natural Language Processing) modules, voice analysis models, and image analysis models.

[0930] 4. Data Analysis

[0931] Text data analysis: The server uses NLP models to tokenize the text data and perform grammatical analysis, which identifies primary and secondary symptoms, such as "cough" and "sore throat."

[0932] Voice data analysis: The server performs noise reduction and filtering on the voice data, and then analyzes the cough pattern using a voice analysis model.

[0933] Image data analysis: The server preprocesses the image data and uses image analysis models to detect patterns of inflammation and swelling.

[0934] 5. Emotion analysis

[0935] The server's emotion engine analyzes text, voice, and image data to recognize the user's emotional state.

[0936] Sentiment analysis from text: Identify emotions based on the user's writing style and expressions. For example, analyze stress from the emphasis on a period such as "3 days" in the text.

[0937] Emotion analysis from audio: Analyzes tone and pitch fluctuations in audio data to detect tension.

[0938] Emotion analysis from images: Analyzing facial expressions and body movements to recognize feelings of sadness or fatigue.

[0939] 6. Generating diagnostic results

[0940] The server integrates the data analysis results with the emotion analysis results to generate a comprehensive diagnosis, which also reflects the user's emotional state.

[0941] For example, it generates a diagnosis such as, "You may have mild bronchitis, but you appear to be under stress. We recommend you drink plenty of fluids and get plenty of rest. Also, if you need psychological support, consult a specialist."

[0942] 7. Sending and displaying diagnostic results

[0943] The server transmits the generated diagnostic results to the terminal, and an application on the terminal displays the diagnostic results to the user.

[0944] This system allows users to obtain comprehensive health checkup results that take into account not only their physical symptoms but also their emotional state.

[0945] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0946] Program processing flow

[0947] Step 1: User Data Entry

[0948] Using a dedicated application, users input text data about their symptoms. For example, they can describe specific symptoms such as, "I've had a persistent cough for the past three days, and my throat hurts." In addition, the system collects audio and image data by recording the sound of the cough using a microphone and taking images of the affected area using a camera.

[0949] Input: Text data describing symptoms, recorded voice data, photographed image data

[0950] Output: Collected dataset (text data, audio data, image data)

[0951] Step 2: Data collection and transmission by the device

[0952] The device collects text data entered by the user, recorded voice data, and captured image data, then encrypts this data using the HTTPS protocol and securely transmits it to a server.

[0953] Input: Collected datasets (text data, audio data, image data)

[0954] Output: Dataset sent securely to server

[0955] Step 3: Data reception and preprocessing by the server

[0956] The server receives data packets sent from the device. The received data is first classified into text data, voice data, and image data. The text data is then passed to the NLP module, the voice data to the voice analysis module, and the image data to the image analysis module.

[0957] Input: Securely transmitted dataset

[0958] Output: Classified dataset (text data, audio data, image data)

[0959] Step 4: Analyzing the text data

[0960] The server uses an NLP model to tokenize and parse the received text data, extracting key symptoms such as "cough" or "sore throat."

[0961] Input: Text data

[0962] Output: Extracted main and secondary symptoms

[0963] Step 5: Analyzing the audio data

[0964] The server denoises and filters the audio data, then uses a speech analysis model to analyze cough patterns.

[0965] Input: Audio data

[0966] Output: Analyzed cough pattern

[0967] Step 6: Analyzing the image data

[0968] The server preprocesses the image data and uses image analysis models to detect patterns of inflammation and swelling.

[0969] Input: Image data

[0970] Output: Detected inflammation and swelling patterns

[0971] Step 7: Sentiment Analysis

[0972] The server's emotion engine analyzes text data, voice data, and image data to recognize the user's emotional state. Specifically, it identifies emotions from text data based on writing style and expression, analyzes tone and pitch fluctuations from voice data, and analyzes facial expressions and body movements from image data.

[0973] Input: text data, audio data, image data

[0974] Output: User's emotional state

[0975] Step 8: Generate comprehensive diagnostic results

[0976] The server combines the data analysis results with the sentiment analysis results to generate a comprehensive diagnosis. This diagnosis includes recommended treatment and, if necessary, consultation with a medical institution. For example, a diagnosis might read, "You may have mild bronchitis, but you appear to be under stress. We recommend you drink plenty of fluids and rest. We also recommend consulting a specialist if you need psychological support."

[0977] Input: Data analysis results, emotion analysis results

[0978] Output: Comprehensive diagnostic results

[0979] Step 9: Send and view diagnostic results

[0980] The server sends the generated diagnostic results to the terminal, and the terminal application displays the received diagnostic results to the user.

[0981] Input: Comprehensive diagnostic results

[0982] Output: Diagnostic results displayed to the user

[0983] Through the above processing steps, the user can receive a diagnosis result that takes into consideration the user's physical and emotional state comprehensively.

[0984] (Application example 2)

[0985] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0986] Conventional health checkup systems typically analyze symptoms and make diagnoses based on text data, voice data, and image data entered by the user. However, these systems were unable to consider the user's emotional state, making it difficult to provide comprehensive diagnostic results. Furthermore, it was difficult to propose services in physical stores that took into account the customer's health and emotional state, making improving customer satisfaction a challenge. Therefore, there was a need for a system that could analyze not only the user's health condition but also their emotional state, reflect this in the diagnostic results, and then propose appropriate products and services in physical stores.

[0987] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0988] In this invention, the server includes means for a user to input text data, voice data, and image data, means for tokenizing the text data and performing grammatical analysis, means for removing noise and processing the voice data, means for preprocessing the image data, means for generating a diagnostic result based on the analysis results of the text data, voice data, and image data, means for displaying the diagnostic result to the user, means for proposing products and services in a physical store, and means for performing emotion analysis on the text data, voice data, and image data and reflecting the emotion in the diagnostic result. This makes it possible to provide a comprehensive diagnostic result based on the user's health condition and emotional state, and to propose appropriate products and services in a physical store based on the result.

[0989] "User" refers to a person who uses the system to diagnose a health or emotional state.

[0990] "Text data" refers to data that includes character information entered by a user.

[0991] "Voice data" refers to data containing voice information input by a user.

[0992] "Image data" refers to data that includes information about an image captured by a user.

[0993] "Tokenization" refers to the process of dividing text data into smaller units such as words or phrases.

[0994] "Grammar analysis" refers to the process of analyzing the syntax and meaning of text data.

[0995] "Noise reduction" refers to the process of removing unnecessary background sounds and noise from audio data.

[0996] "Preprocessing" refers to the process of shaping or adjusting image data before analysis.

[0997] "Diagnosis results" refer to the evaluation results of the user's health condition and emotional state generated based on the analysis results of text data, audio data, and image data.

[0998] "Display means" refers to a device or method for displaying the diagnostic results to the user or store clerk.

[0999] "Brick and mortar store" refers to a physical location that offers goods and services.

[1000] "Product and service suggestions" refers to suggesting appropriate products and services based on the diagnostic results.

[1001] "Emotion analysis" refers to the process of analyzing a user's emotional state from text data, audio data, and image data.

[1002] This will help the reader to understand the invention more clearly.

[1003] MODE FOR CARRYING OUT THE INVENTION

[1004] The present invention is a system for analyzing the health and emotional state of customers in a physical store and recommending appropriate products and services based on the results. The system includes a user terminal, a server, and a store device that displays the summary results.

[1005] User terminal

[1006] The user terminal has the following functions:

[1007] Data Input Interface:

[1008] The user launches a dedicated application and inputs text data about the symptoms.

[1009] The user uses a microphone to record audio data such as coughing, and a camera to take an image of the affected area.

[1010] Data transmission function:

[1011] The terminal converts the input text data, voice data, and image data into packets and transmits them to the server using a secure communication protocol (for example, HTTPS).

[1012] server

[1013] The server has the following functions:

[1014] Data reception and preprocessing:

[1015] The server receives data packets sent from the terminal and separates them by data type: text data is sent to the text processing module, voice data is sent to the voice processing module, and image data is sent to the image processing module.

[1016] Data Analysis:

[1017] The text data is tokenized and grammatically analyzed using a natural language processing (NLP) model to identify primary symptoms and associated secondary symptoms.

[1018] The audio data is denoised and filtered, and then the cough pattern is analysed through a speech analysis model.

[1019] The image data is preprocessed and an image analysis model is used to detect patterns of inflammation and swelling.

[1020] Emotion analysis:

[1021] The server's emotion engine analyzes text data, voice data, and image data to recognize the user's emotional state.

[1022] From the text data, emotions are identified based on the user's writing style and expression.

[1023] Emotions are analyzed from audio data based on fluctuations in tone and pitch.

[1024] Emotions are recognized from image data based on facial expressions and body movements.

[1025] Generating diagnostic results:

[1026] The server integrates the data analysis results and sentiment analysis results to generate a comprehensive diagnosis.

[1027] The diagnosis will include recommended treatment and medical consultation if necessary.

[1028] Product and service suggestions for physical stores are also generated based on the diagnostic results.

[1029] Sending and viewing diagnostic results:

[1030] The server transmits the generated diagnostic results to the terminal and the store equipment. The application on the terminal displays the diagnostic results to the user, and the store equipment also displays the diagnostic results to the store clerk.

[1031] Specific examples

[1032] For example, a customer visits a physical store and enters text such as, "I've been sleep-deprived these past few days due to work stress. My throat is a little sore," records the sound of a cough, and takes a photo of their face. The device then sends this text, audio, and image data to a server. The server analyzes the data and generates a diagnosis that also takes into account their emotional state. Based on the results, a suggestion is displayed, such as, "We think you may be overworked due to stress. We recommend some relaxing herbal tea or aroma oil to relieve stress."

[1033] Prompt Sentence Examples

[1034] Customer: "I've been sleep deprived these past few days due to work stress. My throat is a bit sore."

[1035] Audio Recording: [Coughing]

[1036] Headshot: [Customer photo]

[1037] Input data:

[1038] Text: "I've been sleep deprived these past few days due to work stress. My throat is a bit sore."

[1039] Audio data: Coughing sound

[1040] Image data: customer photos

[1041] Prompt statement:

[1042] "Analyze the text, voice, and image data entered by the customer and generate a diagnosis that takes into account their health status and emotions. The results should also include recommendations for appropriate products and services."

[1043] As described above, the present invention can comprehensively diagnose a user's health and emotional state, and can also suggest appropriate products and services to customers in physical stores.

[1044] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1045] Processing Steps

[1046] Step 1: Data entry

[1047] The user launches a dedicated application and inputs information about their symptoms and emotional state. The input includes text data, voice data, and image data, and is input using the smartphone's keyboard, microphone, and camera, respectively.

[1048] Input: Text, audio, and image data about symptoms and emotional states

[1049] Output: Input dataset in the device

[1050] Step 2: Send data

[1051] The device then packets the input data and sends it to the server using the HTTPS protocol. The data is then organized into text, audio, and image formats and securely sent to the server.

[1052] Input: Input dataset in the device

[1053] Output: Sending a data packet to the server

[1054] Step 3: Data reception and preprocessing

[1055] The server receives data packets sent from the device and separates them into text data, voice data, and image data. Each data type is passed to a different pre-processing module: text data is tokenized, voice data is denoised, and image data is formatted and adjusted.

[1056] Input: Data packet sent

[1057] Output: Separated text data, audio data, and image data

[1058] Step 4: Text data analysis

[1059] The server uses a Natural Language Processing (NLP) model to tokenize and analyze the text data, identifying key and secondary symptoms.

[1060] Input: Text data

[1061] Output: Symptom identification analysis results

[1062] Step 5: Audio data analysis

[1063] The voice processing module filters the voice data and performs noise reduction, then analyzes the cough pattern through a voice analysis model to analyze the user's symptom characteristics and emotional state.

[1064] Input: Denoised audio data

[1065] Output: Cough pattern analysis results

[1066] Step 6: Image data analysis

[1067] The image processing module preprocesses the image data and uses image analysis models to detect patterns of inflammation and swelling, as well as recognize the user's emotional state from facial expressions and body movements.

[1068] Input: Preprocessed image data

[1069] Output: Analysis of inflammation and swelling patterns and emotional state

[1070] Step 7: Sentiment Analysis

[1071] The server's emotion engine analyzes the user's emotional state from text data, voice data, and image data. From the text data, it analyzes emotions based on the user's expression, from voice data, it analyzes emotions based on tone and pitch fluctuations, and from image data, it analyzes emotions based on facial expressions and body movements.

[1072] Input: Text data, audio data, image data

[1073] Output: Overall emotional state analysis

[1074] Step 8: Generate diagnostic results

[1075] The server integrates the results of data analysis and emotion analysis to generate a comprehensive diagnosis that reflects estimated symptoms and emotional state, and also includes recommendations for appropriate products and services.

[1076] Input: Data analysis results and sentiment analysis results

[1077] Output: Comprehensive diagnostic results

[1078] Step 9: Send and view diagnostic results

[1079] The server sends the generated diagnostic results to the terminal and the store equipment. The terminal application displays the diagnostic results to the user, and the store equipment displays them to the store clerk. This allows the user and the store clerk to confirm the appropriate products and services.

[1080] Input: Comprehensive diagnostic results

[1081] Output: Diagnostic results and recommendations displayed to the user and the store clerk

[1082] The above are the specific processing steps of the system that realizes the application example.

[1083] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1084] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1085] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1086] [Third embodiment]

[1087] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1088] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1089] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1090] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1091] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1092] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1093] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1094] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1095] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1096] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1097] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1098] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1099] This system is designed to allow users to easily diagnose their own health condition from home or on the go. The system analyzes the text, voice, and image data entered by the user and provides a diagnosis result. Below, the program processing of this system is explained in natural language and shown in detail with concrete examples.

[1100] System configuration

[1101] User terminal

[1102] 1. Data input interface:

[1103] The user launches a dedicated application and inputs text data about the symptoms.

[1104] The user uses a microphone to record voice data such as coughing.

[1105] The user uses a camera to capture image data of the affected area.

[1106] 2. Data transmission function:

[1107] In order to transmit the input data to the server, the terminal packetizes the data and transmits it to the server using a secure communication protocol (for example, HTTPS).

[1108] server

[1109] 3. Data reception and preprocessing:

[1110] The server receives the data sent from the terminal and identifies the data type of each data.

[1111] The text data is first tokenized and then grammatically analyzed.

[1112] The audio data is soundproofed and analyzed based on specific frequencies.

[1113] The image data is filtered and preprocessed to convert it into a format that is easy to analyze.

[1114] 4. Data Analysis:

[1115] The server passes the preprocessed data to the AI ​​model for further analysis.

[1116] The main symptoms and related symptoms are extracted from the text data.

[1117] The audio data is analyzed for specific acoustic features, and the symptoms are characterized based on the results.

[1118] The image data recognizes specific patterns and identifies the presence or absence of inflammation or swelling.

[1119] 5. Generating diagnostic results:

[1120] The server integrates the analysis results and generates a diagnosis based on the user's symptoms.

[1121] Diagnostic findings may include recommended treatments and further medical consultations.

[1122] 6. Sending and viewing diagnostic results:

[1123] The server transmits the generated diagnosis results to the terminal.

[1124] The terminal displays the received diagnostic results to the user.

[1125] Specific examples

[1126] Diagnosing sore throat and cough

[1127] 1. Data Entry:

[1128] The user enters the text "I've had a persistent cough for the past three days and my throat hurts," then records and uploads an audio recording of themselves coughing.

[1129] 2. Data transmission:

[1130] The terminal converts text and audio files into packets and sends them to the server.

[1131] 3. Data reception and preprocessing:

[1132] The server receives the data, passes the text to an NLP model for analysis, and passes the audio data to an audio processing module for noise reduction.

[1133] 4. Data Analysis:

[1134] The server extracts key symptoms from the text and analyzes cough characteristics from the audio.

[1135] 5. Generating diagnostic results:

[1136] The server aggregates the results and generates a report with a diagnosis of "possible mild bronchitis" and recommended treatment.

[1137] 6. Sending and viewing diagnostic results:

[1138] The server sends the diagnostic report to the device.

[1139] The device displays the message, "You may have mild bronchitis. We recommend you drink plenty of fluids and get plenty of rest. If necessary, seek medical advice."

[1140] The above is a detailed embodiment of this system, which allows users to quickly and effectively check their own health status.

[1141] The processing flow will be explained below.

[1142] Step 1:

[1143] The user launches a dedicated application to describe their symptoms. In the text box, they enter, "I've had a persistent cough for the past three days and my throat hurts." They then use the recording function to record the sound of their cough and, if necessary, take an image of the affected area.

[1144] Step 2:

[1145] The device detects the input text data, voice data, and image data, and packetizes them for transmission to the server using the secure HTTPS communication protocol.

[1146] Step 3:

[1147] The server receives data packets sent from the terminal and separates them by data type: text data is sent to the text processing module, voice data to the voice processing module, and image data to the image processing module.

[1148] Step 4:

[1149] The server uses a Natural Language Processing (NLP) model to analyze the text data. The text data is first tokenized and then grammatically analyzed. The analysis results identify the primary symptom and related secondary symptoms.

[1150] Step 5:

[1151] The server pre-processes the audio data, applying noise reduction filters to bring the audio data into a clean format for analysis. Then, the audio analysis model extracts audio features and analyzes cough patterns.

[1152] Step 6:

[1153] The server preprocesses the image data, filtering and normalizing it, and then passes it to the image analysis model, which detects patterns of inflammation and swelling from the image data.

[1154] Step 7:

[1155] The server combines the results of the text, audio, and image analysis to generate a comprehensive diagnosis of the user's health, including recommended treatments and further medical consultations.

[1156] Step 8:

[1157] The server sends the generated diagnostic results to the device, where they are transmitted in a secure format and read by the device application.

[1158] Step 9:

[1159] The device displays the diagnosis results it received to the user. For example, it might say, "You may have mild bronchitis. We recommend you drink plenty of fluids and get plenty of rest. If necessary, consult a doctor."

[1160] Example 1

[1161] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1162] In modern society, users need to be able to easily diagnose their own health status from home or on the go. However, there is no system that can integrate and effectively analyze multiple data types (e.g., text data, voice data, and image data) to provide diagnostic results. Secure transmission of this data and accurate preprocessing are also challenges.

[1163] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1164] In this invention, the server includes means for a user to input text data, voice data, and image data, means for tokenizing the text data and performing grammatical analysis, means for denoising and processing the voice data, means for preprocessing the image data, means for passing the text data, voice data, and image data to an AI model for analysis, means for generating a diagnosis result based on the analysis result, means for displaying the diagnosis result to the user, means for transmitting the input data to the server using a secure communication protocol, and means for identifying the received data and preprocessing each data type, thereby enabling a user to quickly and effectively diagnose their health condition.

[1165] A "user" is an individual who utilizes the system to diagnose their own health condition.

[1166] "Text data" is data that represents character information entered by a user.

[1167] "Audio data" is data representing acoustic information recorded by a user.

[1168] "Image data" is data that represents visual information captured by a user.

[1169] "Tokenization" is the process of dividing text data into words and phrases.

[1170] "Grammar analysis" is a process of analyzing the grammatical structure of tokenized text data.

[1171] "Noise reduction" is a process of removing unnecessary background sounds from audio data.

[1172] "Preprocessing" is a series of steps to prepare data in a format that is easy to analyze.

[1173] An "AI model" is a system that uses artificial intelligence techniques to analyze data and recognize patterns.

[1174] "Analysis" is the process of examining data in detail and extracting its meaning and characteristics.

[1175] A "diagnostic result" is an assessment of health status or identification of symptoms generated based on the results of the analysis.

[1176] A "secure communication protocol" is a communication protocol for securely transmitting data.

[1177] A "server" is a computer that receives, preprocesses, and analyzes data, and generates and transmits diagnostic results.

[1178] A "user terminal" is a device through which a user inputs and transmits data.

[1179] The term "said" refers to something that has already been mentioned in the text.

[1180] A "means" is a device or process for achieving a particular function.

[1181] This invention is a system that allows users to easily diagnose their own health condition from home or on the go. In this system, users input text data, voice data, and image data, and generate diagnostic results by analyzing the data. Detailed embodiments of this system are described below.

[1182] System configuration

[1183] User terminal

[1184] 1. Data input interface:

[1185] The user launches a dedicated application and inputs text data about the symptoms.

[1186] The user uses a microphone to record voice data such as coughing.

[1187] The user uses a camera to capture image data of the affected area.

[1188] 2. Data transmission function:

[1189] The terminal packetizes the input data and transmits it to the server using a secure communication protocol (for example, HTTPS).

[1190] server

[1191] 3. Data reception and preprocessing:

[1192] The server receives the data sent from the terminal and identifies the data type of each data.

[1193] The text data is first tokenized and then grammatically analyzed.

[1194] The audio data is subjected to noise reduction and analyzed based on specific frequencies.

[1195] The image data is filtered and preprocessed to convert it into a format that is easy to analyze.

[1196] 4. Data Analysis:

[1197] The server passes the pre-processed data to a generative AI model for further analysis.

[1198] The main symptoms and related symptoms are extracted from the text data.

[1199] The audio data is analyzed for specific acoustic features, and the symptoms are characterized based on the results.

[1200] The image data recognizes specific patterns and identifies the presence or absence of inflammation or swelling.

[1201] 5. Generating diagnostic results:

[1202] The server integrates the analysis results and generates a diagnosis based on the user's symptoms.

[1203] Diagnostic findings may include recommended treatments and further medical consultations.

[1204] 6. Sending and viewing diagnostic results:

[1205] The server transmits the generated diagnosis results to the terminal.

[1206] The terminal displays the received diagnostic results to the user.

[1207] Specific examples

[1208] Diagnosing sore throat and cough

[1209] 1. Data Entry:

[1210] The user enters the text "I've had a persistent cough for the past three days and my throat hurts," then records and uploads an audio recording of themselves coughing.

[1211] 2. Data transmission:

[1212] The terminal converts text and audio files into packets and sends them to the server.

[1213] 3. Data reception and preprocessing:

[1214] The server receives the data, passes the text to a natural language processing (NLP) model for analysis, and passes the audio data to an audio processing module for noise reduction.

[1215] 4. Data Analysis:

[1216] The server extracts key symptoms from the text and analyzes cough characteristics from the audio.

[1217] 5. Generating diagnostic results:

[1218] The server aggregates the results and generates a report with a diagnosis of "possible mild bronchitis" and recommended treatment.

[1219] 6. Sending and viewing diagnostic results:

[1220] The server sends the diagnostic report to the device.

[1221] The device displays the message, "You may have mild bronchitis. We recommend you drink plenty of fluids and get plenty of rest. If necessary, seek medical advice."

[1222] The above is a detailed embodiment of this system, which allows users to quickly and effectively check their own health status.

[1223] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1224] Step 1:

[1225] Use of a data entry interface

[1226] User:

[1227] Start the dedicated application.

[1228] In the application's text entry field, enter your symptoms: "I've had a persistent cough and sore throat for the past three days."

[1229] Use the application's voice recording feature to record the sound of your cough using your microphone.

[1230] Use the application's camera function to take a picture of your throat, for example if you have a sore throat.

[1231] Input: Text data, audio data, and image data about the user's symptoms

[1232] Output: Symptom data stored within the application

[1233] Step 2:

[1234] Data transmission

[1235] Device:

[1236] Text data entered by the user, recorded voice data, and captured image data are packetized.

[1237] This data is sent to the server using a secure communication protocol such as HTTPS.

[1238] Input: Text data, voice data, and image data entered by the user

[1239] Output: Packetized data sent to the server

[1240] Step 3:

[1241] Data reception and preprocessing

[1242] server:

[1243] Receives data sent from the device and identifies the data type (text, audio, image).

[1244] The text data is first tokenized and then grammatically analyzed using natural language processing (NLP) techniques.

[1245] For example, extract key keywords such as "cough" and "sore throat."

[1246] The audio data is subjected to noise removal processing and specific frequencies are analyzed to extract cough patterns and characteristics.

[1247] For example, the intensity and frequency of coughing sounds are analyzed.

[1248] The image data is subjected to filtering and pre-processing to improve the image quality.

[1249] A Gaussian filter is used for filtering, and image resolution is adjusted for preprocessing.

[1250] Input: Text data, audio data, and image data sent from the device

[1251] Output: Preprocessed text data, audio data, and image data

[1252] Step 4:

[1253] Data analysis

[1254] server:

[1255] Preprocessed text data, audio data, and image data are passed to a generative AI model.

[1256] A generative AI model analyzes each piece of data in detail.

[1257] From the text data, major symptoms such as "last three days," "cough," and "sore throat" are detected.

[1258] Cough characteristics (e.g., frequency, intensity) are analyzed from the audio data.

[1259] The image data is used to evaluate the condition of the throat and determine whether inflammation is present.

[1260] Input: Preprocessed text data, audio data, and image data

[1261] Output: Analysis results for each data

[1262] Step 5:

[1263] Generating diagnostic results

[1264] server:

[1265] The results of each data analysis are integrated to generate a comprehensive diagnostic result.

[1266] For example, it generates a diagnosis such as "You may have mild bronchitis" and compiles a report with recommended treatment methods.

[1267] Input: Analysis results of text data, audio data, and image data

[1268] Output: A report containing the diagnosis and recommended treatment.

[1269] Step 6:

[1270] Sending and displaying diagnostic results

[1271] server:

[1272] The generated diagnostic results are transmitted to the terminal via secure communication.

[1273] Device:

[1274] The received diagnostic results are displayed to the user.

[1275] For example, it might say, "You may have mild bronchitis. We recommend you drink plenty of fluids and get plenty of rest. If necessary, seek medical advice."

[1276] Input: Diagnostic results sent from the server

[1277] Output: Diagnostic results displayed on the user's terminal

[1278] (Application example 1)

[1279] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1280] In recent years, there has been an increasing need for systems that allow individuals to easily diagnose their health conditions from home or on the go. However, there is a lack of systems that can comprehensively analyze text data, voice data, and image data to provide highly reliable diagnoses. There is also a need for systems that can efficiently process user-entered data and quickly display diagnostic results. A technical method is needed to solve these issues.

[1281] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1282] In this invention, the server includes means for a user to input text data, voice data, and image data, means for tokenizing the text data and performing grammatical analysis, means for removing noise and processing the voice data, means for preprocessing the image data, means for generating a diagnosis result based on the analysis results of the text data, voice data, and image data, means for displaying the diagnosis result to the user, means for collecting data from a smartphone terminal and transmitting it to the server, and means for generating a diagnosis result based on a prompt sentence using a generative AI model, thereby enabling a user to quickly and accurately diagnose their own health condition from home or on the go.

[1283] A "user terminal" is an electronic device that allows a user to input, send, and receive text data, voice data, and image data.

[1284] "Text data" refers to text information entered by a user, and is data for performing natural language processing.

[1285] "Audio data" refers to audio information recorded by a user, and is data for performing acoustic analysis and noise removal processes.

[1286] "Image data" is visual information captured by a user using a camera, and is data for preprocessing.

[1287] "Tokenization" is a process of dividing text data into small units, and is a preprocessing step for grammatical analysis.

[1288] "Grammar analysis" is a process of analyzing the grammatical structure of tokenized text data.

[1289] "Noise reduction" is a process that removes unnecessary sounds such as background noise from audio data.

[1290] "Preprocessing" is a process for converting image data into a format that is easy to analyze.

[1291] The "diagnosis result" is estimated information about the health condition that is generated based on the analyzed text data, voice data, and image data.

[1292] The "means for displaying to the user" is a function or device for providing the diagnostic results to the user in a format that is easy to view.

[1293] A "smartphone terminal" is a small, portable electronic device that is used to collect and transmit user data.

[1294] A "server" is a computer system that receives, analyzes, and transmits data over a network.

[1295] A "generative AI model" is an artificial intelligence model that automatically generates diagnostic results based on input data.

[1296] A "prompt sentence" is an instruction sentence used as input to a generative AI model.

[1297] This system is designed to allow users to easily diagnose their own health condition from home or on the go. It analyzes text, voice, and image data entered by the user and provides diagnostic results. The components of the system and specific processing steps are shown below.

[1298] System configuration

[1299] User terminal

[1300] The user terminal includes a portable device such as a smartphone or tablet. The user inputs the following data through the application:

[1301] 1. Text data input: The user inputs information about symptoms and physical condition in text format.

[1302] 2. Recording voice data: The user records coughing, breathing, etc. and inputs it as voice data.

[1303] 3. Capturing image data: The user uses a camera to capture an image of the affected area.

[1304] server

[1305] The server is a central computing device that receives and analyzes data sent from the terminals. The server is configured as follows:

[1306] 1. Data reception: Receives text data, voice data, and image data sent from the user terminal.

[1307] 2. Preprocessing: The received data is processed by various preprocessing modules. Text data is tokenized and grammatically analyzed, audio data is denoised, and image data is filtered and format converted.

[1308] 3. Data analysis: The preprocessed data is then further analyzed by generative AI models: text data is analyzed for key symptoms and related symptoms, audio data is analyzed for specific acoustic features, and image data is analyzed for specific patterns.

[1309] 4. Generating a diagnosis: The analysis results are combined to generate a diagnosis based on the user's symptoms, which may include recommended treatments or further medical consultations.

[1310] 5. Sending the diagnosis result: Send the generated diagnosis result to the user terminal.

[1311] Hardware and software used

[1312] User devices: smartphones, tablets

[1313] Speech processing software: speech_recognition library

[1314] Image processing software: OpenCV library

[1315] Sending data: the requests library

[1316] Generative AI models: various AI diagnostic models, such as those provided by the transformers library

[1317] Specific examples

[1318] The user launches a smartphone application, enters the text "I've had a persistent cough for the past three days and my throat hurts," records the coughing sound, and takes an image of their throat. The user's device sends this data to the server. The server receives the data, performs grammatical analysis on the text data, removes noise from the audio data, and filters the image data. It then analyzes the data based on the generative AI model and generates a diagnosis: "You may have mild bronchitis. We recommend you drink plenty of fluids and rest. If necessary, consult a doctor." This is then sent to the user's device.

[1319] Prompt Sentence Examples

[1320] User input: "I've had a constant cough and sore throat for the past three days."

[1321] Audio Data: [Recorded coughing sound]

[1322] Image Data: [User-taken image of throat]

[1323] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1324] Step 1:

[1325] A user starts the application using a smartphone and inputs text, voice, and image data. Specifically, the user inputs text such as "I've had a persistent cough for the past three days and my throat hurts," records the sound of their cough, and takes an image of their throat.

[1326] Input: Text data, audio data, image data

[1327] Output: Input text, recorded audio files, captured image files

[1328] Step 2:

[1329] The device prepares the input data and sends it to the server using a secure communication protocol (e.g., HTTPS). The data is packetized as text, audio files, or image files.

[1330] Input: text, audio files, image files

[1331] Output: Packetized data

[1332] Step 3:

[1333] The server receives packetized data sent from the terminal, identifies the received data, and initiates appropriate pre-processing for each data type.

[1334] Input: Packetized data

[1335] Output: Received text data, audio data, and image data

[1336] Step 4:

[1337] The server tokenizes the received text data and performs grammatical analysis, extracting the grammatical structure and semantic features of the text data, for example, identifying key symptoms such as "cough" or "sore throat."

[1338] Input: Text data

[1339] Output: Tokenized text, grammar analysis results

[1340] Step 5:

[1341] The server processes the audio data through noise reduction and analyzes its acoustic features to identify coughs and other distinctive acoustic patterns, such as the frequency range and duration of a cough.

[1342] Input: Audio data

[1343] Output: Noise-removed audio data, acoustic analysis results

[1344] Step 6:

[1345] The server performs preprocessing on the image data, such as filtering and format conversion, to convert the image data into a format that is easier to analyze. For example, contrast adjustment and edge detection are performed on the throat image.

[1346] Input: Image data

[1347] Output: Preprocessed image data

[1348] Step 7:

[1349] The server passes the preprocessed text, audio, and image data to the generative AI model for detailed data analysis, such as analyzing the prompt sentences and generating a comprehensive diagnosis.

[1350] Input: Tokenized text, acoustic analysis results, preprocessed image data

[1351] Output: Diagnostic results

[1352] Step 8:

[1353] The server integrates the generated diagnostic results and generates appropriate feedback to the user, including recommended treatments and next steps (e.g., consulting a medical professional).

[1354] Input: Diagnostic results

[1355] Output: Consolidated diagnostic report

[1356] Step 9:

[1357] The server sends the diagnosis results to the user's device, which then displays the received diagnosis results in an easy-to-read format. For example, it might say, "You may have mild bronchitis. We recommend you drink plenty of fluids and get plenty of rest."

[1358] Input: Consolidated diagnostic report

[1359] Output: Diagnostic results displayed to the user

[1360] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1361] This invention combines a system that allows users to diagnose their own health condition using text data, voice data, and image data with an emotion engine that recognizes the user's emotions, thereby providing comprehensive diagnostic results that take the user's emotional state into consideration.

[1362] System configuration

[1363] User terminal

[1364] 1. Data input interface:

[1365] The user launches a dedicated application and inputs text data about the symptoms.

[1366] The user uses a microphone to record audio data such as coughing, and a camera to take an image of the affected area.

[1367] 2. Data transmission function:

[1368] The terminal converts the input text data, voice data, and image data into packets and transmits them to the server using a secure communication protocol (for example, HTTPS).

[1369] server

[1370] 3. Data reception and preprocessing:

[1371] The server receives data packets sent from the terminal and separates them by data type: text data is sent to the text processing module, voice data is sent to the voice processing module, and image data is sent to the image processing module.

[1372] 4. Data Analysis:

[1373] The text data is tokenized and grammatically analyzed using a natural language processing (NLP) model to identify primary symptoms and associated secondary symptoms.

[1374] The audio data is denoised and filtered, and then the cough pattern is analysed through a speech analysis model.

[1375] The image data is preprocessed and an image analysis model is used to detect patterns of inflammation and swelling.

[1376] 5. Emotion analysis:

[1377] The server's emotion engine analyzes text data, voice data, and image data to recognize the user's emotional state.

[1378] From the text data, emotions are identified based on the user's writing style and expression.

[1379] Emotions are analyzed from audio data based on fluctuations in tone and pitch.

[1380] Emotions are recognized from image data based on facial expressions and body movements.

[1381] 6. Generating diagnostic results:

[1382] The server integrates the data analysis results with the emotion analysis results to generate a comprehensive diagnosis, which also reflects the user's emotional state.

[1383] The diagnosis will include recommended treatment and medical consultation if necessary.

[1384] 7. Sending and viewing diagnostic results:

[1385] The server sends the generated diagnostic results to the terminal, where an application displays the diagnostic results to the user.

[1386] Specific examples

[1387] Diagnosis by combining sore throat, cough, and emotion analysis

[1388] 1. Data Entry:

[1389] The user enters the text "I've had a persistent cough for the past three days and my throat hurts," records and sends an audio recording of the cough, and also takes a photo of their face.

[1390] 2. Data transmission:

[1391] The device sends text, audio, and image files to the server.

[1392] 3. Data reception and preprocessing:

[1393] The server receives the data, analyzes the text using NLP models, denoises the audio data, and preprocesses the image data.

[1394] 4. Data Analysis:

[1395] It extracts key symptoms from text, analyzes cough characteristics from audio, and detects patterns of inflammation and swelling from images.

[1396] 5. Emotion analysis:

[1397] The server uses an emotion engine to analyze stress and anxiety from text, tension from audio, and sadness and fatigue from images.

[1398] 6. Generating diagnostic results:

[1399] The server combines the results of the data analysis and the emotion analysis and generates a diagnosis that reads, "You may have mild bronchitis, but you appear to be under stress. We recommend that you drink plenty of fluids and get plenty of rest. Also, if you need psychological support, please consult a specialist."

[1400] 7. Sending and viewing diagnostic results:

[1401] The server sends the diagnostic results to the terminal, which displays them to the user.

[1402] The above is a specific embodiment of the present invention. By using this system, users can receive diagnostic results that take into account their physical and emotional state comprehensively, and obtain appropriate treatment and countermeasures.

[1403] The processing flow will be explained below.

[1404] Step 1:

[1405] The user starts a dedicated symptom recording application and enters text data about the symptoms. For example, the user might write "I've had a persistent cough and a sore throat for the past three days" in the text box.

[1406] Step 2:

[1407] The user uses a microphone to record the sound of their cough, as well as take a photo of their face and the affected area, all of which is uploaded to the application.

[1408] Step 3:

[1409] The device combines the input text, voice, and image data into a single data packet, which is then sent to the server using a secure communication protocol (e.g., HTTPS).

[1410] Step 4:

[1411] The server receives data packets sent from the device and separates them by data type (text, audio, image), and passes each data to the corresponding analysis module.

[1412] Step 5:

[1413] The server passes the text data to a natural language processing (NLP) model for tokenization and grammatical analysis, which identifies the primary symptom and associated secondary symptoms.

[1414] Step 6:

[1415] The server passes the audio data to a speech processing module for noise reduction and filtering, and then a speech analysis model analyzes the audio characteristics (e.g., coughing patterns).

[1416] Step 7:

[1417] The server passes the image data to the image processing module for preprocessing (filtering, normalization), after which an image analysis model detects patterns of inflammation and swelling.

[1418] Step 8:

[1419] The server passes the analyzed text data to an emotion engine to identify the user's emotions. For example, if there are many negative expressions, it can detect stress or anxiety.

[1420] Step 9:

[1421] The server also analyzes emotions from the voice data, identifying emotions such as tension or impatience based on the tone and pitch of the voice.

[1422] Step 10:

[1423] The server uses the image data to analyze the user's facial expressions and physical state to identify emotions, such as sadness or fatigue from facial distortion or puffy eyes.

[1424] Step 11:

[1425] The server integrates the data obtained from each analysis module (text, audio, image analysis results, and emotion analysis results) to generate a comprehensive diagnosis that also reflects the user's emotional state.

[1426] Step 12:

[1427] The server sends the generated diagnostic results to the device, where they are transmitted in a secure format and displayed in the device application.

[1428] Step 13:

[1429] The device will display the diagnosis results it receives to the user, such as "You may have mild bronchitis, but you appear to be under stress. We recommend you drink plenty of fluids and get plenty of rest. Also, if you need psychological support, please consult a specialist."

[1430] Example 2

[1431] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1432] Conventional health checkup systems provide diagnostic results based on the analysis of text data, voice data, and image data collected from users, but do not take the user's emotional state into consideration. As a result, comprehensive health diagnosis is insufficient and advice on the user's emotional health is lacking. The present invention aims to provide more accurate comprehensive diagnostic results by analyzing data including the user's emotional state.

[1433] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1434] In this invention, the server includes a means for tokenizing and syntactically analyzing text data, a means for denoising and processing audio data, a means for preprocessing image data, and a means for analyzing emotional states, thereby enabling the generation and display of comprehensive diagnostic results based on the analysis results of the text data, audio data, and image data, as well as the emotional states.

[1435] "User" refers to a person who uses the system to input data to diagnose their own health condition.

[1436] "Text data" refers to character information such as symptoms and emotions entered by the user.

[1437] "Audio data" refers to acoustic information such as coughs and voices recorded by a user.

[1438] "Image data" refers to image information such as the affected area and facial expression photographed by the user.

[1439] "Tokenization" refers to the process of breaking down text data into smaller units such as words or phrases.

[1440] "Syntax analysis" refers to the process of analyzing the grammatical structure of tokenized text data.

[1441] "Noise reduction" refers to the process of removing unnecessary noise from audio data.

[1442] "Preprocessing" refers to the process of formatting data to make it easier to analyze.

[1443] "Diagnostic results" refer to health assessments and recommended treatments generated based on data and sentiment analysis.

[1444] "Emotional state" refers to a user's mental state, such as stress, anxiety, tension, sadness, etc.

[1445] "Comprehensive diagnostic results" refer to diagnostic information obtained by taking into account both physical symptoms and emotional state.

[1446] "Means for displaying" refers to a method or device for delivering the generated diagnostic results to a user.

[1447] MODE FOR CARRYING OUT THE INVENTION

[1448] The present invention is a system that allows users to diagnose their own health condition using text data, voice data, and image data, and also provides comprehensive diagnostic results that take into account the user's emotional state. This system is composed of a user terminal and a server.

[1449] User terminal

[1450] 1. Data Entry Interface

[1451] The user launches a dedicated application and enters text data about their symptoms. For example, they can enter, "I've had a persistent cough and a sore throat for the past three days."

[1452] The user uses a microphone to record audio data such as coughing, and a camera to take an image of the affected area.

[1453] 2. Data transmission function

[1454] The terminal converts the input text data, voice data, and image data into packets and transmits them to the server using a secure communication protocol (for example, HTTPS).

[1455] server

[1456] 3. Data Reception and Preprocessing

[1457] The server receives data packets sent from the terminal and separates them by data type (passing text data to the text processing module, voice data to the voice processing module, and image data to the image processing module).

[1458] The main technologies used in this case include NLP (Natural Language Processing) modules, voice analysis models, and image analysis models.

[1459] 4. Data Analysis

[1460] Text data analysis: The server uses NLP models to tokenize the text data and perform grammatical analysis, which identifies primary and secondary symptoms, such as "cough" and "sore throat."

[1461] Voice data analysis: The server performs noise reduction and filtering on the voice data, and then analyzes the cough pattern using a voice analysis model.

[1462] Image data analysis: The server preprocesses the image data and uses image analysis models to detect patterns of inflammation and swelling.

[1463] 5. Emotion analysis

[1464] The server's emotion engine analyzes text, voice, and image data to recognize the user's emotional state.

[1465] Sentiment analysis from text: Identify emotions based on the user's writing style and expressions. For example, analyze stress from the emphasis on a period such as "3 days" in the text.

[1466] Emotion analysis from audio: Analyzes tone and pitch fluctuations in audio data to detect tension.

[1467] Emotion analysis from images: Analyzing facial expressions and body movements to recognize feelings of sadness or fatigue.

[1468] 6. Generating diagnostic results

[1469] The server integrates the data analysis results with the emotion analysis results to generate a comprehensive diagnosis, which also reflects the user's emotional state.

[1470] For example, it generates a diagnosis such as, "You may have mild bronchitis, but you appear to be under stress. We recommend you drink plenty of fluids and get plenty of rest. Also, if you need psychological support, consult a specialist."

[1471] 7. Sending and displaying diagnostic results

[1472] The server transmits the generated diagnostic results to the terminal, and an application on the terminal displays the diagnostic results to the user.

[1473] This system allows users to obtain comprehensive health checkup results that take into account not only their physical symptoms but also their emotional state.

[1474] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1475] Program processing flow

[1476] Step 1: User Data Entry

[1477] Using a dedicated application, users input text data about their symptoms. For example, they can describe specific symptoms such as, "I've had a persistent cough for the past three days, and my throat hurts." In addition, the system collects audio and image data by recording the sound of the cough using a microphone and taking images of the affected area using a camera.

[1478] Input: Text data describing symptoms, recorded voice data, photographed image data

[1479] Output: Collected dataset (text data, audio data, image data)

[1480] Step 2: Data collection and transmission by the device

[1481] The device collects text data entered by the user, recorded voice data, and captured image data, then encrypts this data using the HTTPS protocol and securely transmits it to a server.

[1482] Input: Collected datasets (text data, audio data, image data)

[1483] Output: Dataset sent securely to server

[1484] Step 3: Data reception and preprocessing by the server

[1485] The server receives data packets sent from the device. The received data is first classified into text data, voice data, and image data. The text data is then passed to the NLP module, the voice data to the voice analysis module, and the image data to the image analysis module.

[1486] Input: Securely transmitted dataset

[1487] Output: Classified dataset (text data, audio data, image data)

[1488] Step 4: Analyzing the text data

[1489] The server uses an NLP model to tokenize and parse the received text data, extracting key symptoms such as "cough" or "sore throat."

[1490] Input: Text data

[1491] Output: Extracted main and secondary symptoms

[1492] Step 5: Analyzing the audio data

[1493] The server denoises and filters the audio data, then uses a speech analysis model to analyze cough patterns.

[1494] Input: Audio data

[1495] Output: Analyzed cough pattern

[1496] Step 6: Analyzing the image data

[1497] The server preprocesses the image data and uses image analysis models to detect patterns of inflammation and swelling.

[1498] Input: Image data

[1499] Output: Detected inflammation and swelling patterns

[1500] Step 7: Sentiment Analysis

[1501] The server's emotion engine analyzes text data, voice data, and image data to recognize the user's emotional state. Specifically, it identifies emotions from text data based on writing style and expression, analyzes tone and pitch fluctuations from voice data, and analyzes facial expressions and body movements from image data.

[1502] Input: text data, audio data, image data

[1503] Output: User's emotional state

[1504] Step 8: Generate comprehensive diagnostic results

[1505] The server combines the data analysis results with the sentiment analysis results to generate a comprehensive diagnosis. This diagnosis includes recommended treatment and, if necessary, consultation with a medical institution. For example, a diagnosis might read, "You may have mild bronchitis, but you appear to be under stress. We recommend you drink plenty of fluids and rest. We also recommend consulting a specialist if you need psychological support."

[1506] Input: Data analysis results, emotion analysis results

[1507] Output: Comprehensive diagnostic results

[1508] Step 9: Send and view diagnostic results

[1509] The server sends the generated diagnostic results to the terminal, and the terminal application displays the received diagnostic results to the user.

[1510] Input: Comprehensive diagnostic results

[1511] Output: Diagnostic results displayed to the user

[1512] Through the above processing steps, the user can receive a diagnosis result that takes into consideration the user's physical and emotional state comprehensively.

[1513] (Application example 2)

[1514] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1515] Conventional health checkup systems typically analyze symptoms and make diagnoses based on text data, voice data, and image data entered by the user. However, these systems were unable to consider the user's emotional state, making it difficult to provide comprehensive diagnostic results. Furthermore, it was difficult to propose services in physical stores that took into account the customer's health and emotional state, making improving customer satisfaction a challenge. Therefore, there was a need for a system that could analyze not only the user's health condition but also their emotional state, reflect this in the diagnostic results, and then propose appropriate products and services in physical stores.

[1516] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1517] In this invention, the server includes means for a user to input text data, voice data, and image data, means for tokenizing the text data and performing grammatical analysis, means for removing noise and processing the voice data, means for preprocessing the image data, means for generating a diagnostic result based on the analysis results of the text data, voice data, and image data, means for displaying the diagnostic result to the user, means for proposing products and services in a physical store, and means for performing emotion analysis on the text data, voice data, and image data and reflecting the emotion in the diagnostic result. This makes it possible to provide a comprehensive diagnostic result based on the user's health condition and emotional state, and to propose appropriate products and services in a physical store based on the result.

[1518] "User" refers to a person who uses the system to diagnose a health or emotional state.

[1519] "Text data" refers to data that includes character information entered by a user.

[1520] "Voice data" refers to data containing voice information input by a user.

[1521] "Image data" refers to data that includes information about an image captured by a user.

[1522] "Tokenization" refers to the process of dividing text data into smaller units such as words or phrases.

[1523] "Grammar analysis" refers to the process of analyzing the syntax and meaning of text data.

[1524] "Noise reduction" refers to the process of removing unnecessary background sounds and noise from audio data.

[1525] "Preprocessing" refers to the process of shaping or adjusting image data before analysis.

[1526] "Diagnosis results" refer to the evaluation results of the user's health condition and emotional state generated based on the analysis results of text data, audio data, and image data.

[1527] "Display means" refers to a device or method for displaying the diagnostic results to the user or store clerk.

[1528] "Brick and mortar store" refers to a physical location that offers goods and services.

[1529] "Product and service suggestions" refers to suggesting appropriate products and services based on the diagnostic results.

[1530] "Emotion analysis" refers to the process of analyzing a user's emotional state from text data, audio data, and image data.

[1531] This will help the reader to understand the invention more clearly.

[1532] MODE FOR CARRYING OUT THE INVENTION

[1533] The present invention is a system for analyzing the health and emotional state of customers in a physical store and recommending appropriate products and services based on the results. The system includes a user terminal, a server, and a store device that displays the summary results.

[1534] User terminal

[1535] The user terminal has the following functions:

[1536] Data Input Interface:

[1537] The user launches a dedicated application and inputs text data about the symptoms.

[1538] The user uses a microphone to record audio data such as coughing, and a camera to take an image of the affected area.

[1539] Data transmission function:

[1540] The terminal converts the input text data, voice data, and image data into packets and transmits them to the server using a secure communication protocol (for example, HTTPS).

[1541] server

[1542] The server has the following functions:

[1543] Data reception and preprocessing:

[1544] The server receives data packets sent from the terminal and separates them by data type: text data is sent to the text processing module, voice data is sent to the voice processing module, and image data is sent to the image processing module.

[1545] Data Analysis:

[1546] The text data is tokenized and grammatically analyzed using a natural language processing (NLP) model to identify primary symptoms and associated secondary symptoms.

[1547] The audio data is denoised and filtered, and then the cough pattern is analysed through a speech analysis model.

[1548] The image data is preprocessed and an image analysis model is used to detect patterns of inflammation and swelling.

[1549] Emotion analysis:

[1550] The server's emotion engine analyzes text data, voice data, and image data to recognize the user's emotional state.

[1551] From the text data, emotions are identified based on the user's writing style and expression.

[1552] Emotions are analyzed from audio data based on fluctuations in tone and pitch.

[1553] Emotions are recognized from image data based on facial expressions and body movements.

[1554] Generating diagnostic results:

[1555] The server integrates the data analysis results and sentiment analysis results to generate a comprehensive diagnosis.

[1556] The diagnosis will include recommended treatment and medical consultation if necessary.

[1557] Product and service suggestions for physical stores are also generated based on the diagnostic results.

[1558] Sending and viewing diagnostic results:

[1559] The server transmits the generated diagnostic results to the terminal and the store equipment. The application on the terminal displays the diagnostic results to the user, and the store equipment also displays the diagnostic results to the store clerk.

[1560] Specific examples

[1561] For example, a customer visits a physical store and enters text such as, "I've been sleep-deprived these past few days due to work stress. My throat is a little sore," records the sound of a cough, and takes a photo of their face. The device then sends this text, audio, and image data to a server. The server analyzes the data and generates a diagnosis that also takes into account their emotional state. Based on the results, a suggestion is displayed, such as, "We think you may be overworked due to stress. We recommend some relaxing herbal tea or aroma oil to relieve stress."

[1562] Prompt Sentence Examples

[1563] Customer: "I've been sleep deprived these past few days due to work stress. My throat is a bit sore."

[1564] Audio Recording: [Coughing]

[1565] Headshot: [Customer photo]

[1566] Input data:

[1567] Text: "I've been sleep deprived these past few days due to work stress. My throat is a bit sore."

[1568] Audio data: Coughing sound

[1569] Image data: customer photos

[1570] Prompt statement:

[1571] "Analyze the text, voice, and image data entered by the customer and generate a diagnosis that takes into account their health status and emotions. The results should also include recommendations for appropriate products and services."

[1572] As described above, the present invention can comprehensively diagnose a user's health and emotional state, and can also suggest appropriate products and services to customers in physical stores.

[1573] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1574] Processing Steps

[1575] Step 1: Data entry

[1576] The user launches a dedicated application and inputs information about their symptoms and emotional state. The input includes text data, voice data, and image data, and is input using the smartphone's keyboard, microphone, and camera, respectively.

[1577] Input: Text, audio, and image data about symptoms and emotional states

[1578] Output: Input dataset in the device

[1579] Step 2: Send data

[1580] The device then packets the input data and sends it to the server using the HTTPS protocol. The data is then organized into text, audio, and image formats and securely sent to the server.

[1581] Input: Input dataset in the device

[1582] Output: Sending a data packet to the server

[1583] Step 3: Data reception and preprocessing

[1584] The server receives data packets sent from the device and separates them into text data, voice data, and image data. Each data type is passed to a different pre-processing module: text data is tokenized, voice data is denoised, and image data is formatted and adjusted.

[1585] Input: Data packet sent

[1586] Output: Separated text data, audio data, and image data

[1587] Step 4: Text data analysis

[1588] The server uses a Natural Language Processing (NLP) model to tokenize and analyze the text data, identifying key and secondary symptoms.

[1589] Input: Text data

[1590] Output: Symptom identification analysis results

[1591] Step 5: Audio data analysis

[1592] The voice processing module filters the voice data and performs noise reduction, then analyzes the cough pattern through a voice analysis model to analyze the user's symptom characteristics and emotional state.

[1593] Input: Denoised audio data

[1594] Output: Cough pattern analysis results

[1595] Step 6: Image data analysis

[1596] The image processing module preprocesses the image data and uses image analysis models to detect patterns of inflammation and swelling, as well as recognize the user's emotional state from facial expressions and body movements.

[1597] Input: Preprocessed image data

[1598] Output: Analysis of inflammation and swelling patterns and emotional state

[1599] Step 7: Sentiment Analysis

[1600] The server's emotion engine analyzes the user's emotional state from text data, voice data, and image data. From the text data, it analyzes emotions based on the user's expression, from voice data, it analyzes emotions based on tone and pitch fluctuations, and from image data, it analyzes emotions based on facial expressions and body movements.

[1601] Input: Text data, audio data, image data

[1602] Output: Overall emotional state analysis

[1603] Step 8: Generate diagnostic results

[1604] The server integrates the results of data analysis and emotion analysis to generate a comprehensive diagnosis that reflects estimated symptoms and emotional state, and also includes recommendations for appropriate products and services.

[1605] Input: Data analysis results and sentiment analysis results

[1606] Output: Comprehensive diagnostic results

[1607] Step 9: Send and view diagnostic results

[1608] The server sends the generated diagnostic results to the terminal and the store equipment. The terminal application displays the diagnostic results to the user, and the store equipment displays them to the store clerk. This allows the user and the store clerk to confirm the appropriate products and services.

[1609] Input: Comprehensive diagnostic results

[1610] Output: Diagnostic results and recommendations displayed to the user and the store clerk

[1611] The above are the specific processing steps of the system that realizes the application example.

[1612] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1613] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1614] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1615] [Fourth embodiment]

[1616] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1617] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1618] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1619] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1620] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1621] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1622] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1623] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1624] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1625] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1626] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1627] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1628] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1629] This system is designed to allow users to easily diagnose their own health condition from home or on the go. The system analyzes the text, voice, and image data entered by the user and provides a diagnosis result. Below, the program processing of this system is explained in natural language and shown in detail with concrete examples.

[1630] System configuration

[1631] User terminal

[1632] 1. Data input interface:

[1633] The user launches a dedicated application and inputs text data about the symptoms.

[1634] The user uses a microphone to record voice data such as coughing.

[1635] The user uses a camera to capture image data of the affected area.

[1636] 2. Data transmission function:

[1637] In order to transmit the input data to the server, the terminal packetizes the data and transmits it to the server using a secure communication protocol (for example, HTTPS).

[1638] server

[1639] 3. Data reception and preprocessing:

[1640] The server receives the data sent from the terminal and identifies the data type of each data.

[1641] The text data is first tokenized and then grammatically analyzed.

[1642] The audio data is soundproofed and analyzed based on specific frequencies.

[1643] The image data is filtered and preprocessed to convert it into a format that is easy to analyze.

[1644] 4. Data Analysis:

[1645] The server passes the preprocessed data to the AI ​​model for further analysis.

[1646] The main symptoms and related symptoms are extracted from the text data.

[1647] The audio data is analyzed for specific acoustic features, and the symptoms are characterized based on the results.

[1648] The image data recognizes specific patterns and identifies the presence or absence of inflammation or swelling.

[1649] 5. Generating diagnostic results:

[1650] The server integrates the analysis results and generates a diagnosis based on the user's symptoms.

[1651] Diagnostic findings may include recommended treatments and further medical consultations.

[1652] 6. Sending and viewing diagnostic results:

[1653] The server transmits the generated diagnosis results to the terminal.

[1654] The terminal displays the received diagnostic results to the user.

[1655] Specific examples

[1656] Diagnosing sore throat and cough

[1657] 1. Data Entry:

[1658] The user enters the text "I've had a persistent cough for the past three days and my throat hurts," then records and uploads an audio recording of themselves coughing.

[1659] 2. Data transmission:

[1660] The terminal converts text and audio files into packets and sends them to the server.

[1661] 3. Data reception and preprocessing:

[1662] The server receives the data, passes the text to an NLP model for analysis, and passes the audio data to an audio processing module for noise reduction.

[1663] 4. Data Analysis:

[1664] The server extracts key symptoms from the text and analyzes cough characteristics from the audio.

[1665] 5. Generating diagnostic results:

[1666] The server aggregates the results and generates a report with a diagnosis of "possible mild bronchitis" and recommended treatment.

[1667] 6. Sending and viewing diagnostic results:

[1668] The server sends the diagnostic report to the device.

[1669] The device displays the message, "You may have mild bronchitis. We recommend you drink plenty of fluids and get plenty of rest. If necessary, seek medical advice."

[1670] The above is a detailed embodiment of this system, which allows users to quickly and effectively check their own health status.

[1671] The processing flow will be explained below.

[1672] Step 1:

[1673] The user launches a dedicated application to describe their symptoms. In the text box, they enter, "I've had a persistent cough for the past three days and my throat hurts." They then use the recording function to record the sound of their cough and, if necessary, take an image of the affected area.

[1674] Step 2:

[1675] The device detects the input text data, voice data, and image data, and packetizes them for transmission to the server using the secure HTTPS communication protocol.

[1676] Step 3:

[1677] The server receives data packets sent from the terminal and separates them by data type: text data is sent to the text processing module, voice data to the voice processing module, and image data to the image processing module.

[1678] Step 4:

[1679] The server uses a Natural Language Processing (NLP) model to analyze the text data. The text data is first tokenized and then grammatically analyzed. The analysis results identify the primary symptom and related secondary symptoms.

[1680] Step 5:

[1681] The server pre-processes the audio data, applying noise reduction filters to bring the audio data into a clean format for analysis. Then, the audio analysis model extracts audio features and analyzes cough patterns.

[1682] Step 6:

[1683] The server preprocesses the image data, filtering and normalizing it, and then passes it to the image analysis model, which detects patterns of inflammation and swelling from the image data.

[1684] Step 7:

[1685] The server combines the results of the text, audio, and image analysis to generate a comprehensive diagnosis of the user's health, including recommended treatments and further medical consultations.

[1686] Step 8:

[1687] The server sends the generated diagnostic results to the device, where they are transmitted in a secure format and read by the device application.

[1688] Step 9:

[1689] The device displays the diagnosis results it received to the user. For example, it might say, "You may have mild bronchitis. We recommend you drink plenty of fluids and get plenty of rest. If necessary, consult a doctor."

[1690] Example 1

[1691] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1692] In modern society, users need to be able to easily diagnose their own health status from home or on the go. However, there is no system that can integrate and effectively analyze multiple data types (e.g., text data, voice data, and image data) to provide diagnostic results. Secure transmission of this data and accurate preprocessing are also challenges.

[1693] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1694] In this invention, the server includes means for a user to input text data, voice data, and image data, means for tokenizing the text data and performing grammatical analysis, means for denoising and processing the voice data, means for preprocessing the image data, means for passing the text data, voice data, and image data to an AI model for analysis, means for generating a diagnosis result based on the analysis result, means for displaying the diagnosis result to the user, means for transmitting the input data to the server using a secure communication protocol, and means for identifying the received data and preprocessing each data type, thereby enabling a user to quickly and effectively diagnose their health condition.

[1695] A "user" is an individual who utilizes the system to diagnose their own health condition.

[1696] "Text data" is data that represents character information entered by a user.

[1697] "Audio data" is data representing acoustic information recorded by a user.

[1698] "Image data" is data that represents visual information captured by a user.

[1699] "Tokenization" is the process of dividing text data into words and phrases.

[1700] "Grammar analysis" is a process of analyzing the grammatical structure of tokenized text data.

[1701] "Noise reduction" is a process of removing unnecessary background sounds from audio data.

[1702] "Preprocessing" is a series of steps to prepare data in a format that is easy to analyze.

[1703] An "AI model" is a system that uses artificial intelligence techniques to analyze data and recognize patterns.

[1704] "Analysis" is the process of examining data in detail and extracting its meaning and characteristics.

[1705] A "diagnostic result" is an assessment of health status or identification of symptoms generated based on the results of the analysis.

[1706] A "secure communication protocol" is a communication protocol for securely transmitting data.

[1707] A "server" is a computer that receives, preprocesses, and analyzes data, and generates and transmits diagnostic results.

[1708] A "user terminal" is a device through which a user inputs and transmits data.

[1709] The term "said" refers to something that has already been mentioned in the text.

[1710] A "means" is a device or process for achieving a particular function.

[1711] This invention is a system that allows users to easily diagnose their own health condition from home or on the go. In this system, users input text data, voice data, and image data, and generate diagnostic results by analyzing the data. Detailed embodiments of this system are described below.

[1712] System configuration

[1713] User terminal

[1714] 1. Data input interface:

[1715] The user launches a dedicated application and inputs text data about the symptoms.

[1716] The user uses a microphone to record voice data such as coughing.

[1717] The user uses a camera to capture image data of the affected area.

[1718] 2. Data transmission function:

[1719] The terminal packetizes the input data and transmits it to the server using a secure communication protocol (for example, HTTPS).

[1720] server

[1721] 3. Data reception and preprocessing:

[1722] The server receives the data sent from the terminal and identifies the data type of each data.

[1723] The text data is first tokenized and then grammatically analyzed.

[1724] The audio data is subjected to noise reduction and analyzed based on specific frequencies.

[1725] The image data is filtered and preprocessed to convert it into a format that is easy to analyze.

[1726] 4. Data Analysis:

[1727] The server passes the pre-processed data to a generative AI model for further analysis.

[1728] The main symptoms and related symptoms are extracted from the text data.

[1729] The audio data is analyzed for specific acoustic features, and the symptoms are characterized based on the results.

[1730] The image data recognizes specific patterns and identifies the presence or absence of inflammation or swelling.

[1731] 5. Generating diagnostic results:

[1732] The server integrates the analysis results and generates a diagnosis based on the user's symptoms.

[1733] Diagnostic findings may include recommended treatments and further medical consultations.

[1734] 6. Sending and viewing diagnostic results:

[1735] The server transmits the generated diagnosis results to the terminal.

[1736] The terminal displays the received diagnostic results to the user.

[1737] Specific examples

[1738] Diagnosing sore throat and cough

[1739] 1. Data Entry:

[1740] The user enters the text "I've had a persistent cough for the past three days and my throat hurts," then records and uploads an audio recording of themselves coughing.

[1741] 2. Data transmission:

[1742] The terminal converts text and audio files into packets and sends them to the server.

[1743] 3. Data reception and preprocessing:

[1744] The server receives the data, passes the text to a natural language processing (NLP) model for analysis, and passes the audio data to an audio processing module for noise reduction.

[1745] 4. Data Analysis:

[1746] The server extracts key symptoms from the text and analyzes cough characteristics from the audio.

[1747] 5. Generating diagnostic results:

[1748] The server aggregates the results and generates a report with a diagnosis of "possible mild bronchitis" and recommended treatment.

[1749] 6. Sending and viewing diagnostic results:

[1750] The server sends the diagnostic report to the device.

[1751] The device displays the message, "You may have mild bronchitis. We recommend you drink plenty of fluids and get plenty of rest. If necessary, seek medical advice."

[1752] The above is a detailed embodiment of this system, which allows users to quickly and effectively check their own health status.

[1753] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1754] Step 1:

[1755] Use of a data entry interface

[1756] User:

[1757] Start the dedicated application.

[1758] In the application's text entry field, enter your symptoms: "I've had a persistent cough and sore throat for the past three days."

[1759] Use the application's voice recording feature to record the sound of your cough using your microphone.

[1760] Use the application's camera function to take a picture of your throat, for example if you have a sore throat.

[1761] Input: Text data, audio data, and image data about the user's symptoms

[1762] Output: Symptom data stored within the application

[1763] Step 2:

[1764] Data transmission

[1765] Device:

[1766] Text data entered by the user, recorded voice data, and captured image data are packetized.

[1767] This data is sent to the server using a secure communication protocol such as HTTPS.

[1768] Input: Text data, voice data, and image data entered by the user

[1769] Output: Packetized data sent to the server

[1770] Step 3:

[1771] Data reception and preprocessing

[1772] server:

[1773] Receives data sent from the device and identifies the data type (text, audio, image).

[1774] The text data is first tokenized and then grammatically analyzed using natural language processing (NLP) techniques.

[1775] For example, extract key keywords such as "cough" and "sore throat."

[1776] The audio data is subjected to noise removal processing and specific frequencies are analyzed to extract cough patterns and characteristics.

[1777] For example, the intensity and frequency of coughing sounds are analyzed.

[1778] The image data is subjected to filtering and pre-processing to improve the image quality.

[1779] A Gaussian filter is used for filtering, and image resolution is adjusted for preprocessing.

[1780] Input: Text data, audio data, and image data sent from the device

[1781] Output: Preprocessed text data, audio data, and image data

[1782] Step 4:

[1783] Data analysis

[1784] server:

[1785] Preprocessed text data, audio data, and image data are passed to a generative AI model.

[1786] A generative AI model analyzes each piece of data in detail.

[1787] From the text data, major symptoms such as "last three days," "cough," and "sore throat" are detected.

[1788] Cough characteristics (e.g., frequency, intensity) are analyzed from the audio data.

[1789] The image data is used to evaluate the condition of the throat and determine whether inflammation is present.

[1790] Input: Preprocessed text data, audio data, and image data

[1791] Output: Analysis results for each data

[1792] Step 5:

[1793] Generating diagnostic results

[1794] server:

[1795] The results of each data analysis are integrated to generate a comprehensive diagnostic result.

[1796] For example, it generates a diagnosis such as "You may have mild bronchitis" and compiles a report with recommended treatment methods.

[1797] Input: Analysis results of text data, audio data, and image data

[1798] Output: A report containing the diagnosis and recommended treatment.

[1799] Step 6:

[1800] Sending and displaying diagnostic results

[1801] server:

[1802] The generated diagnostic results are transmitted to the terminal via secure communication.

[1803] Device:

[1804] The received diagnostic results are displayed to the user.

[1805] For example, it might say, "You may have mild bronchitis. We recommend you drink plenty of fluids and get plenty of rest. If necessary, seek medical advice."

[1806] Input: Diagnostic results sent from the server

[1807] Output: Diagnostic results displayed on the user's terminal

[1808] (Application example 1)

[1809] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1810] In recent years, there has been an increasing need for systems that allow individuals to easily diagnose their health conditions from home or on the go. However, there is a lack of systems that can comprehensively analyze text data, voice data, and image data to provide highly reliable diagnoses. There is also a need for systems that can efficiently process user-entered data and quickly display diagnostic results. A technical method is needed to solve these issues.

[1811] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1812] In this invention, the server includes means for a user to input text data, voice data, and image data, means for tokenizing the text data and performing grammatical analysis, means for removing noise and processing the voice data, means for preprocessing the image data, means for generating a diagnosis result based on the analysis results of the text data, voice data, and image data, means for displaying the diagnosis result to the user, means for collecting data from a smartphone terminal and transmitting it to the server, and means for generating a diagnosis result based on a prompt sentence using a generative AI model, thereby enabling a user to quickly and accurately diagnose their own health condition from home or on the go.

[1813] A "user terminal" is an electronic device that allows a user to input, send, and receive text data, voice data, and image data.

[1814] "Text data" refers to text information entered by a user, and is data for performing natural language processing.

[1815] "Audio data" refers to audio information recorded by a user, and is data for performing acoustic analysis and noise removal processes.

[1816] "Image data" is visual information captured by a user using a camera, and is data for preprocessing.

[1817] "Tokenization" is a process of dividing text data into small units, and is a preprocessing step for grammatical analysis.

[1818] "Grammar analysis" is a process of analyzing the grammatical structure of tokenized text data.

[1819] "Noise reduction" is a process that removes unnecessary sounds such as background noise from audio data.

[1820] "Preprocessing" is a process for converting image data into a format that is easy to analyze.

[1821] The "diagnosis result" is estimated information about the health condition that is generated based on the analyzed text data, voice data, and image data.

[1822] The "means for displaying to the user" is a function or device for providing the diagnostic results to the user in a format that is easy to view.

[1823] A "smartphone terminal" is a small, portable electronic device that is used to collect and transmit user data.

[1824] A "server" is a computer system that receives, analyzes, and transmits data over a network.

[1825] A "generative AI model" is an artificial intelligence model that automatically generates diagnostic results based on input data.

[1826] A "prompt sentence" is an instruction sentence used as input to a generative AI model.

[1827] This system is designed to allow users to easily diagnose their own health condition from home or on the go. It analyzes text, voice, and image data entered by the user and provides diagnostic results. The components of the system and specific processing steps are shown below.

[1828] System configuration

[1829] User terminal

[1830] The user terminal includes a portable device such as a smartphone or tablet. The user inputs the following data through the application:

[1831] 1. Text data input: The user inputs information about symptoms and physical condition in text format.

[1832] 2. Recording voice data: The user records coughing, breathing, etc. and inputs it as voice data.

[1833] 3. Capturing image data: The user uses a camera to capture an image of the affected area.

[1834] server

[1835] The server is a central computing device that receives and analyzes data sent from the terminals. The server is configured as follows:

[1836] 1. Data reception: Receives text data, voice data, and image data sent from the user terminal.

[1837] 2. Preprocessing: The received data is processed by various preprocessing modules. Text data is tokenized and grammatically analyzed, audio data is denoised, and image data is filtered and format converted.

[1838] 3. Data analysis: The preprocessed data is then further analyzed by generative AI models: text data is analyzed for key symptoms and related symptoms, audio data is analyzed for specific acoustic features, and image data is analyzed for specific patterns.

[1839] 4. Generating a diagnosis: The analysis results are combined to generate a diagnosis based on the user's symptoms, which may include recommended treatments or further medical consultations.

[1840] 5. Sending the diagnosis result: Send the generated diagnosis result to the user terminal.

[1841] Hardware and software used

[1842] User devices: smartphones, tablets

[1843] Speech processing software: speech_recognition library

[1844] Image processing software: OpenCV library

[1845] Sending data: the requests library

[1846] Generative AI models: various AI diagnostic models, such as those provided by the transformers library

[1847] Specific examples

[1848] The user launches a smartphone application, enters the text "I've had a persistent cough for the past three days and my throat hurts," records the coughing sound, and takes an image of their throat. The user's device sends this data to the server. The server receives the data, performs grammatical analysis on the text data, removes noise from the audio data, and filters the image data. It then analyzes the data based on the generative AI model and generates a diagnosis: "You may have mild bronchitis. We recommend you drink plenty of fluids and rest. If necessary, consult a doctor." This is then sent to the user's device.

[1849] Prompt Sentence Examples

[1850] User input: "I've had a constant cough and sore throat for the past three days."

[1851] Audio Data: [Recorded coughing sound]

[1852] Image Data: [User-taken image of throat]

[1853] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1854] Step 1:

[1855] A user starts the application using a smartphone and inputs text, voice, and image data. Specifically, the user inputs text such as "I've had a persistent cough for the past three days and my throat hurts," records the sound of their cough, and takes an image of their throat.

[1856] Input: Text data, audio data, image data

[1857] Output: Input text, recorded audio files, captured image files

[1858] Step 2:

[1859] The device prepares the input data and sends it to the server using a secure communication protocol (e.g., HTTPS). The data is packetized as text, audio files, or image files.

[1860] Input: text, audio files, image files

[1861] Output: Packetized data

[1862] Step 3:

[1863] The server receives packetized data sent from the terminal, identifies the received data, and initiates appropriate pre-processing for each data type.

[1864] Input: Packetized data

[1865] Output: Received text data, audio data, and image data

[1866] Step 4:

[1867] The server tokenizes the received text data and performs grammatical analysis, extracting the grammatical structure and semantic features of the text data, for example, identifying key symptoms such as "cough" or "sore throat."

[1868] Input: Text data

[1869] Output: Tokenized text, grammar analysis results

[1870] Step 5:

[1871] The server processes the audio data through noise reduction and analyzes its acoustic features to identify coughs and other distinctive acoustic patterns, such as the frequency range and duration of a cough.

[1872] Input: Audio data

[1873] Output: Noise-removed audio data, acoustic analysis results

[1874] Step 6:

[1875] The server performs preprocessing on the image data, such as filtering and format conversion, to convert the image data into a format that is easier to analyze. For example, contrast adjustment and edge detection are performed on the throat image.

[1876] Input: Image data

[1877] Output: Preprocessed image data

[1878] Step 7:

[1879] The server passes the preprocessed text, audio, and image data to the generative AI model for detailed data analysis, such as analyzing the prompt sentences and generating a comprehensive diagnosis.

[1880] Input: Tokenized text, acoustic analysis results, preprocessed image data

[1881] Output: Diagnostic results

[1882] Step 8:

[1883] The server integrates the generated diagnostic results and generates appropriate feedback to the user, including recommended treatments and next steps (e.g., consulting a medical professional).

[1884] Input: Diagnostic results

[1885] Output: Consolidated diagnostic report

[1886] Step 9:

[1887] The server sends the diagnosis results to the user's device, which then displays the received diagnosis results in an easy-to-read format. For example, it might say, "You may have mild bronchitis. We recommend you drink plenty of fluids and get plenty of rest."

[1888] Input: Consolidated diagnostic report

[1889] Output: Diagnostic results displayed to the user

[1890] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1891] This invention combines a system that allows users to diagnose their own health condition using text data, voice data, and image data with an emotion engine that recognizes the user's emotions, thereby providing comprehensive diagnostic results that take the user's emotional state into consideration.

[1892] System configuration

[1893] User terminal

[1894] 1. Data input interface:

[1895] The user launches a dedicated application and inputs text data about the symptoms.

[1896] The user uses a microphone to record audio data such as coughing, and a camera to take an image of the affected area.

[1897] 2. Data transmission function:

[1898] The terminal converts the input text data, voice data, and image data into packets and transmits them to the server using a secure communication protocol (for example, HTTPS).

[1899] server

[1900] 3. Data reception and preprocessing:

[1901] The server receives data packets sent from the terminal and separates them by data type: text data is sent to the text processing module, voice data is sent to the voice processing module, and image data is sent to the image processing module.

[1902] 4. Data Analysis:

[1903] The text data is tokenized and grammatically analyzed using a natural language processing (NLP) model to identify primary symptoms and associated secondary symptoms.

[1904] The audio data is denoised and filtered, and then the cough pattern is analysed through a speech analysis model.

[1905] The image data is preprocessed and an image analysis model is used to detect patterns of inflammation and swelling.

[1906] 5. Emotion analysis:

[1907] The server's emotion engine analyzes text data, voice data, and image data to recognize the user's emotional state.

[1908] From the text data, emotions are identified based on the user's writing style and expression.

[1909] Emotions are analyzed from audio data based on fluctuations in tone and pitch.

[1910] Emotions are recognized from image data based on facial expressions and body movements.

[1911] 6. Generating diagnostic results:

[1912] The server integrates the data analysis results with the emotion analysis results to generate a comprehensive diagnosis, which also reflects the user's emotional state.

[1913] The diagnosis will include recommended treatment and medical consultation if necessary.

[1914] 7. Sending and viewing diagnostic results:

[1915] The server sends the generated diagnostic results to the terminal, where an application displays the diagnostic results to the user.

[1916] Specific examples

[1917] Diagnosis by combining sore throat, cough, and emotion analysis

[1918] 1. Data Entry:

[1919] The user enters the text "I've had a persistent cough for the past three days and my throat hurts," records and sends an audio recording of the cough, and also takes a photo of their face.

[1920] 2. Data transmission:

[1921] The device sends text, audio, and image files to the server.

[1922] 3. Data reception and preprocessing:

[1923] The server receives the data, analyzes the text using NLP models, denoises the audio data, and preprocesses the image data.

[1924] 4. Data Analysis:

[1925] It extracts key symptoms from text, analyzes cough characteristics from audio, and detects patterns of inflammation and swelling from images.

[1926] 5. Emotion analysis:

[1927] The server uses an emotion engine to analyze stress and anxiety from text, tension from audio, and sadness and fatigue from images.

[1928] 6. Generating diagnostic results:

[1929] The server combines the results of the data analysis and the emotion analysis and generates a diagnosis that reads, "You may have mild bronchitis, but you appear to be under stress. We recommend that you drink plenty of fluids and get plenty of rest. Also, if you need psychological support, please consult a specialist."

[1930] 7. Sending and viewing diagnostic results:

[1931] The server sends the diagnostic results to the terminal, which displays them to the user.

[1932] The above is a specific embodiment of the present invention. By using this system, users can receive diagnostic results that take into account their physical and emotional state comprehensively, and obtain appropriate treatment and countermeasures.

[1933] The processing flow will be explained below.

[1934] Step 1:

[1935] The user starts a dedicated symptom recording application and enters text data about the symptoms. For example, the user might write "I've had a persistent cough and a sore throat for the past three days" in the text box.

[1936] Step 2:

[1937] The user uses a microphone to record the sound of their cough, as well as take a photo of their face and the affected area, all of which is uploaded to the application.

[1938] Step 3:

[1939] The device combines the input text, voice, and image data into a single data packet, which is then sent to the server using a secure communication protocol (e.g., HTTPS).

[1940] Step 4:

[1941] The server receives data packets sent from the device and separates them by data type (text, audio, image), and passes each data to the corresponding analysis module.

[1942] Step 5:

[1943] The server passes the text data to a natural language processing (NLP) model for tokenization and grammatical analysis, which identifies the primary symptom and associated secondary symptoms.

[1944] Step 6:

[1945] The server passes the audio data to a speech processing module for noise reduction and filtering, and then a speech analysis model analyzes the audio characteristics (e.g., coughing patterns).

[1946] Step 7:

[1947] The server passes the image data to the image processing module for preprocessing (filtering, normalization), after which an image analysis model detects patterns of inflammation and swelling.

[1948] Step 8:

[1949] The server passes the analyzed text data to an emotion engine to identify the user's emotions. For example, if there are many negative expressions, it can detect stress or anxiety.

[1950] Step 9:

[1951] The server also analyzes emotions from the voice data, identifying emotions such as tension or impatience based on the tone and pitch of the voice.

[1952] Step 10:

[1953] The server uses the image data to analyze the user's facial expressions and physical state to identify emotions, such as sadness or fatigue from facial distortion or puffy eyes.

[1954] Step 11:

[1955] The server integrates the data obtained from each analysis module (text, audio, image analysis results, and emotion analysis results) to generate a comprehensive diagnosis that also reflects the user's emotional state.

[1956] Step 12:

[1957] The server sends the generated diagnostic results to the device, where they are transmitted in a secure format and displayed in the device application.

[1958] Step 13:

[1959] The device will display the diagnosis results it receives to the user, such as "You may have mild bronchitis, but you appear to be under stress. We recommend you drink plenty of fluids and get plenty of rest. Also, if you need psychological support, please consult a specialist."

[1960] Example 2

[1961] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1962] Conventional health checkup systems provide diagnostic results based on the analysis of text data, voice data, and image data collected from users, but do not take the user's emotional state into consideration. As a result, comprehensive health diagnosis is insufficient and advice on the user's emotional health is lacking. The present invention aims to provide more accurate comprehensive diagnostic results by analyzing data including the user's emotional state.

[1963] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1964] In this invention, the server includes a means for tokenizing and syntactically analyzing text data, a means for denoising and processing audio data, a means for preprocessing image data, and a means for analyzing emotional states, thereby enabling the generation and display of comprehensive diagnostic results based on the analysis results of the text data, audio data, and image data, as well as the emotional states.

[1965] "User" refers to a person who uses the system to input data to diagnose their own health condition.

[1966] "Text data" refers to character information such as symptoms and emotions entered by the user.

[1967] "Audio data" refers to acoustic information such as coughs and voices recorded by a user.

[1968] "Image data" refers to image information such as the affected area and facial expression photographed by the user.

[1969] "Tokenization" refers to the process of breaking down text data into smaller units such as words or phrases.

[1970] "Syntax analysis" refers to the process of analyzing the grammatical structure of tokenized text data.

[1971] "Noise reduction" refers to the process of removing unnecessary noise from audio data.

[1972] "Preprocessing" refers to the process of formatting data to make it easier to analyze.

[1973] "Diagnostic results" refer to health assessments and recommended treatments generated based on data and sentiment analysis.

[1974] "Emotional state" refers to a user's mental state, such as stress, anxiety, tension, sadness, etc.

[1975] "Comprehensive diagnostic results" refer to diagnostic information obtained by taking into account both physical symptoms and emotional state.

[1976] "Means for displaying" refers to a method or device for delivering the generated diagnostic results to a user.

[1977] MODE FOR CARRYING OUT THE INVENTION

[1978] The present invention is a system that allows users to diagnose their own health condition using text data, voice data, and image data, and also provides comprehensive diagnostic results that take into account the user's emotional state. This system is composed of a user terminal and a server.

[1979] User terminal

[1980] 1. Data Entry Interface

[1981] The user launches a dedicated application and enters text data about their symptoms. For example, they can enter, "I've had a persistent cough and a sore throat for the past three days."

[1982] The user uses a microphone to record audio data such as coughing, and a camera to take an image of the affected area.

[1983] 2. Data transmission function

[1984] The terminal converts the input text data, voice data, and image data into packets and transmits them to the server using a secure communication protocol (for example, HTTPS).

[1985] server

[1986] 3. Data Reception and Preprocessing

[1987] The server receives data packets sent from the terminal and separates them by data type (passing text data to the text processing module, voice data to the voice processing module, and image data to the image processing module).

[1988] The main technologies used in this case include NLP (Natural Language Processing) modules, voice analysis models, and image analysis models.

[1989] 4. Data Analysis

[1990] Text data analysis: The server uses NLP models to tokenize the text data and perform grammatical analysis, which identifies primary and secondary symptoms, such as "cough" and "sore throat."

[1991] Voice data analysis: The server performs noise reduction and filtering on the voice data, and then analyzes the cough pattern using a voice analysis model.

[1992] Image data analysis: The server preprocesses the image data and uses image analysis models to detect patterns of inflammation and swelling.

[1993] 5. Emotion analysis

[1994] The server's emotion engine analyzes text, voice, and image data to recognize the user's emotional state.

[1995] Sentiment analysis from text: Identify emotions based on the user's writing style and expressions. For example, analyze stress from the emphasis on a period such as "3 days" in the text.

[1996] Emotion analysis from audio: Analyzes tone and pitch fluctuations in audio data to detect tension.

[1997] Emotion analysis from images: Analyzing facial expressions and body movements to recognize feelings of sadness or fatigue.

[1998] 6. Generating diagnostic results

[1999] The server integrates the data analysis results with the emotion analysis results to generate a comprehensive diagnosis, which also reflects the user's emotional state.

[2000] For example, it generates a diagnosis such as, "You may have mild bronchitis, but you appear to be under stress. We recommend you drink plenty of fluids and get plenty of rest. Also, if you need psychological support, consult a specialist."

[2001] 7. Sending and displaying diagnostic results

[2002] The server transmits the generated diagnostic results to the terminal, and an application on the terminal displays the diagnostic results to the user.

[2003] This system allows users to obtain comprehensive health checkup results that take into account not only their physical symptoms but also their emotional state.

[2004] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2005] Program processing flow

[2006] Step 1: User Data Entry

[2007] Using a dedicated application, users input text data about their symptoms. For example, they can describe specific symptoms such as, "I've had a persistent cough for the past three days, and my throat hurts." In addition, the system collects audio and image data by recording the sound of the cough using a microphone and taking images of the affected area using a camera.

[2008] Input: Text data describing symptoms, recorded voice data, photographed image data

[2009] Output: Collected dataset (text data, audio data, image data)

[2010] Step 2: Data collection and transmission by the device

[2011] The device collects text data entered by the user, recorded voice data, and captured image data, then encrypts this data using the HTTPS protocol and securely transmits it to a server.

[2012] Input: Collected datasets (text data, audio data, image data)

[2013] Output: Dataset sent securely to server

[2014] Step 3: Data reception and preprocessing by the server

[2015] The server receives data packets sent from the device. The received data is first classified into text data, voice data, and image data. The text data is then passed to the NLP module, the voice data to the voice analysis module, and the image data to the image analysis module.

[2016] Input: Securely transmitted dataset

[2017] Output: Classified dataset (text data, audio data, image data)

[2018] Step 4: Analyzing the text data

[2019] The server uses an NLP model to tokenize and parse the received text data, extracting key symptoms such as "cough" or "sore throat."

[2020] Input: Text data

[2021] Output: Extracted main and secondary symptoms

[2022] Step 5: Analyzing the audio data

[2023] The server denoises and filters the audio data, then uses a speech analysis model to analyze cough patterns.

[2024] Input: Audio data

[2025] Output: Analyzed cough pattern

[2026] Step 6: Analyzing the image data

[2027] The server preprocesses the image data and uses image analysis models to detect patterns of inflammation and swelling.

[2028] Input: Image data

[2029] Output: Detected inflammation and swelling patterns

[2030] Step 7: Sentiment Analysis

[2031] The server's emotion engine analyzes text data, voice data, and image data to recognize the user's emotional state. Specifically, it identifies emotions from text data based on writing style and expression, analyzes tone and pitch fluctuations from voice data, and analyzes facial expressions and body movements from image data.

[2032] Input: text data, audio data, image data

[2033] Output: User's emotional state

[2034] Step 8: Generate comprehensive diagnostic results

[2035] The server combines the data analysis results with the sentiment analysis results to generate a comprehensive diagnosis. This diagnosis includes recommended treatment and, if necessary, consultation with a medical institution. For example, a diagnosis might read, "You may have mild bronchitis, but you appear to be under stress. We recommend you drink plenty of fluids and rest. We also recommend consulting a specialist if you need psychological support."

[2036] Input: Data analysis results, emotion analysis results

[2037] Output: Comprehensive diagnostic results

[2038] Step 9: Send and view diagnostic results

[2039] The server sends the generated diagnostic results to the terminal, and the terminal application displays the received diagnostic results to the user.

[2040] Input: Comprehensive diagnostic results

[2041] Output: Diagnostic results displayed to the user

[2042] Through the above processing steps, the user can receive a diagnosis result that takes into consideration the user's physical and emotional state comprehensively.

[2043] (Application example 2)

[2044] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2045] Conventional health checkup systems typically analyze symptoms and make diagnoses based on text data, voice data, and image data entered by the user. However, these systems were unable to consider the user's emotional state, making it difficult to provide comprehensive diagnostic results. Furthermore, it was difficult to propose services in physical stores that took into account the customer's health and emotional state, making improving customer satisfaction a challenge. Therefore, there was a need for a system that could analyze not only the user's health condition but also their emotional state, reflect this in the diagnostic results, and then propose appropriate products and services in physical stores.

[2046] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2047] In this invention, the server includes means for a user to input text data, voice data, and image data, means for tokenizing the text data and performing grammatical analysis, means for removing noise and processing the voice data, means for preprocessing the image data, means for generating a diagnostic result based on the analysis results of the text data, voice data, and image data, means for displaying the diagnostic result to the user, means for proposing products and services in a physical store, and means for performing emotion analysis on the text data, voice data, and image data and reflecting the emotion in the diagnostic result. This makes it possible to provide a comprehensive diagnostic result based on the user's health condition and emotional state, and to propose appropriate products and services in a physical store based on the result.

[2048] "User" refers to a person who uses the system to diagnose a health or emotional state.

[2049] "Text data" refers to data that includes character information entered by a user.

[2050] "Voice data" refers to data containing voice information input by a user.

[2051] "Image data" refers to data that includes information about an image captured by a user.

[2052] "Tokenization" refers to the process of dividing text data into smaller units such as words or phrases.

[2053] "Grammar analysis" refers to the process of analyzing the syntax and meaning of text data.

[2054] "Noise reduction" refers to the process of removing unnecessary background sounds and noise from audio data.

[2055] "Preprocessing" refers to the process of shaping or adjusting image data before analysis.

[2056] "Diagnosis results" refer to the evaluation results of the user's health condition and emotional state generated based on the analysis results of text data, audio data, and image data.

[2057] "Display means" refers to a device or method for displaying the diagnostic results to the user or store clerk.

[2058] "Brick and mortar store" refers to a physical location that offers goods and services.

[2059] "Product and service suggestions" refers to suggesting appropriate products and services based on the diagnostic results.

[2060] "Emotion analysis" refers to the process of analyzing a user's emotional state from text data, audio data, and image data.

[2061] This will help the reader to understand the invention more clearly.

[2062] MODE FOR CARRYING OUT THE INVENTION

[2063] The present invention is a system for analyzing the health and emotional state of customers in a physical store and recommending appropriate products and services based on the results. The system includes a user terminal, a server, and a store device that displays the summary results.

[2064] User terminal

[2065] The user terminal has the following functions:

[2066] Data Input Interface:

[2067] The user launches a dedicated application and inputs text data about the symptoms.

[2068] The user uses a microphone to record audio data such as coughing, and a camera to take an image of the affected area.

[2069] Data transmission function:

[2070] The terminal converts the input text data, voice data, and image data into packets and transmits them to the server using a secure communication protocol (for example, HTTPS).

[2071] server

[2072] The server has the following functions:

[2073] Data reception and preprocessing:

[2074] The server receives data packets sent from the terminal and separates them by data type: text data is sent to the text processing module, voice data is sent to the voice processing module, and image data is sent to the image processing module.

[2075] Data Analysis:

[2076] The text data is tokenized and grammatically analyzed using a natural language processing (NLP) model to identify primary symptoms and associated secondary symptoms.

[2077] The audio data is denoised and filtered, and then the cough pattern is analysed through a speech analysis model.

[2078] The image data is preprocessed and an image analysis model is used to detect patterns of inflammation and swelling.

[2079] Emotion analysis:

[2080] The server's emotion engine analyzes text data, voice data, and image data to recognize the user's emotional state.

[2081] From the text data, emotions are identified based on the user's writing style and expression.

[2082] Emotions are analyzed from audio data based on fluctuations in tone and pitch.

[2083] Emotions are recognized from image data based on facial expressions and body movements.

[2084] Generating diagnostic results:

[2085] The server integrates the data analysis results and sentiment analysis results to generate a comprehensive diagnosis.

[2086] The diagnosis will include recommended treatment and medical consultation if necessary.

[2087] Product and service suggestions for physical stores are also generated based on the diagnostic results.

[2088] Sending and viewing diagnostic results:

[2089] The server transmits the generated diagnostic results to the terminal and the store equipment. The application on the terminal displays the diagnostic results to the user, and the store equipment also displays the diagnostic results to the store clerk.

[2090] Specific examples

[2091] For example, a customer visits a physical store and enters text such as, "I've been sleep-deprived these past few days due to work stress. My throat is a little sore," records the sound of a cough, and takes a photo of their face. The device then sends this text, audio, and image data to a server. The server analyzes the data and generates a diagnosis that also takes into account their emotional state. Based on the results, a suggestion is displayed, such as, "We think you may be overworked due to stress. We recommend some relaxing herbal tea or aroma oil to relieve stress."

[2092] Prompt Sentence Examples

[2093] Customer: "I've been sleep deprived these past few days due to work stress. My throat is a bit sore."

[2094] Audio Recording: [Coughing]

[2095] Headshot: [Customer photo]

[2096] Input data:

[2097] Text: "I've been sleep deprived these past few days due to work stress. My throat is a bit sore."

[2098] Audio data: Coughing sound

[2099] Image data: customer photos

[2100] Prompt statement:

[2101] "Analyze the text, voice, and image data entered by the customer and generate a diagnosis that takes into account their health status and emotions. The results should also include recommendations for appropriate products and services."

[2102] As described above, the present invention can comprehensively diagnose a user's health and emotional state, and can also suggest appropriate products and services to customers in physical stores.

[2103] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2104] Processing Steps

[2105] Step 1: Data entry

[2106] The user launches a dedicated application and inputs information about their symptoms and emotional state. The input includes text data, voice data, and image data, and is input using the smartphone's keyboard, microphone, and camera, respectively.

[2107] Input: Text, audio, and image data about symptoms and emotional states

[2108] Output: Input dataset in the device

[2109] Step 2: Send data

[2110] The device then packets the input data and sends it to the server using the HTTPS protocol. The data is then organized into text, audio, and image formats and securely sent to the server.

[2111] Input: Input dataset in the device

[2112] Output: Sending a data packet to the server

[2113] Step 3: Data reception and preprocessing

[2114] The server receives data packets sent from the device and separates them into text data, voice data, and image data. Each data type is passed to a different pre-processing module: text data is tokenized, voice data is denoised, and image data is formatted and adjusted.

[2115] Input: Data packet sent

[2116] Output: Separated text data, audio data, and image data

[2117] Step 4: Text data analysis

[2118] The server uses a Natural Language Processing (NLP) model to tokenize and analyze the text data, identifying key and secondary symptoms.

[2119] Input: Text data

[2120] Output: Symptom identification analysis results

[2121] Step 5: Audio data analysis

[2122] The voice processing module filters the voice data and performs noise reduction, then analyzes the cough pattern through a voice analysis model to analyze the user's symptom characteristics and emotional state.

[2123] Input: Denoised audio data

[2124] Output: Cough pattern analysis results

[2125] Step 6: Image data analysis

[2126] The image processing module preprocesses the image data and uses image analysis models to detect patterns of inflammation and swelling, as well as recognize the user's emotional state from facial expressions and body movements.

[2127] Input: Preprocessed image data

[2128] Output: Analysis of inflammation and swelling patterns and emotional state

[2129] Step 7: Sentiment Analysis

[2130] The server's emotion engine analyzes the user's emotional state from text data, voice data, and image data. From the text data, it analyzes emotions based on the user's expression, from voice data, it analyzes emotions based on tone and pitch fluctuations, and from image data, it analyzes emotions based on facial expressions and body movements.

[2131] Input: Text data, audio data, image data

[2132] Output: Overall emotional state analysis

[2133] Step 8: Generate diagnostic results

[2134] The server integrates the results of data analysis and emotion analysis to generate a comprehensive diagnosis that reflects estimated symptoms and emotional state, and also includes recommendations for appropriate products and services.

[2135] Input: Data analysis results and sentiment analysis results

[2136] Output: Comprehensive diagnostic results

[2137] Step 9: Send and view diagnostic results

[2138] The server sends the generated diagnostic results to the terminal and the store equipment. The terminal application displays the diagnostic results to the user, and the store equipment displays them to the store clerk. This allows the user and the store clerk to confirm the appropriate products and services.

[2139] Input: Comprehensive diagnostic results

[2140] Output: Diagnostic results and recommendations displayed to the user and the store clerk

[2141] The above are the specific processing steps of the system that realizes the application example.

[2142] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2143] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2144] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2145] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2146] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2147] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2148] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2149] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2150] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2151] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2152] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2153] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2154] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2155] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2156] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2157] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2158] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2159] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2160] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2161] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2162] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2163] The following is further disclosed regarding the above embodiment.

[2164] (Claim 1)

[2165] A means for a user to input text data, voice data, and image data;

[2166] means for tokenizing the text data and performing grammatical analysis;

[2167] means for denoising and processing the audio data;

[2168] means for preprocessing the image data;

[2169] means for generating a diagnosis result based on the analysis results of the text data, voice data, and image data;

[2170] means for displaying the diagnostic results to a user;

[2171] A system including:

[2172] (Claim 2)

[2173] The system according to claim 1, wherein the system analyzes the symptom characteristics of the user from the voice data.

[2174] (Claim 3)

[2175] The system of claim 1, wherein the image data is used to identify inflammation and swelling.

[2176] "Example 1"

[2177] (Claim 1)

[2178] A means for a user to input text data, voice data, and image data;

[2179] means for tokenizing the text data and performing grammatical analysis;

[2180] means for denoising and processing the audio data;

[2181] means for preprocessing the image data;

[2182] A means for passing the text data, voice data, and image data to an AI model for analysis;

[2183] means for generating a diagnosis result based on the analysis result;

[2184] means for displaying the diagnostic results to a user;

[2185] means for transmitting the input data to a server using a secure communication protocol;

[2186] means for identifying the received data and preprocessing each data type;

[2187] A system including:

[2188] (Claim 2)

[2189] The system according to claim 1, wherein the system analyzes the user's symptom characteristics from the voice data and identifies the symptom based on specific acoustic features.

[2190] (Claim 3)

[2191] 10. The system of claim 1, wherein the image data is used to identify inflammation and swelling and analyze the results.

[2192] "Application Example 1"

[2193] (Claim 1)

[2194] A means for a user to input text data, voice data, and image data;

[2195] means for tokenizing the text data and performing grammatical analysis;

[2196] means for denoising and processing the audio data;

[2197] means for preprocessing the image data;

[2198] means for generating a diagnosis result based on the analysis results of the text data, voice data, and image data;

[2199] means for displaying the diagnostic results to a user;

[2200] A means for collecting data from the smartphone terminal and transmitting it to a server;

[2201] a means for generating a diagnosis based on the prompt using a generative AI model;

[2202] A system including:

[2203] (Claim 2)

[2204] The system according to claim 1, wherein the system analyzes the symptom characteristics of the user from the voice data.

[2205] (Claim 3)

[2206] The system of claim 1, wherein the image data is used to identify inflammation and swelling.

[2207] "Example 2: Combining Emotion Engines"

[2208] (Claim 1)

[2209] A means for a user to input text data, voice data, and image data;

[2210] means for tokenizing and parsing the text data;

[2211] means for denoising and processing the audio data;

[2212] means for preprocessing the image data;

[2213] means for generating a diagnosis result based on the analysis results of the text data, voice data, and image data and the emotional state;

[2214] means for displaying the diagnostic results to a user;

[2215] A system including:

[2216] (Claim 2)

[2217] The system of claim 1, further comprising: analyzing symptom characteristics and emotional state of the user from the voice data.

[2218] (Claim 3)

[2219] The system of claim 1, wherein the image data is used to identify inflammation, swelling, and the user's emotional state.

[2220] "Application example 2 when combining emotion engines"

[2221] Claim

[2222] (Claim 1)

[2223] A means for a user to input text data, voice data, and image data;

[2224] means for tokenizing the text data and performing grammatical analysis;

[2225] means for denoising and processing the audio data;

[2226] means for preprocessing the image data;

[2227] means for generating a diagnosis result based on the analysis results of the text data, voice data, and image data;

[2228] means for displaying the diagnostic results to a user;

[2229] A means of proposing products and services in physical stores,

[2230] means for performing emotion analysis from the text data, voice data, and image data and reflecting the analysis results in the diagnosis;

[2231] A system including:

[2232] (Claim 2)

[2233] The system of claim 1, further comprising analyzing symptom characteristics and emotional state of the user from the voice data.

[2234] (Claim 3)

[2235] 10. The system of claim 1, wherein the image data is used to identify inflammation and swelling and to recognize the user's emotional state. [Explanation of symbols]

[2236] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for a user to input text data, voice data, and image data; means for tokenizing the text data and performing grammatical analysis; means for denoising and processing the audio data; means for preprocessing the image data; means for generating a diagnosis result based on the analysis results of the text data, voice data, and image data; means for displaying the diagnostic results to a user; A system including:

2. The system according to claim 1 , further comprising: analyzing the symptom characteristics of the user from the voice data.

3. The system of claim 1 , wherein the image data is used to identify inflammation and swelling.

Images