System

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting spoken audio of a user; means for pre-processing the collected audio; means for extracting features from the pre-processed audio; means for using a generative AI model to detect anomalies using the extracted features; and means for sending a notification to the user when an anomaly is detected.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] As the elderly population increases, so do the number of elderly people living alone and those providing care while working. In this modern age, early symptoms of illness are often overlooked, making early detection of illnesses such as cerebral infarction particularly difficult. Currently, medical examinations are required at medical facilities, and self-diagnosis is difficult, leading to a high risk of the illness progressing rapidly and becoming severe. This invention aims to solve the above problem by providing a system that combines voice services with medical data to quickly detect early symptoms of illnesses such as cerebral infarction that affect voice and language. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides the following system. First, a means is provided for collecting speech sounds from the user's smartphone or landline phone that they use daily. This collected speech data is then transferred to a server, where it is subjected to preprocessing such as noise filtering. Next, a means is provided for extracting features such as wavelength and frequency from the preprocessed speech data. The features are then input into a generative AI model to detect abnormalities in the speech data. Finally, if an abnormality is detected, a means is provided for sending a notification to the user's device, encouraging the user to seek medical attention early. This makes it possible to detect symptoms of diseases such as cerebral infarction in the early stages and prevent them from becoming severe.

[0006] "Voice data" is a digital recording of the sound waves produced when a user speaks.

[0007] "Preprocessing" refers to the process of performing processes such as noise filtering and normalization to convert the collected audio data into an analyzable format.

[0008] A "feature" is a specific attribute (for example, fundamental frequency, formant frequency, energy spectrum, etc.) extracted from speech data.

[0009] A "generative AI model" is a model that uses artificial intelligence technology to analyze voice data and detect and predict anomalies.

[0010] "Anomaly detection" is the process of identifying abnormal voice patterns and determining the likelihood that a particular medical risk may exist.

[0011] "User notification" is a function that sends warnings and recommended messages to the user's device when an abnormality is detected. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0013] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0016] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0017] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0018] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0020] [First embodiment]

[0021] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0022] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0025] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0028] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0032] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0033] The "Voice Doctor" of this invention is a system that uses voice data to detect the early stages of symptoms that affect voice and language, such as cerebral infarction. This system is realized through a series of processes, including voice data collection, preprocessing, feature extraction, anomaly detection using a generative AI model, and notification to the user. Below, we will explain the details of each element and the operation of the entire system in detail.

[0034] Basic system configuration

[0035] 1. User provides voice data

[0036] Users provide voice data through everyday conversations or recording notes on their smartphones, etc. This voice data is automatically collected and temporarily stored by the device.

[0037] 2. Transfer of audio data by the terminal

[0038] The device transfers the collected voice data to a server, which transmits the data using a secure communication protocol. This communication is necessary to protect privacy and ensure data integrity.

[0039] 3. Preprocessing of audio data by the server

[0040] The server pre-processes the received audio data, including noise filtering and normalization, to obtain clean data suitable for analysis.

[0041] 4. Feature extraction by the server

[0042] The server extracts features such as wavelength, frequency, and energy spectrum from the preprocessed audio data. These features are important data that are input into the generative AI model.

[0043] 5. Anomaly detection using generative AI models

[0044] The server then runs a generative AI model based on the extracted features to detect abnormal patterns in the voice data. This AI model uses a pre-trained dataset and is capable of identifying anomalies with high accuracy.

[0045] 6. Server-Initiated User Notification

[0046] If an abnormality is detected, the server sends a notification to the user's device, which includes details of the abnormality and recommended actions (e.g., early consultation with a medical institution).

[0047] Specific examples of program processing

[0048] Example 1: Detecting anomalies during everyday conversation

[0049] 1. Audio data collection and transmission

[0050] The terminal collects the voice of the user talking to a friend over the phone and transfers it to the server.

[0051] 2. Audio data preprocessing

[0052] The server performs noise filtering and volume level normalization on the received audio data.

[0053] 3. Feature Extraction

[0054] The server extracts features such as fundamental frequency, formant frequency, and energy spectrum from the preprocessed speech data.

[0055] 4. Anomaly Detection Using AI Models

[0056] The server inputs the extracted features into a generative AI model to determine whether early symptoms of a stroke are suspected.

[0057] 5. User Notices

[0058] If an abnormality is detected, the server sends a notification to the user's smartphone and displays a message saying, "This is suspected to be an early symptom of a stroke. Please seek medical attention immediately."

[0059] Example 2: Expressing your thoughts on everyday life

[0060] 1. Audio data collection and transmission

[0061] The device collects audio data recorded by the user on their smartphone about their thoughts about the day and transfers it to a server.

[0062] 2. Audio data preprocessing

[0063] The server noise filters and normalizes the received audio data.

[0064] 3. Feature Extraction

[0065] The server extracts features such as wavelength and frequency from the preprocessed audio data.

[0066] 4. Anomaly Detection Using AI Models

[0067] The server inputs the extracted features into a generative AI model to determine whether an abnormal pattern exists.

[0068] 5. User Notices

[0069] If an abnormality is detected, the server will send a notification to the user's smartphone and display a message such as "An abnormality has been detected. Please seek medical attention immediately."

[0070] In this way, the "Voice Doctor" of the present invention is a system that combines voice data and medical data to detect early symptoms of cerebral infarction and other conditions early on and prevent the condition from becoming severe.

[0071] The processing flow will be explained below.

[0072] Step 1:

[0073] User provided voice data:

[0074] Users provide voice data through everyday conversations, recording notes on their smartphones, etc. For example, if a user is talking on the phone with a friend, the content of that conversation will be the subject of the service.

[0075] Step 2:

[0076] Device audio collection:

[0077] The device collects the user's speech in real time, and the speech data is temporarily stored in the device's memory.

[0078] Step 3:

[0079] Transferring data from device to server:

[0080] The device transfers the collected voice data to a server at regular intervals using a secure communication protocol.

[0081] Step 4:

[0082] Server receives data:

[0083] The server receives the voice data transferred from the terminal and stores it in storage.

[0084] Step 5:

[0085] Server preprocessing of audio data:

[0086] The server performs noise filtering on the received audio data, including Gaussian and bandpass filters.

[0087] Step 6:

[0088] Server-generated audio normalization:

[0089] The server normalizes the volume level of the noise-filtered audio data, a step that ensures data consistency.

[0090] Step 7:

[0091] Server-based feature extraction:

[0092] The server extracts features such as fundamental frequency, formant frequency, and energy spectrum from the preprocessed audio data. The extracted features are used as input data for the AI ​​model.

[0093] Step 8:

[0094] Server-based feature normalization:

[0095] The server normalizes the extracted features to a consistent scale and prepares them for analysis by the AI ​​model.

[0096] Step 9:

[0097] The server runs the AI ​​model:

[0098] The server inputs the normalized features into a generative AI model to detect abnormalities. This AI model has been trained in advance using data on the early symptoms of cerebral infarction.

[0099] Step 10:

[0100] Server's assessment of the anomaly:

[0101] The server evaluates the anomaly detection results output by the generative AI model, specifically determining whether the anomaly score exceeds a set threshold.

[0102] Step 11:

[0103] Server prepares user notification:

[0104] If an anomaly is detected, the server generates a notification message for the user, including the specific anomaly and recommended action.

[0105] Step 12:

[0106] Sending notifications from the server to the device:

[0107] The server sends the generated notification message to the user's terminal using a communication protocol that allows real-time notification.

[0108] Step 13:

[0109] Displaying notifications by device:

[0110] The user's device will then display the notification message received from the server on its screen, such as "You are suspected of having the early symptoms of a stroke. Please seek medical attention immediately."

[0111] Step 14:

[0112] User Action:

[0113] Based on the received notification, the user considers visiting a medical institution early and takes the necessary measures.

[0114] Step 15:

[0115] Continuous monitoring by server:

[0116] The server continues to monitor new voice data sent periodically by the user to provide ongoing health assessment.

[0117] Example 1

[0118] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0119] There is a lack of adequate tools for early detection of conditions that affect voice and language, such as cerebral infarction. This can delay necessary medical treatment, posing a risk of worsening the condition. There is also a need for systems that can efficiently and effectively collect, process, and analyze voice data.

[0120] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0121] In this invention, the server includes means for collecting user speech, means for preprocessing the collected speech data, means for extracting features from the preprocessed speech data, means for using a generative AI model to detect anomalies using the extracted features, means for sending a notification to the user when an anomaly is detected, means for using a secure communication protocol in transferring the speech data, means for notifying the user of the detected anomaly and recommended actions, and means for normalizing the volume level in preprocessing the speech data. This enables effective collection, processing, and analysis of speech data, early detection of early symptoms of cerebral infarction and other conditions, and prompt medical treatment.

[0122] "User" refers to an individual or organization that uses the system and provides voice data.

[0123] "Speech" refers to the voice uttered by the user, and refers to voice data collected in the form of everyday conversation or recording.

[0124] "Means for collection" refers to a method or apparatus for recording and temporarily storing a user's spoken voice using a device or application.

[0125] "Preprocessing means" refers to a method or device that performs processes such as noise filtering and volume normalization to convert collected audio data into a format suitable for analysis.

[0126] "Means for extracting features" refers to a method or device for extracting data characteristics necessary for analysis, such as fundamental frequency and energy spectrum, from preprocessed speech data.

[0127] "Means using a generative AI model" refers to a method or apparatus that executes a pre-trained artificial intelligence model to detect anomalies in audio data using extracted features.

[0128] "Means for detecting anomalies" refers to a method or device that uses a generative AI model to identify unhealthy conditions or abnormal patterns in audio data.

[0129] "Means for sending a notification to a user" refers to a method or apparatus for sending detected anomaly information to a user's device as a message or alert.

[0130] "Secure communications protocol" refers to a method of transmitting voice data while ensuring data privacy and integrity, such as through TLS or SSL.

[0131] "Volume level normalization" refers to a method or device that performs processing to adjust the volume of audio data to a certain standard.

[0132] "Content of anomaly" refers to information indicating the type and degree of anomaly detected by the generative AI model.

[0133] "Recommended actions" refers to instructions that indicate specific actions or measures that a user should take in response to a detected anomaly.

[0134] "Early symptoms of cerebral infarction" refers to symptoms that appear in the early stages of cerebral infarction, and refers to conditions that can be detected early by analyzing voice data.

[0135] The "Voice Doctor" system of the present invention uses voice data to detect the early stages of symptoms that affect voice and language, such as cerebral infarction. This system is built with the following hardware and software and performs specific data processing and calculations.

[0136] Overall system configuration

[0137] The system consists of a device (smartphone or voice input device) that collects the user's speech, a server that processes the data, software to detect anomalies using generative AI models, and a communication protocol that sends notifications to the user.

[0138] Hardware and software used

[0139] 1. Audio data collection

[0140] Hardware: Smartphones, devices with microphones

[0141] Software: Mobile applications and voice recording applications

[0142] 2. Data Transfer

[0143] Hardware: Internet-connected devices

[0144] Software: Communication protocols (TLS, SSL)

[0145] 3. Server Processing

[0146] Hardware: Server (cloud server, on-premise server)

[0147] Software: Python libraries (Librosa, TensorFlow, PyTorch), database systems

[0148] 4. Generative AI Models

[0149] Software: TensorFlow, PyTorch

[0150] Model: A pre-trained audio anomaly detection model

[0151] 5. Notification System

[0152] Hardware: Smartphones, computers

[0153] Software: Push notification API, mobile notification application

[0154] Detailed explanation of each element

[0155] 1. User provides voice data

[0156] Users provide voice data using a smartphone or other voice input device. When users record their daily conversations or thoughts about the day, this voice data is temporarily stored within the application.

[0157] 2. Transfer of audio data by the terminal

[0158] The device transmits this voice data to the server using a secure communication protocol (e.g., TLS or SSL), which protects the integrity and privacy of the data.

[0159] 3. Preprocessing of audio data by the server

[0160] The server performs preprocessing on the received audio data, including noise filtering and volume normalization, to obtain clean data suitable for analysis.

[0161] 4. Feature extraction by the server

[0162] The server extracts features such as fundamental frequency and energy spectrum from the preprocessed audio data using Librosa or similar software.

[0163] 5. Anomaly detection using generative AI models

[0164] The server inputs the extracted features into a generative AI model to detect abnormal patterns. This AI model uses a pre-trained dataset to identify anomalies with high accuracy. It uses the TensorFlow and PyTorch frameworks.

[0165] 6. Server-Initiated User Notification

[0166] If an abnormality is detected, the server uses the notification API to send a push notification to the user's device. The notification will include details of the detected abnormality and recommended actions. For example, the notification may contain a message such as, "This is suspected to be an early symptom of a stroke. Please seek medical attention immediately."

[0167] Specific examples

[0168] Example 1: Detecting anomalies during everyday conversation

[0169] Voice data collection and transmission

[0170] The terminal collects the audio of the user talking to a friend over the phone and transfers it to a server.

[0171] Audio data preprocessing

[0172] The server performs noise filtering and volume level normalization on the received audio data.

[0173] Feature extraction

[0174] The server extracts features such as fundamental frequency, formant frequency, and energy spectrum from the preprocessed audio data.

[0175] Anomaly detection using AI models

[0176] The server inputs the extracted features into a generative AI model to determine whether early symptoms of a stroke are suspected.

[0177] User Notifications

[0178] If an abnormality is detected, the server sends a notification to the user's smartphone and displays a message saying, "This is suspected to be an early symptom of a stroke. Please seek medical attention immediately."

[0179] Example 2: Expressing your thoughts on everyday life

[0180] Voice data collection and transmission

[0181] The device collects audio data recorded by the user on their smartphone about their thoughts about the day and transfers it to a server.

[0182] Audio data preprocessing

[0183] The server noise filters (e.g., removes background noise from) and normalizes the received audio data.

[0184] Feature extraction

[0185] The server extracts features such as wavelength and frequency from the preprocessed audio data (e.g., using Librosa).

[0186] Anomaly detection using AI models

[0187] The server inputs the extracted features into a generative AI model to determine whether an abnormal pattern exists.

[0188] User Notifications

[0189] If an abnormality is detected, the server sends a notification to the user's smartphone (e.g., notifying the user of the detailed analysis results of the call) and displays a message such as "An abnormality has been detected. Please seek medical attention immediately."

[0190] In this way, the "Voice Doctor" of the present invention is a system that efficiently collects, preprocesses, and analyzes voice data, detects abnormalities early, and prompts the user to take appropriate action. By clearly showing the types of data processing and data calculations that are performed through concrete examples, the operation of the system can be clearly understood.

[0191] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0192] Step 1:

[0193] Providing audio data

[0194] Users provide voice data using a smartphone or voice input device. For example, users can record their daily conversations or their thoughts about the day into the app. This data is temporarily stored within the app.

[0195] Input: User speech

[0196] Output: Audio data temporarily saved on the device

[0197] Specific behavior:

[0198] The user launches the smartphone application and taps the record button.

[0199] Once recording is complete, the audio file will be automatically saved to your device storage.

[0200] Step 2:

[0201] Transferring audio data

[0202] The device transmits the voice data to the server, and the communication is secured by using a secure communication protocol such as TLS or SSL.

[0203] Input: Audio data stored on the device

[0204] Output: Audio data sent to the server

[0205] Specific behavior:

[0206] The application on your device will begin uploading the audio file.

[0207] The data is sent to the server using a secure communication protocol.

[0208] The server confirms receipt of the data and returns a success response to the terminal.

[0209] Step 3:

[0210] Audio data preprocessing

[0211] The server preprocesses the received audio data, performing noise filtering and volume normalization to obtain clean data suitable for analysis.

[0212] Input: raw audio data transferred to the server

[0213] Output: Preprocessed, clean audio data

[0214] Specific behavior:

[0215] The server applies a noise filtering algorithm to remove background noise.

[0216] Next, a volume normalization process is performed to adjust the volume to a consistent level.

[0217] Step 4:

[0218] Feature extraction

[0219] The server extracts features from the preprocessed audio data, using a speech analysis library such as Librosa for this process.

[0220] Input: Preprocessed clean audio data

[0221] Output: Extracted feature data (fundamental frequency, energy spectrum, etc.)

[0222] Specific behavior:

[0223] The server uses Librosa to calculate features such as fundamental frequency and spectrograms from the audio data.

[0224] The extracted features are stored in a database.

[0225] Step 5:

[0226] Detecting Anomalies with Generative AI Models

[0227] The server inputs the extracted features into a generative AI model to detect anomalies. This AI model uses a pre-trained dataset to identify anomalies with high accuracy.

[0228] Input: Extracted feature data

[0229] Output: Judgment result on whether or not there is an abnormality and the type of abnormality

[0230] Specific behavior:

[0231] The server uses the TensorFlow or PyTorch framework to run generative AI models.

[0232] When the model detects an anomaly, the result is recorded in a log on the server.

[0233] Step 6:

[0234] User Notifications

[0235] If an abnormality is detected, the server sends a notification to the user's device, which includes the details of the abnormality and a recommended course of action (e.g., early medical consultation).

[0236] Input: Anomaly detection results

[0237] Output: Notification message sent to the user's terminal

[0238] Specific behavior:

[0239] The server uses a notification API to send the anomaly detection results and recommended actions to the user's smartphone.

[0240] A message will appear on the user's smartphone saying, "This is suspected to be an early symptom of a stroke. Please seek medical attention immediately."

[0241] Through these steps, Voice Doctor is able to collect, process, and analyze the user's voice data, enabling early detection of abnormalities.

[0242] (Application example 1)

[0243] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0244] Conventional voice recognition technologies and health management systems separate voice authentication and health status screening, resulting in inefficiency and reduced user convenience. Furthermore, there are cases where detection of health status abnormalities is delayed, preventing timely medical consultations. The present invention aims to solve these problems by simultaneously performing voice authentication and health status screening, improving user convenience and enabling early detection of health status abnormalities.

[0245] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0246] In this invention, the server includes means for collecting a user's speech, means for preprocessing the collected speech data, means for extracting features from the preprocessed speech data, means for using a generative AI model to detect anomalies using the extracted features, means for sending a notification to the user when an anomaly is detected, and means for simultaneously performing voice authentication and health screening, thereby making it possible to screen the user's health during the voice authentication process and immediately send a notification when an anomaly is detected.

[0247] "User" means an individual or corporation that uses the system.

[0248] "Speech" refers to data of words or voices uttered by a user.

[0249] "Means for collection" refers to devices or software that record and store the user's spoken voice.

[0250] The "preprocessing means" refers to a device or software that performs processing to remove noise from collected voice data and standardize the data.

[0251] A "means for extracting features" is a device or software that extracts important data such as frequency components and energy spectra from preprocessed audio data.

[0252] "Means for using a generative AI model" refers to devices or software that execute a trained artificial intelligence model to detect anomalies based on extracted features.

[0253] The "means for sending a notification" is a device or software that sends a warning or information to the user when an abnormality is detected.

[0254] "Means for simultaneously performing voice authentication and health screening" refers to devices or software that use a user's voice to verify their identity while simultaneously assessing their health.

[0255] The "VoiceAuth Doctor" system of the present invention can simultaneously perform voice authentication and health screening of users. This system is implemented as an application installed on smartphones, and aims to increase user convenience and detect health abnormalities early.

[0256] Basic system configuration

[0257] 1. User provides voice data

[0258] Users provide voice data when conducting voice authentication during everyday conversations or when making payments. When a user performs voice authentication on their smartphone, the voice data is automatically collected.

[0259] 2. Transfer of audio data by the terminal

[0260] The device then transfers the collected voice data to a server, where it is transmitted using a secure communication protocol to ensure privacy and data integrity.

[0261] 3. Preprocessing of audio data by the server

[0262] The server pre-processes the received audio data, including noise filtering and data normalization, to obtain clean data suitable for analysis.

[0263] 4. Feature extraction by the server

[0264] The server extracts features such as frequency components and energy spectra from the preprocessed audio data. For feature extraction, the numpy library is used, for example.

[0265] 5. Anomaly detection using generative AI models

[0266] The server runs a generative AI model based on the extracted features to detect abnormal patterns in the voice data. This AI model is pre-trained using, for example, the Keras library.

[0267] 6. Server-Initiated User Notification

[0268] If an abnormality is detected, the server sends a notification to the user's device. The notification includes the details of the abnormality and recommended actions (e.g., early consultation with a medical institution). The notification is sent using, for example, the Twilio library.

[0269] Specific examples

[0270] For example, when a user performs voice authentication during payment, the voice data is sent to the server and analyzed. If the analysis detects an abnormality, a notification is sent to the user's smartphone via SMS via Twilio saying, "This is suspected to be an early symptom of a stroke. Please seek medical attention immediately."

[0271] Example prompts for generative AI models

[0272] Determine whether an anomaly can be detected based on the features extracted from the following audio data. Feature data: [Feature array]

[0273] In this way, the system of the present invention can screen the user's health condition during the voice authentication process and send an immediate notification if an abnormality is detected, thereby achieving both user convenience and health management.

[0274] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0275] Step 1:

[0276] User-provided voice data

[0277] To perform voice authentication, the user speaks into their smartphone, and the voice recording is collected by the system.

[0278] Input: User speech

[0279] Output: Audio data

[0280] Step 2:

[0281] Transfer of audio data by terminal

[0282] The device transfers the collected voice data to a server using a secure communication protocol, which ensures privacy and data integrity.

[0283] Input: Audio data

[0284] Output: Audio data sent to the server

[0285] Step 3:

[0286] Preprocessing of audio data by the server

[0287] The server performs noise filtering and normalization on the received audio data, for example, normalizing the data using sklearn.preprocessing.StandardScaler.

[0288] Input: Transmitted audio data

[0289] Output: Preprocessed, clean audio data

[0290] Step 4:

[0291] Feature extraction by the server

[0292] The server extracts features such as frequency components and energy spectrum from the preprocessed audio data, for example, by performing FFT (Fast Fourier Transform) using the numpy library.

[0293] Input: Preprocessed audio data

[0294] Output: Extracted feature data

[0295] Step 5:

[0296] Server-generated AI model for anomaly detection

[0297] The server runs a generative AI model based on the extracted features to detect abnormal patterns, for example, by running an AI model pre-trained using the keras library.

[0298] Input: Feature data

[0299] Output: Anomaly detection result (normal / abnormal)

[0300] Step 6:

[0301] Server-based user notification

[0302] If an anomaly is detected, the server sends a notification to the user's device, including the details of the anomaly and recommended actions, such as an SMS notification using the Twilio library.

[0303] Input: Anomaly detection results

[0304] Output: Notification message (if an anomaly is detected)

[0305] The above steps realize a system that screens the user's health status during the voice authentication process and sends an immediate notification if an abnormality is detected.

[0306] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0307] This invention is a system that uses voice data to detect the early stages of symptoms that affect voice and language, such as cerebral infarction, and combines it with an emotion engine that recognizes the user's emotions, enabling more accurate anomaly detection and user notification. This system is realized through a series of processes, from collecting and analyzing voice data, to anomaly detection, emotion recognition, and user notification.

[0308] Basic system configuration

[0309] 1. User provides voice data

[0310] Users provide voice data through everyday conversations, recording notes on their smartphones, etc. For example, if a user is talking on the phone with a friend, the content of that conversation will be the subject of the analysis.

[0311] 2. Audio collection by device

[0312] The device collects the user's speech in real time, and the speech data is temporarily stored in the device's memory.

[0313] 3. Transferring data from the device to the server

[0314] The device transfers the collected voice data to a server at regular intervals using a secure communication protocol.

[0315] 4. Preprocessing of audio data by the server

[0316] The server performs preprocessing on the received audio data, such as noise filtering and normalization, to obtain clean data suitable for analysis.

[0317] 5. Feature extraction by the server

[0318] The server extracts features such as wavelength, frequency, and energy spectrum from the preprocessed audio data. These features are important data that are input into the generative AI model.

[0319] 6. Detecting Anomalies with Generative AI Models

[0320] The server then runs a generative AI model based on the extracted features to detect abnormal patterns in the voice data. This AI model uses a pre-trained dataset and is capable of identifying anomalies with high accuracy.

[0321] 7. Emotion Recognition with Emotion Engine

[0322] The server uses an emotion engine to analyze the voice data and recognize the user's emotional state. For example, it can detect when the user is feeling stressed or anxious.

[0323] 8. Reflecting Emotional Information in Anomaly Detection Results

[0324] The emotion engine uses the emotional information recognized by the system to reinforce or correct the anomaly detection results. For example, by taking into account not only the voice features but also the emotional information, anomaly detection can be performed with greater accuracy.

[0325] 9. Server-generated user notifications

[0326] If an anomaly is detected, the server generates a notification message that takes into account emotional information. This message includes the details of the anomaly and recommended actions (e.g., early medical consultation).

[0327] 10. Sending notifications from the server to the device

[0328] The server sends the generated notification message to the user's terminal using a communication protocol that allows real-time notification.

[0329] 11. Display of notifications by device

[0330] The user's device will then display the notification message received from the server on its screen, such as "You are suspected of having the early symptoms of a stroke. Please seek medical attention immediately."

[0331] 12. User Actions

[0332] Based on the received notification, the user considers visiting a medical institution early and takes the necessary measures.

[0333] 13. Continuous monitoring by server

[0334] The server continues to monitor new voice data sent periodically by the user to provide ongoing health assessment.

[0335] Specific examples

[0336] Example 1: Detecting anomalies during everyday conversations and taking emotional information into account

[0337] 1. Audio data collection and transmission

[0338] The terminal collects the voice of the user talking to a friend over the phone and transfers it to the server.

[0339] 2. Audio data preprocessing

[0340] The server performs noise filtering and volume level normalization on the received audio data.

[0341] 3. Feature Extraction

[0342] The server extracts features such as fundamental frequency, formant frequency, and energy spectrum from the preprocessed speech data.

[0343] 4. Anomaly detection using AI models

[0344] The server inputs the extracted features into a generative AI model to determine whether early symptoms of a stroke are suspected.

[0345] 5. Emotion Recognition by Emotion Engine

[0346] The server recognizes the user's emotions from the voice data and detects, for example, whether the user is feeling anxious or stressed.

[0347] 6. Reflecting emotional information

[0348] The server reinforces the anomaly detection results based on the emotional information and adjusts the notification content appropriately.

[0349] 7. User Notices

[0350] If an abnormality is detected, the server sends a notification to the user's smartphone and displays a message saying, "This is suspected to be an early symptom of a stroke. Please seek medical attention immediately." The server also takes into account the user's emotional state.

[0351] Example 2: Expressing your thoughts on everyday life

[0352] 1. Audio data collection and transmission

[0353] The device collects audio data recorded by the user on their smartphone about their thoughts about the day and transfers it to a server.

[0354] 2. Audio data preprocessing

[0355] The server noise filters and normalizes the received audio data.

[0356] 3. Feature Extraction

[0357] The server extracts features such as wavelength and frequency from the preprocessed audio data.

[0358] 4. Anomaly Detection Using AI Models

[0359] The server inputs the extracted features into a generative AI model to determine whether an abnormal pattern exists.

[0360] 5. Emotion Recognition by Emotion Engine

[0361] The server recognizes the user's emotions from the voice data and detects, for example, positive and negative emotions.

[0362] 6. Reflecting emotional information

[0363] The server reinforces the anomaly detection results based on emotional information and adjusts the notification content appropriately.

[0364] 7. User Notices

[0365] If an abnormality is detected, the server sends a notification to the user's smartphone, displaying a message such as "An abnormality has been detected. Please seek medical attention immediately." The content of this notification also takes into account the user's emotional state.

[0366] In this way, the present invention is a system that can detect early symptoms of conditions such as cerebral infarction at an early stage by combining voice data and medical data, and by taking emotional information into account, it can provide more appropriate responses and notifications.

[0367] The processing flow will be explained below.

[0368] Step 1:

[0369] User provided voice data:

[0370] Users provide voice data through everyday conversations, recording notes on their smartphones, etc. For example, if a user is talking on the phone with a friend, the content of that conversation will be the subject of the service.

[0371] Step 2:

[0372] Device audio collection:

[0373] The device collects the user's speech in real time, and the speech data is temporarily stored in the device's memory.

[0374] Step 3:

[0375] Transferring data from device to server:

[0376] The device transfers the collected voice data to a server at regular intervals using a secure communication protocol.

[0377] Step 4:

[0378] Server receives data:

[0379] The server receives the voice data transferred from the terminal and stores it in storage.

[0380] Step 5:

[0381] Server preprocessing of audio data:

[0382] The server performs noise filtering on the received audio data, including Gaussian and bandpass filters.

[0383] Step 6:

[0384] Server-generated audio normalization:

[0385] The server normalizes the volume level of the noise-filtered audio data, a step that ensures data consistency.

[0386] Step 7:

[0387] Server-based feature extraction:

[0388] The server extracts features such as fundamental frequency, formant frequency, and energy spectrum from the preprocessed audio data. The extracted features are used as input data for the AI ​​model.

[0389] Step 8:

[0390] Server-based feature normalization:

[0391] The server normalizes the extracted features to a consistent scale and prepares them for analysis by the AI ​​model.

[0392] Step 9:

[0393] The server runs the AI ​​model:

[0394] The server inputs the normalized features into a generative AI model to detect abnormalities. This AI model has been trained in advance using data on the early symptoms of cerebral infarction.

[0395] Step 10:

[0396] Server execution of emotion engine:

[0397] The server inputs the voice data into an emotion engine to recognize the user's emotional state, and the emotion engine analyzes the tone and rhythm of the voice to identify emotions.

[0398] Step 11:

[0399] Server's assessment of the anomaly:

[0400] The server combines the anomaly detection results output by the generative AI model with the recognition results of the emotion engine to make a final determination of the anomaly score, for example, determining whether the anomaly score exceeds a set threshold.

[0401] Step 12:

[0402] Server prepares user notification:

[0403] If an anomaly is detected, the server generates a notification message for the user, including the specific anomaly and recommended actions. The notification content can also be tailored based on emotion data.

[0404] Step 13:

[0405] Sending notifications from the server to the device:

[0406] The server sends the generated notification message to the user's terminal using a communication protocol that allows real-time notification.

[0407] Step 14:

[0408] Displaying notifications by device:

[0409] The user's device will then display the notification message received from the server on its screen, such as "You are suspected of having the early symptoms of a stroke. Please seek medical attention immediately."

[0410] Step 15:

[0411] User Action:

[0412] Based on the received notification, the user considers visiting a medical institution early and takes the necessary measures.

[0413] Step 16:

[0414] Continuous monitoring by server:

[0415] The server continues to monitor new voice data sent periodically by the user to provide ongoing health assessment.

[0416] Example 2

[0417] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0418] Systems that use voice data to detect early symptoms of neurological diseases such as cerebral infarction typically detect abnormalities based on voice features, but accurate anomaly detection is difficult because the user's emotional state can affect the results. Furthermore, when an abnormality is detected, the user is not always notified appropriately, which can lead to delayed responses. Furthermore, noise in the voice data can reduce detection accuracy.

[0419] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting user speech, means for preprocessing the collected speech data, means for extracting features from the preprocessed speech data, means for using a generative AI model that detects anomalies using the extracted features, means for recognizing emotions based on the speech data, means for correcting anomaly detection results using recognized emotional information, and means for sending a notification to the user taking the emotional information into consideration when an anomaly is detected. This enables highly accurate anomaly detection that takes emotional information into account and appropriate user notification.

[0420] A "user" is an individual who utilizes the system to provide their voice data and receive health monitoring.

[0421] "Speech" refers to voice data generated by a user speaking.

[0422] "Means for collecting" refers to hardware or software that captures the user's speech in real time and temporarily stores it.

[0423] "Preprocessing" refers to data processing to make collected voice data suitable for analysis, and specifically includes noise filtering and normalization.

[0424] "Features" refers to statistical data such as wavelength, frequency, and energy spectrum that are extracted from audio data and are the subject of analysis.

[0425] A "generative AI model" refers to an artificial intelligence algorithm that is trained using large amounts of training data to detect anomalies in voice data.

[0426] "Means for detecting anomalies" refers to the process of using extracted features to run a generative AI model and identify abnormal patterns.

[0427] "Means for recognizing emotions" refers to a computational process or algorithm for analyzing and identifying a user's emotional state from audio data.

[0428] "Emotion information" refers to emotion data recognized based on the user's voice data.

[0429] "Means for correcting anomaly detection results" refers to a process for reinforcing or correcting anomaly detection results based on recognized emotion information.

[0430] "Means for sending a notification" refers to a process for generating a notification message that takes into account emotional information when an anomaly is detected and sending it to the user's device.

[0431] MODE FOR CARRYING OUT THE INVENTION

[0432] The present invention is a system that uses voice data and emotional information to detect the early stages of neurological diseases such as cerebral infarction. This system is realized through a series of processes, from collecting and analyzing voice data, to detecting abnormalities, recognizing emotions, and notifying the user. Details of the hardware and software required to implement the present invention are described below.

[0433] Provision and collection of voice data

[0434] User-provided voice data

[0435] Users provide voice data through everyday conversations, recording notes on their smartphones, etc. For example, a user may be talking to a friend on the phone or recording their thoughts on their smartphone like a diary.

[0436] Audio collection by device

[0437] A device (e.g., a smartphone) uses a built-in microphone to collect the user's voice in real time. This voice data is temporarily stored in the device's memory and then transferred to a server. The collected voice data may be collected every minute or for specific events.

[0438] Audio data preprocessing and feature extraction

[0439] Transferring data from the device to the server

[0440] The device transfers the collected voice data to the server at regular intervals. Data transfer is performed using a secure communication protocol such as SSL / TLS, ensuring data security.

[0441] Preprocessing of audio data by the server

[0442] The server performs preprocessing on the received audio data, such as noise filtering (e.g., Wiener filtering) and normalization, to obtain clean data suitable for analysis.

[0443] Feature extraction by the server

[0444] The server extracts features such as fundamental frequency, formant frequency, and energy spectrum from the preprocessed speech data. These features are important information to input into the generative AI model.

[0445] Anomaly detection and emotion recognition

[0446] Detecting Anomalies with Generative AI Models

[0447] The server runs a generative AI model (e.g., a model implemented in TensorFlow or PyTorch) based on the extracted features to detect anomalous patterns in the audio data. The AI ​​model uses a pre-trained dataset to identify anomalies with high accuracy.

[0448] Emotion recognition by emotion engine

[0449] The server uses an emotion engine (e.g., IBM Watson's emotion analysis service) to analyze the voice data and recognize the user's emotional state. For example, it can determine whether the user is feeling stressed based on the tone and tempo of the voice.

[0450] User notification and continuous monitoring

[0451] Reflecting emotional information in anomaly detection results

[0452] The server reinforces or corrects the anomaly detection results based on the emotion information recognized by the emotion engine. By taking emotion information into account, anomaly detection can be performed with greater accuracy.

[0453] Server-generated user notifications

[0454] If an anomaly is detected, the server generates a notification message that takes into account emotional information. This message includes the details of the anomaly and recommended actions (e.g., early medical consultation).

[0455] Sending notifications from the server to the device

[0456] The server sends the generated notification message to the user's device using a communication protocol that allows real-time notifications (e.g., WebSocket).

[0457] Displaying notifications by device

[0458] The device displays the notification message received from the server to the user. For example, a message such as "You are suspected of having the early symptoms of a cerebral infarction. Please seek medical attention immediately" may be displayed.

[0459] Specific examples

[0460] Example 1: Detecting anomalies during everyday conversations and taking emotional information into account

[0461] The device collects audio of the user's phone conversations with friends and transfers it to the server at regular intervals. The server then noise-filters and normalizes the received audio data before extracting features. The extracted features are input into a generative AI model to detect patterns that may be early symptoms of cerebral infarction. At the same time, an emotion engine is used to recognize the user's emotional state, and this emotional information is used to reinforce the anomaly detection results. If an anomaly is detected, the server generates a notification message that takes the emotional information into account and sends it to the user's device. The device then displays this notification message to the user.

[0462] Example 2: Expressing your thoughts on everyday life

[0463] The device collects audio data recorded by the user on their smartphone about their thoughts about the day and transfers the data to a server. The server then noise-filters the received audio data and normalizes the volume. It then extracts features from the preprocessed audio data and inputs them into a generative AI model to determine whether an abnormal pattern exists. It then uses an emotion engine to recognize the user's emotions and reinforces the anomaly detection results based on that emotional information. If an abnormality is detected, the server sends a notification message to the user's device, such as "An abnormality has been detected. Please seek medical attention immediately," and displays it on the device.

[0464] Example prompts to input to the generative AI model

[0465] Prompt: I had a fun conversation with a friend today, but halfway through I started to feel tired. Then I got a headache. Is this a sign that something is wrong?

[0466] This system integrates voice data and emotional information to achieve more accurate anomaly detection and user notification, providing an effective means of monitoring users' health status.

[0467] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0468] Step 1:

[0469] User-provided voice data

[0470] Users provide voice data through everyday conversations or recording notes on their smartphones. For example, consider a situation where you are talking on the phone with a friend. In this case, the user's voice is the input. The voice is collected in real time by the device via the built-in microphone.

[0471] Step 2:

[0472] Audio collection by device

[0473] The device collects the user's speech in real time using a built-in microphone, and the user's speech is temporarily stored in the device's memory. At this time, the collected speech data is treated as input data, and the stored data is used as output.

[0474] Step 3:

[0475] Transferring data from the device to the server

[0476] The device transfers the collected voice data to the server at regular intervals. This is done based on specific timing or events (e.g., every minute or at the end of recording), and the voice data is sent to the server using a secure communication protocol (e.g., SSL / TLS). The input data is the voice data stored on the device, and the output is the data transferred to the server.

[0477] Step 4:

[0478] Preprocessing of audio data by the server

[0479] The server performs noise filtering (e.g., Wiener filtering) and normalization on the received audio data. Noise filtering removes background noise, and volume normalization equalizes the data. The input is the audio data transferred to the server, and the output is clean audio data that has been denoised and normalized.

[0480] Step 5:

[0481] Feature extraction by the server

[0482] The server extracts features such as fundamental frequency, formant frequency, and energy spectrum from the preprocessed speech data. It analyzes the speech waveform and quantifies the data by calculating these features. The input is clean speech data, and the output is the extracted feature data.

[0483] Step 6:

[0484] Detecting Anomalies with Generative AI Models

[0485] The server inputs the extracted features into a generative AI model to detect abnormal patterns in the voice data. The generative AI model operates based on a pre-trained dataset and identifies abnormal patterns that indicate, for example, the early symptoms of a stroke. The input is feature data, and the output is the result of anomaly detection.

[0486] Step 7:

[0487] Emotion recognition by emotion engine

[0488] The server uses an emotion engine based on the voice data to recognize the user's emotional state. It analyzes the tone, tempo, rhythm, etc. of the voice to identify emotions (e.g., stress, anxiety, joy). The input is voice data or feature data, and the output is the recognized emotional information.

[0489] Step 8:

[0490] Reflecting emotional information in anomaly detection results

[0491] The server corrects or reinforces the anomaly detection results based on the recognized emotional information. By taking emotional information into account, the severity and likelihood of an anomaly can be more accurately determined. The input is the anomaly detection results and emotional information, and the output is the corrected anomaly detection results.

[0492] Step 9:

[0493] Server-generated user notifications

[0494] If an anomaly is detected, the server generates a notification message that takes into account emotional information. This message includes the details of the anomaly and a recommended action (e.g., to seek medical advice). The input is the corrected anomaly detection result, and the output is the generated notification message.

[0495] Step 10:

[0496] Sending notifications from the server to the device

[0497] The server sends the generated notification message to the user's device in real time. The communication protocol used is one that allows real-time notification (e.g., WebSocket). The input is the notification message, and the output is the notification sent to the user's device.

[0498] Step 11:

[0499] Displaying notifications by device

[0500] The terminal displays the notification message received from the server to the user. The notification appears as a pop-up on the screen, and may say something like, "You are suspected of having an early symptom of a cerebral infarction. Please seek medical attention immediately." The input is the notification message, and the output is the notification displayed to the user.

[0501] Step 12:

[0502] User Action

[0503] The user checks the displayed notification message and takes appropriate action, such as immediately visiting a medical institution. The input is the displayed notification, and the output is the specific action taken by the user.

[0504] Step 13:

[0505] Continuous monitoring by server

[0506] The server continues to monitor new voice data periodically sent by the user and performs continuous health status assessments. This allows for continuous monitoring of the user's health status. The input is new voice data, and the output is a continuously updated health status assessment result.

[0507] (Application example 2)

[0508] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0509] Conventional health monitoring systems could detect health abnormalities based on voice data, but it was difficult to simultaneously consider the user's emotional state. This could result in inaccurate detection of abnormalities and insufficient appropriate responses. Furthermore, for occupations such as security guards, real-time notification displays on-site are important, but there was a lack of means to achieve this. Therefore, there was a need for a system that could improve the accuracy of health abnormality detection, consider the user's emotional state, and enable immediate on-site response.

[0510] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0511] In this invention, the server includes means for collecting user speech, means for preprocessing the collected speech data, means for extracting features from the preprocessed speech data, means for using a generative AI model to detect abnormalities using the extracted features, means for sending a notification to the user when an abnormality is detected, means for recognizing the user's emotional state based on the collected speech data, means for reflecting the user's emotional information in the abnormality detection result, and means for displaying a notification to the user via a head-mounted display. This improves the accuracy of detecting health abnormalities, enables appropriate responses that take the user's emotional state into consideration, and enables immediate on-site responses.

[0512] The "means for collecting the user's speech" refers to a device or method for recording the user's speech as digital data using a voice input device such as a microphone or recorder.

[0513] The "means for preprocessing collected voice data" refers to a method or device that performs noise filtering, normalization, etc. on the collected voice data to convert it into a state suitable for analysis.

[0514] The "means for extracting features from preprocessed speech data" refers to a method or device for extracting important parameters necessary for analysis, such as fundamental frequency, formant frequency, and energy spectrum, from preprocessed speech data.

[0515] "Means using a generative AI model to detect anomalies using extracted features" refers to algorithms or methods that analyze the features extracted by the generative AI model and compare them with normal values ​​to determine anomalies.

[0516] "Means for sending a notification to the user when an abnormality is detected" refers to a method or system for notifying the user of an abnormality when it is detected as a message on the screen of a mobile terminal or computer.

[0517] "Emotion recognition means for recognizing the emotional state of a user based on collected voice data" refers to an algorithm or system that analyzes the tone and pattern of a user's voice from voice data and determines the user's emotions (e.g., anger, sadness, joy, etc.).

[0518] "Means for reflecting user emotional information in anomaly detection results" refers to a method or system that takes into account the user's emotional state in response to detected anomaly information and adjusts the content and urgency of the notification message based on that.

[0519] "Means for displaying notifications to a user via a head-mounted display" refers to a method or system that visually displays abnormality detection results and other important information to a user using a wearable display device.

[0520] The system for implementing this invention includes a series of processes that collect and analyze voice data, detect health abnormalities and emotional states, and notify the user. The system includes the following components:

[0521] Acquisition and preprocessing of audio data

[0522] 1. Collecting user speech

[0523] The terminal collects the user's voice in real time using a voice input device such as a microphone or recorder.

[0524] 2. Preprocessing of audio data

[0525] The server performs preprocessing on the collected audio data, such as noise filtering and normalization, to obtain clean data suitable for analysis.

[0526] Feature extraction and anomaly detection

[0527] 3. Feature Extraction

[0528] From the preprocessed audio data, features such as fundamental frequency and energy spectrum are extracted, and these features are used as input data for generative AI models.

[0529] 4. Anomaly Detection

[0530] Based on the extracted features, the server uses a generative AI model to detect abnormal health patterns in the voice data. This generative AI model uses a pre-trained dataset to identify abnormalities with high accuracy.

[0531] Recognizing emotional states and reflecting them in anomaly detection results

[0532] 5. Recognizing emotional states

[0533] During the process of analyzing the voice data, the server uses emotion recognition means to recognize the user's emotional state, thereby determining whether the user is feeling stressed or relaxed.

[0534] 6. Reflecting emotional information

[0535] The recognized emotion information is used to reinforce or correct the anomaly detection results, which enables more accurate anomaly detection.

[0536] Notification method for extension

[0537] 7. Generating and Sending User Notifications

[0538] If an abnormality is detected, the server generates a notification message that takes into account emotional information and sends it to the user's device. This notification message includes the details of the abnormality and recommended actions (e.g., early consultation with a medical institution).

[0539] 8. Displaying notifications using a head-mounted display

[0540] The device (for example, a head-mounted display) displays the received notification message in the user's field of vision, allowing the user to immediately check the abnormality notification.

[0541] The specific hardware and software used

[0542] Hardware: Microphone, recorder, head-mounted display

[0543] Software: sounddevice, requests library, custom generative AI model (AISymptomDetector) and emotion engine (EmotionEngine)

[0544] Specific examples

[0545] Example 1: Security guard on patrol

[0546] Security guards wear head-mounted displays, and their speech while patrolling is collected and analyzed in real time. If an abnormal pattern is detected, a message appears on the head-mounted display stating, "An abnormality has been detected. Please seek medical attention immediately." If the guard is feeling stressed, the message "The user is feeling extremely stressed" is also displayed.

[0547] Examples of prompt statements

[0548] The voice data is 10 seconds of conversation data. Please detect the early symptoms of cerebral infarction from this data. Also, please use the emotion engine to recognize the emotional state and reflect it in the anomaly detection results.

[0549] This configuration improves the accuracy of detecting health abnormalities, and enables appropriate responses that take into account the user's emotional state and immediate responses on the spot.

[0550] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0551] Step 1:

[0552] Collecting user speech

[0553] The terminal collects the user's speech in real time. The user's speech is used as input. This is recorded as digital data using a voice input device such as a microphone or recorder. The output is digitized voice data.

[0554] Step 2:

[0555] Audio data preprocessing

[0556] The server receives the collected audio data and performs noise filtering and normalization on it. The input is the digitized audio data obtained in step 1. Noise filtering removes known external noise, and normalization ensures a consistent volume level. The output is clean, pre-processed audio data suitable for analysis.

[0557] Step 3:

[0558] Feature extraction

[0559] The server extracts features such as fundamental frequency and energy spectrum from the preprocessed speech data. The preprocessed speech data obtained in step 2 is used as input. Data analysis techniques are used to extract the characteristics of the speech data in the time and frequency domains. The extracted feature data is obtained as output.

[0560] Step 4:

[0561] Anomaly detection

[0562] The server runs a generative AI model using the extracted feature data to detect abnormal health patterns. The feature data obtained in step 3 is used as input. This data is input into the generative AI model to identify abnormal patterns (e.g., early symptoms of cerebral infarction). The output is an abnormality detection result.

[0563] Step 5:

[0564] Recognition of emotional states

[0565] The server uses emotion recognition means during the voice data analysis process to recognize the user's emotional state. The voice data collected in step 1 is used as input. The emotion recognition means analyzes the tone and pattern of the voice to determine the user's emotion (e.g., anger, sadness, joy, etc.). The output is the recognition result of the emotional state.

[0566] Step 6:

[0567] Reflecting emotional information

[0568] The server integrates the anomaly detection results and emotion recognition results to reinforce or correct the anomaly detection results. The inputs are the anomaly detection results obtained in step 4 and the emotional state recognition results obtained in step 5. Emotional information is added to the anomaly information and the notification content is adjusted. The output is the reinforced or corrected anomaly detection results.

[0569] Step 7:

[0570] Generate and send user notifications

[0571] The server generates a notification message for the user based on the corrected anomaly detection results. The input is the augmented or corrected anomaly detection results obtained in step 6. The server generates an appropriate notification message (e.g., recommended actions) and sends it to the user's device. The output is the notification message.

[0572] Step 8:

[0573] Displaying notifications using a head-mounted display

[0574] The device (head-mounted display) displays the received notification message in a form that can be seen by the user. The notification message generated in step 7 is used as input. The message is displayed in the user's field of vision in real time, prompting them to take an appropriate action. The output is a notification message that the user can visually confirm.

[0575] This will improve the system's accuracy in detecting health abnormalities, allowing for appropriate responses that take into account the user's emotional state and immediate on-site response.

[0576] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0577] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0578] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0579] [Second embodiment]

[0580] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0581] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0582] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0583] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0584] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0585] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0586] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0587] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0588] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0589] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0590] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0591] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0592] The "Voice Doctor" of this invention is a system that uses voice data to detect the early stages of symptoms that affect voice and language, such as cerebral infarction. This system is realized through a series of processes, including voice data collection, preprocessing, feature extraction, anomaly detection using a generative AI model, and notification to the user. Below, we will explain the details of each element and the operation of the entire system in detail.

[0593] Basic system configuration

[0594] 1. User provides voice data

[0595] Users provide voice data through everyday conversations or recording notes on their smartphones, etc. This voice data is automatically collected and temporarily stored by the device.

[0596] 2. Transfer of audio data by the terminal

[0597] The device transfers the collected voice data to a server, which transmits the data using a secure communication protocol. This communication is necessary to protect privacy and ensure data integrity.

[0598] 3. Preprocessing of audio data by the server

[0599] The server pre-processes the received audio data, including noise filtering and normalization, to obtain clean data suitable for analysis.

[0600] 4. Feature extraction by the server

[0601] The server extracts features such as wavelength, frequency, and energy spectrum from the preprocessed audio data. These features are important data that are input into the generative AI model.

[0602] 5. Anomaly detection using generative AI models

[0603] The server then runs a generative AI model based on the extracted features to detect abnormal patterns in the voice data. This AI model uses a pre-trained dataset and is capable of identifying anomalies with high accuracy.

[0604] 6. Server-Initiated User Notification

[0605] If an abnormality is detected, the server sends a notification to the user's device, which includes details of the abnormality and recommended actions (e.g., early consultation with a medical institution).

[0606] Specific examples of program processing

[0607] Example 1: Detecting anomalies during everyday conversation

[0608] 1. Audio data collection and transmission

[0609] The terminal collects the voice of the user talking to a friend over the phone and transfers it to the server.

[0610] 2. Audio data preprocessing

[0611] The server performs noise filtering and volume level normalization on the received audio data.

[0612] 3. Feature Extraction

[0613] The server extracts features such as fundamental frequency, formant frequency, and energy spectrum from the preprocessed speech data.

[0614] 4. Anomaly detection using AI models

[0615] The server inputs the extracted features into a generative AI model to determine whether early symptoms of a stroke are suspected.

[0616] 5. User Notices

[0617] If an abnormality is detected, the server sends a notification to the user's smartphone and displays a message saying, "This is suspected to be an early symptom of a stroke. Please seek medical attention immediately."

[0618] Example 2: Expressing your thoughts on everyday life

[0619] 1. Audio data collection and transmission

[0620] The device collects audio data recorded by the user on their smartphone about their thoughts about the day and transfers it to a server.

[0621] 2. Audio data preprocessing

[0622] The server noise filters and normalizes the received audio data.

[0623] 3. Feature Extraction

[0624] The server extracts features such as wavelength and frequency from the preprocessed audio data.

[0625] 4. Anomaly Detection Using AI Models

[0626] The server inputs the extracted features into a generative AI model to determine whether an abnormal pattern exists.

[0627] 5. User Notices

[0628] If an abnormality is detected, the server will send a notification to the user's smartphone and display a message such as "An abnormality has been detected. Please seek medical attention immediately."

[0629] In this way, the "Voice Doctor" of the present invention is a system that combines voice data and medical data to detect early symptoms of cerebral infarction and other conditions early on and prevent the condition from becoming severe.

[0630] The processing flow will be explained below.

[0631] Step 1:

[0632] User provided voice data:

[0633] Users provide voice data through everyday conversations, recording notes on their smartphones, etc. For example, if a user is talking on the phone with a friend, the content of that conversation will be the subject of the service.

[0634] Step 2:

[0635] Device audio collection:

[0636] The device collects the user's speech in real time, and the speech data is temporarily stored in the device's memory.

[0637] Step 3:

[0638] Transferring data from device to server:

[0639] The device transfers the collected voice data to a server at regular intervals using a secure communication protocol.

[0640] Step 4:

[0641] Server receives data:

[0642] The server receives the voice data transferred from the terminal and stores it in storage.

[0643] Step 5:

[0644] Server preprocessing of audio data:

[0645] The server performs noise filtering on the received audio data, including Gaussian and bandpass filters.

[0646] Step 6:

[0647] Server-generated audio normalization:

[0648] The server normalizes the volume level of the noise-filtered audio data, a step that ensures data consistency.

[0649] Step 7:

[0650] Server-based feature extraction:

[0651] The server extracts features such as fundamental frequency, formant frequency, and energy spectrum from the preprocessed audio data. The extracted features are used as input data for the AI ​​model.

[0652] Step 8:

[0653] Server-based feature normalization:

[0654] The server normalizes the extracted features to a consistent scale and prepares them for analysis by the AI ​​model.

[0655] Step 9:

[0656] The server runs the AI ​​model:

[0657] The server inputs the normalized features into a generative AI model to detect abnormalities. This AI model has been trained in advance using data on the early symptoms of cerebral infarction.

[0658] Step 10:

[0659] Server's assessment of the anomaly:

[0660] The server evaluates the anomaly detection results output by the generative AI model, specifically determining whether the anomaly score exceeds a set threshold.

[0661] Step 11:

[0662] Server prepares user notification:

[0663] If an anomaly is detected, the server generates a notification message for the user, including the specific anomaly and recommended action.

[0664] Step 12:

[0665] Sending notifications from the server to the device:

[0666] The server sends the generated notification message to the user's terminal using a communication protocol that allows real-time notification.

[0667] Step 13:

[0668] Displaying notifications by device:

[0669] The user's device will then display the notification message received from the server on its screen, such as "You are suspected of having the early symptoms of a stroke. Please seek medical attention immediately."

[0670] Step 14:

[0671] User Action:

[0672] Based on the received notification, the user considers visiting a medical institution early and takes the necessary measures.

[0673] Step 15:

[0674] Continuous monitoring by server:

[0675] The server continues to monitor new voice data sent periodically by the user to provide ongoing health assessment.

[0676] Example 1

[0677] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0678] There is a lack of adequate tools for early detection of conditions affecting voice and language, such as cerebral infarction. This can delay necessary medical treatment, posing a risk of worsening the condition. There is also a need for systems that can efficiently and effectively collect, process, and analyze voice data.

[0679] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0680] In this invention, the server includes means for collecting user speech, means for preprocessing the collected speech data, means for extracting features from the preprocessed speech data, means for using a generative AI model to detect anomalies using the extracted features, means for sending a notification to the user when an anomaly is detected, means for using a secure communication protocol in transferring the speech data, means for notifying the user of the detected anomaly and recommended actions, and means for normalizing the volume level in preprocessing the speech data. This enables effective collection, processing, and analysis of speech data, early detection of early symptoms of cerebral infarction and other conditions, and prompt medical treatment.

[0681] "User" refers to an individual or organization that uses the system and provides voice data.

[0682] "Speech" refers to the voice uttered by the user, and refers to voice data collected in the form of everyday conversation or recording.

[0683] "Means for collection" refers to a method or apparatus for recording and temporarily storing a user's spoken voice using a device or application.

[0684] "Preprocessing means" refers to a method or device that performs processes such as noise filtering and volume normalization to convert collected audio data into a format suitable for analysis.

[0685] "Means for extracting features" refers to a method or device for extracting data characteristics necessary for analysis, such as fundamental frequency and energy spectrum, from preprocessed speech data.

[0686] "Means using a generative AI model" refers to a method or apparatus that executes a pre-trained artificial intelligence model to detect anomalies in audio data using extracted features.

[0687] "Means for detecting anomalies" refers to a method or device that uses a generative AI model to identify unhealthy conditions or abnormal patterns in audio data.

[0688] "Means for sending a notification to a user" refers to a method or apparatus for sending detected anomaly information to a user's device as a message or alert.

[0689] "Secure communications protocol" refers to a method of transmitting voice data while ensuring data privacy and integrity, such as through TLS or SSL.

[0690] "Volume level normalization" refers to a method or device that performs processing to adjust the volume of audio data to a certain standard.

[0691] "Content of anomaly" refers to information indicating the type and degree of anomaly detected by the generative AI model.

[0692] "Recommended actions" refers to instructions that indicate specific actions or measures that a user should take in response to a detected anomaly.

[0693] "Early symptoms of cerebral infarction" refers to symptoms that appear in the early stages of cerebral infarction, and refers to conditions that can be detected early by analyzing voice data.

[0694] The "Voice Doctor" system of the present invention uses voice data to detect the early stages of symptoms that affect voice and language, such as cerebral infarction. This system is built with the following hardware and software and performs specific data processing and calculations.

[0695] Overall system configuration

[0696] The system consists of a device (smartphone or voice input device) that collects the user's speech, a server that processes the data, software to detect anomalies using generative AI models, and a communication protocol that sends notifications to the user.

[0697] Hardware and software used

[0698] 1. Audio data collection

[0699] Hardware: Smartphones, devices with microphones

[0700] Software: Mobile applications and voice recording applications

[0701] 2. Data Transfer

[0702] Hardware: Internet-connected devices

[0703] Software: Communication protocols (TLS, SSL)

[0704] 3. Server Processing

[0705] Hardware: Server (cloud server, on-premise server)

[0706] Software: Python libraries (Librosa, TensorFlow, PyTorch), database systems

[0707] 4. Generative AI Models

[0708] Software: TensorFlow, PyTorch

[0709] Model: A pre-trained audio anomaly detection model

[0710] 5. Notification System

[0711] Hardware: Smartphones, computers

[0712] Software: Push notification API, mobile notification application

[0713] Detailed explanation of each element

[0714] 1. User provides voice data

[0715] Users provide voice data using a smartphone or other voice input device. When users record their daily conversations or thoughts about the day, this voice data is temporarily stored within the application.

[0716] 2. Transfer of audio data by the terminal

[0717] The device transmits this voice data to the server using a secure communication protocol (e.g., TLS or SSL), which protects the integrity and privacy of the data.

[0718] 3. Preprocessing of audio data by the server

[0719] The server performs preprocessing on the received audio data, including noise filtering and volume normalization, to obtain clean data suitable for analysis.

[0720] 4. Feature extraction by the server

[0721] The server extracts features such as fundamental frequency and energy spectrum from the preprocessed audio data using Librosa or similar software.

[0722] 5. Anomaly detection using generative AI models

[0723] The server inputs the extracted features into a generative AI model to detect abnormal patterns. This AI model uses a pre-trained dataset to identify anomalies with high accuracy. It uses the TensorFlow and PyTorch frameworks.

[0724] 6. Server-Initiated User Notification

[0725] If an abnormality is detected, the server uses the notification API to send a push notification to the user's device. The notification will include details of the detected abnormality and recommended actions. For example, the notification may contain a message such as, "This is suspected to be an early symptom of a stroke. Please seek medical attention immediately."

[0726] Specific examples

[0727] Example 1: Detecting anomalies during everyday conversation

[0728] Voice data collection and transmission

[0729] The terminal collects the audio of the user talking to a friend over the phone and transfers it to a server.

[0730] Audio data preprocessing

[0731] The server performs noise filtering and volume level normalization on the received audio data.

[0732] Feature extraction

[0733] The server extracts features such as fundamental frequency, formant frequency, and energy spectrum from the preprocessed audio data.

[0734] Anomaly detection using AI models

[0735] The server inputs the extracted features into a generative AI model to determine whether early symptoms of a stroke are suspected.

[0736] User Notifications

[0737] If an abnormality is detected, the server sends a notification to the user's smartphone and displays a message saying, "This is suspected to be an early symptom of a stroke. Please seek medical attention immediately."

[0738] Example 2: Expressing your thoughts on everyday life

[0739] Voice data collection and transmission

[0740] The device collects audio data recorded by the user on their smartphone about their thoughts about the day and transfers it to a server.

[0741] Audio data preprocessing

[0742] The server noise filters (e.g., removes background noise from) and normalizes the received audio data.

[0743] Feature extraction

[0744] The server extracts features such as wavelength and frequency from the preprocessed audio data (e.g., using Librosa).

[0745] Anomaly detection using AI models

[0746] The server inputs the extracted features into a generative AI model to determine whether an abnormal pattern exists.

[0747] User Notifications

[0748] If an abnormality is detected, the server sends a notification to the user's smartphone (e.g., notifying the user of the detailed analysis results of the call) and displays a message such as "An abnormality has been detected. Please seek medical attention immediately."

[0749] In this way, the "Voice Doctor" of the present invention is a system that efficiently collects, preprocesses, and analyzes voice data, detects abnormalities early, and prompts the user to take appropriate action. By clearly showing the types of data processing and data calculations that are performed through concrete examples, the operation of the system can be clearly understood.

[0750] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0751] Step 1:

[0752] Providing audio data

[0753] Users provide voice data using a smartphone or voice input device. For example, users can record their daily conversations or their thoughts about the day into the app. This data is temporarily stored within the app.

[0754] Input: User speech

[0755] Output: Audio data temporarily saved on the device

[0756] Specific behavior:

[0757] The user launches the smartphone application and taps the record button.

[0758] Once recording is complete, the audio file will be automatically saved to your device storage.

[0759] Step 2:

[0760] Transferring audio data

[0761] The device transmits the voice data to the server, and the communication is secured by using a secure communication protocol such as TLS or SSL.

[0762] Input: Audio data stored on the device

[0763] Output: Audio data sent to the server

[0764] Specific behavior:

[0765] The application on your device will begin uploading the audio file.

[0766] The data is sent to the server using a secure communication protocol.

[0767] The server confirms receipt of the data and returns a success response to the terminal.

[0768] Step 3:

[0769] Audio data preprocessing

[0770] The server preprocesses the received audio data, performing noise filtering and volume normalization to obtain clean data suitable for analysis.

[0771] Input: raw audio data transferred to the server

[0772] Output: Preprocessed, clean audio data

[0773] Specific behavior:

[0774] The server applies a noise filtering algorithm to remove background noise.

[0775] Next, a volume normalization process is performed to adjust the volume to a consistent level.

[0776] Step 4:

[0777] Feature extraction

[0778] The server extracts features from the preprocessed audio data, using a speech analysis library such as Librosa for this process.

[0779] Input: Preprocessed clean audio data

[0780] Output: Extracted feature data (fundamental frequency, energy spectrum, etc.)

[0781] Specific behavior:

[0782] The server uses Librosa to calculate features such as fundamental frequency and spectrograms from the audio data.

[0783] The extracted features are stored in a database.

[0784] Step 5:

[0785] Detecting Anomalies with Generative AI Models

[0786] The server inputs the extracted features into a generative AI model to detect anomalies. This AI model uses a pre-trained dataset and can identify anomalies with high accuracy.

[0787] Input: Extracted feature data

[0788] Output: Judgment result on whether or not there is an abnormality and the type of abnormality

[0789] Specific behavior:

[0790] The server uses the TensorFlow or PyTorch framework to run generative AI models.

[0791] When the model detects an anomaly, the result is recorded in a log on the server.

[0792] Step 6:

[0793] User Notifications

[0794] If an abnormality is detected, the server sends a notification to the user's device, which includes the details of the abnormality and a recommended course of action (e.g., early medical consultation).

[0795] Input: Anomaly detection results

[0796] Output: Notification message sent to the user's terminal

[0797] Specific behavior:

[0798] The server uses a notification API to send the anomaly detection results and recommended actions to the user's smartphone.

[0799] A message will appear on the user's smartphone saying, "This is suspected to be an early symptom of a stroke. Please seek medical attention immediately."

[0800] Through these steps, Voice Doctor is able to collect, process, and analyze the user's voice data, enabling early detection of abnormalities.

[0801] (Application example 1)

[0802] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0803] Conventional voice recognition technologies and health management systems separate voice authentication and health status screening, resulting in inefficiency and reduced user convenience. Furthermore, there are cases where detection of health status abnormalities is delayed, preventing timely medical consultations. The present invention aims to solve these problems by simultaneously performing voice authentication and health status screening, improving user convenience and enabling early detection of health status abnormalities.

[0804] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0805] In this invention, the server includes means for collecting a user's speech, means for preprocessing the collected speech data, means for extracting features from the preprocessed speech data, means for using a generative AI model to detect anomalies using the extracted features, means for sending a notification to the user when an anomaly is detected, and means for simultaneously performing voice authentication and health screening, thereby making it possible to screen the user's health during the voice authentication process and immediately send a notification when an anomaly is detected.

[0806] "User" means an individual or corporation that uses the system.

[0807] "Speech" refers to data of words or voices uttered by a user.

[0808] "Means for collection" refers to devices or software that record and store the user's spoken voice.

[0809] The "preprocessing means" refers to a device or software that performs processing to remove noise from collected voice data and standardize the data.

[0810] A "means for extracting features" is a device or software that extracts important data such as frequency components and energy spectra from preprocessed audio data.

[0811] "Means for using a generative AI model" refers to devices or software that execute a trained artificial intelligence model to detect anomalies based on extracted features.

[0812] The "means for sending a notification" is a device or software that sends a warning or information to the user when an abnormality is detected.

[0813] "Means for simultaneously performing voice authentication and health screening" refers to devices or software that use a user's voice to verify their identity while simultaneously assessing their health.

[0814] The "VoiceAuth Doctor" system of the present invention can simultaneously perform voice authentication and health screening of users. This system is implemented as an application installed on smartphones, and aims to increase user convenience and detect health abnormalities early.

[0815] Basic system configuration

[0816] 1. User provides voice data

[0817] Users provide voice data when conducting voice authentication during everyday conversations or when making payments. When a user performs voice authentication on their smartphone, the voice data is automatically collected.

[0818] 2. Transfer of audio data by the terminal

[0819] The device then transfers the collected voice data to a server, where it is transmitted using a secure communication protocol to ensure privacy and data integrity.

[0820] 3. Preprocessing of audio data by the server

[0821] The server pre-processes the received audio data, including noise filtering and data normalization, to obtain clean data suitable for analysis.

[0822] 4. Feature extraction by the server

[0823] The server extracts features such as frequency components and energy spectra from the preprocessed audio data. For feature extraction, the numpy library is used, for example.

[0824] 5. Anomaly detection using generative AI models

[0825] The server runs a generative AI model based on the extracted features to detect abnormal patterns in the voice data. This AI model is pre-trained using, for example, the Keras library.

[0826] 6. Server-Initiated User Notification

[0827] If an abnormality is detected, the server sends a notification to the user's device. The notification includes the details of the abnormality and recommended actions (e.g., early consultation with a medical institution). The notification is sent using, for example, the Twilio library.

[0828] Specific examples

[0829] For example, when a user performs voice authentication during payment, the voice data is sent to the server and analyzed. If the analysis detects an abnormality, a notification is sent to the user's smartphone via SMS via Twilio saying, "This is suspected to be an early symptom of a stroke. Please seek medical attention immediately."

[0830] Example prompts for generative AI models

[0831] Determine whether an anomaly can be detected based on the features extracted from the following audio data. Feature data: [Feature array]

[0832] In this way, the system of the present invention can screen the user's health condition during the voice authentication process and send an immediate notification if an abnormality is detected, thereby achieving both user convenience and health management.

[0833] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0834] Step 1:

[0835] User-provided voice data

[0836] To perform voice authentication, the user speaks into their smartphone, and the voice recording is collected by the system.

[0837] Input: User speech

[0838] Output: Audio data

[0839] Step 2:

[0840] Transfer of audio data by terminal

[0841] The device transfers the collected voice data to a server using a secure communication protocol, which ensures privacy and data integrity.

[0842] Input: Audio data

[0843] Output: Audio data sent to the server

[0844] Step 3:

[0845] Preprocessing of audio data by the server

[0846] The server performs noise filtering and normalization on the received audio data, for example, normalizing the data using sklearn.preprocessing.StandardScaler.

[0847] Input: Transmitted audio data

[0848] Output: Preprocessed, clean audio data

[0849] Step 4:

[0850] Feature extraction by the server

[0851] The server extracts features such as frequency components and energy spectrum from the preprocessed audio data, for example, by performing FFT (Fast Fourier Transform) using the numpy library.

[0852] Input: Preprocessed audio data

[0853] Output: Extracted feature data

[0854] Step 5:

[0855] Server-generated AI model for anomaly detection

[0856] The server runs a generative AI model based on the extracted features to detect abnormal patterns, for example, by running an AI model pre-trained using the keras library.

[0857] Input: Feature data

[0858] Output: Anomaly detection result (normal / abnormal)

[0859] Step 6:

[0860] Server-based user notification

[0861] If an anomaly is detected, the server sends a notification to the user's device, including the details of the anomaly and recommended actions, such as an SMS notification using the Twilio library.

[0862] Input: Anomaly detection results

[0863] Output: Notification message (if an anomaly is detected)

[0864] The above steps realize a system that screens the user's health status during the voice authentication process and sends an immediate notification if an abnormality is detected.

[0865] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0866] This invention is a system that uses voice data to detect the early stages of symptoms that affect voice and language, such as cerebral infarction, and combines it with an emotion engine that recognizes the user's emotions, enabling more accurate anomaly detection and user notification. This system is realized through a series of processes, from collecting and analyzing voice data, to anomaly detection, emotion recognition, and user notification.

[0867] Basic system configuration

[0868] 1. User provides voice data

[0869] Users provide voice data through everyday conversations, recording notes on their smartphones, etc. For example, if a user is talking on the phone with a friend, the content of that conversation will be the subject of the analysis.

[0870] 2. Audio collection by device

[0871] The device collects the user's speech in real time, and the speech data is temporarily stored in the device's memory.

[0872] 3. Transferring data from the device to the server

[0873] The device transfers the collected voice data to a server at regular intervals using a secure communication protocol.

[0874] 4. Preprocessing of audio data by the server

[0875] The server performs preprocessing on the received audio data, such as noise filtering and normalization, to obtain clean data suitable for analysis.

[0876] 5. Feature extraction by the server

[0877] The server extracts features such as wavelength, frequency, and energy spectrum from the preprocessed audio data. These features are important data that are input into the generative AI model.

[0878] 6. Detecting Anomalies with Generative AI Models

[0879] The server then runs a generative AI model based on the extracted features to detect abnormal patterns in the voice data. This AI model uses a pre-trained dataset and is capable of identifying anomalies with high accuracy.

[0880] 7. Emotion Recognition with Emotion Engine

[0881] The server uses an emotion engine to analyze the voice data and recognize the user's emotional state. For example, it can detect when the user is feeling stressed or anxious.

[0882] 8. Reflecting Emotional Information in Anomaly Detection Results

[0883] The emotion engine uses the emotional information recognized by the system to reinforce or correct the anomaly detection results. For example, by taking into account not only the voice features but also the emotional information, anomaly detection can be performed with greater accuracy.

[0884] 9. Server-generated user notifications

[0885] If an anomaly is detected, the server generates a notification message that takes into account emotional information. This message includes the details of the anomaly and recommended actions (e.g., early medical consultation).

[0886] 10. Sending notifications from the server to the device

[0887] The server sends the generated notification message to the user's terminal using a communication protocol that allows real-time notification.

[0888] 11. Display of notifications by device

[0889] The user's device will then display the notification message received from the server on its screen, such as "You are suspected of having the early symptoms of a stroke. Please seek medical attention immediately."

[0890] 12. User Actions

[0891] Based on the received notification, the user considers visiting a medical institution early and takes the necessary measures.

[0892] 13. Continuous monitoring by server

[0893] The server continues to monitor new voice data sent periodically by the user to provide ongoing health assessment.

[0894] Specific examples

[0895] Example 1: Detecting anomalies during everyday conversations and taking emotional information into account

[0896] 1. Audio data collection and transmission

[0897] The terminal collects the voice of the user talking to a friend over the phone and transfers it to the server.

[0898] 2. Audio data preprocessing

[0899] The server performs noise filtering and volume level normalization on the received audio data.

[0900] 3. Feature Extraction

[0901] The server extracts features such as fundamental frequency, formant frequency, and energy spectrum from the preprocessed speech data.

[0902] 4. Anomaly Detection Using AI Models

[0903] The server inputs the extracted features into a generative AI model to determine whether early symptoms of a stroke are suspected.

[0904] 5. Emotion Recognition by Emotion Engine

[0905] The server recognizes the user's emotions from the voice data and detects, for example, whether the user is feeling anxious or stressed.

[0906] 6. Reflecting emotional information

[0907] The server reinforces the anomaly detection results based on emotional information and adjusts the notification content appropriately.

[0908] 7. User Notices

[0909] If an abnormality is detected, the server sends a notification to the user's smartphone and displays a message saying, "This is suspected to be an early symptom of a stroke. Please seek medical attention immediately." The server also takes into account the user's emotional state.

[0910] Example 2: Expressing your thoughts on everyday life

[0911] 1. Audio data collection and transmission

[0912] The device collects audio data recorded by the user on their smartphone about their thoughts about the day and transfers it to a server.

[0913] 2. Audio data preprocessing

[0914] The server noise filters and normalizes the received audio data.

[0915] 3. Feature Extraction

[0916] The server extracts features such as wavelength and frequency from the preprocessed audio data.

[0917] 4. Anomaly Detection Using AI Models

[0918] The server inputs the extracted features into a generative AI model to determine whether an abnormal pattern exists.

[0919] 5. Emotion Recognition by Emotion Engine

[0920] The server recognizes the user's emotions from the voice data and detects, for example, positive and negative emotions.

[0921] 6. Reflecting emotional information

[0922] The server reinforces the anomaly detection results based on emotional information and adjusts the notification content appropriately.

[0923] 7. User Notices

[0924] If an abnormality is detected, the server sends a notification to the user's smartphone, displaying a message such as "An abnormality has been detected. Please seek medical attention immediately." The content of this notification also takes into account the user's emotional state.

[0925] In this way, the present invention is a system that can detect early symptoms of conditions such as cerebral infarction at an early stage by combining voice data and medical data, and by taking emotional information into account, it can provide more appropriate responses and notifications.

[0926] The processing flow will be explained below.

[0927] Step 1:

[0928] User provided voice data:

[0929] Users provide voice data through everyday conversations, recording notes on their smartphones, etc. For example, if a user is talking on the phone with a friend, the content of that conversation will be the subject of the service.

[0930] Step 2:

[0931] Device audio collection:

[0932] The device collects the user's speech in real time, and the speech data is temporarily stored in the device's memory.

[0933] Step 3:

[0934] Transferring data from device to server:

[0935] The device transfers the collected voice data to a server at regular intervals using a secure communication protocol.

[0936] Step 4:

[0937] Server receives data:

[0938] The server receives the voice data transferred from the terminal and stores it in storage.

[0939] Step 5:

[0940] Server preprocessing of audio data:

[0941] The server performs noise filtering on the received audio data, including Gaussian and bandpass filters.

[0942] Step 6:

[0943] Server-generated audio normalization:

[0944] The server normalizes the volume level of the noise-filtered audio data, a step that ensures data consistency.

[0945] Step 7:

[0946] Server-based feature extraction:

[0947] The server extracts features such as fundamental frequency, formant frequency, and energy spectrum from the preprocessed audio data. The extracted features are used as input data for the AI ​​model.

[0948] Step 8:

[0949] Server-based feature normalization:

[0950] The server normalizes the extracted features to a consistent scale and prepares them for analysis by the AI ​​model.

[0951] Step 9:

[0952] The server runs the AI ​​model:

[0953] The server inputs the normalized features into a generative AI model to detect abnormalities. This AI model has been trained in advance using data on the early symptoms of cerebral infarction.

[0954] Step 10:

[0955] Server execution of emotion engine:

[0956] The server inputs the voice data into an emotion engine to recognize the user's emotional state, and the emotion engine analyzes the tone and rhythm of the voice to identify emotions.

[0957] Step 11:

[0958] Server's assessment of the anomaly:

[0959] The server combines the anomaly detection results output by the generative AI model with the recognition results of the emotion engine to make a final determination of the anomaly score, for example, determining whether the anomaly score exceeds a set threshold.

[0960] Step 12:

[0961] Server prepares user notification:

[0962] If an anomaly is detected, the server generates a notification message for the user, including the specific anomaly and recommended actions. The notification content can also be tailored based on emotion data.

[0963] Step 13:

[0964] Sending notifications from the server to the device:

[0965] The server sends the generated notification message to the user's terminal using a communication protocol that allows real-time notification.

[0966] Step 14:

[0967] Displaying notifications by device:

[0968] The user's device will then display the notification message received from the server on its screen, such as "You are suspected of having the early symptoms of a stroke. Please seek medical attention immediately."

[0969] Step 15:

[0970] User Action:

[0971] Based on the received notification, the user considers visiting a medical institution early and takes the necessary measures.

[0972] Step 16:

[0973] Continuous monitoring by server:

[0974] The server continues to monitor new voice data sent periodically by the user to provide ongoing health assessment.

[0975] Example 2

[0976] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0977] Systems that use voice data to detect early symptoms of neurological diseases such as cerebral infarction typically detect abnormalities based on voice features, but accurate anomaly detection is difficult because the user's emotional state can affect the results. Furthermore, when an abnormality is detected, the user is not always notified appropriately, which can lead to delayed responses. Furthermore, noise in the voice data can reduce detection accuracy.

[0978] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting user utterances, means for preprocessing the collected voice data, means for extracting features from the preprocessed voice data, means for using a generative AI model that detects anomalies using the extracted features, means for recognizing emotions based on the voice data, means for correcting anomaly detection results using recognized emotional information, and means for sending a notification to the user taking the emotional information into consideration when an anomaly is detected. This enables highly accurate anomaly detection that takes emotional information into account and appropriate user notification.

[0979] A "user" is an individual who utilizes the system to provide their voice data and receive health monitoring.

[0980] "Speech" refers to voice data generated by a user speaking.

[0981] "Means for collecting" refers to hardware or software that captures the user's speech in real time and temporarily stores it.

[0982] "Preprocessing" refers to data processing to make collected voice data suitable for analysis, and specifically includes noise filtering and normalization.

[0983] "Features" refers to statistical data such as wavelength, frequency, and energy spectrum that are extracted from audio data and are the subject of analysis.

[0984] A "generative AI model" refers to an artificial intelligence algorithm that is trained using large amounts of training data to detect anomalies in voice data.

[0985] "Means for detecting anomalies" refers to the process of using extracted features to run a generative AI model and identify abnormal patterns.

[0986] "Means for recognizing emotions" refers to a computational process or algorithm for analyzing and identifying a user's emotional state from audio data.

[0987] "Emotion information" refers to emotion data recognized based on the user's voice data.

[0988] "Means for correcting anomaly detection results" refers to a process for reinforcing or correcting anomaly detection results based on recognized emotion information.

[0989] "Means for sending a notification" refers to a process for generating a notification message that takes into account emotional information when an anomaly is detected and sending it to the user's device.

[0990] MODE FOR CARRYING OUT THE INVENTION

[0991] The present invention is a system that uses voice data and emotional information to detect the early stages of neurological diseases such as cerebral infarction. This system is realized through a series of processes, from collecting and analyzing voice data, to detecting abnormalities, recognizing emotions, and notifying the user. Details of the hardware and software required to implement the present invention are described below.

[0992] Provision and collection of voice data

[0993] User-provided voice data

[0994] Users provide voice data through everyday conversations, recording notes on their smartphones, etc. For example, a user may be talking to a friend on the phone or recording their thoughts on their smartphone like a diary.

[0995] Audio collection by device

[0996] A device (e.g., a smartphone) uses a built-in microphone to collect the user's voice in real time. This voice data is temporarily stored in the device's memory and then transferred to a server. The collected voice data may be collected every minute or for specific events.

[0997] Audio data preprocessing and feature extraction

[0998] Transferring data from the device to the server

[0999] The device transfers the collected voice data to the server at regular intervals. Data transfer is performed using a secure communication protocol such as SSL / TLS, ensuring data security.

[1000] Preprocessing of audio data by the server

[1001] The server performs preprocessing on the received audio data, such as noise filtering (e.g., Wiener filtering) and normalization, to obtain clean data suitable for analysis.

[1002] Feature extraction by the server

[1003] The server extracts features such as fundamental frequency, formant frequency, and energy spectrum from the preprocessed speech data. These features are important information to input into the generative AI model.

[1004] Anomaly detection and emotion recognition

[1005] Detecting Anomalies with Generative AI Models

[1006] The server runs a generative AI model (e.g., a model implemented in TensorFlow or PyTorch) based on the extracted features to detect anomalous patterns in the audio data. The AI ​​model uses a pre-trained dataset to identify anomalies with high accuracy.

[1007] Emotion recognition by emotion engine

[1008] The server uses an emotion engine (e.g., IBM Watson's emotion analysis service) to analyze the voice data and recognize the user's emotional state. For example, it can determine whether the user is feeling stressed based on the tone and tempo of the voice.

[1009] User notification and continuous monitoring

[1010] Reflecting emotional information in anomaly detection results

[1011] The server reinforces or corrects the anomaly detection results based on the emotion information recognized by the emotion engine. By taking emotion information into account, anomaly detection can be performed with greater accuracy.

[1012] Server-generated user notifications

[1013] If an anomaly is detected, the server generates a notification message that takes into account emotional information. This message includes the details of the anomaly and recommended actions (e.g., early medical consultation).

[1014] Sending notifications from the server to the device

[1015] The server sends the generated notification message to the user's device using a communication protocol that allows real-time notifications (e.g., WebSocket).

[1016] Displaying notifications by device

[1017] The device displays the notification message received from the server to the user. For example, a message such as "You are suspected of having the early symptoms of a cerebral infarction. Please seek medical attention immediately" may be displayed.

[1018] Specific examples

[1019] Example 1: Detecting anomalies during everyday conversations and taking emotional information into account

[1020] The device collects audio of the user's phone conversations with friends and transfers it to the server at regular intervals. The server then noise-filters and normalizes the received audio data before extracting features. The extracted features are input into a generative AI model to detect patterns that may be early symptoms of cerebral infarction. At the same time, an emotion engine is used to recognize the user's emotional state, and this emotional information is used to reinforce the anomaly detection results. If an anomaly is detected, the server generates a notification message that takes the emotional information into account and sends it to the user's device. The device then displays this notification message to the user.

[1021] Example 2: Expressing your thoughts on everyday life

[1022] The device collects audio data recorded by the user on their smartphone about their thoughts about the day and transfers the data to a server. The server then noise-filters the received audio data and normalizes the volume. It then extracts features from the preprocessed audio data and inputs them into a generative AI model to determine whether an abnormal pattern exists. It then uses an emotion engine to recognize the user's emotions and reinforces the anomaly detection results based on that emotional information. If an abnormality is detected, the server sends a notification message to the user's device, such as "An abnormality has been detected. Please seek medical attention immediately," and displays it on the device.

[1023] Example prompts to input to the generative AI model

[1024] Prompt: I had a fun conversation with a friend today, but halfway through I started to feel tired. Then I got a headache. Is this a sign that something is wrong?

[1025] This system integrates voice data and emotional information to achieve more accurate anomaly detection and user notification, providing an effective means of monitoring users' health status.

[1026] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1027] Step 1:

[1028] User-provided voice data

[1029] Users provide voice data through everyday conversations or recording notes on their smartphones. For example, consider a situation where you are talking on the phone with a friend. In this case, the user's voice is the input. The voice is collected in real time by the device via the built-in microphone.

[1030] Step 2:

[1031] Audio collection by device

[1032] The device collects the user's speech in real time using a built-in microphone, and the user's speech is temporarily stored in the device's memory. At this time, the collected speech data is treated as input data, and the stored data is used as output.

[1033] Step 3:

[1034] Transferring data from the device to the server

[1035] The device transfers the collected voice data to the server at regular intervals. This is done based on specific timing or events (e.g., every minute or at the end of recording), and the voice data is sent to the server using a secure communication protocol (e.g., SSL / TLS). The input data is the voice data stored on the device, and the output is the data transferred to the server.

[1036] Step 4:

[1037] Preprocessing of audio data by the server

[1038] The server performs noise filtering (e.g., Wiener filtering) and normalization on the received audio data. Noise filtering removes background noise, and volume normalization equalizes the data. The input is the audio data transferred to the server, and the output is clean audio data that has been denoised and normalized.

[1039] Step 5:

[1040] Feature extraction by the server

[1041] The server extracts features such as fundamental frequency, formant frequency, and energy spectrum from the preprocessed speech data. It analyzes the speech waveform and quantifies the data by calculating these features. The input is clean speech data, and the output is the extracted feature data.

[1042] Step 6:

[1043] Detecting Anomalies with Generative AI Models

[1044] The server inputs the extracted features into a generative AI model to detect abnormal patterns in the voice data. The generative AI model operates based on a pre-trained dataset and identifies abnormal patterns that indicate, for example, the early symptoms of a stroke. The input is feature data, and the output is the result of anomaly detection.

[1045] Step 7:

[1046] Emotion recognition by emotion engine

[1047] The server uses an emotion engine based on the voice data to recognize the user's emotional state. It analyzes the tone, tempo, rhythm, etc. of the voice to identify emotions (e.g., stress, anxiety, joy). The input is voice data or feature data, and the output is the recognized emotional information.

[1048] Step 8:

[1049] Reflecting emotional information in anomaly detection results

[1050] The server corrects or reinforces the anomaly detection results based on the recognized emotional information. By taking emotional information into account, the severity and likelihood of an anomaly can be more accurately determined. The input is the anomaly detection results and emotional information, and the output is the corrected anomaly detection results.

[1051] Step 9:

[1052] Server-generated user notifications

[1053] If an anomaly is detected, the server generates a notification message that takes into account emotional information. This message includes the details of the anomaly and a recommended action (e.g., to seek medical advice). The input is the corrected anomaly detection result, and the output is the generated notification message.

[1054] Step 10:

[1055] Sending notifications from the server to the device

[1056] The server sends the generated notification message to the user's device in real time. The communication protocol used is one that allows real-time notifications (e.g., WebSocket). The input is the notification message, and the output is the notification sent to the user's device.

[1057] Step 11:

[1058] Displaying notifications by device

[1059] The terminal displays the notification message received from the server to the user. The notification appears as a pop-up on the screen, and may say something like, "You are suspected of having an early symptom of a cerebral infarction. Please seek medical attention immediately." The input is the notification message, and the output is the notification displayed to the user.

[1060] Step 12:

[1061] User Action

[1062] The user checks the displayed notification message and takes appropriate action, such as immediately visiting a medical institution. The input is the displayed notification, and the output is the specific action taken by the user.

[1063] Step 13:

[1064] Continuous monitoring by server

[1065] The server continues to monitor new voice data periodically sent by the user and performs continuous health status assessments. This allows for continuous monitoring of the user's health status. The input is new voice data, and the output is a continuously updated health status assessment result.

[1066] (Application example 2)

[1067] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1068] Conventional health monitoring systems could detect health abnormalities based on voice data, but it was difficult to simultaneously consider the user's emotional state. This could result in inaccurate detection of abnormalities and insufficient appropriate responses. Furthermore, for occupations such as security guards, real-time notification displays on-site are important, but there was a lack of means to achieve this. Therefore, there was a need for a system that could improve the accuracy of health abnormality detection, consider the user's emotional state, and enable immediate on-site response.

[1069] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1070] In this invention, the server includes means for collecting user speech, means for preprocessing the collected speech data, means for extracting features from the preprocessed speech data, means for using a generative AI model to detect abnormalities using the extracted features, means for sending a notification to the user when an abnormality is detected, means for recognizing the user's emotional state based on the collected speech data, means for reflecting the user's emotional information in the abnormality detection result, and means for displaying a notification to the user via a head-mounted display. This improves the accuracy of detecting health abnormalities, enables appropriate responses that take the user's emotional state into consideration, and enables immediate on-site responses.

[1071] The "means for collecting the user's speech" refers to a device or method for recording the user's speech as digital data using a voice input device such as a microphone or recorder.

[1072] The "means for preprocessing collected voice data" refers to a method or device that performs noise filtering, normalization, etc. on the collected voice data to convert it into a state suitable for analysis.

[1073] The "means for extracting features from preprocessed speech data" refers to a method or device for extracting important parameters necessary for analysis, such as fundamental frequency, formant frequency, and energy spectrum, from preprocessed speech data.

[1074] "Means using a generative AI model to detect anomalies using extracted features" refers to algorithms or methods that analyze the features extracted by the generative AI model and compare them with normal values ​​to determine anomalies.

[1075] "Means for sending a notification to the user when an abnormality is detected" refers to a method or system for notifying the user of an abnormality when it is detected as a message on the screen of a mobile terminal or computer.

[1076] "Emotion recognition means for recognizing the emotional state of a user based on collected voice data" refers to an algorithm or system that analyzes the tone and pattern of a user's voice from voice data and determines the user's emotions (e.g., anger, sadness, joy, etc.).

[1077] "Means for reflecting user emotional information in anomaly detection results" refers to a method or system that takes into account the user's emotional state in response to detected anomaly information and adjusts the content and urgency of the notification message based on that.

[1078] "Means for displaying notifications to a user via a head-mounted display" refers to a method or system that visually displays abnormality detection results and other important information to a user using a wearable display device.

[1079] The system for implementing this invention includes a series of processes that collect and analyze voice data, detect health abnormalities and emotional states, and notify the user. The system includes the following components:

[1080] Acquisition and preprocessing of audio data

[1081] 1. Collecting user speech

[1082] The terminal collects the user's voice in real time using a voice input device such as a microphone or recorder.

[1083] 2. Preprocessing of audio data

[1084] The server performs preprocessing on the collected audio data, such as noise filtering and normalization, to obtain clean data suitable for analysis.

[1085] Feature extraction and anomaly detection

[1086] 3. Feature Extraction

[1087] From the preprocessed audio data, features such as fundamental frequency and energy spectrum are extracted, and these features are used as input data for generative AI models.

[1088] 4. Anomaly Detection

[1089] Based on the extracted features, the server uses a generative AI model to detect abnormal health patterns in the voice data. This generative AI model uses a pre-trained dataset to identify abnormalities with high accuracy.

[1090] Recognizing emotional states and reflecting them in anomaly detection results

[1091] 5. Recognizing emotional states

[1092] During the process of analyzing the voice data, the server uses emotion recognition means to recognize the user's emotional state, thereby determining whether the user is feeling stressed or relaxed.

[1093] 6. Reflecting emotional information

[1094] The recognized emotion information is used to reinforce or correct the anomaly detection results, which enables more accurate anomaly detection.

[1095] Notification method for extension

[1096] 7. Generating and Sending User Notifications

[1097] If an abnormality is detected, the server generates a notification message that takes into account emotional information and sends it to the user's device. This notification message includes the details of the abnormality and recommended actions (e.g., early consultation with a medical institution).

[1098] 8. Displaying notifications using a head-mounted display

[1099] The device (for example, a head-mounted display) displays the received notification message in the user's field of vision, allowing the user to immediately check the abnormality notification.

[1100] The specific hardware and software used

[1101] Hardware: Microphone, recorder, head-mounted display

[1102] Software: sounddevice, requests library, custom generative AI model (AISymptomDetector) and emotion engine (EmotionEngine)

[1103] Specific examples

[1104] Example 1: Security guard on patrol

[1105] Security guards wear head-mounted displays, and their speech while patrolling is collected and analyzed in real time. If an abnormal pattern is detected, a message appears on the head-mounted display stating, "An abnormality has been detected. Please seek medical attention immediately." If the guard is feeling stressed, the message "The user is feeling extremely stressed" is also displayed.

[1106] Examples of prompt statements

[1107] The voice data is 10 seconds of conversation data. Please detect the early symptoms of cerebral infarction from this data. Also, please use the emotion engine to recognize the emotional state and reflect it in the anomaly detection results.

[1108] This configuration improves the accuracy of detecting health abnormalities, and enables appropriate responses that take into account the user's emotional state and immediate responses on the spot.

[1109] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1110] Step 1:

[1111] Collecting user speech

[1112] The terminal collects the user's speech in real time. The user's speech is used as input. This is recorded as digital data using a voice input device such as a microphone or recorder. The output is digitized voice data.

[1113] Step 2:

[1114] Audio data preprocessing

[1115] The server receives the collected audio data and performs noise filtering and normalization on it. The input is the digitized audio data obtained in step 1. Noise filtering removes known external noise, and normalization ensures a consistent volume level. The output is clean, pre-processed audio data suitable for analysis.

[1116] Step 3:

[1117] Feature extraction

[1118] The server extracts features such as fundamental frequency and energy spectrum from the preprocessed speech data. The preprocessed speech data obtained in step 2 is used as input. Data analysis techniques are used to extract the characteristics of the speech data in the time and frequency domains. The extracted feature data is obtained as output.

[1119] Step 4:

[1120] Anomaly detection

[1121] The server runs a generative AI model using the extracted feature data to detect abnormal health patterns. The feature data obtained in step 3 is used as input. This data is input into the generative AI model to identify abnormal patterns (e.g., early symptoms of cerebral infarction). The output is an abnormality detection result.

[1122] Step 5:

[1123] Recognition of emotional states

[1124] The server uses emotion recognition means during the voice data analysis process to recognize the user's emotional state. The voice data collected in step 1 is used as input. The emotion recognition means analyzes the tone and pattern of the voice to determine the user's emotion (e.g., anger, sadness, joy, etc.). The output is the recognition result of the emotional state.

[1125] Step 6:

[1126] Reflecting emotional information

[1127] The server integrates the anomaly detection results and emotion recognition results to reinforce or correct the anomaly detection results. The inputs are the anomaly detection results obtained in step 4 and the emotional state recognition results obtained in step 5. Emotional information is added to the anomaly information and the notification content is adjusted. The output is the reinforced or corrected anomaly detection results.

[1128] Step 7:

[1129] Generate and send user notifications

[1130] The server generates a notification message for the user based on the corrected anomaly detection results. The input is the augmented or corrected anomaly detection results obtained in step 6. The server generates an appropriate notification message (e.g., recommended actions) and sends it to the user's device. The output is the notification message.

[1131] Step 8:

[1132] Displaying notifications using a head-mounted display

[1133] The device (head-mounted display) displays the received notification message in a form that can be seen by the user. The notification message generated in step 7 is used as input. The message is displayed in the user's field of vision in real time, prompting them to take an appropriate action. The output is a notification message that the user can visually confirm.

[1134] This will improve the system's accuracy in detecting health abnormalities, allowing for appropriate responses that take into account the user's emotional state and immediate on-site response.

[1135] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1136] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1137] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1138] [Third embodiment]

[1139] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1140] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1141] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1142] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1143] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1144] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1145] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1146] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1147] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1148] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1149] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1150] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1151] The "Voice Doctor" of this invention is a system that uses voice data to detect the early stages of symptoms that affect voice and language, such as cerebral infarction. This system is realized through a series of processes, including voice data collection, preprocessing, feature extraction, anomaly detection using a generative AI model, and notification to the user. Below, we will explain the details of each element and the operation of the entire system in detail.

[1152] Basic system configuration

[1153] 1. User provides voice data

[1154] Users provide voice data through everyday conversations or recording notes on their smartphones, etc. This voice data is automatically collected and temporarily stored by the device.

[1155] 2. Transfer of audio data by the terminal

[1156] The device transfers the collected voice data to a server, which transmits the data using a secure communication protocol. This communication is necessary to protect privacy and ensure data integrity.

[1157] 3. Preprocessing of audio data by the server

[1158] The server pre-processes the received audio data, including noise filtering and normalization, to obtain clean data suitable for analysis.

[1159] 4. Feature extraction by the server

[1160] The server extracts features such as wavelength, frequency, and energy spectrum from the preprocessed audio data. These features are important data that are input into the generative AI model.

[1161] 5. Anomaly detection using generative AI models

[1162] The server then runs a generative AI model based on the extracted features to detect abnormal patterns in the voice data. This AI model uses a pre-trained dataset and is capable of identifying anomalies with high accuracy.

[1163] 6. Server-Initiated User Notification

[1164] If an abnormality is detected, the server sends a notification to the user's device, which includes details of the abnormality and recommended actions (e.g., early consultation with a medical institution).

[1165] Specific examples of program processing

[1166] Example 1: Detecting anomalies during everyday conversation

[1167] 1. Audio data collection and transmission

[1168] The terminal collects the voice of the user talking to a friend over the phone and transfers it to the server.

[1169] 2. Audio data preprocessing

[1170] The server performs noise filtering and volume level normalization on the received audio data.

[1171] 3. Feature Extraction

[1172] The server extracts features such as fundamental frequency, formant frequency, and energy spectrum from the preprocessed speech data.

[1173] 4. Anomaly detection using AI models

[1174] The server inputs the extracted features into a generative AI model to determine whether early symptoms of a stroke are suspected.

[1175] 5. User Notices

[1176] If an abnormality is detected, the server sends a notification to the user's smartphone and displays a message saying, "This is suspected to be an early symptom of a stroke. Please seek medical attention immediately."

[1177] Example 2: Expressing your thoughts on everyday life

[1178] 1. Audio data collection and transmission

[1179] The device collects audio data recorded by the user on their smartphone about their thoughts about the day and transfers it to a server.

[1180] 2. Audio data preprocessing

[1181] The server noise filters and normalizes the received audio data.

[1182] 3. Feature Extraction

[1183] The server extracts features such as wavelength and frequency from the preprocessed audio data.

[1184] 4. Anomaly Detection Using AI Models

[1185] The server inputs the extracted features into a generative AI model to determine whether an abnormal pattern exists.

[1186] 5. User Notices

[1187] If an abnormality is detected, the server will send a notification to the user's smartphone and display a message such as "An abnormality has been detected. Please seek medical attention immediately."

[1188] In this way, the "Voice Doctor" of the present invention is a system that combines voice data and medical data to detect early symptoms of cerebral infarction and other conditions early on and prevent the condition from becoming severe.

[1189] The processing flow will be explained below.

[1190] Step 1:

[1191] User provided voice data:

[1192] Users provide voice data through everyday conversations, recording notes on their smartphones, etc. For example, if a user is talking on the phone with a friend, the content of that conversation will be the subject of the service.

[1193] Step 2:

[1194] Device audio collection:

[1195] The device collects the user's speech in real time, and the speech data is temporarily stored in the device's memory.

[1196] Step 3:

[1197] Transferring data from device to server:

[1198] The device transfers the collected voice data to a server at regular intervals using a secure communication protocol.

[1199] Step 4:

[1200] Server receives data:

[1201] The server receives the voice data transferred from the terminal and stores it in storage.

[1202] Step 5:

[1203] Server preprocessing of audio data:

[1204] The server performs noise filtering on the received audio data, including Gaussian and bandpass filters.

[1205] Step 6:

[1206] Server-generated audio normalization:

[1207] The server normalizes the volume level of the noise-filtered audio data, a step that ensures data consistency.

[1208] Step 7:

[1209] Server-based feature extraction:

[1210] The server extracts features such as fundamental frequency, formant frequency, and energy spectrum from the preprocessed audio data. The extracted features are used as input data for the AI ​​model.

[1211] Step 8:

[1212] Server-based feature normalization:

[1213] The server normalizes the extracted features to a consistent scale and prepares them for analysis by the AI ​​model.

[1214] Step 9:

[1215] The server runs the AI ​​model:

[1216] The server inputs the normalized features into a generative AI model to detect abnormalities. This AI model has been trained in advance using data on the early symptoms of cerebral infarction.

[1217] Step 10:

[1218] Server's assessment of the anomaly:

[1219] The server evaluates the anomaly detection results output by the generative AI model, specifically determining whether the anomaly score exceeds a set threshold.

[1220] Step 11:

[1221] Server prepares user notification:

[1222] If an anomaly is detected, the server generates a notification message for the user, including the specific anomaly and recommended action.

[1223] Step 12:

[1224] Sending notifications from the server to the device:

[1225] The server sends the generated notification message to the user's terminal using a communication protocol that allows real-time notification.

[1226] Step 13:

[1227] Displaying notifications by device:

[1228] The user's device will then display the notification message received from the server on its screen, such as "You are suspected of having the early symptoms of a stroke. Please seek medical attention immediately."

[1229] Step 14:

[1230] User Action:

[1231] Based on the received notification, the user considers visiting a medical institution early and takes the necessary measures.

[1232] Step 15:

[1233] Continuous monitoring by server:

[1234] The server continues to monitor new voice data sent periodically by the user to provide ongoing health assessment.

[1235] Example 1

[1236] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1237] There is a lack of adequate tools for early detection of conditions that affect voice and language, such as cerebral infarction. This can delay necessary medical treatment, posing a risk of worsening the condition. There is also a need for systems that can efficiently and effectively collect, process, and analyze voice data.

[1238] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1239] In this invention, the server includes means for collecting user speech, means for preprocessing the collected speech data, means for extracting features from the preprocessed speech data, means for using a generative AI model to detect anomalies using the extracted features, means for sending a notification to the user when an anomaly is detected, means for using a secure communication protocol in transferring the speech data, means for notifying the user of the detected anomaly and recommended actions, and means for normalizing the volume level in preprocessing the speech data. This enables effective collection, processing, and analysis of speech data, early detection of early symptoms of cerebral infarction and other conditions, and prompt medical treatment.

[1240] "User" refers to an individual or organization that uses the system and provides voice data.

[1241] "Speech" refers to the voice uttered by the user, and refers to voice data collected in the form of everyday conversation or recording.

[1242] "Means for collection" refers to a method or apparatus for recording and temporarily storing a user's spoken voice using a device or application.

[1243] "Preprocessing means" refers to a method or device that performs processes such as noise filtering and volume normalization to convert collected audio data into a format suitable for analysis.

[1244] "Means for extracting features" refers to a method or device for extracting data characteristics necessary for analysis, such as fundamental frequency and energy spectrum, from preprocessed speech data.

[1245] "Means using a generative AI model" refers to a method or apparatus that executes a pre-trained artificial intelligence model to detect anomalies in audio data using extracted features.

[1246] "Means for detecting anomalies" refers to a method or device that uses a generative AI model to identify unhealthy conditions or abnormal patterns in audio data.

[1247] "Means for sending a notification to a user" refers to a method or apparatus for sending detected anomaly information to a user's device as a message or alert.

[1248] "Secure communications protocol" refers to a method of transmitting voice data while ensuring data privacy and integrity, such as through TLS or SSL.

[1249] "Volume level normalization" refers to a method or device that performs processing to adjust the volume of audio data to a certain standard.

[1250] "Content of anomaly" refers to information indicating the type and degree of anomaly detected by the generative AI model.

[1251] "Recommended actions" refers to instructions that indicate specific actions or measures that a user should take in response to a detected anomaly.

[1252] "Early symptoms of cerebral infarction" refers to symptoms that appear in the early stages of cerebral infarction, and refers to conditions that can be detected early by analyzing voice data.

[1253] The "Voice Doctor" system of the present invention uses voice data to detect the early stages of symptoms that affect voice and language, such as cerebral infarction. This system is built with the following hardware and software and performs specific data processing and calculations.

[1254] Overall system configuration

[1255] The system consists of a device (smartphone or voice input device) that collects the user's speech, a server that processes the data, software to detect anomalies using generative AI models, and a communication protocol that sends notifications to the user.

[1256] Hardware and software used

[1257] 1. Audio data collection

[1258] Hardware: Smartphones, devices with microphones

[1259] Software: Mobile applications and voice recording applications

[1260] 2. Data Transfer

[1261] Hardware: Internet-connected devices

[1262] Software: Communication protocols (TLS, SSL)

[1263] 3. Server Processing

[1264] Hardware: Server (cloud server, on-premise server)

[1265] Software: Python libraries (Librosa, TensorFlow, PyTorch), database systems

[1266] 4. Generative AI Models

[1267] Software: TensorFlow, PyTorch

[1268] Model: A pre-trained audio anomaly detection model

[1269] 5. Notification System

[1270] Hardware: Smartphones, computers

[1271] Software: Push notification API, mobile notification application

[1272] Detailed explanation of each element

[1273] 1. User provides voice data

[1274] Users provide voice data using a smartphone or other voice input device. When users record their daily conversations or thoughts about the day, this voice data is temporarily stored within the application.

[1275] 2. Transfer of audio data by the terminal

[1276] The device transmits this voice data to the server using a secure communication protocol (e.g., TLS or SSL), which protects the integrity and privacy of the data.

[1277] 3. Preprocessing of audio data by the server

[1278] The server performs preprocessing on the received audio data, including noise filtering and volume normalization, to obtain clean data suitable for analysis.

[1279] 4. Feature extraction by the server

[1280] The server extracts features such as fundamental frequency and energy spectrum from the preprocessed audio data using Librosa or similar software.

[1281] 5. Anomaly detection using generative AI models

[1282] The server inputs the extracted features into a generative AI model to detect abnormal patterns. This AI model uses a pre-trained dataset to identify anomalies with high accuracy. It uses the TensorFlow and PyTorch frameworks.

[1283] 6. Server-Initiated User Notification

[1284] If an abnormality is detected, the server uses the notification API to send a push notification to the user's device. The notification will include details of the detected abnormality and recommended actions. For example, the notification may contain a message such as, "This is suspected to be an early symptom of a stroke. Please seek medical attention immediately."

[1285] Specific examples

[1286] Example 1: Detecting anomalies during everyday conversation

[1287] Voice data collection and transmission

[1288] The terminal collects the audio of the user talking to a friend over the phone and transfers it to a server.

[1289] Audio data preprocessing

[1290] The server performs noise filtering and volume level normalization on the received audio data.

[1291] Feature extraction

[1292] The server extracts features such as fundamental frequency, formant frequency, and energy spectrum from the preprocessed audio data.

[1293] Anomaly detection using AI models

[1294] The server inputs the extracted features into a generative AI model to determine whether early symptoms of a stroke are suspected.

[1295] User Notifications

[1296] If an abnormality is detected, the server sends a notification to the user's smartphone and displays a message saying, "This is suspected to be an early symptom of a stroke. Please seek medical attention immediately."

[1297] Example 2: Expressing your thoughts on everyday life

[1298] Voice data collection and transmission

[1299] The device collects audio data recorded by the user on their smartphone about their thoughts about the day and transfers it to a server.

[1300] Audio data preprocessing

[1301] The server noise filters (e.g., removes background noise from) and normalizes the received audio data.

[1302] Feature extraction

[1303] The server extracts features such as wavelength and frequency from the preprocessed audio data (e.g., using Librosa).

[1304] Anomaly detection using AI models

[1305] The server inputs the extracted features into a generative AI model to determine whether an abnormal pattern exists.

[1306] User Notifications

[1307] If an abnormality is detected, the server sends a notification to the user's smartphone (e.g., notifying the user of the detailed analysis results of the call) and displays a message such as "An abnormality has been detected. Please seek medical attention immediately."

[1308] In this way, the "Voice Doctor" of the present invention is a system that efficiently collects, preprocesses, and analyzes voice data, detects abnormalities early, and prompts the user to take appropriate action. By clearly showing the types of data processing and data calculations that are performed through concrete examples, the operation of the system can be clearly understood.

[1309] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1310] Step 1:

[1311] Providing audio data

[1312] Users provide voice data using a smartphone or voice input device. For example, users can record their daily conversations or their thoughts about the day into the app. This data is temporarily stored within the app.

[1313] Input: User speech

[1314] Output: Audio data temporarily saved on the device

[1315] Specific behavior:

[1316] The user launches the smartphone application and taps the record button.

[1317] Once recording is complete, the audio file will be automatically saved to your device storage.

[1318] Step 2:

[1319] Transferring audio data

[1320] The device transmits the voice data to the server, and the communication is secured by using a secure communication protocol such as TLS or SSL.

[1321] Input: Audio data stored on the device

[1322] Output: Audio data sent to the server

[1323] Specific behavior:

[1324] The application on your device will begin uploading the audio file.

[1325] The data is sent to the server using a secure communication protocol.

[1326] The server confirms receipt of the data and returns a success response to the terminal.

[1327] Step 3:

[1328] Audio data preprocessing

[1329] The server preprocesses the received audio data, performing noise filtering and volume normalization to obtain clean data suitable for analysis.

[1330] Input: raw audio data transferred to the server

[1331] Output: Preprocessed, clean audio data

[1332] Specific behavior:

[1333] The server applies a noise filtering algorithm to remove background noise.

[1334] Next, a volume normalization process is performed to adjust the volume to a consistent level.

[1335] Step 4:

[1336] Feature extraction

[1337] The server extracts features from the preprocessed audio data, using a speech analysis library such as Librosa for this process.

[1338] Input: Preprocessed clean audio data

[1339] Output: Extracted feature data (fundamental frequency, energy spectrum, etc.)

[1340] Specific behavior:

[1341] The server uses Librosa to calculate features such as fundamental frequency and spectrograms from the audio data.

[1342] The extracted features are stored in a database.

[1343] Step 5:

[1344] Detecting Anomalies with Generative AI Models

[1345] The server inputs the extracted features into a generative AI model to detect anomalies. This AI model uses a pre-trained dataset and can identify anomalies with high accuracy.

[1346] Input: Extracted feature data

[1347] Output: Judgment result on whether or not there is an abnormality and the type of abnormality

[1348] Specific behavior:

[1349] The server uses the TensorFlow or PyTorch framework to run generative AI models.

[1350] When the model detects an anomaly, the result is recorded in a log on the server.

[1351] Step 6:

[1352] User Notifications

[1353] If an abnormality is detected, the server sends a notification to the user's device, which includes the details of the abnormality and a recommended course of action (e.g., early medical consultation).

[1354] Input: Anomaly detection results

[1355] Output: Notification message sent to the user's terminal

[1356] Specific behavior:

[1357] The server uses a notification API to send the anomaly detection results and recommended actions to the user's smartphone.

[1358] A message will appear on the user's smartphone saying, "This is suspected to be an early symptom of a stroke. Please seek medical attention immediately."

[1359] Through these steps, Voice Doctor is able to collect, process, and analyze the user's voice data, enabling early detection of abnormalities.

[1360] (Application example 1)

[1361] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1362] Conventional voice recognition technologies and health management systems separate voice authentication and health status screening, resulting in inefficiency and reduced user convenience. Furthermore, there are cases where detection of health status abnormalities is delayed, preventing timely medical consultations. The present invention aims to solve these problems by simultaneously performing voice authentication and health status screening, improving user convenience and enabling early detection of health status abnormalities.

[1363] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1364] In this invention, the server includes means for collecting a user's speech, means for preprocessing the collected speech data, means for extracting features from the preprocessed speech data, means for using a generative AI model to detect anomalies using the extracted features, means for sending a notification to the user when an anomaly is detected, and means for simultaneously performing voice authentication and health screening, thereby making it possible to screen the user's health during the voice authentication process and immediately send a notification when an anomaly is detected.

[1365] "User" means an individual or corporation that uses the system.

[1366] "Speech" refers to data of words or voices uttered by a user.

[1367] "Means for collection" refers to devices or software that record and store the user's spoken voice.

[1368] The "preprocessing means" refers to a device or software that performs processing to remove noise from collected voice data and standardize the data.

[1369] A "means for extracting features" is a device or software that extracts important data such as frequency components and energy spectra from preprocessed audio data.

[1370] "Means for using a generative AI model" refers to devices or software that execute a trained artificial intelligence model to detect anomalies based on extracted features.

[1371] The "means for sending a notification" is a device or software that sends a warning or information to the user when an abnormality is detected.

[1372] "Means for simultaneously performing voice authentication and health screening" refers to devices or software that use a user's voice to verify their identity while simultaneously assessing their health.

[1373] The "VoiceAuth Doctor" system of the present invention can simultaneously perform voice authentication and health screening of users. This system is implemented as an application installed on smartphones, and aims to increase user convenience and detect health abnormalities early.

[1374] Basic system configuration

[1375] 1. User provides voice data

[1376] Users provide voice data when conducting voice authentication during everyday conversations or when making payments. When a user performs voice authentication on their smartphone, the voice data is automatically collected.

[1377] 2. Transfer of audio data by the terminal

[1378] The device then transfers the collected voice data to a server, where it is transmitted using a secure communication protocol to ensure privacy and data integrity.

[1379] 3. Preprocessing of audio data by the server

[1380] The server pre-processes the received audio data, including noise filtering and data normalization, to obtain clean data suitable for analysis.

[1381] 4. Feature extraction by the server

[1382] The server extracts features such as frequency components and energy spectra from the preprocessed audio data. For feature extraction, the numpy library is used, for example.

[1383] 5. Anomaly detection using generative AI models

[1384] The server runs a generative AI model based on the extracted features to detect abnormal patterns in the voice data. This AI model is pre-trained using, for example, the Keras library.

[1385] 6. Server-Initiated User Notification

[1386] If an abnormality is detected, the server sends a notification to the user's device. The notification includes details of the abnormality and recommended actions (e.g., early consultation with a medical institution). Notifications are sent using, for example, the Twilio library.

[1387] Specific examples

[1388] For example, when a user performs voice authentication during payment, the voice data is sent to the server and analyzed. If the analysis detects an abnormality, a notification is sent to the user's smartphone via SMS via Twilio saying, "This is suspected to be an early symptom of a stroke. Please seek medical attention immediately."

[1389] Example prompts for generative AI models

[1390] Determine whether an anomaly can be detected based on the features extracted from the following audio data. Feature data: [Feature array]

[1391] In this way, the system of the present invention can screen the user's health condition during the voice authentication process and send an immediate notification if an abnormality is detected, thereby achieving both user convenience and health management.

[1392] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1393] Step 1:

[1394] User provided voice data

[1395] To perform voice authentication, the user speaks into their smartphone, and the voice recording is collected by the system.

[1396] Input: User speech

[1397] Output: Audio data

[1398] Step 2:

[1399] Transfer of audio data by terminal

[1400] The device transfers the collected voice data to a server using a secure communication protocol, which ensures privacy and data integrity.

[1401] Input: Audio data

[1402] Output: Audio data sent to the server

[1403] Step 3:

[1404] Preprocessing of audio data by the server

[1405] The server performs noise filtering and normalization on the received audio data, for example, normalizing the data using sklearn.preprocessing.StandardScaler.

[1406] Input: Transmitted audio data

[1407] Output: Preprocessed, clean audio data

[1408] Step 4:

[1409] Feature extraction by the server

[1410] The server extracts features such as frequency components and energy spectrum from the preprocessed audio data, for example, by performing FFT (Fast Fourier Transform) using the numpy library.

[1411] Input: Preprocessed audio data

[1412] Output: Extracted feature data

[1413] Step 5:

[1414] Server-generated AI model for anomaly detection

[1415] The server runs a generative AI model based on the extracted features to detect abnormal patterns, for example, by running an AI model pre-trained using the keras library.

[1416] Input: Feature data

[1417] Output: Anomaly detection result (normal / abnormal)

[1418] Step 6:

[1419] Server-based user notification

[1420] If an anomaly is detected, the server sends a notification to the user's device, including the details of the anomaly and recommended actions, such as an SMS notification using the Twilio library.

[1421] Input: Anomaly detection results

[1422] Output: Notification message (if an anomaly is detected)

[1423] The above steps realize a system that screens the user's health status during the voice authentication process and sends an immediate notification if an abnormality is detected.

[1424] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1425] This invention is a system that uses voice data to detect the early stages of symptoms that affect voice and language, such as cerebral infarction, and combines it with an emotion engine that recognizes the user's emotions, enabling more accurate anomaly detection and user notification. This system is realized through a series of processes, from collecting and analyzing voice data, to anomaly detection, emotion recognition, and user notification.

[1426] Basic system configuration

[1427] 1. User provides voice data

[1428] Users provide voice data through everyday conversations, recording notes on their smartphones, etc. For example, if a user is talking on the phone with a friend, the content of that conversation will be the subject of the analysis.

[1429] 2. Audio collection by device

[1430] The device collects the user's speech in real time, and the speech data is temporarily stored in the device's memory.

[1431] 3. Transferring data from the device to the server

[1432] The device transfers the collected voice data to a server at regular intervals using a secure communication protocol.

[1433] 4. Preprocessing of audio data by the server

[1434] The server performs preprocessing on the received audio data, such as noise filtering and normalization, to obtain clean data suitable for analysis.

[1435] 5. Feature extraction by the server

[1436] The server extracts features such as wavelength, frequency, and energy spectrum from the preprocessed audio data. These features are important data that are input into the generative AI model.

[1437] 6. Detecting Anomalies with Generative AI Models

[1438] The server then runs a generative AI model based on the extracted features to detect abnormal patterns in the voice data. This AI model uses a pre-trained dataset and is capable of identifying anomalies with high accuracy.

[1439] 7. Emotion Recognition with Emotion Engine

[1440] The server uses an emotion engine to analyze the voice data and recognize the user's emotional state. For example, it can detect when the user is feeling stressed or anxious.

[1441] 8. Reflecting Emotional Information in Anomaly Detection Results

[1442] The emotion engine uses the emotional information recognized by the system to reinforce or correct the anomaly detection results. For example, by taking into account not only the voice features but also the emotional information, anomaly detection can be performed with greater accuracy.

[1443] 9. Server-generated user notifications

[1444] If an anomaly is detected, the server generates a notification message that takes into account emotional information. This message includes the details of the anomaly and recommended actions (e.g., early medical consultation).

[1445] 10. Sending notifications from the server to the device

[1446] The server sends the generated notification message to the user's terminal using a communication protocol that allows real-time notification.

[1447] 11. Display of notifications by device

[1448] The user's device will then display the notification message received from the server on its screen, such as "You are suspected of having the early symptoms of a stroke. Please seek medical attention immediately."

[1449] 12. User Actions

[1450] Based on the received notification, the user considers visiting a medical institution early and takes the necessary measures.

[1451] 13. Continuous monitoring by server

[1452] The server continues to monitor new voice data sent periodically by the user to provide ongoing health assessment.

[1453] Specific examples

[1454] Example 1: Detecting anomalies during everyday conversations and taking emotional information into account

[1455] 1. Audio data collection and transmission

[1456] The terminal collects the voice of the user talking to a friend over the phone and transfers it to the server.

[1457] 2. Audio data preprocessing

[1458] The server performs noise filtering and volume level normalization on the received audio data.

[1459] 3. Feature Extraction

[1460] The server extracts features such as fundamental frequency, formant frequency, and energy spectrum from the preprocessed speech data.

[1461] 4. Anomaly Detection Using AI Models

[1462] The server inputs the extracted features into a generative AI model to determine whether early symptoms of a stroke are suspected.

[1463] 5. Emotion Recognition by Emotion Engine

[1464] The server recognizes the user's emotions from the voice data and detects, for example, whether the user is feeling anxious or stressed.

[1465] 6. Reflecting emotional information

[1466] The server reinforces the anomaly detection results based on emotional information and adjusts the notification content appropriately.

[1467] 7. User Notices

[1468] If an abnormality is detected, the server sends a notification to the user's smartphone and displays a message saying, "This is suspected to be an early symptom of a stroke. Please seek medical attention immediately." The server also takes into account the user's emotional state.

[1469] Example 2: Expressing your thoughts on everyday life

[1470] 1. Audio data collection and transmission

[1471] The device collects audio data recorded by the user on their smartphone about their thoughts about the day and transfers it to a server.

[1472] 2. Audio data preprocessing

[1473] The server noise filters and normalizes the received audio data.

[1474] 3. Feature Extraction

[1475] The server extracts features such as wavelength and frequency from the preprocessed audio data.

[1476] 4. Anomaly Detection Using AI Models

[1477] The server inputs the extracted features into a generative AI model to determine whether an abnormal pattern exists.

[1478] 5. Emotion Recognition by Emotion Engine

[1479] The server recognizes the user's emotions from the voice data and detects, for example, positive and negative emotions.

[1480] 6. Reflecting emotional information

[1481] The server reinforces the anomaly detection results based on emotional information and adjusts the notification content appropriately.

[1482] 7. User Notices

[1483] If an abnormality is detected, the server sends a notification to the user's smartphone, displaying a message such as "An abnormality has been detected. Please seek medical attention immediately." The content of this notification also takes into account the user's emotional state.

[1484] In this way, the present invention is a system that can detect early symptoms of conditions such as cerebral infarction at an early stage by combining voice data and medical data, and by taking emotional information into account, it can provide more appropriate responses and notifications.

[1485] The processing flow will be explained below.

[1486] Step 1:

[1487] User provided voice data:

[1488] Users provide voice data through everyday conversations, recording notes on their smartphones, etc. For example, if a user is talking on the phone with a friend, the content of that conversation will be the subject of the service.

[1489] Step 2:

[1490] Device audio collection:

[1491] The device collects the user's speech in real time, and the speech data is temporarily stored in the device's memory.

[1492] Step 3:

[1493] Transferring data from device to server:

[1494] The device transfers the collected voice data to a server at regular intervals using a secure communication protocol.

[1495] Step 4:

[1496] Server receives data:

[1497] The server receives the voice data transferred from the terminal and stores it in storage.

[1498] Step 5:

[1499] Server preprocessing of audio data:

[1500] The server performs noise filtering on the received audio data, including Gaussian and bandpass filters.

[1501] Step 6:

[1502] Server-generated audio normalization:

[1503] The server normalizes the volume level of the noise-filtered audio data, a step that ensures data consistency.

[1504] Step 7:

[1505] Server-based feature extraction:

[1506] The server extracts features such as fundamental frequency, formant frequency, and energy spectrum from the preprocessed audio data. The extracted features are used as input data for the AI ​​model.

[1507] Step 8:

[1508] Server-based feature normalization:

[1509] The server normalizes the extracted features to a consistent scale and prepares them for analysis by the AI ​​model.

[1510] Step 9:

[1511] The server runs the AI ​​model:

[1512] The server inputs the normalized features into a generative AI model to detect abnormalities. This AI model has been trained in advance using data on the early symptoms of cerebral infarction.

[1513] Step 10:

[1514] Server execution of emotion engine:

[1515] The server inputs the voice data into an emotion engine to recognize the user's emotional state, and the emotion engine analyzes the tone and rhythm of the voice to identify emotions.

[1516] Step 11:

[1517] Server's assessment of the anomaly:

[1518] The server combines the anomaly detection results output by the generative AI model with the recognition results of the emotion engine to make a final determination of the anomaly score, for example, determining whether the anomaly score exceeds a set threshold.

[1519] Step 12:

[1520] Server prepares user notification:

[1521] If an anomaly is detected, the server generates a notification message for the user, including the specific anomaly and recommended actions. The notification content can also be tailored based on emotion data.

[1522] Step 13:

[1523] Sending notifications from the server to the device:

[1524] The server sends the generated notification message to the user's terminal using a communication protocol that allows real-time notification.

[1525] Step 14:

[1526] Displaying notifications by device:

[1527] The user's device will then display the notification message received from the server on its screen, such as "You are suspected of having the early symptoms of a stroke. Please seek medical attention immediately."

[1528] Step 15:

[1529] User Action:

[1530] Based on the received notification, the user considers visiting a medical institution early and takes the necessary measures.

[1531] Step 16:

[1532] Continuous monitoring by server:

[1533] The server continues to monitor new voice data sent periodically by the user to provide ongoing health assessment.

[1534] Example 2

[1535] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1536] Systems that use voice data to detect early symptoms of neurological diseases such as cerebral infarction typically detect abnormalities based on voice features, but accurate anomaly detection is difficult because the user's emotional state can affect the results. Furthermore, when an abnormality is detected, the user is not always notified appropriately, which can lead to delayed responses. Furthermore, noise in the voice data can reduce detection accuracy.

[1537] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting user speech, means for preprocessing the collected speech data, means for extracting features from the preprocessed speech data, means for using a generative AI model that detects anomalies using the extracted features, means for recognizing emotions based on the speech data, means for correcting anomaly detection results using recognized emotional information, and means for sending a notification to the user taking the emotional information into consideration when an anomaly is detected. This enables highly accurate anomaly detection that takes emotional information into account and appropriate user notification.

[1538] A "user" is an individual who utilizes the system to provide their voice data and receive health monitoring.

[1539] "Speech" refers to voice data generated by a user speaking.

[1540] "Means for collecting" refers to hardware or software that captures the user's speech in real time and temporarily stores it.

[1541] "Preprocessing" refers to data processing to make collected voice data suitable for analysis, and specifically includes noise filtering and normalization.

[1542] "Features" refers to statistical data such as wavelength, frequency, and energy spectrum that are extracted from audio data and are the subject of analysis.

[1543] A "generative AI model" refers to an artificial intelligence algorithm that is trained using large amounts of training data to detect anomalies in voice data.

[1544] "Means for detecting anomalies" refers to the process of using extracted features to run a generative AI model and identify abnormal patterns.

[1545] "Means for recognizing emotions" refers to a computational process or algorithm for analyzing and identifying a user's emotional state from audio data.

[1546] "Emotion information" refers to emotion data recognized based on the user's voice data.

[1547] "Means for correcting anomaly detection results" refers to a process for reinforcing or correcting anomaly detection results based on recognized emotion information.

[1548] "Means for sending a notification" refers to a process for generating a notification message that takes into account emotional information when an anomaly is detected and sending it to the user's device.

[1549] MODE FOR CARRYING OUT THE INVENTION

[1550] The present invention is a system that uses voice data and emotional information to detect the early stages of neurological diseases such as cerebral infarction. This system is realized through a series of processes, from collecting and analyzing voice data, to detecting abnormalities, recognizing emotions, and notifying the user. Details of the hardware and software required to implement the present invention are described below.

[1551] Provision and collection of voice data

[1552] User provided voice data

[1553] Users provide voice data through everyday conversations, recording notes on their smartphones, etc. For example, a user may be talking to a friend on the phone or recording their thoughts on their smartphone like a diary.

[1554] Audio collection by device

[1555] A device (e.g., a smartphone) uses a built-in microphone to collect the user's voice in real time. This voice data is temporarily stored in the device's memory and then transferred to a server. The collected voice data may be collected every minute or for specific events.

[1556] Audio data preprocessing and feature extraction

[1557] Transferring data from the device to the server

[1558] The device transfers the collected voice data to the server at regular intervals. Data transfer is performed using a secure communication protocol such as SSL / TLS, ensuring data security.

[1559] Preprocessing of audio data by the server

[1560] The server performs preprocessing on the received audio data, such as noise filtering (e.g., Wiener filtering) and normalization, to obtain clean data suitable for analysis.

[1561] Feature extraction by the server

[1562] The server extracts features such as fundamental frequency, formant frequency, and energy spectrum from the preprocessed speech data. These features are important information to input into the generative AI model.

[1563] Anomaly detection and emotion recognition

[1564] Detecting Anomalies with Generative AI Models

[1565] The server runs a generative AI model (e.g., a model implemented in TensorFlow or PyTorch) based on the extracted features to detect anomalous patterns in the audio data. The AI ​​model uses a pre-trained dataset to identify anomalies with high accuracy.

[1566] Emotion recognition by emotion engine

[1567] The server uses an emotion engine (e.g., IBM Watson's emotion analysis service) to analyze the voice data and recognize the user's emotional state. For example, it can determine whether the user is feeling stressed based on the tone and tempo of the voice.

[1568] User notification and continuous monitoring

[1569] Reflecting emotional information in anomaly detection results

[1570] The server reinforces or corrects the anomaly detection results based on the emotion information recognized by the emotion engine. By taking emotion information into account, anomaly detection can be performed with greater accuracy.

[1571] Server-generated user notifications

[1572] If an anomaly is detected, the server generates a notification message that takes into account emotional information. This message includes the details of the anomaly and recommended actions (e.g., early medical consultation).

[1573] Sending notifications from the server to the device

[1574] The server sends the generated notification message to the user's device using a communication protocol that allows real-time notifications (e.g., WebSocket).

[1575] Displaying notifications by device

[1576] The device displays the notification message received from the server to the user. For example, a message such as "You are suspected of having the early symptoms of a cerebral infarction. Please seek medical attention immediately" may be displayed.

[1577] Specific examples

[1578] Example 1: Detecting anomalies during everyday conversations and taking emotional information into account

[1579] The device collects audio of the user's phone conversations with friends and transfers it to the server at regular intervals. The server then noise-filters and normalizes the received audio data before extracting features. The extracted features are input into a generative AI model to detect patterns that may be early symptoms of cerebral infarction. At the same time, an emotion engine is used to recognize the user's emotional state, and this emotional information is used to reinforce the anomaly detection results. If an anomaly is detected, the server generates a notification message that takes the emotional information into account and sends it to the user's device. The device then displays this notification message to the user.

[1580] Example 2: Expressing your thoughts on everyday life

[1581] The device collects audio data recorded by the user on their smartphone about their thoughts about the day and transfers the data to a server. The server then noise-filters the received audio data and normalizes the volume. It then extracts features from the preprocessed audio data and inputs them into a generative AI model to determine whether an abnormal pattern exists. It then uses an emotion engine to recognize the user's emotions and reinforces the anomaly detection results based on that emotional information. If an abnormality is detected, the server sends a notification message to the user's device, such as "An abnormality has been detected. Please seek medical attention immediately," and displays it on the device.

[1582] Example prompts to input to the generative AI model

[1583] Prompt: I had a fun conversation with a friend today, but halfway through I started to feel tired. Then I got a headache. Is this a sign that something is wrong?

[1584] This system integrates voice data and emotional information to achieve more accurate anomaly detection and user notification, providing an effective means of monitoring users' health status.

[1585] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1586] Step 1:

[1587] User-provided voice data

[1588] Users provide voice data through everyday conversations or recording notes on their smartphones. For example, consider a situation where you are talking on the phone with a friend. In this case, the user's voice is the input. The voice is collected in real time by the device via the built-in microphone.

[1589] Step 2:

[1590] Audio collection by device

[1591] The device collects the user's speech in real time using a built-in microphone, and the user's speech is temporarily stored in the device's memory. At this time, the collected speech data is treated as input data, and the stored data is used as output.

[1592] Step 3:

[1593] Transferring data from the device to the server

[1594] The device transfers the collected voice data to the server at regular intervals. This is done based on specific timing or events (e.g., every minute or at the end of recording), and the voice data is sent to the server using a secure communication protocol (e.g., SSL / TLS). The input data is the voice data stored on the device, and the output is the data transferred to the server.

[1595] Step 4:

[1596] Preprocessing of audio data by the server

[1597] The server performs noise filtering (e.g., Wiener filtering) and normalization on the received audio data. Noise filtering removes background noise, and volume normalization equalizes the data. The input is the audio data transferred to the server, and the output is clean audio data that has been denoised and normalized.

[1598] Step 5:

[1599] Feature extraction by the server

[1600] The server extracts features such as fundamental frequency, formant frequency, and energy spectrum from the preprocessed speech data. It analyzes the speech waveform and quantifies the data by calculating these features. The input is clean speech data, and the output is the extracted feature data.

[1601] Step 6:

[1602] Detecting Anomalies with Generative AI Models

[1603] The server inputs the extracted features into a generative AI model to detect abnormal patterns in the voice data. The generative AI model operates based on a pre-trained dataset and identifies abnormal patterns that indicate, for example, the early symptoms of a stroke. The input is feature data, and the output is the result of anomaly detection.

[1604] Step 7:

[1605] Emotion recognition by emotion engine

[1606] The server uses an emotion engine based on the voice data to recognize the user's emotional state. It analyzes the tone, tempo, rhythm, etc. of the voice to identify emotions (e.g., stress, anxiety, joy). The input is voice data or feature data, and the output is the recognized emotional information.

[1607] Step 8:

[1608] Reflecting emotional information in anomaly detection results

[1609] The server corrects or reinforces the anomaly detection results based on the recognized emotional information. By taking emotional information into account, the severity and likelihood of an anomaly can be more accurately determined. The input is the anomaly detection results and emotional information, and the output is the corrected anomaly detection results.

[1610] Step 9:

[1611] Server-generated user notifications

[1612] If an anomaly is detected, the server generates a notification message that takes into account emotional information. This message includes the details of the anomaly and a recommended action (e.g., to seek medical advice). The input is the corrected anomaly detection result, and the output is the generated notification message.

[1613] Step 10:

[1614] Sending notifications from the server to the device

[1615] The server sends the generated notification message to the user's device in real time. The communication protocol used is one that allows real-time notifications (e.g., WebSocket). The input is the notification message, and the output is the notification sent to the user's device.

[1616] Step 11:

[1617] Displaying notifications by device

[1618] The terminal displays the notification message received from the server to the user. The notification appears as a pop-up on the screen, and may say something like, "You are suspected of having an early symptom of a cerebral infarction. Please seek medical attention immediately." The input is the notification message, and the output is the notification displayed to the user.

[1619] Step 12:

[1620] User Action

[1621] The user checks the displayed notification message and takes appropriate action, such as immediately visiting a medical institution. The input is the displayed notification, and the output is the specific action taken by the user.

[1622] Step 13:

[1623] Continuous monitoring by server

[1624] The server continues to monitor new voice data periodically sent by the user and performs continuous health status assessments. This allows for continuous monitoring of the user's health status. The input is new voice data, and the output is a continuously updated health status assessment result.

[1625] (Application example 2)

[1626] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1627] Conventional health monitoring systems could detect health abnormalities based on voice data, but it was difficult to simultaneously consider the user's emotional state. This could result in inaccurate detection of abnormalities and insufficient appropriate responses. Furthermore, for occupations such as security guards, real-time notification displays on-site are important, but there was a lack of means to achieve this. Therefore, there was a need for a system that could improve the accuracy of health abnormality detection, consider the user's emotional state, and enable immediate on-site response.

[1628] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1629] In this invention, the server includes means for collecting user speech, means for preprocessing the collected speech data, means for extracting features from the preprocessed speech data, means for using a generative AI model to detect abnormalities using the extracted features, means for sending a notification to the user when an abnormality is detected, means for recognizing the user's emotional state based on the collected speech data, means for reflecting the user's emotional information in the abnormality detection result, and means for displaying a notification to the user via a head-mounted display. This improves the accuracy of detecting health abnormalities, enables appropriate responses that take the user's emotional state into consideration, and enables immediate on-site responses.

[1630] The "means for collecting the user's speech" refers to a device or method for recording the user's speech as digital data using a voice input device such as a microphone or recorder.

[1631] The "means for preprocessing collected voice data" refers to a method or device that performs noise filtering, normalization, etc. on the collected voice data to convert it into a state suitable for analysis.

[1632] The "means for extracting features from preprocessed speech data" refers to a method or device for extracting important parameters necessary for analysis, such as fundamental frequency, formant frequency, and energy spectrum, from preprocessed speech data.

[1633] "Means using a generative AI model to detect anomalies using extracted features" refers to algorithms or methods that analyze the features extracted by the generative AI model and compare them with normal values ​​to determine anomalies.

[1634] "Means for sending a notification to the user when an abnormality is detected" refers to a method or system for notifying the user of an abnormality when it is detected as a message on the screen of a mobile terminal or computer.

[1635] "Emotion recognition means for recognizing the emotional state of a user based on collected voice data" refers to an algorithm or system that analyzes the tone and pattern of a user's voice from voice data and determines the user's emotions (e.g., anger, sadness, joy, etc.).

[1636] "Means for reflecting user emotional information in anomaly detection results" refers to a method or system that takes into account the user's emotional state in response to detected anomaly information and adjusts the content and urgency of the notification message based on that.

[1637] "Means for displaying notifications to a user via a head-mounted display" refers to a method or system that visually displays abnormality detection results and other important information to a user using a wearable display device.

[1638] The system for implementing this invention includes a series of processes that collect and analyze voice data, detect health abnormalities and emotional states, and notify the user. The system includes the following components:

[1639] Acquisition and preprocessing of audio data

[1640] 1. Collecting user speech

[1641] The terminal collects the user's voice in real time using a voice input device such as a microphone or recorder.

[1642] 2. Preprocessing of audio data

[1643] The server performs preprocessing on the collected audio data, such as noise filtering and normalization, to obtain clean data suitable for analysis.

[1644] Feature extraction and anomaly detection

[1645] 3. Feature Extraction

[1646] From the preprocessed audio data, features such as fundamental frequency and energy spectrum are extracted, and these features are used as input data for generative AI models.

[1647] 4. Anomaly Detection

[1648] Based on the extracted features, the server uses a generative AI model to detect abnormal health patterns in the voice data. This generative AI model uses a pre-trained dataset to identify abnormalities with high accuracy.

[1649] Recognizing emotional states and reflecting them in anomaly detection results

[1650] 5. Recognizing emotional states

[1651] During the process of analyzing the voice data, the server uses emotion recognition means to recognize the user's emotional state, thereby determining whether the user is feeling stressed or relaxed.

[1652] 6. Reflecting emotional information

[1653] The recognized emotion information is used to reinforce or correct the anomaly detection results, which enables more accurate anomaly detection.

[1654] Notification method for extension

[1655] 7. Generating and Sending User Notifications

[1656] If an abnormality is detected, the server generates a notification message that takes into account emotional information and sends it to the user's device. This notification message includes the details of the abnormality and recommended actions (e.g., early consultation with a medical institution).

[1657] 8. Displaying notifications using a head-mounted display

[1658] The device (for example, a head-mounted display) displays the received notification message in the user's field of vision, allowing the user to immediately check the abnormality notification.

[1659] The specific hardware and software used

[1660] Hardware: Microphone, recorder, head-mounted display

[1661] Software: sounddevice, requests library, custom generative AI model (AISymptomDetector) and emotion engine (EmotionEngine)

[1662] Specific examples

[1663] Example 1: Security guard on patrol

[1664] Security guards wear head-mounted displays, and their speech while patrolling is collected and analyzed in real time. If an abnormal pattern is detected, a message appears on the head-mounted display stating, "An abnormality has been detected. Please seek medical attention immediately." If the guard is feeling stressed, the message "The user is feeling extremely stressed" is also displayed.

[1665] Examples of prompt statements

[1666] The voice data is 10 seconds of conversation data. Please detect the early symptoms of cerebral infarction from this data. Also, please use the emotion engine to recognize the emotional state and reflect it in the anomaly detection results.

[1667] This configuration improves the accuracy of detecting health abnormalities, and enables appropriate responses that take into account the user's emotional state and immediate responses on the spot.

[1668] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1669] Step 1:

[1670] Collecting user speech

[1671] The terminal collects the user's speech in real time. The user's speech is used as input. This is recorded as digital data using a voice input device such as a microphone or recorder. The output is digitized voice data.

[1672] Step 2:

[1673] Audio data preprocessing

[1674] The server receives the collected audio data and performs noise filtering and normalization on it. The input is the digitized audio data obtained in step 1. Noise filtering removes known external noise, and normalization ensures a consistent volume level. The output is clean, pre-processed audio data suitable for analysis.

[1675] Step 3:

[1676] Feature extraction

[1677] The server extracts features such as fundamental frequency and energy spectrum from the preprocessed speech data. The preprocessed speech data obtained in step 2 is used as input. Data analysis techniques are used to extract the characteristics of the speech data in the time and frequency domains. The extracted feature data is obtained as output.

[1678] Step 4:

[1679] Anomaly detection

[1680] The server runs a generative AI model using the extracted feature data to detect abnormal health patterns. The feature data obtained in step 3 is used as input. This data is input into the generative AI model to identify abnormal patterns (e.g., early symptoms of cerebral infarction). The output is an abnormality detection result.

[1681] Step 5:

[1682] Recognition of emotional states

[1683] The server uses emotion recognition means during the voice data analysis process to recognize the user's emotional state. The voice data collected in step 1 is used as input. The emotion recognition means analyzes the tone and pattern of the voice to determine the user's emotion (e.g., anger, sadness, joy, etc.). The output is the recognition result of the emotional state.

[1684] Step 6:

[1685] Reflecting emotional information

[1686] The server integrates the anomaly detection results and emotion recognition results to reinforce or correct the anomaly detection results. The inputs are the anomaly detection results obtained in step 4 and the emotional state recognition results obtained in step 5. Emotional information is added to the anomaly information and the notification content is adjusted. The output is the reinforced or corrected anomaly detection results.

[1687] Step 7:

[1688] Generate and send user notifications

[1689] The server generates a notification message for the user based on the corrected anomaly detection results. The input is the augmented or corrected anomaly detection results obtained in step 6. The server generates an appropriate notification message (e.g., recommended actions) and sends it to the user's device. The output is the notification message.

[1690] Step 8:

[1691] Displaying notifications using a head-mounted display

[1692] The device (head-mounted display) displays the received notification message in a form that can be seen by the user. The notification message generated in step 7 is used as input. The message is displayed in the user's field of vision in real time, prompting them to take an appropriate action. The output is a notification message that the user can visually confirm.

[1693] This will improve the system's accuracy in detecting health abnormalities, allowing for appropriate responses that take into account the user's emotional state and immediate on-site response.

[1694] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1695] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1696] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1697] [Fourth embodiment]

[1698] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1699] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1700] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1701] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1702] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1703] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1704] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1705] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1706] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1707] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1708] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1709] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1710] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1711] The "Voice Doctor" of this invention is a system that uses voice data to detect the early stages of symptoms that affect voice and language, such as cerebral infarction. This system is realized through a series of processes, including voice data collection, preprocessing, feature extraction, anomaly detection using a generative AI model, and notification to the user. Below, we will explain the details of each element and the operation of the entire system in detail.

[1712] Basic system configuration

[1713] 1. User provides voice data

[1714] Users provide voice data through everyday conversations or recording notes on their smartphones, etc. This voice data is automatically collected and temporarily stored by the device.

[1715] 2. Transfer of audio data by the terminal

[1716] The device transfers the collected voice data to a server, which transmits the data using a secure communication protocol. This communication is necessary to protect privacy and ensure data integrity.

[1717] 3. Preprocessing of audio data by the server

[1718] The server pre-processes the received audio data, including noise filtering and normalization, to obtain clean data suitable for analysis.

[1719] 4. Feature extraction by the server

[1720] The server extracts features such as wavelength, frequency, and energy spectrum from the preprocessed audio data. These features are important data that are input into the generative AI model.

[1721] 5. Anomaly detection using generative AI models

[1722] The server then runs a generative AI model based on the extracted features to detect abnormal patterns in the voice data. This AI model uses a pre-trained dataset and is capable of identifying anomalies with high accuracy.

[1723] 6. Server-Initiated User Notification

[1724] If an abnormality is detected, the server sends a notification to the user's device, which includes details of the abnormality and recommended actions (e.g., early consultation with a medical institution).

[1725] Specific examples of program processing

[1726] Example 1: Detecting anomalies during everyday conversation

[1727] 1. Audio data collection and transmission

[1728] The terminal collects the voice of the user talking to a friend over the phone and transfers it to the server.

[1729] 2. Audio data preprocessing

[1730] The server performs noise filtering and volume level normalization on the received audio data.

[1731] 3. Feature Extraction

[1732] The server extracts features such as fundamental frequency, formant frequency, and energy spectrum from the preprocessed speech data.

[1733] 4. Anomaly detection using AI models

[1734] The server inputs the extracted features into a generative AI model to determine whether early symptoms of a stroke are suspected.

[1735] 5. User Notices

[1736] If an abnormality is detected, the server sends a notification to the user's smartphone and displays a message saying, "This is suspected to be an early symptom of a stroke. Please seek medical attention immediately."

[1737] Example 2: Expressing your thoughts on everyday life

[1738] 1. Audio data collection and transmission

[1739] The device collects audio data recorded by the user on their smartphone about their thoughts about the day and transfers it to a server.

[1740] 2. Audio data preprocessing

[1741] The server noise filters and normalizes the received audio data.

[1742] 3. Feature Extraction

[1743] The server extracts features such as wavelength and frequency from the preprocessed audio data.

[1744] 4. Anomaly Detection Using AI Models

[1745] The server inputs the extracted features into a generative AI model to determine whether an abnormal pattern exists.

[1746] 5. User Notices

[1747] If an abnormality is detected, the server will send a notification to the user's smartphone and display a message such as "An abnormality has been detected. Please seek medical attention immediately."

[1748] In this way, the "Voice Doctor" of the present invention is a system that combines voice data and medical data to detect early symptoms of cerebral infarction and other conditions early on and prevent the condition from becoming severe.

[1749] The processing flow will be explained below.

[1750] Step 1:

[1751] User provided voice data:

[1752] Users provide voice data through everyday conversations, recording notes on their smartphones, etc. For example, if a user is talking on the phone with a friend, the content of that conversation will be the subject of the service.

[1753] Step 2:

[1754] Device audio collection:

[1755] The device collects the user's speech in real time, and the speech data is temporarily stored in the device's memory.

[1756] Step 3:

[1757] Transferring data from device to server:

[1758] The device transfers the collected voice data to a server at regular intervals using a secure communication protocol.

[1759] Step 4:

[1760] Server receives data:

[1761] The server receives the voice data transferred from the terminal and stores it in storage.

[1762] Step 5:

[1763] Server preprocessing of audio data:

[1764] The server performs noise filtering on the received audio data, including Gaussian and bandpass filters.

[1765] Step 6:

[1766] Server-generated audio normalization:

[1767] The server normalizes the volume level of the noise-filtered audio data, a step that ensures data consistency.

[1768] Step 7:

[1769] Server-based feature extraction:

[1770] The server extracts features such as fundamental frequency, formant frequency, and energy spectrum from the preprocessed audio data. The extracted features are used as input data for the AI ​​model.

[1771] Step 8:

[1772] Server-based feature normalization:

[1773] The server normalizes the extracted features to a consistent scale and prepares them for analysis by the AI ​​model.

[1774] Step 9:

[1775] The server runs the AI ​​model:

[1776] The server inputs the normalized features into a generative AI model to detect abnormalities. This AI model has been trained in advance using data on the early symptoms of cerebral infarction.

[1777] Step 10:

[1778] Server's assessment of the anomaly:

[1779] The server evaluates the anomaly detection results output by the generative AI model, specifically determining whether the anomaly score exceeds a set threshold.

[1780] Step 11:

[1781] Server prepares user notification:

[1782] If an anomaly is detected, the server generates a notification message for the user, including the specific anomaly and recommended action.

[1783] Step 12:

[1784] Sending notifications from the server to the device:

[1785] The server sends the generated notification message to the user's terminal using a communication protocol that allows real-time notification.

[1786] Step 13:

[1787] Displaying notifications by device:

[1788] The user's device will then display the notification message received from the server on its screen, such as "You are suspected of having the early symptoms of a stroke. Please seek medical attention immediately."

[1789] Step 14:

[1790] User Action:

[1791] Based on the received notification, the user considers visiting a medical institution early and takes the necessary measures.

[1792] Step 15:

[1793] Continuous monitoring by server:

[1794] The server continues to monitor new voice data sent periodically by the user to provide ongoing health assessment.

[1795] Example 1

[1796] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1797] There is a lack of adequate tools for early detection of conditions that affect voice and language, such as cerebral infarction. This can delay necessary medical treatment, posing a risk of worsening the condition. There is also a need for systems that can efficiently and effectively collect, process, and analyze voice data.

[1798] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1799] In this invention, the server includes means for collecting user speech, means for preprocessing the collected speech data, means for extracting features from the preprocessed speech data, means for using a generative AI model to detect anomalies using the extracted features, means for sending a notification to the user when an anomaly is detected, means for using a secure communication protocol in transferring the speech data, means for notifying the user of the detected anomaly and recommended actions, and means for normalizing the volume level in preprocessing the speech data. This enables effective collection, processing, and analysis of speech data, early detection of early symptoms of cerebral infarction and other conditions, and prompt medical treatment.

[1800] "User" refers to an individual or organization that uses the system and provides voice data.

[1801] "Speech" refers to the voice uttered by the user, and refers to voice data collected in the form of everyday conversation or recording.

[1802] "Means for collection" refers to a method or apparatus for recording and temporarily storing a user's spoken voice using a device or application.

[1803] "Preprocessing means" refers to a method or device that performs processes such as noise filtering and volume normalization to convert collected audio data into a format suitable for analysis.

[1804] "Means for extracting features" refers to a method or device for extracting data characteristics necessary for analysis, such as fundamental frequency and energy spectrum, from preprocessed speech data.

[1805] "Means using a generative AI model" refers to a method or apparatus that executes a pre-trained artificial intelligence model to detect anomalies in audio data using extracted features.

[1806] "Means for detecting anomalies" refers to a method or device that uses a generative AI model to identify unhealthy conditions or abnormal patterns in audio data.

[1807] "Means for sending a notification to a user" refers to a method or apparatus for sending detected anomaly information to a user's device as a message or alert.

[1808] "Secure communications protocol" refers to a method of transmitting voice data while ensuring data privacy and integrity, such as through TLS or SSL.

[1809] "Volume level normalization" refers to a method or device that performs processing to adjust the volume of audio data to a certain standard.

[1810] "Content of anomaly" refers to information indicating the type and degree of anomaly detected by the generative AI model.

[1811] "Recommended actions" refers to instructions that indicate specific actions or measures that a user should take in response to a detected anomaly.

[1812] "Early symptoms of cerebral infarction" refers to symptoms that appear in the early stages of cerebral infarction, and refers to conditions that can be detected early by analyzing voice data.

[1813] The "Voice Doctor" system of the present invention uses voice data to detect the early stages of symptoms that affect voice and language, such as cerebral infarction. This system is built with the following hardware and software and performs specific data processing and calculations.

[1814] Overall system configuration

[1815] The system consists of a device (smartphone or voice input device) that collects the user's speech, a server that processes the data, software to detect anomalies using generative AI models, and a communication protocol that sends notifications to the user.

[1816] Hardware and software used

[1817] 1. Audio data collection

[1818] Hardware: Smartphones, devices with microphones

[1819] Software: Mobile applications and voice recording applications

[1820] 2. Data Transfer

[1821] Hardware: Internet-connected devices

[1822] Software: Communication protocols (TLS, SSL)

[1823] 3. Server Processing

[1824] Hardware: Server (cloud server, on-premise server)

[1825] Software: Python libraries (Librosa, TensorFlow, PyTorch), database systems

[1826] 4. Generative AI Models

[1827] Software: TensorFlow, PyTorch

[1828] Model: A pre-trained audio anomaly detection model

[1829] 5. Notification System

[1830] Hardware: Smartphones, computers

[1831] Software: Push notification API, mobile notification application

[1832] Detailed explanation of each element

[1833] 1. User provides voice data

[1834] Users provide voice data using a smartphone or other voice input device. When users record their daily conversations or thoughts about the day, this voice data is temporarily stored within the application.

[1835] 2. Transfer of audio data by the terminal

[1836] The device transmits this voice data to the server using a secure communication protocol (e.g., TLS or SSL), which protects the integrity and privacy of the data.

[1837] 3. Preprocessing of audio data by the server

[1838] The server performs preprocessing on the received audio data, including noise filtering and volume normalization, to obtain clean data suitable for analysis.

[1839] 4. Feature extraction by the server

[1840] The server extracts features such as fundamental frequency and energy spectrum from the preprocessed audio data using Librosa or similar software.

[1841] 5. Anomaly detection using generative AI models

[1842] The server inputs the extracted features into a generative AI model to detect abnormal patterns. This AI model uses a pre-trained dataset to identify anomalies with high accuracy. It uses the TensorFlow and PyTorch frameworks.

[1843] 6. Server-Initiated User Notification

[1844] If an abnormality is detected, the server uses the notification API to send a push notification to the user's device. The notification will include details of the detected abnormality and recommended actions. For example, the notification may contain a message such as, "This is suspected to be an early symptom of a stroke. Please seek medical attention immediately."

[1845] Specific examples

[1846] Example 1: Detecting anomalies during everyday conversation

[1847] Voice data collection and transmission

[1848] The terminal collects the audio of the user talking to a friend over the phone and transfers it to a server.

[1849] Audio data preprocessing

[1850] The server performs noise filtering and volume level normalization on the received audio data.

[1851] Feature extraction

[1852] The server extracts features such as fundamental frequency, formant frequency, and energy spectrum from the preprocessed audio data.

[1853] Anomaly detection using AI models

[1854] The server inputs the extracted features into a generative AI model to determine whether early symptoms of a stroke are suspected.

[1855] User Notifications

[1856] If an abnormality is detected, the server sends a notification to the user's smartphone and displays a message saying, "This is suspected to be an early symptom of a stroke. Please seek medical attention immediately."

[1857] Example 2: Expressing your thoughts on everyday life

[1858] Voice data collection and transmission

[1859] The device collects audio data recorded by the user on their smartphone about their thoughts about the day and transfers it to a server.

[1860] Audio data preprocessing

[1861] The server noise filters (e.g., removes background noise from) and normalizes the received audio data.

[1862] Feature extraction

[1863] The server extracts features such as wavelength and frequency from the preprocessed audio data (e.g., using Librosa).

[1864] Anomaly detection using AI models

[1865] The server inputs the extracted features into a generative AI model to determine whether an abnormal pattern exists.

[1866] User Notifications

[1867] If an abnormality is detected, the server sends a notification to the user's smartphone (e.g., notifying the user of the detailed analysis results of the call) and displays a message such as "An abnormality has been detected. Please seek medical attention immediately."

[1868] In this way, the "Voice Doctor" of the present invention is a system that efficiently collects, preprocesses, and analyzes voice data, detects abnormalities early, and prompts the user to take appropriate action. By clearly showing the types of data processing and data calculations that are performed through concrete examples, the operation of the system can be clearly understood.

[1869] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1870] Step 1:

[1871] Providing audio data

[1872] Users provide voice data using a smartphone or voice input device. For example, users can record their daily conversations or their thoughts about the day into the app. This data is temporarily stored within the app.

[1873] Input: User speech

[1874] Output: Audio data temporarily saved on the device

[1875] Specific behavior:

[1876] The user launches the smartphone application and taps the record button.

[1877] Once recording is complete, the audio file will be automatically saved to your device storage.

[1878] Step 2:

[1879] Transferring audio data

[1880] The device transmits the voice data to the server, and the communication is secured by using a secure communication protocol such as TLS or SSL.

[1881] Input: Audio data stored on the device

[1882] Output: Audio data sent to the server

[1883] Specific behavior:

[1884] The application on your device will begin uploading the audio file.

[1885] The data is sent to the server using a secure communication protocol.

[1886] The server confirms receipt of the data and returns a success response to the terminal.

[1887] Step 3:

[1888] Audio data preprocessing

[1889] The server preprocesses the received audio data, performing noise filtering and volume normalization to obtain clean data suitable for analysis.

[1890] Input: raw audio data transferred to the server

[1891] Output: Preprocessed, clean audio data

[1892] Specific behavior:

[1893] The server applies a noise filtering algorithm to remove background noise.

[1894] Next, a volume normalization process is performed to adjust the volume to a consistent level.

[1895] Step 4:

[1896] Feature extraction

[1897] The server extracts features from the preprocessed audio data, using a speech analysis library such as Librosa for this process.

[1898] Input: Preprocessed clean audio data

[1899] Output: Extracted feature data (fundamental frequency, energy spectrum, etc.)

[1900] Specific behavior:

[1901] The server uses Librosa to calculate features such as fundamental frequency and spectrograms from the audio data.

[1902] The extracted features are stored in a database.

[1903] Step 5:

[1904] Detecting Anomalies with Generative AI Models

[1905] The server inputs the extracted features into a generative AI model to detect anomalies. This AI model uses a pre-trained dataset to identify anomalies with high accuracy.

[1906] Input: Extracted feature data

[1907] Output: Judgment result on whether or not there is an abnormality and the type of abnormality

[1908] Specific behavior:

[1909] The server uses the TensorFlow or PyTorch framework to run generative AI models.

[1910] When the model detects an anomaly, the result is recorded in a log on the server.

[1911] Step 6:

[1912] User Notifications

[1913] If an abnormality is detected, the server sends a notification to the user's device, which includes the details of the abnormality and a recommended course of action (e.g., early medical consultation).

[1914] Input: Anomaly detection results

[1915] Output: Notification message sent to the user's terminal

[1916] Specific behavior:

[1917] The server uses a notification API to send the anomaly detection results and recommended actions to the user's smartphone.

[1918] A message will appear on the user's smartphone saying, "This is suspected to be an early symptom of a stroke. Please seek medical attention immediately."

[1919] Through these steps, Voice Doctor is able to collect, process, and analyze the user's voice data, enabling early detection of abnormalities.

[1920] (Application example 1)

[1921] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1922] Conventional voice recognition technologies and health management systems separate voice authentication and health status screening, resulting in inefficiency and reduced user convenience. Furthermore, there are cases where detection of health status abnormalities is delayed, preventing timely medical consultations. The present invention aims to solve these problems by simultaneously performing voice authentication and health status screening, improving user convenience and enabling early detection of health status abnormalities.

[1923] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1924] In this invention, the server includes means for collecting a user's speech, means for preprocessing the collected speech data, means for extracting features from the preprocessed speech data, means for using a generative AI model to detect anomalies using the extracted features, means for sending a notification to the user when an anomaly is detected, and means for simultaneously performing voice authentication and health screening, thereby making it possible to screen the user's health during the voice authentication process and immediately send a notification when an anomaly is detected.

[1925] "User" means an individual or corporation that uses the system.

[1926] "Speech" refers to data of words or voices uttered by a user.

[1927] "Means for collection" refers to devices or software that record and store the user's spoken voice.

[1928] The "preprocessing means" refers to a device or software that performs processing to remove noise from collected voice data and standardize the data.

[1929] A "means for extracting features" is a device or software that extracts important data such as frequency components and energy spectra from preprocessed audio data.

[1930] "Means for using a generative AI model" refers to devices or software that execute a trained artificial intelligence model to detect anomalies based on extracted features.

[1931] The "means for sending a notification" is a device or software that sends a warning or information to the user when an abnormality is detected.

[1932] "Means for simultaneously performing voice authentication and health screening" refers to devices or software that use a user's voice to verify their identity while simultaneously assessing their health.

[1933] The "VoiceAuth Doctor" system of the present invention can simultaneously perform voice authentication and health screening of users. This system is implemented as an application installed on smartphones, and aims to increase user convenience and detect health abnormalities early.

[1934] Basic system configuration

[1935] 1. User provides voice data

[1936] Users provide voice data when conducting voice authentication during everyday conversations or when making payments. When a user performs voice authentication on their smartphone, the voice data is automatically collected.

[1937] 2. Transfer of audio data by the terminal

[1938] The device then transfers the collected voice data to a server, where it is transmitted using a secure communication protocol to ensure privacy and data integrity.

[1939] 3. Preprocessing of audio data by the server

[1940] The server pre-processes the received audio data, including noise filtering and data normalization, to obtain clean data suitable for analysis.

[1941] 4. Feature extraction by the server

[1942] The server extracts features such as frequency components and energy spectra from the preprocessed audio data. For feature extraction, the numpy library is used, for example.

[1943] 5. Anomaly detection using generative AI models

[1944] The server runs a generative AI model based on the extracted features to detect abnormal patterns in the voice data. This AI model is pre-trained using, for example, the Keras library.

[1945] 6. Server-Initiated User Notification

[1946] If an abnormality is detected, the server sends a notification to the user's device. The notification includes the details of the abnormality and recommended actions (e.g., early consultation with a medical institution). The notification is sent using, for example, the Twilio library.

[1947] Specific examples

[1948] For example, when a user performs voice authentication during payment, the voice data is sent to the server and analyzed. If the analysis detects an abnormality, a notification is sent to the user's smartphone via SMS via Twilio saying, "This is suspected to be an early symptom of a stroke. Please seek medical attention immediately."

[1949] Example prompts for generative AI models

[1950] Determine whether an anomaly can be detected based on the features extracted from the following audio data. Feature data: [Feature array]

[1951] In this way, the system of the present invention can screen the user's health condition during the voice authentication process and send an immediate notification if an abnormality is detected, thereby achieving both user convenience and health management.

[1952] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1953] Step 1:

[1954] User-provided voice data

[1955] To perform voice authentication, the user speaks into their smartphone, and the voice recording is collected by the system.

[1956] Input: User speech

[1957] Output: Audio data

[1958] Step 2:

[1959] Transfer of audio data by terminal

[1960] The device transfers the collected voice data to a server using a secure communication protocol, which ensures privacy and data integrity.

[1961] Input: Audio data

[1962] Output: Audio data sent to the server

[1963] Step 3:

[1964] Preprocessing of audio data by the server

[1965] The server performs noise filtering and normalization on the received audio data, for example, normalizing the data using sklearn.preprocessing.StandardScaler.

[1966] Input: Transmitted audio data

[1967] Output: Preprocessed, clean audio data

[1968] Step 4:

[1969] Feature extraction by the server

[1970] The server extracts features such as frequency components and energy spectrum from the preprocessed audio data, for example, by performing FFT (Fast Fourier Transform) using the numpy library.

[1971] Input: Preprocessed audio data

[1972] Output: Extracted feature data

[1973] Step 5:

[1974] Server-generated AI model for anomaly detection

[1975] The server runs a generative AI model based on the extracted features to detect abnormal patterns, for example, by running an AI model pre-trained using the keras library.

[1976] Input: Feature data

[1977] Output: Anomaly detection result (normal / abnormal)

[1978] Step 6:

[1979] Server-based user notification

[1980] If an anomaly is detected, the server sends a notification to the user's device, including the details of the anomaly and recommended actions, such as an SMS notification using the Twilio library.

[1981] Input: Anomaly detection results

[1982] Output: Notification message (if an anomaly is detected)

[1983] The above steps realize a system that screens the user's health status during the voice authentication process and sends an immediate notification if an abnormality is detected.

[1984] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1985] This invention is a system that uses voice data to detect the early stages of symptoms that affect voice and language, such as cerebral infarction, and combines it with an emotion engine that recognizes the user's emotions, enabling more accurate anomaly detection and user notification. This system is realized through a series of processes, from collecting and analyzing voice data, to anomaly detection, emotion recognition, and user notification.

[1986] Basic system configuration

[1987] 1. User provides voice data

[1988] Users provide voice data through everyday conversations, recording notes on their smartphones, etc. For example, if a user is talking on the phone with a friend, the content of that conversation will be the subject of the analysis.

[1989] 2. Audio collection by device

[1990] The device collects the user's speech in real time, and the speech data is temporarily stored in the device's memory.

[1991] 3. Transferring data from the device to the server

[1992] The device transfers the collected voice data to a server at regular intervals using a secure communication protocol.

[1993] 4. Preprocessing of audio data by the server

[1994] The server performs preprocessing on the received audio data, such as noise filtering and normalization, to obtain clean data suitable for analysis.

[1995] 5. Feature extraction by the server

[1996] The server extracts features such as wavelength, frequency, and energy spectrum from the preprocessed audio data. These features are important data that are input into the generative AI model.

[1997] 6. Detecting Anomalies with Generative AI Models

[1998] The server then runs a generative AI model based on the extracted features to detect abnormal patterns in the voice data. This AI model uses a pre-trained dataset and is capable of identifying anomalies with high accuracy.

[1999] 7. Emotion Recognition with Emotion Engine

[2000] The server uses an emotion engine to analyze the voice data and recognize the user's emotional state. For example, it can detect when the user is feeling stressed or anxious.

[2001] 8. Reflecting Emotional Information in Anomaly Detection Results

[2002] The emotion engine uses the emotional information recognized by the system to reinforce or correct the anomaly detection results. For example, by taking into account not only the voice features but also the emotional information, anomaly detection can be performed with greater accuracy.

[2003] 9. Server-generated user notifications

[2004] If an anomaly is detected, the server generates a notification message that takes into account emotional information. This message includes the details of the anomaly and recommended actions (e.g., early medical consultation).

[2005] 10. Sending notifications from the server to the device

[2006] The server sends the generated notification message to the user's terminal using a communication protocol that allows real-time notification.

[2007] 11. Display of notifications by device

[2008] The user's device will then display the notification message received from the server on its screen, such as "You are suspected of having the early symptoms of a stroke. Please seek medical attention immediately."

[2009] 12. User Actions

[2010] Based on the received notification, the user considers visiting a medical institution early and takes the necessary measures.

[2011] 13. Continuous monitoring by server

[2012] The server continues to monitor new voice data sent periodically by the user to provide ongoing health assessment.

[2013] Specific examples

[2014] Example 1: Detecting anomalies during everyday conversations and taking emotional information into account

[2015] 1. Audio data collection and transmission

[2016] The terminal collects the voice of the user talking to a friend over the phone and transfers it to the server.

[2017] 2. Audio data preprocessing

[2018] The server performs noise filtering and volume level normalization on the received audio data.

[2019] 3. Feature Extraction

[2020] The server extracts features such as fundamental frequency, formant frequency, and energy spectrum from the preprocessed speech data.

[2021] 4. Anomaly Detection Using AI Models

[2022] The server inputs the extracted features into a generative AI model to determine whether early symptoms of a stroke are suspected.

[2023] 5. Emotion Recognition by Emotion Engine

[2024] The server recognizes the user's emotions from the voice data and detects, for example, whether the user is feeling anxious or stressed.

[2025] 6. Reflecting emotional information

[2026] The server reinforces the anomaly detection results based on emotional information and adjusts the notification content appropriately.

[2027] 7. User Notices

[2028] If an abnormality is detected, the server sends a notification to the user's smartphone and displays a message saying, "This is suspected to be an early symptom of a stroke. Please seek medical attention immediately." The server also takes into account the user's emotional state.

[2029] Example 2: Expressing your thoughts on everyday life

[2030] 1. Audio data collection and transmission

[2031] The device collects audio data recorded by the user on their smartphone about their thoughts about the day and transfers it to a server.

[2032] 2. Audio data preprocessing

[2033] The server noise filters and normalizes the received audio data.

[2034] 3. Feature Extraction

[2035] The server extracts features such as wavelength and frequency from the preprocessed audio data.

[2036] 4. Anomaly Detection Using AI Models

[2037] The server inputs the extracted features into a generative AI model to determine whether an abnormal pattern exists.

[2038] 5. Emotion Recognition by Emotion Engine

[2039] The server recognizes the user's emotions from the voice data and detects, for example, positive and negative emotions.

[2040] 6. Reflecting emotional information

[2041] The server reinforces the anomaly detection results based on emotional information and adjusts the notification content appropriately.

[2042] 7. User Notices

[2043] If an abnormality is detected, the server sends a notification to the user's smartphone, displaying a message such as "An abnormality has been detected. Please seek medical attention immediately." The content of this notification also takes into account the user's emotional state.

[2044] In this way, the present invention is a system that can detect early symptoms of conditions such as cerebral infarction at an early stage by combining voice data and medical data, and by taking emotional information into account, it can provide more appropriate responses and notifications.

[2045] The processing flow will be explained below.

[2046] Step 1:

[2047] User provided voice data:

[2048] Users provide voice data through everyday conversations, recording notes on their smartphones, etc. For example, if a user is talking on the phone with a friend, the content of that conversation will be the subject of the service.

[2049] Step 2:

[2050] Device audio collection:

[2051] The device collects the user's speech in real time, and the speech data is temporarily stored in the device's memory.

[2052] Step 3:

[2053] Transferring data from device to server:

[2054] The device transfers the collected voice data to a server at regular intervals using a secure communication protocol.

[2055] Step 4:

[2056] Server receives data:

[2057] The server receives the voice data transferred from the terminal and stores it in storage.

[2058] Step 5:

[2059] Server preprocessing of audio data:

[2060] The server performs noise filtering on the received audio data, including Gaussian and bandpass filters.

[2061] Step 6:

[2062] Server-generated audio normalization:

[2063] The server normalizes the volume level of the noise-filtered audio data, a step that ensures data consistency.

[2064] Step 7:

[2065] Server-based feature extraction:

[2066] The server extracts features such as fundamental frequency, formant frequency, and energy spectrum from the preprocessed audio data. The extracted features are used as input data for the AI ​​model.

[2067] Step 8:

[2068] Server-based feature normalization:

[2069] The server normalizes the extracted features to a consistent scale and prepares them for analysis by the AI ​​model.

[2070] Step 9:

[2071] The server runs the AI ​​model:

[2072] The server inputs the normalized features into a generative AI model to detect abnormalities. This AI model has been trained in advance using data on the early symptoms of cerebral infarction.

[2073] Step 10:

[2074] Server execution of emotion engine:

[2075] The server inputs the voice data into an emotion engine to recognize the user's emotional state, and the emotion engine analyzes the tone and rhythm of the voice to identify emotions.

[2076] Step 11:

[2077] Server's assessment of the anomaly:

[2078] The server combines the anomaly detection results output by the generative AI model with the recognition results of the emotion engine to make a final determination of the anomaly score, for example, determining whether the anomaly score exceeds a set threshold.

[2079] Step 12:

[2080] Server prepares user notification:

[2081] If an anomaly is detected, the server generates a notification message for the user, including the specific anomaly and recommended actions. The notification content can also be tailored based on emotion data.

[2082] Step 13:

[2083] Sending notifications from the server to the device:

[2084] The server sends the generated notification message to the user's terminal using a communication protocol that allows real-time notification.

[2085] Step 14:

[2086] Displaying notifications by device:

[2087] The user's device will then display the notification message received from the server on its screen, such as "You are suspected of having the early symptoms of a stroke. Please seek medical attention immediately."

[2088] Step 15:

[2089] User Action:

[2090] Based on the received notification, the user considers visiting a medical institution early and takes the necessary measures.

[2091] Step 16:

[2092] Continuous monitoring by server:

[2093] The server continues to monitor new voice data sent periodically by the user to provide ongoing health assessment.

[2094] Example 2

[2095] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2096] Systems that use voice data to detect early symptoms of neurological diseases such as cerebral infarction typically detect abnormalities based on voice features, but accurate anomaly detection is difficult because the user's emotional state can affect the results. Furthermore, when an abnormality is detected, the user is not always notified appropriately, which can lead to delayed responses. Furthermore, noise in the voice data can reduce detection accuracy.

[2097] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting user speech, means for preprocessing the collected speech data, means for extracting features from the preprocessed speech data, means for using a generative AI model that detects anomalies using the extracted features, means for recognizing emotions based on the speech data, means for correcting anomaly detection results using recognized emotional information, and means for sending a notification to the user taking the emotional information into consideration when an anomaly is detected. This enables highly accurate anomaly detection that takes emotional information into account and appropriate user notification.

[2098] A "user" is an individual who utilizes the system to provide their voice data and receive health monitoring.

[2099] "Speech" refers to voice data generated by a user speaking.

[2100] "Means for collecting" refers to hardware or software that captures the user's speech in real time and temporarily stores it.

[2101] "Preprocessing" refers to data processing to make collected voice data suitable for analysis, and specifically includes noise filtering and normalization.

[2102] "Features" refers to statistical data such as wavelength, frequency, and energy spectrum that are extracted from audio data and are the subject of analysis.

[2103] A "generative AI model" refers to an artificial intelligence algorithm that is trained using large amounts of training data to detect anomalies in voice data.

[2104] "Means for detecting anomalies" refers to the process of using extracted features to run a generative AI model and identify abnormal patterns.

[2105] "Means for recognizing emotions" refers to a computational process or algorithm for analyzing and identifying a user's emotional state from audio data.

[2106] "Emotion information" refers to emotion data recognized based on the user's voice data.

[2107] "Means for correcting anomaly detection results" refers to a process for reinforcing or correcting anomaly detection results based on recognized emotion information.

[2108] "Means for sending a notification" refers to a process for generating a notification message that takes into account emotional information when an anomaly is detected and sending it to the user's device.

[2109] MODE FOR CARRYING OUT THE INVENTION

[2110] The present invention is a system that uses voice data and emotional information to detect the early stages of neurological diseases such as cerebral infarction. This system is realized through a series of processes, from collecting and analyzing voice data, to detecting abnormalities, recognizing emotions, and notifying the user. Details of the hardware and software required to implement the present invention are described below.

[2111] Provision and collection of voice data

[2112] User-provided voice data

[2113] Users provide voice data through everyday conversations, recording notes on their smartphones, etc. For example, a user may be talking to a friend on the phone or recording their thoughts on their smartphone like a diary.

[2114] Audio collection by device

[2115] A device (e.g., a smartphone) uses a built-in microphone to collect the user's voice in real time. This voice data is temporarily stored in the device's memory and then transferred to a server. The collected voice data may be collected every minute or for specific events.

[2116] Audio data preprocessing and feature extraction

[2117] Transferring data from the device to the server

[2118] The device transfers the collected voice data to the server at regular intervals. Data transfer is performed using a secure communication protocol such as SSL / TLS, ensuring data security.

[2119] Preprocessing of audio data by the server

[2120] The server performs preprocessing on the received audio data, such as noise filtering (e.g., Wiener filtering) and normalization, to obtain clean data suitable for analysis.

[2121] Feature extraction by the server

[2122] The server extracts features such as fundamental frequency, formant frequency, and energy spectrum from the preprocessed speech data. These features are important information to input into the generative AI model.

[2123] Anomaly detection and emotion recognition

[2124] Detecting Anomalies with Generative AI Models

[2125] The server runs a generative AI model (e.g., a model implemented in TensorFlow or PyTorch) based on the extracted features to detect anomalous patterns in the audio data. The AI ​​model uses a pre-trained dataset to identify anomalies with high accuracy.

[2126] Emotion recognition by emotion engine

[2127] The server uses an emotion engine (e.g., IBM Watson's emotion analysis service) to analyze the voice data and recognize the user's emotional state. For example, it can determine whether the user is feeling stressed based on the tone and tempo of the voice.

[2128] User notification and continuous monitoring

[2129] Reflecting emotional information in anomaly detection results

[2130] The server reinforces or corrects the anomaly detection results based on the emotion information recognized by the emotion engine. By taking emotion information into account, anomaly detection can be performed with greater accuracy.

[2131] Server-generated user notifications

[2132] If an anomaly is detected, the server generates a notification message that takes into account emotional information. This message includes the details of the anomaly and recommended actions (e.g., early medical consultation).

[2133] Sending notifications from the server to the device

[2134] The server sends the generated notification message to the user's device using a communication protocol that allows real-time notifications (e.g., WebSocket).

[2135] Displaying notifications by device

[2136] The device displays the notification message received from the server to the user. For example, a message such as "You are suspected of having the early symptoms of a cerebral infarction. Please seek medical attention immediately" may be displayed.

[2137] Specific examples

[2138] Example 1: Detecting anomalies during everyday conversations and taking emotional information into account

[2139] The device collects audio of the user's phone conversations with friends and transfers it to the server at regular intervals. The server then noise-filters and normalizes the received audio data before extracting features. The extracted features are input into a generative AI model to detect patterns that may be early symptoms of cerebral infarction. At the same time, an emotion engine is used to recognize the user's emotional state, and this emotional information is used to reinforce the anomaly detection results. If an anomaly is detected, the server generates a notification message that takes the emotional information into account and sends it to the user's device. The device then displays this notification message to the user.

[2140] Example 2: Expressing your thoughts on everyday life

[2141] The device collects audio data recorded by the user on their smartphone about their thoughts about the day and transfers the data to a server. The server then noise-filters the received audio data and normalizes the volume. It then extracts features from the preprocessed audio data and inputs them into a generative AI model to determine whether an abnormal pattern exists. It then uses an emotion engine to recognize the user's emotions and reinforces the anomaly detection results based on that emotional information. If an abnormality is detected, the server sends a notification message to the user's device, such as "An abnormality has been detected. Please seek medical attention immediately," and displays it on the device.

[2142] Example prompts to input to the generative AI model

[2143] Prompt: I had a fun conversation with a friend today, but halfway through I started to feel tired. Then I got a headache. Is this a sign that something is wrong?

[2144] This system integrates voice data and emotional information to achieve more accurate anomaly detection and user notification, providing an effective means of monitoring users' health status.

[2145] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2146] Step 1:

[2147] User-provided voice data

[2148] Users provide voice data through everyday conversations or recording notes on their smartphones. For example, consider a situation where you are talking on the phone with a friend. In this case, the user's voice is the input. The voice is collected in real time by the device via the built-in microphone.

[2149] Step 2:

[2150] Audio collection by device

[2151] The device collects the user's speech in real time using a built-in microphone, and the user's speech is temporarily stored in the device's memory. At this time, the collected speech data is treated as input data, and the stored data is used as output.

[2152] Step 3:

[2153] Transferring data from the device to the server

[2154] The device transfers the collected voice data to the server at regular intervals. This is done based on specific timing or events (e.g., every minute or at the end of recording), and the voice data is sent to the server using a secure communication protocol (e.g., SSL / TLS). The input data is the voice data stored on the device, and the output is the data transferred to the server.

[2155] Step 4:

[2156] Preprocessing of audio data by the server

[2157] The server performs noise filtering (e.g., Wiener filtering) and normalization on the received audio data. Noise filtering removes background noise, and volume normalization equalizes the data. The input is the audio data transferred to the server, and the output is clean audio data that has been denoised and normalized.

[2158] Step 5:

[2159] Feature extraction by the server

[2160] The server extracts features such as fundamental frequency, formant frequency, and energy spectrum from the preprocessed speech data. It analyzes the speech waveform and quantifies the data by calculating these features. The input is clean speech data, and the output is the extracted feature data.

[2161] Step 6:

[2162] Detecting Anomalies with Generative AI Models

[2163] The server inputs the extracted features into a generative AI model to detect abnormal patterns in the voice data. The generative AI model operates based on a pre-trained dataset and identifies abnormal patterns that indicate, for example, the early symptoms of a stroke. The input is feature data, and the output is the result of anomaly detection.

[2164] Step 7:

[2165] Emotion recognition by emotion engine

[2166] The server uses an emotion engine based on the voice data to recognize the user's emotional state. It analyzes the tone, tempo, rhythm, etc. of the voice to identify emotions (e.g., stress, anxiety, joy). The input is voice data or feature data, and the output is the recognized emotional information.

[2167] Step 8:

[2168] Reflecting emotional information in anomaly detection results

[2169] The server corrects or reinforces the anomaly detection results based on the recognized emotional information. By taking emotional information into account, the severity and likelihood of an anomaly can be more accurately determined. The input is the anomaly detection results and emotional information, and the output is the corrected anomaly detection results.

[2170] Step 9:

[2171] Server-generated user notifications

[2172] If an anomaly is detected, the server generates a notification message that takes into account emotional information. This message includes the details of the anomaly and a recommended action (e.g., to seek medical advice). The input is the corrected anomaly detection result, and the output is the generated notification message.

[2173] Step 10:

[2174] Sending notifications from the server to the device

[2175] The server sends the generated notification message to the user's device in real time. The communication protocol used is one that allows real-time notifications (e.g., WebSocket). The input is the notification message, and the output is the notification sent to the user's device.

[2176] Step 11:

[2177] Displaying notifications by device

[2178] The terminal displays the notification message received from the server to the user. The notification appears as a pop-up on the screen, and may say something like, "You are suspected of having an early symptom of a cerebral infarction. Please seek medical attention immediately." The input is the notification message, and the output is the notification displayed to the user.

[2179] Step 12:

[2180] User Action

[2181] The user checks the displayed notification message and takes appropriate action, such as immediately visiting a medical institution. The input is the displayed notification, and the output is the specific action taken by the user.

[2182] Step 13:

[2183] Continuous monitoring by server

[2184] The server continues to monitor new voice data periodically sent by the user and performs continuous health status assessments. This allows for continuous monitoring of the user's health status. The input is new voice data, and the output is a continuously updated health status assessment result.

[2185] (Application example 2)

[2186] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2187] Conventional health monitoring systems could detect health abnormalities based on voice data, but it was difficult to simultaneously consider the user's emotional state. This could result in inaccurate detection of abnormalities and insufficient appropriate responses. Furthermore, for occupations such as security guards, real-time notification displays on-site are important, but there was a lack of means to achieve this. Therefore, there was a need for a system that could improve the accuracy of health abnormality detection, consider the user's emotional state, and enable immediate on-site response.

[2188] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2189] In this invention, the server includes means for collecting user speech, means for preprocessing the collected speech data, means for extracting features from the preprocessed speech data, means for using a generative AI model to detect abnormalities using the extracted features, means for sending a notification to the user when an abnormality is detected, means for recognizing the user's emotional state based on the collected speech data, means for reflecting the user's emotional information in the abnormality detection result, and means for displaying a notification to the user via a head-mounted display. This improves the accuracy of detecting health abnormalities, enables appropriate responses that take the user's emotional state into consideration, and enables immediate on-site responses.

[2190] The "means for collecting the user's speech" refers to a device or method for recording the user's speech as digital data using a voice input device such as a microphone or recorder.

[2191] The "means for preprocessing collected voice data" refers to a method or device that performs noise filtering, normalization, etc. on the collected voice data to convert it into a state suitable for analysis.

[2192] The "means for extracting features from preprocessed speech data" refers to a method or device for extracting important parameters necessary for analysis, such as fundamental frequency, formant frequency, and energy spectrum, from preprocessed speech data.

[2193] "Means using a generative AI model to detect anomalies using extracted features" refers to algorithms or methods that analyze the features extracted by the generative AI model and compare them with normal values ​​to determine anomalies.

[2194] "Means for sending a notification to the user when an abnormality is detected" refers to a method or system for notifying the user of an abnormality when it is detected as a message on the screen of a mobile terminal or computer.

[2195] "Emotion recognition means for recognizing the emotional state of a user based on collected voice data" refers to an algorithm or system that analyzes the tone and pattern of a user's voice from voice data and determines the user's emotions (e.g., anger, sadness, joy, etc.).

[2196] "Means for reflecting user emotional information in anomaly detection results" refers to a method or system that takes into account the user's emotional state in response to detected anomaly information and adjusts the content and urgency of the notification message based on that.

[2197] "Means for displaying notifications to a user via a head-mounted display" refers to a method or system that visually displays abnormality detection results and other important information to a user using a wearable display device.

[2198] The system for implementing this invention includes a series of processes that collect and analyze voice data, detect health abnormalities and emotional states, and notify the user. The system includes the following components:

[2199] Acquisition and preprocessing of audio data

[2200] 1. Collecting user speech

[2201] The terminal collects the user's voice in real time using a voice input device such as a microphone or recorder.

[2202] 2. Preprocessing of audio data

[2203] The server performs preprocessing on the collected audio data, such as noise filtering and normalization, to obtain clean data suitable for analysis.

[2204] Feature extraction and anomaly detection

[2205] 3. Feature Extraction

[2206] From the preprocessed audio data, features such as fundamental frequency and energy spectrum are extracted, and these features are used as input data for generative AI models.

[2207] 4. Anomaly Detection

[2208] Based on the extracted features, the server uses a generative AI model to detect abnormal health patterns in the voice data. This generative AI model uses a pre-trained dataset to identify abnormalities with high accuracy.

[2209] Recognizing emotional states and reflecting them in anomaly detection results

[2210] 5. Recognizing emotional states

[2211] During the process of analyzing the voice data, the server uses emotion recognition means to recognize the user's emotional state, thereby determining whether the user is feeling stressed or relaxed.

[2212] 6. Reflecting emotional information

[2213] The recognized emotion information is used to reinforce or correct the anomaly detection results, which enables more accurate anomaly detection.

[2214] Notification method for extension

[2215] 7. Generating and Sending User Notifications

[2216] If an abnormality is detected, the server generates a notification message that takes into account emotional information and sends it to the user's device. This notification message includes the details of the abnormality and recommended actions (e.g., early consultation with a medical institution).

[2217] 8. Displaying notifications using a head-mounted display

[2218] The device (for example, a head-mounted display) displays the received notification message in the user's field of vision, allowing the user to immediately check the abnormality notification.

[2219] The specific hardware and software used

[2220] Hardware: Microphone, recorder, head-mounted display

[2221] Software: sounddevice, requests library, custom generative AI model (AISymptomDetector) and emotion engine (EmotionEngine)

[2222] Specific examples

[2223] Example 1: Security guard on patrol

[2224] Security guards wear head-mounted displays, and their speech while patrolling is collected and analyzed in real time. If an abnormal pattern is detected, a message appears on the head-mounted display stating, "An abnormality has been detected. Please seek medical attention immediately." If the guard is feeling stressed, the message "The user is feeling extremely stressed" is also displayed.

[2225] Examples of prompt statements

[2226] The voice data is 10 seconds of conversation data. Please detect the early symptoms of cerebral infarction from this data. Also, please use the emotion engine to recognize the emotional state and reflect it in the anomaly detection results.

[2227] This configuration improves the accuracy of detecting health abnormalities, and enables appropriate responses that take into account the user's emotional state and immediate responses on the spot.

[2228] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2229] Step 1:

[2230] Collecting user speech

[2231] The terminal collects the user's speech in real time. The user's speech is used as input. This is recorded as digital data using a voice input device such as a microphone or recorder. The output is digitized voice data.

[2232] Step 2:

[2233] Audio data preprocessing

[2234] The server receives the collected audio data and performs noise filtering and normalization on it. The input is the digitized audio data obtained in step 1. Noise filtering removes known external noise, and normalization ensures a consistent volume level. The output is clean, pre-processed audio data suitable for analysis.

[2235] Step 3:

[2236] Feature extraction

[2237] The server extracts features such as fundamental frequency and energy spectrum from the preprocessed speech data. The preprocessed speech data obtained in step 2 is used as input. Data analysis techniques are used to extract the characteristics of the speech data in the time and frequency domains. The extracted feature data is obtained as output.

[2238] Step 4:

[2239] Anomaly detection

[2240] The server runs a generative AI model using the extracted feature data to detect abnormal health patterns. The feature data obtained in step 3 is used as input. This data is input into the generative AI model to identify abnormal patterns (e.g., early symptoms of cerebral infarction). The output is an abnormality detection result.

[2241] Step 5:

[2242] Recognition of emotional states

[2243] The server uses emotion recognition means during the voice data analysis process to recognize the user's emotional state. The voice data collected in step 1 is used as input. The emotion recognition means analyzes the tone and pattern of the voice to determine the user's emotion (e.g., anger, sadness, joy, etc.). The output is the recognition result of the emotional state.

[2244] Step 6:

[2245] Reflecting emotional information

[2246] The server integrates the anomaly detection results and emotion recognition results to reinforce or correct the anomaly detection results. The inputs are the anomaly detection results obtained in step 4 and the emotional state recognition results obtained in step 5. Emotional information is added to the anomaly information and the notification content is adjusted. The output is the reinforced or corrected anomaly detection results.

[2247] Step 7:

[2248] Generate and send user notifications

[2249] The server generates a notification message for the user based on the corrected anomaly detection results. The input is the augmented or corrected anomaly detection results obtained in step 6. The server generates an appropriate notification message (e.g., recommended actions) and sends it to the user's device. The output is the notification message.

[2250] Step 8:

[2251] Displaying notifications using a head-mounted display

[2252] The device (head-mounted display) displays the received notification message in a form that can be seen by the user. The notification message generated in step 7 is used as input. The message is displayed in the user's field of vision in real time, prompting them to take an appropriate action. The output is a notification message that the user can visually confirm.

[2253] This will improve the system's accuracy in detecting health abnormalities, allowing for appropriate responses that take into account the user's emotional state and immediate on-site response.

[2254] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2255] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2256] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2257] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using t...

Claims

1. A means for collecting a user's speech; means for pre-processing the collected audio data; means for extracting features from the preprocessed speech data; A means for using a generative AI model to detect anomalies using the extracted features; means for sending a notification to a user when an anomaly is detected; A system including:

2. The system of claim 1 further comprising means for noise filtering the collected audio data.

3. The system according to claim 1 , further comprising means for identifying an early symptom of cerebral infarction based on the extracted feature amount.

Images