System

Abstract

To provide a system.SOLUTION: A system that places a call to a specific subject to conduct a safety check and alleviate loneliness through a conversation, and can implement physical condition management using a tone of voice, comprises means for utilizing a conversion system that mimics human voices, converting voice into text, generating responses using generative AI, converting the responses into voice and enabling a conversation over the telephone.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Elderly people living alone, people with physical disabilities, and people who have lost their partners face feelings of loneliness and problems with checking on the safety of others. Managing the health of these people is also an important issue. [Means for solving the problem]

[0005] This invention provides a system that calls specific individuals to check on their safety and ease loneliness through conversation, while also enabling health management through voice tone. This system utilizes a conversion system that mimics the human voice, converts speech into text, and then uses generative AI to create responses, converting them back into speech, resulting in a conversation over the phone. Furthermore, the voices used are voice actors (male and female) and recordings of relatives before they pass away. This system is targeted at elderly people living alone, people with physical disabilities, and those who have lost their partners. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15]FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] One aspect of the present invention is a system that calls specific individuals to check on their safety, alleviating loneliness through conversation and enabling health management through voice tone. This system utilizes a conversion system that mimics the human voice, converts speech to text, and then uses a generation AI to create a response, which is then converted back into speech, resulting in a conversation over the phone. Specifically, speech recognition technology is used to convert the target's speech into text, which is then input into the generation AI. The generation AI analyzes the text and generates an appropriate response. The generated response is converted from text to speech and communicated to the target via phone.

[0029] "Example 2"

[0030] In another embodiment of the present invention, the voices of voice actors (male and female) or relatives recorded before they died are used. Specifically, voice data of the voice actor or voice data of a relative recorded before they died is input into a voice conversion system, and a voice that imitates it is generated. In this embodiment, the target person can enjoy conversations with familiar voices, resulting in higher satisfaction.

[0031] "Example 3"

[0032] A further embodiment of the present invention targets elderly people living alone, people with physical disabilities, and those who have lost their partners. Specifically, a list of these people is created and a regular call schedule is set up. The call times can be adjusted to suit the target person's lifestyle and preferences. This embodiment can reduce the amount of time the target person feels lonely and provide a sense of security.

[0033] The processing flow of each embodiment will be described below.

[0034] "Example 1"

[0035] Step 1: The system places calls to specific individuals, scheduled to fit their lifestyles and preferences.

[0036] Step 2: When the subject answers the phone, the system uses voice recognition technology to convert the subject's speech into text.

[0037] Step 3: The text is fed into the generative AI, which analyzes the text and generates an appropriate answer.

[0038] Step 4: The generated responses are converted from text to speech and communicated to the target person over the phone.

[0039] "Example 2"

[0040] Step 1: Input voice data from a voice actor or a recording of a loved one before they pass away into the voice conversion system.

[0041] Step 2: The voice conversion system generates a voice that mimics the input voice data.

[0042] Step 3: The generated audio is transmitted to the subject via telephone and used in a conversation with the subject.

[0043] "Example 3"

[0044] Step 1: Create a list of target groups, including elderly people living alone, people with physical disabilities, and those who have lost their partners.

[0045] Step 2: Schedule regular phone calls for each person. Adjust the call times to suit each person's lifestyle and preferences.

[0046] Step 3: According to the set schedule, the system calls the target person and conducts a conversation.

[0047] Example 1

[0048] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0049] In modern society, elderly people living alone, people with physical disabilities, and those who have lost their partners often feel lonely, which can have a negative impact on their mental health. Furthermore, it is difficult to quickly detect changes in the physical condition of these individuals and take appropriate action. Conventional safety confirmation systems simply make phone calls and lack the functionality to analyze the tone of the individual's voice and manage their physical condition. Therefore, more advanced safety confirmation and health management systems are needed.

[0050] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0051] In this invention, the server includes a speech recognition means for converting speech to text, a means for inputting the converted text into a generative AI model to generate an appropriate response, and a speech synthesis means for converting the generated response from text to speech. This makes it possible to call a specific target person to check on their safety and ease loneliness through conversation, as well as to analyze the target person's tone of voice to detect changes in their physical condition and respond promptly.

[0052] The server of the present disclosure includes: a means for confirming the safety and health of a specific target by making a call to the target and confirming the target's response; a means for detecting abnormalities in the target's physical condition based on changes in the target's voice by analyzing the tone and quality of the target's voice; an emotion engine for recognizing the target's emotions by inputting the target's voice and facial expressions; a means for generating dialogue appropriate to the emotions by inputting specific prompt sentences into a generative AI model; and a providing means for converting the generated dialogue into imitation voice and providing it to the target. The imitation voice may include a voice that the target is familiar with or that the target likes. The providing means may convert the generated dialogue into imitation voice using voice data of a voice actor or a relative.

[0053] "Specific target groups" refer to individuals who meet certain conditions, such as elderly people living alone, people with physical disabilities, and those who have lost their partners.

[0054] "Making a phone call to check on safety" refers to the act of calling a person and checking their response to check their safety and health.

[0055] "Relieving loneliness through conversation" refers to reducing the sense of loneliness felt by the subject by communicating with them verbally.

[0056] "Health management through voice tone" refers to analyzing the tone and quality of a person's voice and detecting abnormalities in their health based on these changes.

[0057] "Speech recognition means for converting speech into text" refers to technology that collects the speech of a person as digital voice data and converts it into text information.

[0058] A "generative AI model" refers to an artificial intelligence model that analyzes input text data and generates an appropriate response.

[0059] "Text-to-speech synthesis means" refers to a technology that converts generated text data into speech data.

[0060] "Communication means for transmitting voice to a target person via telephone" refers to technology for transmitting generated voice data to a target person via a telephone line.

[0061] "A voice analysis means that analyzes the tone of a subject's voice and detects changes in their physical condition" refers to a technology that analyzes the characteristics of a subject's voice and detects abnormalities in their physical condition based on those changes.

[0062] This invention is a system that can call specific individuals to check on their safety, alleviate loneliness through conversation, and also manage their health by monitoring their tone of voice. This system is realized by combining voice recognition technology, generative AI models, voice synthesis technology, communication technology, and voice analysis technology.

[0063] The server first uses a communication method to call the target person. Specifically, it uses a cloud communication platform to make the call. For example, it uses the Twilio API to call the target person's phone number.

[0064] The device collects the voice of the target person after answering the phone. The collected voice is converted into text using a speech recognition means. Specifically, the speech is converted into text using the Google® Cloud Speech-to-Text API.

[0065] The server inputs the converted text into a generative AI model, which uses OpenAI's GPT-4 (registered trademark). The generative AI model analyzes the input text and generates an appropriate answer.

[0066] The generated answer is then converted from text to speech again using a speech synthesis tool, specifically Amazon Polly, to convert text to speech.

[0067] The server transmits the generated voice to the subject via telephone. The communication means is used to transmit the generated voice data to the subject via a telephone line.

[0068] Furthermore, the device analyzes the subject's tone of voice to detect changes in their physical condition. Using a voice analysis tool, the device analyzes the tone and quality of the subject's voice and detects abnormalities in their physical condition based on these changes. The analysis results are then notified to a medical institution or family member as necessary.

[0069] As a concrete example, the following scenario can be considered.

[0070] When a user wants to check on the safety of an elderly person, the system operates as follows:

[0071] 1. The server uses the Twilio API to call the senior citizen.

[0072] 2. The device collects the elderly person's voice saying "Hello, how are you?" and converts it into text using the Google Cloud Speech-to-Text API.

[0073] 3. The server inputs the converted text "Hello, how are you?" into GPT-4 and generates an appropriate answer.

[0074] 4. The server generates the answer, "Hello, how are you? How was your day?" and converts it into speech using Amazon Polly.

[0075] 5. The server transmits the generated audio to the elderly person via the Twilio API.

[0076] 6. The device analyzes the elderly person's voice tone and notifies medical institutions or family members if any abnormalities are detected.

[0077] An example prompt is, "If an older adult says, 'Hello, how are you?' please generate an appropriate response."

[0078] This system allows users to check the safety of specific individuals, alleviating feelings of loneliness and managing their health.

[0079] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0080] Step 1:

[0081] The server places a call to the target person.

[0082] Input: Subject's phone number

[0083] Specific operation: The server calls the API of the cloud communication platform (e.g., Twilio API) and makes a call to the target person's phone number.

[0084] Output: The call to the target person is successful and the call starts.

[0085] Step 2:

[0086] The terminal collects the subject's voice and converts it into text using speech recognition means.

[0087] Input: Subject's voice

[0088] What it does: The device collects the target person's voice saying "Hello, how are you?" and converts it into text using speech recognition software (e.g., Google Cloud Speech-to-Text API).

[0089] Output: The converted text "Hello, how are you?"

[0090] Step 3:

[0091] The server inputs the converted text into a generative AI model to generate an appropriate answer.

[0092] Input: Transformed text "Hello, how are you?"

[0093] Specific operation: The server inputs the converted text into a generative AI model (e.g., GPT-4), sets a prompt sentence, and generates an appropriate answer.

[0094] Output: Generated answer "Hello, I'm fine. How was your day?"

[0095] Step 4:

[0096] The server converts the generated answers from text to speech.

[0097] Input: Generated answer "Hi, I'm fine. How was your day?"

[0098] What happens: The server uses speech synthesis software (e.g., Amazon Polly) to convert the generated text into speech.

[0099] Output: Generated audio data

[0100] Step 5:

[0101] The server transmits the generated voice to the target person via telephone.

[0102] Input: Generated audio data

[0103] Specific operation: The server uses a communication means (e.g., Twilio API) to transmit the generated voice data to the target person via a telephone line.

[0104] Output: Audio delivered to subject

[0105] Step 6:

[0106] The device analyzes the subject's tone of voice and detects changes in their physical condition.

[0107] Input: Subject's voice data

[0108] Specific operation: The device uses voice analysis software to analyze the tone and quality of the subject's voice, and detects abnormalities in their physical condition based on changes.

[0109] Output: Notification data when abnormalities in health are detected (notification to medical institutions and family members)

[0110] In this way, the system can check the safety of specific individuals, alleviating loneliness and also managing their health.

[0111] (Application example 1)

[0112] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0113] In modern society, the number of elderly people living alone, physically disabled people, and people who have lost their partners is increasing, and confirming the safety of these people and reducing their feelings of loneliness has become an important issue. Managing the health of these people is also important, and a quick response is required, especially in emergencies. However, the current system lacks the means to effectively solve these issues.

[0114] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0115] In this invention, the server includes: a means for calling specific individuals to check on their safety, alleviating loneliness through conversation and enabling health management based on tone of voice; a means for converting speech to text using a conversion system that mimics the human voice, creating a response using a generation AI, converting the response back to speech, and conducting a conversation over the phone; a means for converting the target's speech to text using speech recognition technology; a means for analyzing the text using a generation AI and generating an appropriate response; a means for converting the generated response from text to speech and communicating it to the target over the phone; and a means for inferring the target's health from tone of voice and content of conversation and notifying family or medical institutions as necessary. This makes it possible to check the safety of elderly people living alone, people with disabilities, and those who have lost their partners, reduce loneliness, and manage their health.

[0116] "Specific target groups" refer to people whose safety needs to be confirmed and whose feelings of loneliness need to be alleviated, such as elderly people living alone, people with physical disabilities, and those who have lost their partners.

[0117] "Safety check" refers to the act of checking the current situation and health condition of a person.

[0118] "Conversational" refers to a format in which a conversation is held with the subject over the telephone.

[0119] "Health management based on voice tone" refers to inferring a person's health condition from the tone and quality of their voice and taking the necessary measures.

[0120] "Conversion system" refers to technology that mimics the human voice and converts speech into text.

[0121] "Generative AI" refers to artificial intelligence that analyzes input text and generates appropriate answers.

[0122] "Speech recognition technology" refers to technology that converts speech into text.

[0123] "Text analysis" refers to the process of understanding input text and generating appropriate responses.

[0124] "Speech synthesis" refers to the technology of converting text into speech.

[0125] "Notification" refers to the act of conveying information to family members or medical institutions based on the subject's physical condition and situation.

[0126] The system for implementing this invention makes phone calls to specific individuals to check on their safety, alleviating their feelings of loneliness through conversation and managing their physical condition through voice tone. This system is composed of a combination of voice recognition technology, generation AI, and voice synthesis technology.

[0127] First, the server periodically calls a specific target person. When the call connects, it captures the target person's voice and converts it into text using speech recognition technology. This speech recognition technology uses the speech_recognition library and Google's speech recognition API.

[0128] The server then inputs the converted text into a generation AI, which uses OpenAI's API to analyze the input text and generate an appropriate response. The generated response is then converted from text to speech again, using the gTTS library for speech synthesis.

[0129] The generated voice is transmitted to the target person via telephone. The server then infers the target person's physical condition from the tone of voice and content of the conversation, and notifies family members or medical institutions as necessary. This notification function enables rapid response in emergencies.

[0130] For example, consider the following conversation:

[0131] Elderly person: "I'm feeling a bit tired today."

[0132] Generator: "That's worrying. How are you feeling these days?"

[0133] Example prompt sentence:

[0134] Conversation with an elderly person: I'm feeling a little tired today.

[0135] Generate an appropriate response.

[0136] This system will enable elderly people living alone, people with disabilities, and those who have lost their partners to effectively check on their safety, reduce feelings of loneliness, and manage their health.

[0137] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0138] Step 1:

[0139] The server periodically calls a specific target person. When the call is connected, the target person's voice is acquired. The input is the target person's voice, and the output is voice data. This voice data is collected through a microphone.

[0140] Step 2:

[0141] The server converts the acquired voice data into text using speech recognition technology. Specifically, it uses the speech_recognition library and Google's speech recognition API. The input is voice data, and the output is text data. This conversion allows the subject's speech to be obtained in text format.

[0142] Step 3:

[0143] The server inputs the converted text data into the generation AI. The generation AI uses OpenAI's API to analyze the input text and generate an appropriate answer. The input is text data, and the output is the generated answer text data. This analysis results in an appropriate response to the subject's utterance.

[0144] Step 4:

[0145] The server converts the text data of the generated answer into speech using speech synthesis technology. Specifically, it uses the gTTS library. The input is the text data of the generated answer, and the output is speech data. This conversion results in the generated response in speech format.

[0146] Step 5:

[0147] The server transmits the generated voice data to the subject via telephone. The input is voice data, and the output is the voice transmitted to the subject. This transmission allows the subject to hear the response from the generating AI.

[0148] Step 6:

[0149] The server infers the subject's physical condition from the tone of voice and content of the conversation. If necessary, it notifies family members or medical institutions. The input is the subject's tone of voice and content of the conversation, and the output is notification information. This inference and notification enables a rapid response in the event of an emergency.

[0150] Example 2

[0151] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0152] In modern society, elderly people living alone, people with physical disabilities, and those who have lost their partners often feel lonely, which can lead to poor mental health. Furthermore, there is a lack of efficient ways to check on the safety and health of these people. Furthermore, there is a need for a system that allows these people to enjoy conversations with a friendly voice.

[0153] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0154] In this invention, the server is a system that communicates with specific individuals to check their safety, alleviate loneliness through conversation, and also manage their health through tone of voice, and includes: a means for utilizing a voice-imitating conversion system to convert voice to text, create a response using a generative AI model, convert the response back to voice, and communicate in a conversational format; a means for uploading voice data from a terminal to the server; a means for analyzing the uploaded voice data and extracting features of the voice data; a means for inputting the features of the analyzed voice data into the generative AI model; a means for the generative AI model to generate new voice based on the features of the input voice data; and a means for providing the generated voice data to a user. This allows the individual to enjoy a conversation in a friendly voice, reducing loneliness and enabling efficient safety confirmation and health management.

[0155] "Specific target groups" refer to people who are prone to feeling lonely, such as elderly people living alone, people with physical disabilities, and those who have lost their partners, and who need to be checked on for safety and health management.

[0156] "Communications" refers to the means of sending and receiving voice and data via telephone and the Internet.

[0157] "Safety check" refers to the act of checking the safety and health of a specific person.

[0158] "Conversational" refers to a form of two-way communication using voice.

[0159] "Vocal timbre" refers to characteristics such as the quality, tone, and rhythm of a voice.

[0160] "Health management" refers to the act of monitoring the health status of a specific subject and responding as necessary.

[0161] A "voice imitation conversion system" refers to a system that generates new voices based on input voice data.

[0162] A "generative AI model" refers to an artificial intelligence model that generates new data based on input data.

[0163] "Audio data" refers to data that records audio in digital format.

[0164] "Terminal" refers to a device, such as a PC or smartphone, used to upload audio data to a server.

[0165] "Server" refers to the computer system that analyzes voice data and runs generative AI models.

[0166] "Analysis" refers to the process of extracting features from audio data.

[0167] "Features" refers to characteristics of the audio data such as pitch, rhythm, and timbre.

[0168] A "prompt sentence" refers to an instruction sentence to be input into a generative AI model.

[0169] "New voice" refers to a voice generated by a generative AI model that mimics the original voice data.

[0170] "Providing" refers to the act of transmitting the generated voice data to the user.

[0171] This invention is a system that can communicate with specific individuals to check their safety, alleviate loneliness through conversation, and manage their physical condition through tone of voice. This system utilizes a voice mimicking conversion system to convert speech into text, generate responses using a generative AI model, convert the responses into speech, and communicate in a conversational format.

[0172] Generating a Program

[0173] The server generates a program for the voice conversion system, which includes an algorithm for receiving voice data as input and generating a new voice that mimics it.

[0174] Program processing explanation

[0175] The server causes the speech conversion system to perform the following processes.

[0176] 1. Input audio data:

[0177] Users upload audio data of voice actors or loved ones recorded before they died to the server.

[0178] As specific hardware, users use computers and smartphones.

[0179] As for software, a web browser or dedicated application is used to upload audio data.

[0180] 2. Analysis of audio data:

[0181] The server analyzes the uploaded audio data.

[0182] Analysis is performed using voice recognition software (e.g., Google Cloud Speech-to-Text API).

[0183] Extract features of audio data (pitch, rhythm, timbre, etc.).

[0184] 3. Input to the generative AI model:

[0185] The server inputs the extracted features of the voice data into a generative AI model.

[0186] Generative AI models used include OpenAI's GPT-3 (registered trademark) and Google's WaveNet.

[0187] 4. Imitation voice generation:

[0188] The generative AI model generates new speech based on the characteristics of the input speech data.

[0189] This new voice mimics the original voice data.

[0190] 5. Output of generated speech:

[0191] The server provides the generated voice data to the user.

[0192] The user can download the generated audio or play it via streaming.

[0193] Adding specific examples

[0194] As a concrete example, the following prompt sentence is input into the generative AI model:

[0195] Example prompt sentence:

[0196] "This voice data is the voice of my deceased grandfather. Please generate a new voice based on this voice."

[0197] Examples:

[0198] The user uploads audio data of their grandfather recorded before his death to the server. The server analyzes the audio data and inputs it into the generative AI model. The generative AI model generates a new voice that imitates the grandfather's voice, and the server provides that voice to the user. The user can listen to the generated voice and enjoy a conversation with their grandfather.

[0199] In this way, a specific target person can enjoy a conversation in a friendly voice, reducing feelings of loneliness and enabling efficient safety confirmation and health management.

[0200] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0201] Step 1:

[0202] Inputting audio data

[0203] Users upload audio data of voice actors or loved ones recorded before they passed away from their device (PC or smartphone) to the server.

[0204] Specific operation: The user uses a web browser or dedicated application to select an audio file and click the upload button.

[0205] Input: Audio data file (e.g. .wav, .mp3)

[0206] Output: Audio data stored on the server

[0207] Step 2:

[0208] Analysis of audio data

[0209] The server receives the uploaded voice data and analyzes it using voice recognition software (e.g., Google Cloud Speech-to-Text API).

[0210] Specific operation: The server extracts characteristics of the audio data, such as pitch, rhythm, and timbre, and stores them in a database.

[0211] Input: Audio data stored on the server

[0212] Output: Extracted audio features (e.g., pitch, rhythm, timbre)

[0213] Step 3:

[0214] Input to generative AI models

[0215] The server inputs the features of the analyzed voice data into a generative AI model (e.g., OpenAI's GPT-3 or Google's WaveNet).

[0216] Specific operation: The server sends the features of the voice data to the generative AI model as a prompt sentence.

[0217] Input: Extracted audio features

[0218] Output: The prompt input to the generative AI model

[0219] Step 4:

[0220] Imitation voice generation

[0221] The generative AI model generates new speech based on the characteristics of the input speech data.

[0222] How it works: The generative AI model analyzes the characteristics of the voice data and generates imitation voices based on that.

[0223] Input: The prompt given to the generative AI model

[0224] Output: Generated imitation voice data

[0225] Step 5:

[0226] Generated speech output

[0227] The server provides the generated voice data to the user.

[0228] Specific operation: The server sends the generated audio data to the user's device, where the user can download it or play it via streaming.

[0229] Input: Generated imitation voice data

[0230] Output: The audio data provided to the user (e.g., download link, streaming URL)

[0231] (Application example 2)

[0232] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0233] In modern society, elderly people living alone, people with physical disabilities, and those who have lost their partners are prone to feeling lonely and have difficulty managing their health. Furthermore, food delivery services also face the issue of not being able to provide guidance or confirmation in a friendly voice, which hinders the user experience.

[0234] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for utilizing a conversion system that mimics a human voice, converting speech to text, creating a response using a generation AI, converting the response to speech, and communicating in a conversational format, as well as a means for providing guidance and confirmation in a friendly voice. This helps elderly people living alone, people with physical disabilities, and those who have lost their partners to feel less lonely and allows them to manage their health. Furthermore, in food delivery services, users can receive guidance and confirmation in a friendly voice, improving the user experience.

[0235] "Specific eligible individuals" refer to individuals who meet certain conditions, such as elderly people living alone, people with physical disabilities, and those who have lost their partners.

[0236] "Communications" refers to the means of sending and receiving voice and data via telephone and the Internet.

[0237] "Safety check" refers to the act of checking the health and safety of a specific person.

[0238] "Vocal timbre" refers to the characteristics and tone of a voice, which can be used to determine the speaker's emotions and physical condition.

[0239] "Health management" refers to monitoring the health status of a specific subject and taking appropriate measures as necessary.

[0240] "Conversion system" refers to technology that converts speech to text and text to speech.

[0241] "Generative AI" refers to technology that uses artificial intelligence to generate appropriate answers from text data.

[0242] A "familiar voice" refers to a voice that is familiar to or well-liked by the target audience.

[0243] "Guidance and confirmation" refers to actions to provide information to the user or confirm their actions.

[0244] An "audio professional" refers to someone whose job involves working exclusively with audio, such as a voice actor or narrator.

[0245] "Immediate family" refers to family members related by blood.

[0246] The following system configuration will be described as an embodiment of the present invention.

[0247] System Configuration

[0248] This system consists of three main components: a server, a terminal, and a user. The server includes a speech conversion system and a generative AI, and the terminal is a communication device such as a smartphone or tablet. The user is a specific target group, such as elderly people living alone, people with physical disabilities, or those who have lost their partner.

[0249] Program processing

[0250] The server uses a speech-to-text system to convert the user's speech into text and inputs that text into the generation AI. The generation AI generates an appropriate answer based on the input text and converts that answer back into speech. The converted speech is used to provide guidance and confirmation to the user in a friendly voice.

[0251] Hardware and software used

[0252] Hardware: Servers, smartphones, tablets

[0253] Software: Python, pyttsx3 (speech synthesis library), soundfile (audio file reading library), scipy (scientific calculation library)

[0254] Data processing and calculation

[0255] 1. Reading audio data: The server reads the user's audio data using the soundfile library.

[0256] 2. Audio data conversion: The loaded audio data is converted into text using a speech conversion system.

[0257] 3. Answer generation using generative AI: Text data is input into a generative AI model to generate appropriate answers.

[0258] 4. Re-conversion to speech: The generated text response is converted to speech using the pyttsx3 library.

[0259] 5. Play audio: The converted audio is sent to the user's device, providing guidance and confirmation in a friendly voice.

[0260] Specific examples

[0261] For example, if a user is ordering a pizza from a food delivery app, hearing their deceased mother's voice ask, "Are you sure you want to confirm your order?" before confirming the order can create a sense of familiarity and security for the user.

[0262] Prompt Sentence Examples

[0263] "In your food delivery app, implement a friendly voice confirmation feature before the user confirms their order. Use pre-recorded voice data and translate it into the target voice for playback."

[0264] In this way, the embodiments for carrying out the invention have been specifically shown.

[0265] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0266] Step 1:

[0267] The server receives the user's voice data. The input is the user's voice data, and the output is the voice data stored in the server. Specifically, the server reads the voice data sent from the smartphone or tablet using the soundfile library.

[0268] Step 2:

[0269] The server converts the voice data into text using a speech conversion system. The input is voice data and the output is text data. Specifically, the server analyzes the voice data using speech recognition technology and generates corresponding text data.

[0270] Step 3:

[0271] The server inputs text data into the generative AI model and generates an appropriate answer. The input is text data, and the output is the generated answer text. Specifically, the server inputs a prompt sentence into the generative AI model and generates an appropriate answer to the user's question or request.

[0272] Step 4:

[0273] The server converts the generated answer text into speech. The input is the generated answer text, and the output is speech data. Specifically, the server converts text data into speech data using the pyttsx3 library.

[0274] Step 5:

[0275] The server sends the converted voice data to the user's device. The input is the voice data, and the output is the voice data sent to the user's device. Specifically, the server uses a communication protocol to send the voice data to the user's smartphone or tablet.

[0276] Step 6:

[0277] The user's device plays the received voice data. The input is the voice data sent from the server, and the output is the voice the user hears. Specifically, the user's device uses its built-in voice playback function to play the voice data and provides guidance and confirmation in a friendly voice.

[0278] Example 3

[0279] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0280] There is a need to reduce the amount of time that people who are prone to social isolation, such as elderly people living alone, people with physical disabilities, and those who have lost their partners, feel lonely and provide them with a sense of security. It is also important to regularly check on the safety of these people and manage their health. However, there is not yet a sufficient system in place to implement this efficiently and effectively.

[0281] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0282] In this invention, the server includes means for calling specific subjects to check on their safety and alleviate loneliness through conversation, and also for managing their physical condition through tone of voice, means for collecting subject information and storing it in a database, means for setting a call schedule based on the subject's lifestyle and preferences, means for automatically making calls based on the set schedule, and means for adjusting the schedule and updating the database in response to the subject's feedback and changes in lifestyle. This makes it possible to reduce the amount of time the subject feels lonely and provide a sense of security.

[0283] "Specific target groups" refer to people who are prone to social isolation, such as elderly people living alone, people with disabilities, and those who have lost their partners.

[0284] "Safety check" refers to phone calls or conversations to check on the health and safety of the person in question.

[0285] "Health management through vocal tone" refers to analyzing the tone and quality of a person's voice to understand their physical and mental state.

[0286] A "conversion system" refers to a system that mimics a human voice, converts the speech into text, creates an answer using generative AI, and then converts it back into speech.

[0287] "Generative AI" refers to a system that uses artificial intelligence technology to generate appropriate answers based on input text data.

[0288] "Database" refers to an information system for systematically storing and managing information on subjects.

[0289] A "call schedule" refers to a plan for making calls at the optimal time based on the target person's lifestyle and preferences.

[0290] "Feedback" refers to opinions and reactions obtained from subjects, and is information used to improve and adjust the system.

[0291] "Means for making automatic phone calls" refers to the ability of the system to automatically make phone calls based on a set schedule.

[0292] "Schedule adjustment" refers to changing the time and frequency of calls based on the target person's lifestyle and feedback.

[0293] MODE FOR CARRYING OUT THE INVENTION

[0294] This invention is a system that provides a sense of security and alleviates loneliness by making regular phone calls to people who are prone to social isolation, such as elderly people living alone, people with physical disabilities, and those who have lost their partners. A specific embodiment of this system will be described below.

[0295] 1. Program Generation

[0296] The server generates a program that collects information about the target person and stores it in a database. This program sets a call schedule based on the target person's lifestyle and preferences, and automatically makes calls based on the set schedule.

[0297] 2. Program Processing

[0298] The server provides a web form to collect information about the subjects. Users use this web form to enter information such as the subject's name, phone number, lifestyle, preferences, etc. The collected information is stored in a MySQL® database.

[0299] The server then uses a Python script to schedule calls for each individual. The script calculates the best time to call based on the individual's lifestyle and preferences. For example, if a person prefers to be called in the morning, the server will schedule the script to call at that time.

[0300] The server uses the Twilio API to automatically make calls based on a set schedule. The time of the call is adjusted to suit the recipient's lifestyle and preferences. For example, if recipient A wants to be called at 10:00 a.m. every day, the server will make the call at that time.

[0301] The server also receives feedback from the subject and adjusts the schedule as needed: for example, if the subject decides they prefer to be called in the afternoon, the information in the database is updated and the schedule is rescheduled.

[0302] 3. Examples and prompts

[0303] As a concrete example, consider the following scenario.

[0304] Subject A (an elderly person living alone) requests a phone call every day at 10:00 a.m.

[0305] Subject B (a physically disabled person) requests phone calls three times a week at 2:00 p.m.

[0306] The server stores this information in a database and uses a Python script to schedule calls, which are then automatically placed at the scheduled time using the Twilio API.

[0307] Example prompts to input to a generative AI model:

[0308] "Please schedule a call to elderly person A who lives alone every day at 10:00 AM."

[0309] "Schedule a call with Person B, who is disabled, three times a week at 2:00 PM."

[0310] This system can reduce the amount of time that a subject feels lonely and provide a sense of security. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0311] Step 1:

[0312] The user uses a web form to enter information about the target person, including their name, phone number, lifestyle, preferences, etc. This information is then sent to the server.

[0313] Input: Subject's name, phone number, lifestyle, preferences

[0314] Output: Information of the target person sent to the server

[0315] Step 2:

[0316] The server stores the received information in a MySQL database by inserting the information into the appropriate table.

[0317] Input: Information about the target person sent by the user

[0318] Output: Subject information stored in a database

[0319] Step 3:

[0320] The server schedules calls for each individual using a Python script that calculates the best time to call based on the individual's lifestyle and preferences.

[0321] Input: Subject information stored in the database

[0322] Output: The configured call schedule

[0323] Step 4:

[0324] The server uses the Twilio API to automatically place calls based on a set schedule, with call times tailored to the recipient's lifestyle and preferences.

[0325] Input: Configured call schedule

[0326] Output: Calls made to the target person

[0327] Step 5:

[0328] The server receives feedback from the subject and adjusts the schedule as needed: for example, if the subject decides they prefer a call in the afternoon, it updates the database and reschedules the call.

[0329] Input: Feedback from subjects

[0330] Output: Adjusted call schedule and updated database

[0331] In this way, the system can reduce the amount of time a subject feels lonely and provide a sense of security.

[0332] (Application example 3)

[0333] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0334] Elderly people living alone, people with physical disabilities, and those who have lost their partners often feel lonely and anxious. Furthermore, these individuals may not receive appropriate support in emergencies. Conventional safety confirmation systems struggle to respond flexibly to individuals' lifestyles and preferences, and are insufficient to provide a sense of security. Furthermore, they lack the functionality to analyze individuals' responses and notify emergency contacts as necessary, potentially delaying emergency response.

[0335] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0336] In this invention, the server includes means for calling specific individuals to check on their safety and alleviate loneliness through conversation, and also for managing their physical condition through tone of voice, means for creating a list of individuals and setting a call schedule based on their lifestyle and preferences, means for automatically making calls based on the set schedule, means for analyzing the individual's responses and notifying an emergency contact as necessary, and means for providing the individual with a sense of security through regular calls. This reduces the individual's sense of loneliness, provides a sense of security, and enables a rapid response in emergencies.

[0337] "Specific target groups" refer to individuals who meet certain conditions, such as elderly people living alone, people with physical disabilities, and those who have lost their partners.

[0338] "Safety check" refers to the act of checking whether a person is safe.

[0339] "Health management through voice tone" refers to a method of analyzing the tone and quality of a person's voice to detect changes or abnormalities in their physical condition.

[0340] "Conversion system" refers to a system that mimics the human voice and converts speech into text.

[0341] "Generative AI" refers to technology that uses artificial intelligence to generate appropriate answers from text.

[0342] "Target list" refers to a list that compiles information about specific targets.

[0343] "Scheduling" refers to the act of setting a time to call based on the target person's lifestyle and preferences.

[0344] "Means for making automatic phone calls" refers to the ability to make automatic phone calls based on a set schedule.

[0345] "Means for analyzing responses" refers to the function of analyzing the subject's telephone responses and notifying emergency contacts as necessary.

[0346] "Means of providing a sense of security" refers to the function of providing a sense of security to the target person through regular phone calls.

[0347] The following system configuration is used as an embodiment of the present invention.

[0348] System Configuration

[0349] 1. Hardware:

[0350] Server: Processes data and runs generative AI models.

[0351] Device: A smartphone or robot that makes phone calls to the subject.

[0352] Communications infrastructure: Internet connection and telephone lines.

[0353] 2. Software:

[0354] Generative AI model: Generates appropriate answers based on the subject's responses.

[0355] Speech-to-text system: imitates the human voice and converts speech to text.

[0356] Scheduling system: Schedule calls to suit your target audience's lifestyle and preferences.

[0357] Response analysis system: Analyzes the subject's response and notifies emergency contacts as necessary.

[0358] Program processing explanation

[0359] The server creates a list of contacts and sets a call schedule based on their lifestyles and preferences. The device automatically makes calls based on the set schedule. During the call, the voice conversion system converts the contact's voice into text, and the generative AI model generates an appropriate response. The generated text is then converted into voice and sent back to the contact. The response analysis system analyzes the contact's response and notifies emergency contacts if necessary.

[0360] Specific examples

[0361] For example, if Tanaka, an elderly person living alone, is registered on the target list, the server sets a schedule to call Tanaka every day at 10:00 to suit his lifestyle. At 10:00, the device automatically calls Tanaka. When Tanaka answers the phone, the speech conversion system converts Tanaka's voice into text, and the generative AI model generates an appropriate response such as "How are you spending your day?" This text is then converted into speech and responded to Tanaka. If an abnormality is detected in Tanaka's response, the response analysis system notifies the emergency contact.

[0362] Prompt Sentence Examples

[0363] "Develop a system for making regular phone calls to elderly people living alone, people with disabilities, and those who have lost their partners. Create a list of people to call and schedule calls based on their lifestyles and preferences. Calls should provide reassurance and include the ability to notify emergency contacts if necessary."

[0364] In this way, it is possible to provide a specific embodiment of the invention.

[0365] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0366] Step 1:

[0367] The server creates a target list.

[0368] Input: Subject's personal information (name, phone number, lifestyle, preferences, etc.)

[0369] Data processing: The subject's information is registered in a database and managed in list format.

[0370] Output: List of target users

[0371] Step 2:

[0372] The server schedules calls based on the recipient's lifestyle and preferences.

[0373] Input: List of subjects, information on their lifestyle and preferences

[0374] Data calculation: Calculate the optimal time to call each target person and set a schedule.

[0375] Output: Call Schedule

[0376] Step 3:

[0377] The device will automatically make calls based on a set schedule.

[0378] Input: Call Schedule

[0379] Specific actions: Call the target person according to the schedule.

[0380] Output: Start of phone call

[0381] Step 4:

[0382] The terminal converts the target person's voice into text using a speech conversion system.

[0383] Input: Subject's voice

[0384] Data processing: Using a voice conversion system, convert the voice into text data.

[0385] Output: Text data

[0386] Step 5:

[0387] The server uses a generative AI model to generate appropriate answers from the text data.

[0388] Input: Text data

[0389] Data Computation: Using generative AI models to generate answers based on text data.

[0390] Output: Generated answer text

[0391] Step 6:

[0392] The terminal converts the generated answer text into speech and responds to the subject.

[0393] Input: Generated answer text

[0394] Data processing: Using a speech conversion system, convert the text into audio data.

[0395] Specific operation: Play audio data to the target person.

[0396] Output: Voice response to the target person

[0397] Step 7:

[0398] The server analyzes the subject's response and notifies emergency contacts if necessary.

[0399] Input: Subject's response text data

[0400] Data calculation: Using the response analysis system, the response content is analyzed, and if an abnormality is detected, an emergency contact is notified.

[0401] Specific behavior: Send a notification to emergency contacts.

[0402] Output:Notify emergency contacts

[0403] In this way, each processing step can be concretely explained.

[0404] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0405] "Example 1"

[0406] One embodiment of the present invention is a system that incorporates an emotion engine. This system makes a phone call to a specific target person and engages in a conversation. The conversation is converted from speech to text using a conversion system that mimics the human voice, and a generation AI creates a response. The created response is converted back into speech and transmitted to the target person via phone. At this time, the emotion engine analyzes the target person's emotions from their tone of voice and speaking style, and feeds the results back to the generation AI. Based on this feedback, the generation AI generates a conversation that is adapted to the target person's emotions.

[0407] "Example 2"

[0408] As a specific example, when the target person is sad, the emotion engine detects that sadness and provides feedback to the generation AI. Based on this information, the generation AI generates conversations that comfort the target person. Similarly, when the target person is happy, it generates conversations that share that joy. This allows the system to provide appropriate dialogue that corresponds to the target person's emotions.

[0409] "Example 3"

[0410] Furthermore, the emotion engine can analyze emotions not only from the subject's tone of voice and speaking style, but also from the content of the conversation. For example, if the subject says, "I had fun today," the emotion engine detects this joy and provides feedback to the generation AI. Based on this information, the generation AI generates a conversation that shares the subject's joy. This allows the system to provide a dialogue that more deeply reflects the subject's emotions.

[0411] The processing flow of each embodiment will be described below.

[0412] "Example 1"

[0413] Step 1: The system makes a call to a specific target person.

[0414] Step 2: Once a conversation with the subject begins, the system converts the speech into text using a conversion system that mimics the human voice.

[0415] Step 3: Generative AI analyzes the text and creates an appropriate response.

[0416] Step 4: The created answer is converted back into audio and transmitted to the target person via phone. Step 5: The emotion engine analyzes the target person's emotions from their tone of voice and speaking style, and feeds the results back to the generation AI.

[0417] Step 6: The generative AI uses this feedback to generate conversations that adapt to the subject's emotions.

[0418] "Example 2"

[0419] Step 1: When a subject is sad, the emotion engine detects that sadness.

[0420] Step 2: The detected emotion information is fed back to the generation AI.

[0421] Step 3: The generative AI uses this information to generate a conversation that will comfort the target person.

[0422] Step 4: The generated conversation is converted into audio and transmitted to the target person via telephone.

[0423] "Example 3"

[0424] Step 1: The emotion engine analyzes emotions not only from the tone of voice and speaking style of the subject, but also from the content of what is being said.

[0425] Step 2: For example, if the subject says, "I had fun today," the emotion engine detects that joy.

[0426] Step 3: The detected emotion information is fed back to the generation AI.

[0427] Step 4: Generative AI uses this information to generate conversations that share the target person's joy.

[0428] Step 5: The generated conversation is converted into audio and transmitted to the target person via telephone.

[0429] Example 1

[0430] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0431] In modern society, elderly people living alone, people with physical disabilities, and those who have lost their partners often feel lonely and anxious. These people need effective means to regularly check on their safety and reduce their feelings of loneliness. It is also important to understand the physical and emotional state of the person from their tone of voice and speaking style, and respond appropriately. However, conventional systems have the problem of not being able to fully meet these needs.

[0432] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0433] In this invention, the server includes means for converting the target person's voice into text using speech recognition technology, means for analyzing the text using a generative AI model and generating an appropriate response, means for converting the generated response into speech using speech synthesis technology, means for analyzing the target person's tone of voice and speaking style using an emotion engine and feeding the results back to the generative AI model, and means for the generative AI model to generate a conversation adapted to the target person's emotions based on the feedback. This makes it possible to check the target person's safety through natural conversation, reduce feelings of loneliness, and understand the target person's physical condition and emotional state from the tone of voice and speaking style, enabling appropriate response.

[0434] "Speech recognition technology" is a technology that converts voice data into text data.

[0435] A "generative AI model" is an artificial intelligence model that analyzes input text data and generates appropriate answers.

[0436] "Speech synthesis technology" is a technology that converts text data into voice data.

[0437] The "emotion engine" is a technology that analyzes the subject's tone of voice and speaking style to assess their emotional state.

[0438] "Feedback" is the process of returning information to the generative AI model based on the analysis results.

[0439] A "target person" is a person who the system calls to check on their safety and to have a conversation.

[0440] "Safety check" refers to the act of checking the safety and health of a person.

[0441] "Loneliness reduction" refers to actions aimed at alleviating the sense of loneliness felt by the subject.

[0442] "Vocal timbre" refers to the tone and texture of a speaker's voice.

[0443] "Health management" refers to the act of understanding the subject's health condition and taking appropriate measures.

[0444] This invention is a system that calls specific individuals to check their safety, alleviating feelings of loneliness through conversation and managing their physical condition through tone of voice. This system is composed of a combination of voice recognition technology, generative AI models, voice synthesis technology, and an emotion engine.

[0445] The server first calls the target person. Once the call is connected, the device collects the target person's voice. The collected voice data is converted into text data using speech recognition software (e.g., Google Cloud Speech-to-Text). This text data is then input into a generative AI model (e.g., OpenAI's GPT-4).

[0446] The generative AI model analyzes the input text and generates an appropriate response. This response text is then converted back into audio data using speech synthesis software (e.g., Amazon Polly). The converted audio is then transmitted to the target person via telephone.

[0447] Furthermore, the device uses an emotion engine (e.g., Affectiva's emotion recognition technology) to analyze the subject's tone of voice and speaking style. The analysis results are obtained as data indicating the subject's emotional state. This data is fed back to a generative AI model, which then generates conversation content adapted to the subject's emotions.

[0448] As a concrete example, the following scenario can be considered.

[0449] scenario:

[0450] When the user answers the phone, the server asks, "Hello, how was your day?" If the user replies, "I'm a little tired today," the device collects this speech and uses speech recognition software to convert it into text: "I'm a little tired today." The server inputs this text into a generative AI model, which generates a response: "That's tough. Is there anything I can help you with?" The server inputs this response into speech synthesis software, which the device converts into speech and conveys to the user. The device then uses an emotion engine to analyze the user's level of fatigue from their tone of voice and feeds the results back to the generative AI model. The generative AI model uses this feedback to adjust the content of the next conversation.

[0451] Example prompt sentence:

[0452] "If a subject says, 'I'm a little tired today,' we provide the following prompt to the generative AI model: 'If a subject feels tired, how should we respond?'"

[0453] In this way, the system can confirm the safety of the target person and reduce their sense of loneliness through natural conversation.

[0454] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0455] Step 1:

[0456] The server calls the target person.

[0457] The server automatically makes a call using the target person's pre-registered phone number. Once the call is connected, the process proceeds to the next step. The input is the target person's phone number, and the output is the call connection status.

[0458] Step 2:

[0459] The device collects the subject's voice.

[0460] The device collects the subject's voice in real time via telephone, and temporarily stores the voice data for subsequent processing. The input is the subject's voice, and the output is the collected voice data.

[0461] Step 3:

[0462] The device uses speech recognition software to convert the speech into text.

[0463] The device uses speech recognition software (e.g., Google Cloud Speech-to-Text) to convert the collected voice data into text data. The input is the voice data, and the output is the converted text data.

[0464] Step 4:

[0465] The server inputs text into the generative AI model.

[0466] The server inputs the text data obtained from the speech recognition software into a generative AI model (e.g., OpenAI's GPT-4). The input is the text data, and the output is the analysis result by the generative AI model.

[0467] Step 5:

[0468] A generative AI model generates the appropriate answer.

[0469] A generative AI model generates appropriate answers based on input text. These answers are tailored to the subject's situation and emotions. The input is text data, and the output is the generated answer text.

[0470] Step 6:

[0471] The server inputs the generated response into speech synthesis software.

[0472] The server inputs the answer text obtained from the generative AI model into speech synthesis software (e.g., Amazon Polly). The input is the answer text, and the output is voice data.

[0473] Step 7:

[0474] The device uses speech synthesis software to convert the text to speech.

[0475] The device uses speech synthesis software to convert the response text into audio data, which is then ready to be transmitted to the target person. The input is the response text, and the output is the converted audio data.

[0476] Step 8:

[0477] The device transmits the converted voice to the target person.

[0478] The device transmits the converted voice to the subject via telephone, and the subject can hear the answer generated by the generative AI model. The input is voice data, and the output is voice communication to the subject.

[0479] Step 9:

[0480] The device uses an emotion engine to analyze the subject's tone of voice and speaking style.

[0481] The device uses an emotion engine (e.g., Affectiva's emotion recognition technology) to analyze the subject's tone of voice and speaking style. The analysis results are obtained as data indicating the subject's emotional state. The input is the subject's voice data, and the output is the emotion analysis results.

[0482] Step 10:

[0483] The server feeds back the analysis results of the emotion engine to the generative AI model.

[0484] The server feeds back the analysis results obtained from the emotion engine to the generative AI model. This feedback allows the generative AI model to generate conversation content that takes into account the emotional state of the target person. The input is the emotion analysis results, and the output is the fed-back data.

[0485] Step 11:

[0486] A generative AI model adjusts the conversation based on feedback.

[0487] The generative AI model adjusts the conversation content based on the feedback and generates responses for the next conversation. This process is repeated until the conversation with the target person is completed. The input is the feedback data, and the output is the adjusted conversation content.

[0488] (Application example 1)

[0489] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0490] In modern society, the number of elderly people living alone, physically disabled people, and people who have lost their partners is increasing, and the loneliness and health management of these people are becoming major issues. In particular, regular safety checks and emotional care are necessary, but doing this manually is difficult in terms of cost and effort. In addition, there is a need for a system that can conduct conversations that adapt to the emotions of the target person.

[0491] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server is a system that calls a specific target person to check on their safety and alleviate loneliness through conversation, while also managing their health through tone of voice. The system includes: a means for converting speech to text using a conversion system that mimics a human voice, creating a response using a generation AI, converting the response back to speech, and conducting a conversation over the phone; a means for analyzing the target person's emotions from their tone of voice and speaking style using an emotion analysis engine and feeding the results back to the generation AI; a means for the generation AI to generate a conversation adapted to the target person's emotions based on the emotion analysis results; a means for converting speech to text using a speech recognition API, a generation AI model, and a speech synthesis API, and converting the generated text back to speech; and a means for calling the target person using a smartphone and communicating the generated speech. This reduces the target person's sense of loneliness and enables health management through conversation adapted to their emotions.

[0492] "Specific target groups" refer to people who need regular safety checks and emotional support, such as elderly people living alone, people with disabilities, and those who have lost their partners.

[0493] "Calling to check on safety" refers to the act of contacting the target person by phone to check whether they are safe.

[0494] "Alleviating loneliness through conversation" refers to reducing a subject's sense of loneliness by engaging in a dialogue with them.

[0495] "Health management based on tone of voice" refers to inferring a person's physical condition from their tone of voice and speaking style, and taking the necessary measures.

[0496] "Conversion system" refers to technology that mimics the human voice and converts speech into text.

[0497] "Generative AI" refers to artificial intelligence that analyzes input text and generates appropriate answers.

[0498] An "emotion analysis engine" refers to technology that analyzes emotions from the subject's tone of voice and speaking style.

[0499] "Speech Recognition API" refers to an application programming interface for converting speech to text.

[0500] A "generative AI model" refers to an artificial intelligence model that generates appropriate answers based on input text data.

[0501] "Speech Synthesis API" means an application programming interface for converting text to speech.

[0502] A "smartphone" refers to a mobile information terminal that has telephone functions and can connect to the Internet and run applications.

[0503] The system for carrying out this invention makes telephone calls to specific target individuals to check on their safety, alleviating loneliness through conversation and managing their physical condition through tone of voice. A specific embodiment of this system will be described below.

[0504] Hardware and software used

[0505] Hardware

[0506] Smartphone (iOS or ANDROID (registered trademark))

[0507] software

[0508] Speech Recognition API (Google Cloud Speech-to-Text)

[0509] Generative AI model (OpenAI GPT-4)

[0510] Speech synthesis API (Google Cloud Text-to-Speech)

[0511] Sentiment analysis engine (IBM Watson (registered trademark) Tone Analyzer)

[0512] System configuration

[0513] 1. Voice Recognition

[0514] The subject's voice is collected through the smartphone's microphone.

[0515] Use the speech recognition API (Google Cloud Speech-to-Text) to convert the collected audio into text.

[0516] 2. Text Analysis

[0517] Input text into a generative AI model (OpenAI GPT-4) to generate an appropriate answer.

[0518] 3. Emotion analysis

[0519] An emotion analysis engine (IBM Watson Tone Analyzer) is used to analyze emotions from the subject's tone of voice and speaking style.

[0520] The analysis results are fed back to the generation AI.

[0521] 4. Answer generation

[0522] Based on the results of emotion analysis, the generative AI generates conversations that adapt to the emotions of the target person.

[0523] 5. Speech Synthesis

[0524] Use a speech synthesis API (Google Cloud Text-to-Speech) to convert the generated text response into audio.

[0525] 6. Phone Calls

[0526] The subject is called using a smartphone and the generated audio is transmitted.

[0527] Specific examples

[0528] For example, if an elderly person lives alone, the app can automatically call them at a set time every day to check on their safety. Below are some examples of specific prompts:

[0529] Prompt Sentence Examples

[0530] Subject's audio text: "I'm feeling a little under the weather today."

[0531] Prompt to generator AI: "Subject says they are not feeling well. Please offer kind words of encouragement."

[0532] Generator's response: "That's worrying. Please take it easy and get plenty of rest. Let me know if there's anything I can do to help."

[0533] In this way, the system can reduce the subject's sense of loneliness and help them manage their health through emotionally adaptive conversations.

[0534] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0535] Step 1:

[0536] The smartphone device calls the subject and collects the subject's voice through the microphone. The collected voice data is sent to a voice recognition API (Google Cloud Speech-to-Text). The input is the subject's voice data, and the output is text data. The voice recognition API analyzes the voice data and converts it into corresponding text data.

[0537] Step 2:

[0538] The server inputs the text data received from the speech recognition API into a generative AI model (OpenAI GPT-4). The input is the text data from the speech recognition API, and the output is the generated answer text. The generative AI model analyzes the input text data and generates an appropriate answer.

[0539] Step 3:

[0540] The server sends the generated response text to an emotion analysis engine (IBM Watson Tone Analyzer), which analyzes the emotion of the subject based on their tone of voice and speaking style. The input is the generated response text, and the output is the emotion analysis result. The emotion analysis engine analyzes the text data and identifies the subject's emotional state.

[0541] Step 4:

[0542] The server feeds back the emotion analysis results to the generative AI model and generates a conversation that adapts to the emotions. The input is the emotion analysis results and the generated answer text, and the output is the final answer text that adapts to the emotions. The generative AI model takes the emotion analysis results into account and generates an answer that adapts to the target person's emotions.

[0543] Step 5:

[0544] The server sends the final answer text to a speech synthesis API (Google Cloud Text-to-Speech) and converts it into audio data. The input is the final answer text and the output is audio data. The speech synthesis API analyzes the text data and generates the corresponding audio data.

[0545] Step 6:

[0546] The smartphone device plays back the voice data received from the speech synthesis API and transmits it to the target person. The input is the voice data from the speech synthesis API, and the output is the voice that is transmitted to the target person. The smartphone device plays back the generated voice and transmits it to the target person, completing the conversation.

[0547] Example 2

[0548] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0549] In modern society, elderly people living alone, people with physical disabilities, and those who have lost their partners often feel lonely, which can lead to poor mental health. It is also difficult to remotely manage the health of these individuals. Conventional safety confirmation systems simply check the safety of individuals, and are unable to provide dialogue tailored to the individual's emotions and physical condition.

[0550] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0551] In this invention, the server includes: means for communicating with a specific target person to check on their safety and alleviate loneliness through conversation, and also capable of managing their health through voice; means for utilizing a conversion system that mimics a human voice to convert voice to text, creating a response using a generative AI model, converting the response back into voice, and communicating in a conversational format; means for inputting the target person's voice and facial expressions into an emotion engine to detect emotions; means for inputting prompt sentences into the generative AI model based on feedback from the emotion engine to generate dialogue corresponding to the emotions; and means for converting the generated dialogue into mimic voice and providing it to the target person. This allows for the provision of dialogue that corresponds to the target person's emotions and physical condition, reducing loneliness and enabling remote health management.

[0552] "Specific target groups" refer to individuals who are particularly susceptible to loneliness and need to manage their health, such as elderly people living alone, people with physical disabilities, and those who have lost their partners.

[0553] "Communication" refers to the act of sending or receiving information by telephone, internet, or other digital means of communication.

[0554] "Safety check" refers to the act of checking whether a specific person is safe.

[0555] "Voice-based health management" refers to the act of analyzing a person's tone of voice and speaking style to understand their physical and emotional state.

[0556] A "conversion system" refers to a system that analyzes input voice data and generates imitation voices.

[0557] "Generative AI model" refers to an artificial intelligence model that generates natural-looking dialogue based on input text and prompts.

[0558] An "emotion engine" refers to a system that analyzes a subject's voice and facial expressions to detect their emotional state.

[0559] A "prompt sentence" refers to an instruction sentence entered into a generative AI model to generate a specific dialogue.

[0560] "Imitation voice" refers to a voice that is generated based on specific voice data and is similar to the original voice.

[0561] "Conversational style" refers to the style or method of conducting a dialogue, including two-way communication using voice or text.

[0562] MODE FOR CARRYING OUT THE INVENTION

[0563] This invention is a system that communicates with specific target individuals, checks their safety, and alleviates loneliness through conversation, while also managing their physical condition through voice. This system operates in cooperation with three parties: a server, a terminal, and a user.

[0564] Hardware and software used

[0565] Hardware: Servers, devices (PCs, smartphones, etc.)

[0566] Software: Text-to-speech system, emotion engine, generative AI model

[0567] Specific operation of the system

[0568] 1. Uploading audio data

[0569] Users upload audio data of voice actors or loved ones recorded before they passed away from their device to the server.

[0570] Example: A user opens an app on their smartphone, selects audio data, and presses the "Upload" button. The server receives and stores this audio data.

[0571] 2. Analysis of speech data and generation of imitation speech

[0572] The server inputs the uploaded voice data into the voice conversion system.

[0573] Example: The server calls a voice conversion system (e.g., Google Cloud Text-to-Speech API) and analyzes the voice data. Based on the analysis results, it generates an imitation voice and saves it on the server.

[0574] 3. User Emotion Detection

[0575] The terminal inputs the user's voice and facial expressions into an emotion engine to detect emotions.

[0576] Example: The device's camera and microphone capture the user's facial expressions and voice in real time and send them to an emotion engine (e.g., Affectiva SDK). The emotion engine analyzes the user's emotional state and returns the results to the device.

[0577] 4. Emotion-Based Dialogue Generation

[0578] The server inputs prompt sentences into the generative AI model based on feedback from the emotion engine.

[0579] Example: A server sends a prompt to a generative AI model (e.g., OpenAI GPT-3) saying, "The user is sad, so please generate a comforting conversation." The generative AI model generates an appropriate dialogue and returns it to the server.

[0580] 5. Providing dialogue using imitation voice

[0581] The server converts the generated dialogue into imitation voice and transmits it to the terminal.

[0582] Example: The server inputs the generated text into a speech conversion system to generate an imitation voice, which is then sent to the device, which plays it back to the user.

[0583] Prompt Sentence Examples

[0584] "The user is sad, so generate a comforting conversation."

[0585] "Your users are happy, so create conversations that share that joy."

[0586] This system allows users to enjoy conversations in a friendly voice that responds to their emotions, reducing feelings of loneliness and enabling remote health management.

[0587] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0588] Program processing flow

[0589] Step 1: Upload your audio data

[0590] Users upload audio data of voice actors or loved ones recorded before they passed away from their device to the server.

[0591] Input: User-selected audio data file

[0592] Specific operation: The user opens the app on their smartphone, selects the audio data, and presses the "Upload" button. The device then sends the audio data to the server.

[0593] Output: Audio data file stored on the server

[0594] Step 2: Analyzing the audio data and generating imitation voices

[0595] The server inputs the uploaded voice data into the voice conversion system.

[0596] Input: Audio data file stored on the server

[0597] Specific operation: The server calls a voice conversion system (e.g., Google Cloud Text-to-Speech API) and analyzes the voice data. Based on the analysis results, it generates an imitation voice and saves it on the server.

[0598] Output: Imitation voice data stored on the server

[0599] Step 3: Detecting user emotions

[0600] The terminal inputs the user's voice and facial expressions into an emotion engine to detect emotions.

[0601] Input: Real-time user voice and facial expression data

[0602] How it works: The device's camera and microphone capture the user's facial expressions and voice in real time and send them to an emotion engine (e.g., Affectiva SDK). The emotion engine analyzes the user's emotional state and returns the results to the device.

[0603] Output: User's emotional state data

[0604] Step 4: Emotion-based dialogue generation

[0605] The server inputs prompt sentences into the generative AI model based on feedback from the emotion engine.

[0606] Input: User emotional state data

[0607] Specific operation: The server sends a prompt to the generative AI model (e.g., OpenAI GPT-3) saying, "The user is sad, so please generate a comforting conversation." The generative AI model generates an appropriate dialogue and returns it to the server.

[0608] Output: Generated dialogue text

[0609] Step 5: Providing dialogue through imitation voice

[0610] The server converts the generated dialogue into imitation voice and transmits it to the terminal.

[0611] Input: Generated dialogue text

[0612] Specific operation: The server inputs the generated text into the speech conversion system to generate an imitation voice, which is then sent to the terminal, which plays it back to the user.

[0613] Output: The imitation voice played to the user

[0614] In this way, the system provides dialogue that responds to the user's emotions, reducing feelings of loneliness while enabling remote health management.

[0615] (Application example 2)

[0616] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0617] In modern society, elderly people living alone, people with physical disabilities, and those who have lost their partners are prone to feeling lonely and are in need of psychological support. Food delivery services also need systems that allow users to place orders and make inquiries in a friendly voice. However, conventional systems have difficulty providing appropriate dialogue based on emotions, making it difficult to increase user satisfaction.

[0618] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0619] In this invention, the server includes: means for communicating with a specific target person to check on their safety and alleviate loneliness through conversation, and also for managing their health through voice tone; means for utilizing a conversion system that mimics a human voice to convert speech to text, creating a response using a generation AI, converting it back into speech, and communicating in a conversational format; means for detecting the target person's emotions using an emotion engine and generating a conversation according to those emotions; and means for using voice data from a voice actor or a relative recorded before death so that the target person can enjoy a conversation in a familiar voice. This makes it possible to provide appropriate dialogue according to the target person's emotions, alleviating loneliness, and increasing user satisfaction in food delivery services.

[0620] "Specific target groups" refer to individuals who meet certain conditions, such as elderly people living alone, people with physical disabilities, and those who have lost their partners.

[0621] "Communications" refers to the means of sending and receiving voice and data via telephone and the Internet.

[0622] "Safety check" refers to the act of checking the health and safety of the person in question.

[0623] "Conversational" refers to a form of two-way communication using voice.

[0624] "Health management through vocal tone" refers to a method of understanding a person's physical and emotional state by analyzing the tone and quality of their voice.

[0625] "Conversion system" refers to a system that converts speech to text and text to speech.

[0626] "Generative AI" refers to a system that uses artificial intelligence to generate appropriate responses based on input data.

[0627] An "emotion engine" refers to an algorithm or system for detecting emotions from a subject's voice or text.

[0628] A "voice actor" is a professional who performs the voices of others as a profession.

[0629] "Audio data of immediate family members" refers to audio data recorded by the subject's family members or close friends while they were alive.

[0630] "Food delivery service" refers to a service that allows you to order food and drinks and have them delivered to a designated location.

[0631] "User" refers to an individual who uses a system or service.

[0632] The system for carrying out this invention is a system that can communicate with specific target individuals to check their safety, alleviate loneliness through conversation, and also manage their physical condition by tone of voice. A specific embodiment of this system is described below.

[0633] System Configuration

[0634] The system consists of the following main components:

[0635] 1. Communications: Hardware and software for sending and receiving voice and data over the telephone or the Internet.

[0636] 2. Speech conversion system: Software for converting speech to text and text to speech, specifically using the speech_recognition library and gTTS library.

[0637] 3. Generative AI: Artificial intelligence that generates appropriate responses based on input data. Uses the transformers library.

[0638] 4. Emotion Engine: An algorithm for detecting emotions from a subject's speech or text. It uses the emotion analysis model from the transformers library.

[0639] 5. Audio data: Audio data of voice actors or loved ones recorded before they passed away.

[0640] Processing flow

[0641] 1. Voice input: The subject inputs voice through a microphone, which is then transmitted to the server via a communication means.

[0642] 2. Speech Recognition: The server converts the speech to text using the speech_recognition library.

[0643] 3. Sentiment detection: The converted text is fed into the sentiment analysis model in the transformers library to detect emotions.

[0644] 4. Response generation: Based on the detected sentiment, the generative AI generates an appropriate response text.

[0645] 5. Speech generation: The generated response text is converted into speech using the gTTS library.

[0646] 6. Audio output: The generated audio is sent to the subject via a communication means and played back through a speaker.

[0647] Hardware and software used

[0648] Hardware: Microphones, speakers, servers

[0649] Software: speech_recognition library, transformers library, gTTS library, playsound library

[0650] Specific examples

[0651] For example, if a user says, "I'm very tired today," the emotion detection system will determine this as "NEGATIVE," and the generative AI will generate a response like, "Sorry, is there a problem?" This response is converted into audio and played back to the user.

[0652] Prompt Sentence Examples

[0653] User: I'm very tired today

[0654] System: Sorry, is there a problem?

[0655] In this way, it is possible to provide appropriate dialogue that is tailored to the target person's emotions, alleviating feelings of loneliness and increasing user satisfaction in food delivery services.

[0656] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0657] Step 1:

[0658] The user inputs voice through a microphone.

[0659] Input: User's voice

[0660] Output: Audio data

[0661] What it does: When a user says, "I'm so tired today," the audio is captured by a microphone.

[0662] Step 2:

[0663] The terminal transmits the voice data to the server.

[0664] Input: Audio data

[0665] Output: Audio data sent to the server

[0666] What it does: The captured audio data is sent over the internet to a server.

[0667] Step 3:

[0668] The server converts the audio to text using the speech_recognition library.

[0669] Input: Audio data

[0670] Output: Text data

[0671] What happens: The server receives the audio data and uses the speech_recognition library to convert it into text: "I'm very tired today."

[0672] Step 4:

[0673] The server uses the sentiment analysis model from the transformers library to detect sentiment from text data.

[0674] Input: Text data

[0675] Output: Emotion data (e.g., NEGATIVE)

[0676] Specific operation: The server inputs the converted text data into a sentiment analysis model and detects the sentiment "NEGATIVE."

[0677] Step 5:

[0678] The server uses generative AI to generate an appropriate response text based on the detected emotion.

[0679] Input: Emotion data, text data

[0680] Output: Response text (e.g. "Sorry, is there a problem?")

[0681] Specific operation: The server inputs emotion data and text data into the generation AI, which generates a response text saying, "Sorry, is there a problem?"

[0682] Step 6:

[0683] The server converts the response text into speech using the gTTS library.

[0684] Input: Response text

[0685] Output: Response voice data

[0686] Specific operation: The server inputs the generated response text into the gTTS library and converts it into voice data.

[0687] Step 7:

[0688] The server transmits the response voice data to the terminal.

[0689] Input: Response voice data

[0690] Output: Response audio data sent to the terminal

[0691] Specific operation: The server transmits the generated response voice data to the terminal via the Internet.

[0692] Step 8:

[0693] The device will play a response voice through the speaker.

[0694] Input: Response voice data

[0695] Output: The audio played to the user

[0696] Specific operation: The device plays the received response voice data over the speaker, delivering the voice message "Sorry, is there a problem?" to the user.

[0697] Example 3

[0698] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0699] In modern society, elderly people living alone, people with physical disabilities, and those who have lost their partners often feel lonely, which can have a negative impact on their mental health. Furthermore, there is a lack of efficient ways to check on the safety and health of these people. Furthermore, there is the challenge of providing conversations that are sensitive to the feelings of these people.

[0700] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0701] In this invention, the server is a system that can call specific target people to check on their safety, alleviate loneliness through conversation, and also manage their physical condition through tone of voice. It utilizes a conversion system that imitates human voices, converts voice into text, creates a response using a generative AI model, converts it into voice, and conducts a conversation over the phone. A way to create a list of contacts and schedule calls based on their lifestyle and preferences. An emotion analysis means that analyzes the tone of voice, speaking style, and content of the conversation in real time during a call to detect emotions; A means for the generative AI model to generate a response using a prompt sentence based on feedback from the sentiment analysis means; a means for communicating the generated response to the subject; This will help reduce the subject's sense of loneliness, provide a sense of security, and enable emotionally sensitive dialogue.

[0702] "Specific target groups" refer to individuals who meet certain conditions, such as elderly people living alone, people with physical disabilities, and those who have lost their partners.

[0703] "Safety check" refers to the act of checking whether a person is safe.

[0704] "Health management through voice tone" refers to analyzing the tone of a person's voice and speaking style to understand their physical and emotional state.

[0705] "Conversion system" refers to a system that mimics the human voice and converts speech into text.

[0706] A "generative AI model" refers to an artificial intelligence model that generates appropriate responses based on input text or prompts.

[0707] A "prompt sentence" refers to an instruction or input sentence that is used to generate a response for a generative AI model.

[0708] A "list of subjects" refers to a database or list that compiles information about specific subjects.

[0709] A "calling schedule" refers to a plan that sets the date, time, and frequency of calls to the target person.

[0710] "Emotion analysis means" refers to technology that analyzes the tone of voice, speaking style, and content of the conversation of the person on the phone to detect their emotions.

[0711] "Feedback" refers to the communication of information obtained from sentiment analysis methods to the generative AI model.

[0712] "Means for generating a response" refers to the process by which a generative AI model creates an appropriate response based on a prompt sentence.

[0713] "Means for carrying out a dialogue" refers to the technology used to convey the generated response to the target person and continue the dialogue.

[0714] This invention is a system that makes a call to a specific target person to check on their safety, alleviate feelings of loneliness through conversation, and manage their physical condition by tone of voice. A specific embodiment of this system will be described below.

[0715] System configuration

[0716] Hardware

[0717] The server contains a database, a voice conversion system, and a generative AI model. The terminal has telephone functionality and is equipped with a microphone and speaker for communicating with the target person.

[0718] software

[0719] The server runs the following software:

[0720] Database management system: Manages subject information.

[0721] Scheduling software: Schedule calls to be made.

[0722] Speech-to-text systems: convert speech to text and text to speech.

[0723] Sentiment analysis engine: Analyzes voice data during calls to detect emotions.

[0724] Generative AI model: Generates a response based on the prompt.

[0725] System Operation

[0726] Creating a target list

[0727] The server collects information on elderly people living alone, people with physical disabilities, and people who have lost their partners, and registers it in a database, including the person's name, age, address, phone number, lifestyle, and preferences.

[0728] Setting up a call schedule

[0729] The server schedules calls based on the recipient's lifestyle and preferences. For example, if the recipient is a morning person, the server schedules a call every Monday at 10 a.m.

[0730] Making a call

[0731] The device will call the target person based on the set schedule, and once the call is connected, the device will start the call.

[0732] Emotion analysis

[0733] The device analyzes the tone of voice, speaking style, and content of the conversation in real time during the call. If the person says, "I had a great time today," the emotion analysis engine will detect their joy.

[0734] Response Generation

[0735] The server receives feedback from the sentiment analysis engine and sends a prompt to the generative AI model. For example, it sends a prompt such as, "The subject said that he enjoyed playing with his grandchild. Please generate a response that shares his joy." The generative AI model generates a response based on this prompt.

[0736] Dialogue execution

[0737] The device then communicates the generated response to the subject, for example, "That's great! What did you play?" This keeps the conversation going, reducing the subject's sense of loneliness and providing a sense of security.

[0738] Specific examples

[0739] As a concrete example, the following scenario can be considered.

[0740] Examples:

[0741] Let's say the user is Mr. A, an elderly person living alone. Taking Mr. A's lifestyle into consideration, the server sets a schedule for a call every Monday at 10:00 AM. The device calls Mr. A at 10:00 AM on Monday, and the emotion analysis engine analyzes Mr. A's tone of voice and the content of what he said. When Mr. A says, "My grandchildren came to play with me today and it was fun," the emotion analysis engine detects his joy and sends feedback to the server. The generative AI model generates a response saying, "That's great! What did you play with?"

[0742] Example prompt sentence:

[0743] "The subject said they had fun playing with their grandchild. Generate a response that shares their joy."

[0744] In this way, the system can provide a conversation that is sensitive to the subject's emotions, reduce the subject's sense of loneliness, and provide a sense of security. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0745] Step 1:

[0746] The server creates a list of eligible people. The user enters information about elderly people living alone, people with physical disabilities, and people who have lost their partners into an input form and clicks the submit button. The server saves the entered information (name, age, address, phone number, lifestyle, preferences, etc.) in a database. The input is the target person's information, and the output is a list of eligible people saved in the database.

[0747] Step 2:

[0748] The server sets a call schedule based on the target person's lifestyle and preferences. The server retrieves the target person's information from a database and calculates the optimal time to call. For example, if the target person is a morning person, the server sets 10:00 AM as the optimal time. The input is the target person's lifestyle and preferences, and the output is the set call schedule.

[0749] Step 3:

[0750] The terminal calls the target person based on the set schedule. When the call connects, the terminal greets the target person with a greeting such as "Hello, how are you?" The input is the set call schedule, and the output is the start of a call with the target person.

[0751] Step 4:

[0752] During a call, the device analyzes the target person's tone of voice, speaking style, and content of the conversation in real time. The emotion analysis engine analyzes the target person's voice data and detects their emotions. For example, if the target person says, "I had a great time today," the emotion analysis engine will detect their joy. The input is the target person's voice data, and the output is the detected emotional information.

[0753] Step 5:

[0754] The server receives feedback from the sentiment analysis engine and sends a prompt to the generative AI model. For example, the prompt might read, "The subject said that he enjoyed playing with his grandchild. Please generate a response that shares his joy." The generative AI model generates a response based on this prompt. The input is the feedback from the sentiment analysis engine and the prompt, and the output is the generated response.

[0755] Step 6:

[0756] The device then communicates the generated response to the subject. For example, it might say, "That's great! What did you play?" This allows for a continuation of the dialogue, reducing the subject's sense of loneliness and providing a sense of security. The input is the generated response, and the output is a continuous dialogue with the subject.

[0757] (Application example 3)

[0758] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0759] Elderly people living alone, people with physical disabilities, and those who have lost their partners are prone to feelings of loneliness and anxiety, and require psychological support in their daily lives. However, there are currently no systems in place that regularly check on the safety of these people and provide dialogue that responds to their emotions. Notification functions are also needed to respond quickly to emergencies. Therefore, a system that can solve these issues is needed.

[0760] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes: means for calling a specific target person to check on their safety and alleviate loneliness through conversation, and also for managing their health through tone of voice; means for setting a schedule for regular calls based on the target person's lifestyle and preferences; means including an emotion engine that analyzes emotions from the target person's tone of voice and content of conversation; means for a generation AI to generate conversations based on the results of the emotion analysis; and means for sending a notification to a pre-registered emergency contact in the event of an emergency. This reduces the target person's sense of loneliness, provides a sense of security, and enables a rapid response in the event of an emergency.

[0761] "Specific target groups" refer to individuals who meet certain conditions, such as elderly people living alone, people with physical disabilities, and those who have lost their partners.

[0762] "Making phone calls to check on safety" refers to the act of calling the target person periodically to check on their health and safety.

[0763] "Relieving loneliness through conversation" refers to reducing the sense of loneliness felt by a subject by having a conversation with them.

[0764] "Health management through voice tone" refers to analyzing the subject's tone of voice and speaking style to understand their health and emotional state.

[0765] "Conversion system" refers to a system that mimics the human voice and converts speech into text.

[0766] "Generative AI" refers to artificial intelligence that generates appropriate answers based on input text data.

[0767] An "emotion engine" refers to a system that analyzes emotions from the tone of a person's voice and the content of their speech.

[0768] "Setting a schedule" means deciding on regular times to call based on the person's lifestyle and preferences.

[0769] "Sending a notification to emergency contacts" means that if the target person is in an emergency, a notification will be sent to the contacts registered in advance.

[0770] A system for carrying out the present invention periodically calls a specific target person to check on their safety and provide a dialogue according to their emotions. A specific embodiment of this system will be described below.

[0771] System configuration

[0772] The system consists of the following main components:

[0773] 1. Server: Manages the entire system and processes data.

[0774] 2. Smartphone: A device used by the subject to receive and make phone calls.

[0775] 3. Emotion engine: Analyzes emotions from the subject's tone of voice and content of speech.

[0776] 4. Generative AI: Generates appropriate conversations based on the results of sentiment analysis.

[0777] 5. Emergency notification system: In the event of an emergency, notifications will be sent to pre-registered emergency contacts.

[0778] Program processing

[0779] The server does the following:

[0780] 1. Scheduling regular phone calls: Schedule regular phone calls based on the recipient's lifestyle and preferences. Use Python's Schedule library for this.

[0781] 2. Make a phone call: Use the Twilio API to make a call to the target person.

[0782] 3. Emotion analysis: Analyze the subject's tone of voice and content of speech and use an emotion engine to identify emotions.

[0783] 4. Conversation generation: Based on the results of sentiment analysis, the generative AI generates appropriate conversation.

[0784] 5. Emergency Notification: If the target person experiences an emergency, a notification will be sent to the pre-registered emergency contact.

[0785] Hardware and software used

[0786] Hardware: Smartphone

[0787] software:

[0788] Twilio API: API for making phone calls

[0789] Emotion Analysis Module: A module for analyzing the emotions of the subject

[0790] AI Conversation Module: A module that generates conversations based on the results of sentiment analysis

[0791] Schedule Library: A Python library for scheduling periodic tasks

[0792] Specific examples

[0793] For example, if Tanaka says, "I had fun today," the emotion analysis module will detect the joy, and the generative AI will generate a conversation such as, "That's great! What happened?"

[0794] Prompt Sentence Examples

[0795] If Tanaka says, "I had fun today," the emotion analysis module will detect that joy, and the generative AI will generate a conversation like, "That's great! What happened?"

[0796] In this way, it is possible to provide a dialogue that responds to the subject's emotions, reduce feelings of loneliness, provide a sense of security, and respond quickly in the event of an emergency.

[0797] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0798] Step 1:

[0799] The server schedules regular phone calls based on the target person's lifestyle and preferences. As input, it receives data on the target person's lifestyle and preferences and uses the Python Schedule library to set the schedule. As output, it determines the specific date and time to make the call.

[0800] Step 2:

[0801] The server uses the Twilio API to call the target person according to the set schedule. As input, it receives the scheduled date and time and the target person's phone number, calls the Twilio API to make the call, and as output, the phone call is connected to the target person.

[0802] Step 3:

[0803] The device (smartphone) records the subject's voice and sends it to the server. As input, it receives the subject's voice data and sends it to the server. As output, the voice data is sent to the server.

[0804] Step 4:

[0805] The server inputs the received voice data into the emotion engine and analyzes the emotion of the target person. As input, it receives the voice data and analyzes the emotion using the emotion engine. As output, it obtains the analyzed emotion data.

[0806] Step 5:

[0807] The server inputs the results of the emotion analysis into the generative AI to generate an appropriate conversation. As input, it receives emotion data and generates a conversation using the generative AI. As output, it obtains the generated conversation text.

[0808] Step 6:

[0809] The server converts the generated conversation text into speech and transmits it to the terminal. As input, the server receives the generated conversation text and generates speech data using a speech conversion system. As output, the speech data is transmitted to the terminal.

[0810] Step 7:

[0811] The device plays the received voice data and engages in a dialogue with the subject. As input, the device receives voice data and plays it back through a speaker. As output, a dialogue with the subject takes place.

[0812] Step 8:

[0813] The server sends a notification to pre-registered emergency contacts when a target person is in an emergency. As input, it receives emergency detection data and sends a notification to the emergency contacts. As output, an emergency notification is sent.

[0814] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0815] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0816] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0817] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0818] [Second embodiment]

[0819] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0820] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0821] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0822] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0823] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0824] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0825] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0826] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0827] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0828] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0829] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0830] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0831] "Example 1"

[0832] One aspect of the present invention is a system that calls specific individuals to check on their safety, alleviating loneliness through conversation and enabling health management through voice tone. This system utilizes a conversion system that mimics the human voice, converts speech to text, and then uses a generation AI to create a response, which is then converted back into speech, resulting in a conversation over the phone. Specifically, speech recognition technology is used to convert the target's speech into text, which is then input into the generation AI. The generation AI analyzes the text and generates an appropriate response. The generated response is converted from text to speech and communicated to the target via phone.

[0833] "Example 2"

[0834] In another embodiment of the present invention, the voices of voice actors (male and female) or relatives recorded before they died are used. Specifically, voice data of the voice actor or voice data of a relative recorded before they died is input into a voice conversion system, and a voice that imitates it is generated. In this embodiment, the target person can enjoy conversations with familiar voices, resulting in higher satisfaction.

[0835] "Example 3"

[0836] A further embodiment of the present invention targets elderly people living alone, people with physical disabilities, and those who have lost their partners. Specifically, a list of these people is created and a regular call schedule is set up. The call times can be adjusted to suit the target person's lifestyle and preferences. This embodiment can reduce the amount of time the target person feels lonely and provide a sense of security.

[0837] The processing flow of each embodiment will be described below.

[0838] "Example 1"

[0839] Step 1: The system places calls to specific individuals, scheduled to fit their lifestyles and preferences.

[0840] Step 2: When the subject answers the phone, the system uses voice recognition technology to convert the subject's speech into text.

[0841] Step 3: The text is fed into the generative AI, which analyzes the text and generates an appropriate answer.

[0842] Step 4: The generated responses are converted from text to speech and communicated to the target person over the phone.

[0843] "Example 2"

[0844] Step 1: Input voice data from a voice actor or a recording of a loved one before they pass away into the voice conversion system.

[0845] Step 2: The voice conversion system generates a voice that mimics the input voice data.

[0846] Step 3: The generated audio is transmitted to the subject via telephone and used in a conversation with the subject.

[0847] "Example 3"

[0848] Step 1: Create a list of target groups, including elderly people living alone, people with physical disabilities, and those who have lost their partners.

[0849] Step 2: Schedule regular phone calls for each person. Adjust the call times to suit each person's lifestyle and preferences.

[0850] Step 3: According to the set schedule, the system calls the target person and conducts a conversation.

[0851] Example 1

[0852] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0853] In modern society, elderly people living alone, people with physical disabilities, and those who have lost their partners often feel lonely, which can have a negative impact on their mental health. Furthermore, it is difficult to quickly detect changes in the physical condition of these individuals and take appropriate action. Conventional safety confirmation systems simply make phone calls and lack the functionality to analyze the tone of the individual's voice and manage their physical condition. Therefore, more advanced safety confirmation and health management systems are needed.

[0854] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0855] In this invention, the server includes a speech recognition means for converting speech to text, a means for inputting the converted text into a generative AI model to generate an appropriate response, and a speech synthesis means for converting the generated response from text to speech. This makes it possible to call a specific target person to check on their safety and ease loneliness through conversation, as well as to analyze the target person's tone of voice to detect changes in their physical condition and respond promptly.

[0856] "Specific target groups" refer to individuals who meet certain conditions, such as elderly people living alone, people with physical disabilities, and those who have lost their partners.

[0857] "Making a phone call to check on safety" refers to the act of calling a person and checking their response to check their safety and health.

[0858] "Relieving loneliness through conversation" refers to reducing the sense of loneliness felt by the subject by communicating with them verbally.

[0859] "Health management through voice tone" refers to analyzing the tone and quality of a person's voice and detecting abnormalities in their health based on these changes.

[0860] "Speech recognition means for converting speech into text" refers to technology that collects the speech of a person as digital voice data and converts it into text information.

[0861] A "generative AI model" refers to an artificial intelligence model that analyzes input text data and generates an appropriate response.

[0862] "Text-to-speech synthesis means" refers to a technology that converts generated text data into speech data.

[0863] "Communication means for transmitting voice to a target person via telephone" refers to technology for transmitting generated voice data to a target person via a telephone line.

[0864] "A voice analysis means that analyzes the tone of a subject's voice and detects changes in their physical condition" refers to a technology that analyzes the characteristics of a subject's voice and detects abnormalities in their physical condition based on those changes.

[0865] This invention is a system that can call specific individuals to check on their safety, alleviate loneliness through conversation, and also manage their health by monitoring their tone of voice. This system is realized by combining voice recognition technology, generative AI models, voice synthesis technology, communication technology, and voice analysis technology.

[0866] The server first uses a communication method to call the target person. Specifically, it uses a cloud communication platform to make the call. For example, it uses the Twilio API to call the target person's phone number.

[0867] The device collects the voice after the target person answers the phone, and converts the collected voice into text using speech recognition. Specifically, it uses the Google Cloud Speech-to-Text API to convert voice to text.

[0868] The server inputs the converted text into a generative AI model, which uses OpenAI's GPT-4, and the generative AI model analyzes the input text and generates an appropriate answer.

[0869] The generated answer is then converted from text to speech again using a speech synthesis tool, specifically Amazon Polly, to convert text to speech.

[0870] The server transmits the generated voice to the subject via telephone. The communication means is used to transmit the generated voice data to the subject via a telephone line.

[0871] Furthermore, the device analyzes the subject's tone of voice to detect changes in their physical condition. Using a voice analysis tool, the device analyzes the tone and quality of the subject's voice and detects abnormalities in their physical condition based on these changes. The analysis results are then notified to a medical institution or family member as necessary.

[0872] As a concrete example, the following scenario can be considered.

[0873] When a user wants to check on the safety of an elderly person, the system operates as follows:

[0874] 1. The server uses the Twilio API to call the senior citizen.

[0875] 2. The device collects the elderly person's voice saying "Hello, how are you?" and converts it into text using the Google Cloud Speech-to-Text API.

[0876] 3. The server inputs the converted text "Hello, how are you?" into GPT-4 and generates an appropriate answer.

[0877] 4. The server generates the answer, "Hello, how are you? How was your day?" and converts it into speech using Amazon Polly.

[0878] 5. The server transmits the generated audio to the elderly person via the Twilio API.

[0879] 6. The device analyzes the elderly person's voice tone and notifies medical institutions or family members if any abnormalities are detected.

[0880] An example prompt is, "If an older adult says, 'Hello, how are you?' please generate an appropriate response."

[0881] This system allows users to check the safety of specific individuals, alleviating feelings of loneliness and managing their health.

[0882] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0883] Step 1:

[0884] The server places a call to the target person.

[0885] Input: Subject's phone number

[0886] Specific operation: The server calls the API of the cloud communication platform (e.g., Twilio API) and makes a call to the target person's phone number.

[0887] Output: The call to the target person is successful and the call starts.

[0888] Step 2:

[0889] The terminal collects the subject's voice and converts it into text using speech recognition means.

[0890] Input: Subject's voice

[0891] What it does: The device collects the target person's voice saying "Hello, how are you?" and converts it into text using speech recognition software (e.g., Google Cloud Speech-to-Text API).

[0892] Output: The converted text "Hello, how are you?"

[0893] Step 3:

[0894] The server inputs the converted text into a generative AI model to generate an appropriate answer.

[0895] Input: Transformed text "Hello, how are you?"

[0896] Specific operation: The server inputs the converted text into a generative AI model (e.g., GPT-4), sets a prompt sentence, and generates an appropriate answer.

[0897] Output: Generated answer "Hello, I'm fine. How was your day?"

[0898] Step 4:

[0899] The server converts the generated answers from text to speech.

[0900] Input: Generated answer "Hi, I'm fine. How was your day?"

[0901] What happens: The server uses speech synthesis software (e.g., Amazon Polly) to convert the generated text into speech.

[0902] Output: Generated audio data

[0903] Step 5:

[0904] The server transmits the generated voice to the target person via telephone.

[0905] Input: Generated audio data

[0906] Specific operation: The server uses a communication means (e.g., Twilio API) to transmit the generated voice data to the target person via a telephone line.

[0907] Output: Audio delivered to subject

[0908] Step 6:

[0909] The device analyzes the subject's tone of voice and detects changes in their physical condition.

[0910] Input: Subject's voice data

[0911] Specific operation: The device uses voice analysis software to analyze the tone and quality of the subject's voice, and detects abnormalities in their physical condition based on changes.

[0912] Output: Notification data when abnormalities in health are detected (notification to medical institutions and family members)

[0913] In this way, the system can check the safety of specific individuals, alleviating loneliness and also managing their health.

[0914] (Application example 1)

[0915] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0916] In modern society, the number of elderly people living alone, physically disabled people, and people who have lost their partners is increasing, and confirming the safety of these people and reducing their feelings of loneliness has become an important issue. Managing the health of these people is also important, and a quick response is required, especially in emergencies. However, the current system lacks the means to effectively solve these issues.

[0917] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0918] In this invention, the server includes: a means for calling specific individuals to check on their safety, alleviating loneliness through conversation and enabling health management based on tone of voice; a means for converting speech to text using a conversion system that mimics the human voice, creating a response using a generation AI, converting the response back to speech, and conducting a conversation over the phone; a means for converting the target's speech to text using speech recognition technology; a means for analyzing the text using a generation AI and generating an appropriate response; a means for converting the generated response from text to speech and communicating it to the target over the phone; and a means for inferring the target's health from tone of voice and content of conversation and notifying family or medical institutions as necessary. This makes it possible to check the safety of elderly people living alone, people with disabilities, and those who have lost their partners, reduce loneliness, and manage their health.

[0919] "Specific target groups" refer to people whose safety needs to be confirmed and whose feelings of loneliness need to be alleviated, such as elderly people living alone, people with physical disabilities, and those who have lost their partners.

[0920] "Safety check" refers to the act of checking the current situation and health condition of a person.

[0921] "Conversational" refers to a format in which a conversation is held with the subject over the telephone.

[0922] "Health management through tone of voice" refers to inferring a person's health condition from the tone and quality of their voice and taking the necessary measures.

[0923] "Conversion system" refers to technology that mimics the human voice and converts speech into text.

[0924] "Generative AI" refers to artificial intelligence that analyzes input text and generates appropriate answers.

[0925] "Speech recognition technology" refers to technology that converts speech into text.

[0926] "Text analysis" refers to the process of understanding input text and generating appropriate responses.

[0927] "Speech synthesis" refers to the technology of converting text into speech.

[0928] "Notification" refers to the act of conveying information to family members or medical institutions based on the subject's physical condition and situation.

[0929] The system for implementing this invention makes phone calls to specific individuals to check on their safety, alleviating their feelings of loneliness through conversation and managing their physical condition through voice tone. This system is composed of a combination of voice recognition technology, generation AI, and voice synthesis technology.

[0930] First, the server periodically calls a specific target person. When the call connects, it captures the target person's voice and converts it into text using speech recognition technology. This speech recognition technology uses the speech_recognition library and Google's speech recognition API.

[0931] The server then inputs the converted text into a generation AI, which uses OpenAI's API to analyze the input text and generate an appropriate response. The generated response is then converted from text to speech again, using the gTTS library for speech synthesis.

[0932] The generated voice is transmitted to the target person via telephone. The server then infers the target person's physical condition from the tone of voice and content of the conversation, and notifies family members or medical institutions as necessary. This notification function enables rapid response in emergencies.

[0933] For example, consider the following conversation:

[0934] Elderly person: "I'm feeling a bit tired today."

[0935] Generator: "That's worrying. How are you feeling these days?"

[0936] Example prompt sentence:

[0937] Conversation with an elderly person: I'm feeling a little tired today.

[0938] Generate an appropriate response.

[0939] This system will enable elderly people living alone, people with disabilities, and those who have lost their partners to effectively check on their safety, reduce feelings of loneliness, and manage their health.

[0940] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0941] Step 1:

[0942] The server periodically calls a specific target person. When the call is connected, the target person's voice is acquired. The input is the target person's voice, and the output is voice data. This voice data is collected through a microphone.

[0943] Step 2:

[0944] The server converts the acquired voice data into text using speech recognition technology. Specifically, it uses the speech_recognition library and Google's speech recognition API. The input is voice data, and the output is text data. This conversion allows the subject's speech to be obtained in text format.

[0945] Step 3:

[0946] The server inputs the converted text data into the generation AI. The generation AI uses OpenAI's API to analyze the input text and generate an appropriate answer. The input is text data, and the output is the generated answer text data. This analysis results in an appropriate response to the subject's utterance.

[0947] Step 4:

[0948] The server converts the text data of the generated answer into speech using speech synthesis technology. Specifically, it uses the gTTS library. The input is the text data of the generated answer, and the output is speech data. This conversion results in the generated response in speech format.

[0949] Step 5:

[0950] The server transmits the generated voice data to the subject via telephone. The input is voice data, and the output is the voice transmitted to the subject. This transmission allows the subject to hear the response from the generating AI.

[0951] Step 6:

[0952] The server infers the subject's physical condition from the tone of voice and content of the conversation. If necessary, it notifies family members or medical institutions. The input is the subject's tone of voice and content of the conversation, and the output is notification information. This inference and notification enables a rapid response in the event of an emergency.

[0953] Example 2

[0954] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0955] In modern society, elderly people living alone, people with physical disabilities, and those who have lost their partners often feel lonely, which can lead to poor mental health. Furthermore, there is a lack of efficient ways to check on the safety and health of these people. Furthermore, there is a need for a system that allows these people to enjoy conversations with a friendly voice.

[0956] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0957] In this invention, the server is a system that communicates with specific individuals to check their safety, alleviate loneliness through conversation, and also manage their health through tone of voice, and includes: a means for utilizing a voice-imitating conversion system to convert voice to text, create a response using a generative AI model, convert the response back to voice, and communicate in a conversational format; a means for uploading voice data from a terminal to the server; a means for analyzing the uploaded voice data and extracting features of the voice data; a means for inputting the features of the analyzed voice data into the generative AI model; a means for the generative AI model to generate new voice based on the features of the input voice data; and a means for providing the generated voice data to a user. This allows the individual to enjoy a conversation in a friendly voice, reducing loneliness and enabling efficient safety confirmation and health management.

[0958] "Specific target groups" refer to people who are prone to feeling lonely, such as elderly people living alone, people with physical disabilities, and those who have lost their partners, and who need to be checked on for safety and health management.

[0959] "Communications" refers to the means of sending and receiving voice and data via telephone and the Internet.

[0960] "Safety check" refers to the act of checking the safety and health of a specific person.

[0961] "Conversational" refers to a form of two-way communication using voice.

[0962] "Vocal timbre" refers to characteristics such as the quality, tone, and rhythm of a voice.

[0963] "Health management" refers to the act of monitoring the health status of a specific subject and responding as necessary.

[0964] A "voice imitation conversion system" refers to a system that generates new voices based on input voice data.

[0965] A "generative AI model" refers to an artificial intelligence model that generates new data based on input data.

[0966] "Audio data" refers to data that records audio in digital format.

[0967] "Terminal" refers to a device, such as a PC or smartphone, used to upload audio data to a server.

[0968] "Server" refers to the computer system that analyzes voice data and runs generative AI models.

[0969] "Analysis" refers to the process of extracting features from audio data.

[0970] "Features" refers to characteristics of the audio data such as pitch, rhythm, and timbre.

[0971] A "prompt sentence" refers to an instruction sentence to be input into a generative AI model.

[0972] "New voice" refers to a voice generated by a generative AI model that mimics the original voice data.

[0973] "Providing" refers to the act of transmitting the generated voice data to the user.

[0974] This invention is a system that can communicate with specific individuals to check their safety, alleviate loneliness through conversation, and manage their physical condition through tone of voice. This system utilizes a voice mimicking conversion system to convert speech into text, generate responses using a generative AI model, convert the responses into speech, and communicate in a conversational format.

[0975] Generating a Program

[0976] The server generates a program for the voice conversion system, which includes an algorithm for receiving voice data as input and generating a new voice that imitates it.

[0977] Program processing explanation

[0978] The server causes the speech conversion system to perform the following processes.

[0979] 1. Input audio data:

[0980] Users upload audio data of voice actors or loved ones recorded before they died to the server.

[0981] As specific hardware, users use computers and smartphones.

[0982] As for software, a web browser or dedicated application is used to upload audio data.

[0983] 2. Analysis of audio data:

[0984] The server analyzes the uploaded audio data.

[0985] Analysis is performed using voice recognition software (e.g., Google Cloud Speech-to-Text API).

[0986] Extract features of audio data (pitch, rhythm, timbre, etc.).

[0987] 3. Input to the generative AI model:

[0988] The server inputs the extracted features of the voice data into a generative AI model.

[0989] Generative AI models used include OpenAI's GPT-3 and Google's WaveNet.

[0990] 4. Imitation voice generation:

[0991] The generative AI model generates new speech based on the characteristics of the input speech data.

[0992] This new voice mimics the original voice data.

[0993] 5. Output of generated speech:

[0994] The server provides the generated voice data to the user.

[0995] The user can download the generated audio or play it via streaming.

[0996] Adding specific examples

[0997] As a concrete example, the following prompt sentence is input into the generative AI model:

[0998] Example prompt sentence:

[0999] "This voice data is the voice of my deceased grandfather. Please generate a new voice based on this voice."

[1000] Examples:

[1001] The user uploads audio data of their grandfather recorded before his death to the server. The server analyzes the audio data and inputs it into the generative AI model. The generative AI model generates a new voice that imitates the grandfather's voice, and the server provides that voice to the user. The user can listen to the generated voice and enjoy a conversation with their grandfather.

[1002] In this way, a specific target person can enjoy a conversation in a friendly voice, reducing feelings of loneliness and enabling efficient safety confirmation and health management.

[1003] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1004] Step 1:

[1005] Inputting audio data

[1006] Users upload audio data of voice actors or loved ones recorded before they passed away from their device (PC or smartphone) to the server.

[1007] Specific operation: The user uses a web browser or dedicated application to select an audio file and click the upload button.

[1008] Input: Audio data file (e.g. .wav, .mp3)

[1009] Output: Audio data stored on the server

[1010] Step 2:

[1011] Analysis of audio data

[1012] The server receives the uploaded voice data and analyzes it using voice recognition software (e.g., Google Cloud Speech-to-Text API).

[1013] Specific operation: The server extracts characteristics of the audio data, such as pitch, rhythm, and timbre, and stores them in a database.

[1014] Input: Audio data stored on the server

[1015] Output: Extracted audio features (e.g., pitch, rhythm, timbre)

[1016] Step 3:

[1017] Input to generative AI models

[1018] The server inputs the features of the analyzed voice data into a generative AI model (e.g., OpenAI's GPT-3 or Google's WaveNet).

[1019] Specific operation: The server sends the features of the voice data to the generative AI model as a prompt sentence.

[1020] Input: Extracted audio features

[1021] Output: The prompt input to the generative AI model

[1022] Step 4:

[1023] Imitation voice generation

[1024] The generative AI model generates new speech based on the characteristics of the input speech data.

[1025] How it works: The generative AI model analyzes the characteristics of the voice data and generates imitation voices based on that.

[1026] Input: The prompt given to the generative AI model

[1027] Output: Generated imitation voice data

[1028] Step 5:

[1029] Generated speech output

[1030] The server provides the generated voice data to the user.

[1031] Specific operation: The server sends the generated audio data to the user's device, where the user can download it or play it via streaming.

[1032] Input: Generated imitation voice data

[1033] Output: The audio data provided to the user (e.g., download link, streaming URL)

[1034] (Application example 2)

[1035] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1036] In modern society, elderly people living alone, people with physical disabilities, and those who have lost their partners are prone to feeling lonely and have difficulty managing their health. Furthermore, food delivery services also face the issue of not being able to provide guidance or confirmation in a friendly voice, which hinders the user experience.

[1037] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for utilizing a conversion system that mimics a human voice, converting speech to text, creating a response using a generation AI, converting the response to speech, and communicating in a conversational format, as well as a means for providing guidance and confirmation in a friendly voice. This helps elderly people living alone, people with physical disabilities, and those who have lost their partners to feel less lonely and allows them to manage their health. Furthermore, in food delivery services, users can receive guidance and confirmation in a friendly voice, improving the user experience.

[1038] "Specific eligible individuals" refer to individuals who meet certain conditions, such as elderly people living alone, people with physical disabilities, and those who have lost their partners.

[1039] "Communications" refers to the means of sending and receiving voice and data via telephone and the Internet.

[1040] "Safety check" refers to the act of checking the health and safety of a specific person.

[1041] "Vocal timbre" refers to the characteristics and tone of a voice, which can be used to determine the speaker's emotions and physical condition.

[1042] "Health management" refers to monitoring the health status of a specific subject and taking appropriate measures as necessary.

[1043] "Conversion system" refers to technology that converts speech to text and text to speech.

[1044] "Generative AI" refers to technology that uses artificial intelligence to generate appropriate answers from text data.

[1045] A "familiar voice" refers to a voice that is familiar to or well-liked by the target audience.

[1046] "Guidance and confirmation" refers to actions to provide information to the user or confirm their actions.

[1047] An "audio professional" refers to someone whose job involves working exclusively with audio, such as a voice actor or narrator.

[1048] "Immediate family" refers to family members related by blood.

[1049] The following system configuration will be described as an embodiment of the present invention.

[1050] System Configuration

[1051] This system consists of three main components: a server, a terminal, and a user. The server includes a speech conversion system and a generative AI, and the terminal is a communication device such as a smartphone or tablet. The user is a specific target group, such as elderly people living alone, people with physical disabilities, or those who have lost their partner.

[1052] Program processing

[1053] The server uses a speech-to-text system to convert the user's speech into text and inputs that text into the generation AI. The generation AI generates an appropriate answer based on the input text and converts that answer back into speech. The converted speech is used to provide guidance and confirmation to the user in a friendly voice.

[1054] Hardware and software used

[1055] Hardware: Servers, smartphones, tablets

[1056] Software: Python, pyttsx3 (speech synthesis library), soundfile (audio file reading library), scipy (scientific calculation library)

[1057] Data processing and calculation

[1058] 1. Reading audio data: The server reads the user's audio data using the soundfile library.

[1059] 2. Audio data conversion: The loaded audio data is converted into text using a speech conversion system.

[1060] 3. Answer generation using generative AI: Text data is input into a generative AI model to generate appropriate answers.

[1061] 4. Re-conversion to speech: The generated text response is converted to speech using the pyttsx3 library.

[1062] 5. Play audio: The converted audio is sent to the user's device, providing guidance and confirmation in a friendly voice.

[1063] Specific examples

[1064] For example, if a user is ordering a pizza from a food delivery app, hearing their deceased mother's voice ask, "Are you sure you want to confirm your order?" before confirming the order can create a sense of familiarity and security for the user.

[1065] Prompt Sentence Examples

[1066] "In your food delivery app, implement a friendly voice confirmation feature before the user confirms their order. Use pre-recorded voice data and translate it into the target voice for playback."

[1067] In this way, the embodiments for carrying out the invention have been specifically shown.

[1068] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1069] Step 1:

[1070] The server receives the user's voice data. The input is the user's voice data, and the output is the voice data stored in the server. Specifically, the server reads the voice data sent from the smartphone or tablet using the soundfile library.

[1071] Step 2:

[1072] The server converts the voice data into text using a speech conversion system. The input is voice data and the output is text data. Specifically, the server analyzes the voice data using speech recognition technology and generates corresponding text data.

[1073] Step 3:

[1074] The server inputs text data into the generative AI model and generates an appropriate answer. The input is text data, and the output is the generated answer text. Specifically, the server inputs a prompt sentence into the generative AI model and generates an appropriate answer to the user's question or request.

[1075] Step 4:

[1076] The server converts the generated answer text into speech. The input is the generated answer text, and the output is speech data. Specifically, the server converts text data into speech data using the pyttsx3 library.

[1077] Step 5:

[1078] The server sends the converted voice data to the user's device. The input is the voice data, and the output is the voice data sent to the user's device. Specifically, the server uses a communication protocol to send the voice data to the user's smartphone or tablet.

[1079] Step 6:

[1080] The user's device plays the received voice data. The input is the voice data sent from the server, and the output is the voice the user hears. Specifically, the user's device uses its built-in voice playback function to play the voice data and provides guidance and confirmation in a friendly voice.

[1081] Example 3

[1082] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1083] There is a need to reduce the amount of time that people who are prone to social isolation, such as elderly people living alone, people with physical disabilities, and those who have lost their partners, feel lonely and provide them with a sense of security. It is also important to regularly check on the safety of these people and manage their health. However, there is not yet a sufficient system in place to implement this efficiently and effectively.

[1084] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1085] In this invention, the server includes means for calling specific subjects to check on their safety and alleviate loneliness through conversation, and also for managing their physical condition through tone of voice, means for collecting subject information and storing it in a database, means for setting a call schedule based on the subject's lifestyle and preferences, means for automatically making calls based on the set schedule, and means for adjusting the schedule and updating the database in response to the subject's feedback and changes in lifestyle. This makes it possible to reduce the amount of time the subject feels lonely and provide a sense of security.

[1086] "Specific target groups" refer to people who are prone to social isolation, such as elderly people living alone, people with disabilities, and those who have lost their partners.

[1087] "Safety check" refers to phone calls or conversations to check on the health and safety of the person in question.

[1088] "Health management through vocal tone" refers to analyzing the tone and quality of a person's voice to understand their physical and mental state.

[1089] A "conversion system" refers to a system that mimics a human voice, converts the speech into text, creates an answer using generative AI, and then converts it back into speech.

[1090] "Generative AI" refers to a system that uses artificial intelligence technology to generate appropriate answers based on input text data.

[1091] "Database" refers to an information system for systematically storing and managing information on subjects.

[1092] A "call schedule" refers to a plan for making calls at the optimal time based on the target person's lifestyle and preferences.

[1093] "Feedback" refers to opinions and reactions obtained from subjects, and is information used to improve and adjust the system.

[1094] "Means for making automatic phone calls" refers to the ability of the system to automatically make phone calls based on a set schedule.

[1095] "Schedule adjustment" refers to changing the time and frequency of calls based on the target person's lifestyle and feedback.

[1096] MODE FOR CARRYING OUT THE INVENTION

[1097] This invention is a system that provides a sense of security and alleviates loneliness by making regular phone calls to people who are prone to social isolation, such as elderly people living alone, people with physical disabilities, and those who have lost their partners. A specific embodiment of this system will be described below.

[1098] 1. Program Generation

[1099] The server generates a program that collects information about the target person and stores it in a database. This program sets a call schedule based on the target person's lifestyle and preferences, and automatically makes calls based on the set schedule.

[1100] 2. Program Processing

[1101] The server provides a web form to collect information about the subjects, which users use to enter information such as the subjects' names, phone numbers, lifestyles, preferences, etc. The collected information is stored in a MySQL database.

[1102] The server then uses a Python script to schedule calls for each individual. The script calculates the best time to call based on the individual's lifestyle and preferences. For example, if a person prefers to be called in the morning, the server will schedule the script to call at that time.

[1103] The server uses the Twilio API to automatically make calls based on a set schedule. The time of the call is adjusted to suit the recipient's lifestyle and preferences. For example, if recipient A wants to be called at 10:00 a.m. every day, the server will make the call at that time.

[1104] The server also receives feedback from the subject and adjusts the schedule as needed: for example, if the subject decides they prefer to be called in the afternoon, the information in the database is updated and the schedule is rescheduled.

[1105] 3. Examples and prompts

[1106] As a concrete example, consider the following scenario.

[1107] Subject A (an elderly person living alone) requests a phone call every day at 10:00 a.m.

[1108] Subject B (a physically disabled person) requests phone calls three times a week at 2:00 p.m.

[1109] The server stores this information in a database and uses a Python script to schedule calls, which are then automatically placed at the scheduled time using the Twilio API.

[1110] Example prompts to input to a generative AI model:

[1111] "Please schedule a call to elderly person A who lives alone every day at 10:00 AM."

[1112] "Schedule a call with Person B, who is disabled, three times a week at 2:00 PM."

[1113] This system can reduce the amount of time that a subject feels lonely and provide a sense of security. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1114] Step 1:

[1115] The user uses a web form to enter information about the target person, including their name, phone number, lifestyle, preferences, etc. This information is then sent to the server.

[1116] Input: Subject's name, phone number, lifestyle, preferences

[1117] Output: Information of the target person sent to the server

[1118] Step 2:

[1119] The server stores the received information in a MySQL database by inserting the information into the appropriate table.

[1120] Input: Information about the target person sent by the user

[1121] Output: Subject information stored in a database

[1122] Step 3:

[1123] The server schedules calls for each individual using a Python script that calculates the best time to call based on the individual's lifestyle and preferences.

[1124] Input: Subject information stored in the database

[1125] Output: The configured call schedule

[1126] Step 4:

[1127] The server uses the Twilio API to automatically place calls based on a set schedule, with call times tailored to the recipient's lifestyle and preferences.

[1128] Input: Configured call schedule

[1129] Output: Calls made to the target person

[1130] Step 5:

[1131] The server receives feedback from the subject and adjusts the schedule as needed: for example, if the subject decides they prefer a call in the afternoon, it updates the database and reschedules the call.

[1132] Input: Feedback from subjects

[1133] Output: Adjusted call schedule and updated database

[1134] In this way, the system can reduce the amount of time a subject feels lonely and provide a sense of security.

[1135] (Application example 3)

[1136] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1137] Elderly people living alone, people with physical disabilities, and those who have lost their partners often feel lonely and anxious. Furthermore, these individuals may not receive appropriate support in emergencies. Conventional safety confirmation systems struggle to respond flexibly to individuals' lifestyles and preferences, and are insufficient to provide a sense of security. Furthermore, they lack the functionality to analyze individuals' responses and notify emergency contacts as necessary, potentially delaying emergency response.

[1138] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1139] In this invention, the server includes means for calling specific individuals to check on their safety and alleviate loneliness through conversation, and also for managing their physical condition through tone of voice, means for creating a list of individuals and setting a call schedule based on their lifestyle and preferences, means for automatically making calls based on the set schedule, means for analyzing the individual's responses and notifying an emergency contact as necessary, and means for providing the individual with a sense of security through regular calls. This reduces the individual's sense of loneliness, provides a sense of security, and enables a rapid response in emergencies.

[1140] "Specific target groups" refer to individuals who meet certain conditions, such as elderly people living alone, people with physical disabilities, and those who have lost their partners.

[1141] "Safety check" refers to the act of checking whether a person is safe.

[1142] "Health management through voice tone" refers to a method of analyzing the tone and quality of a person's voice to detect changes or abnormalities in their physical condition.

[1143] "Conversion system" refers to a system that mimics the human voice and converts speech into text.

[1144] "Generative AI" refers to technology that uses artificial intelligence to generate appropriate answers from text.

[1145] "Target list" refers to a list that compiles information about specific targets.

[1146] "Scheduling" refers to the act of setting a time to call based on the target person's lifestyle and preferences.

[1147] "Means for making automatic phone calls" refers to the ability to make automatic phone calls based on a set schedule.

[1148] "Means for analyzing responses" refers to the function of analyzing the subject's telephone responses and notifying emergency contacts as necessary.

[1149] "Means of providing a sense of security" refers to the function of providing a sense of security to the target person through regular phone calls.

[1150] The following system configuration is used as an embodiment of the present invention.

[1151] System Configuration

[1152] 1. Hardware:

[1153] Server: Processes data and runs generative AI models.

[1154] Device: A smartphone or robot that makes phone calls to the subject.

[1155] Communications infrastructure: Internet connection and telephone lines.

[1156] 2. Software:

[1157] Generative AI model: Generates appropriate answers based on the subject's responses.

[1158] Speech-to-text system: imitates the human voice and converts speech to text.

[1159] Scheduling system: Schedule calls to suit your target audience's lifestyle and preferences.

[1160] Response analysis system: Analyzes the subject's response and notifies emergency contacts as necessary.

[1161] Program processing explanation

[1162] The server creates a list of contacts and sets a call schedule based on their lifestyles and preferences. The device automatically makes calls based on the set schedule. During the call, the voice conversion system converts the contact's voice into text, and the generative AI model generates an appropriate response. The generated text is then converted into voice and sent back to the contact. The response analysis system analyzes the contact's response and notifies emergency contacts if necessary.

[1163] Specific examples

[1164] For example, if Tanaka, an elderly person living alone, is registered on the target list, the server sets a schedule to call Tanaka every day at 10:00 to suit his lifestyle. At 10:00, the device automatically calls Tanaka. When Tanaka answers the phone, the speech conversion system converts Tanaka's voice into text, and the generative AI model generates an appropriate response such as "How are you spending your day?" This text is then converted into speech and responded to Tanaka. If an abnormality is detected in Tanaka's response, the response analysis system notifies the emergency contact.

[1165] Prompt Sentence Examples

[1166] "Develop a system for making regular phone calls to elderly people living alone, people with disabilities, and those who have lost their partners. Create a list of people to call and schedule calls based on their lifestyles and preferences. Calls should provide reassurance and include the ability to notify emergency contacts if necessary."

[1167] In this way, it is possible to provide a specific embodiment of the invention.

[1168] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1169] Step 1:

[1170] The server creates a target list.

[1171] Input: Subject's personal information (name, phone number, lifestyle, preferences, etc.)

[1172] Data processing: The subject's information is registered in a database and managed in list format.

[1173] Output: List of target users

[1174] Step 2:

[1175] The server schedules calls based on the recipient's lifestyle and preferences.

[1176] Input: List of subjects, information on their lifestyle and preferences

[1177] Data calculation: Calculate the optimal time to call each target person and set a schedule.

[1178] Output: Call Schedule

[1179] Step 3:

[1180] The device will automatically make calls based on a set schedule.

[1181] Input: Call Schedule

[1182] Specific actions: Call the target person according to the schedule.

[1183] Output: Start of phone call

[1184] Step 4:

[1185] The terminal converts the target person's voice into text using a speech conversion system.

[1186] Input: Subject's voice

[1187] Data processing: Using a voice conversion system, convert the voice into text data.

[1188] Output: Text data

[1189] Step 5:

[1190] The server uses a generative AI model to generate appropriate answers from the text data.

[1191] Input: Text data

[1192] Data Computation: Using generative AI models to generate answers based on text data.

[1193] Output: Generated answer text

[1194] Step 6:

[1195] The terminal converts the generated answer text into speech and responds to the subject.

[1196] Input: Generated answer text

[1197] Data processing: Using a speech conversion system, convert the text into audio data.

[1198] Specific operation: Play audio data to the target person.

[1199] Output: Voice response to the target person

[1200] Step 7:

[1201] The server analyzes the subject's response and notifies emergency contacts if necessary.

[1202] Input: Subject's response text data

[1203] Data calculation: Using the response analysis system, the response content is analyzed, and if an abnormality is detected, an emergency contact is notified.

[1204] Specific behavior: Send a notification to emergency contacts.

[1205] Output:Notify emergency contacts

[1206] In this way, each processing step can be concretely explained.

[1207] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1208] "Example 1"

[1209] One embodiment of the present invention is a system that incorporates an emotion engine. This system makes a phone call to a specific target person and engages in a conversation. The conversation is converted from speech to text using a conversion system that mimics the human voice, and a generation AI creates a response. The created response is converted back into speech and transmitted to the target person via phone. At this time, the emotion engine analyzes the target person's emotions from their tone of voice and speaking style, and feeds the results back to the generation AI. Based on this feedback, the generation AI generates a conversation that is adapted to the target person's emotions.

[1210] "Example 2"

[1211] As a specific example, when the target person is sad, the emotion engine detects that sadness and provides feedback to the generation AI. Based on this information, the generation AI generates conversations that comfort the target person. Similarly, when the target person is happy, it generates conversations that share that joy. This allows the system to provide appropriate dialogue that corresponds to the target person's emotions.

[1212] "Example 3"

[1213] Furthermore, the emotion engine can analyze emotions not only from the subject's tone of voice and speaking style, but also from the content of the conversation. For example, if the subject says, "I had fun today," the emotion engine detects this joy and provides feedback to the generation AI. Based on this information, the generation AI generates a conversation that shares the subject's joy. This allows the system to provide a dialogue that more deeply reflects the subject's emotions.

[1214] The processing flow of each embodiment will be described below.

[1215] "Example 1"

[1216] Step 1: The system makes a call to a specific target person.

[1217] Step 2: Once a conversation with the subject begins, the system converts the speech into text using a conversion system that mimics the human voice.

[1218] Step 3: Generative AI analyzes the text and creates an appropriate response.

[1219] Step 4: The created answer is converted back into audio and transmitted to the target person via phone. Step 5: The emotion engine analyzes the target person's emotions from their tone of voice and speaking style, and feeds the results back to the generation AI.

[1220] Step 6: The generative AI uses this feedback to generate conversations that adapt to the subject's emotions.

[1221] "Example 2"

[1222] Step 1: When a subject is sad, the emotion engine detects that sadness.

[1223] Step 2: The detected emotion information is fed back to the generation AI.

[1224] Step 3: The generative AI uses this information to generate a conversation that will comfort the target person.

[1225] Step 4: The generated conversation is converted into audio and transmitted to the target person via telephone.

[1226] "Example 3"

[1227] Step 1: The emotion engine analyzes emotions not only from the tone of voice and speaking style of the subject, but also from the content of what is being said.

[1228] Step 2: For example, if the subject says, "I had fun today," the emotion engine detects that joy.

[1229] Step 3: The detected emotion information is fed back to the generation AI.

[1230] Step 4: Generative AI uses this information to generate conversations that share the target person's joy.

[1231] Step 5: The generated conversation is converted into audio and transmitted to the target person via telephone.

[1232] Example 1

[1233] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1234] In modern society, elderly people living alone, people with physical disabilities, and those who have lost their partners often feel lonely and anxious. These people need effective means to regularly check on their safety and reduce their feelings of loneliness. It is also important to understand the physical and emotional state of the person from their tone of voice and speaking style, and respond appropriately. However, conventional systems have the problem of not being able to fully meet these needs.

[1235] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1236] In this invention, the server includes means for converting the target person's voice into text using speech recognition technology, means for analyzing the text using a generative AI model and generating an appropriate response, means for converting the generated response into speech using speech synthesis technology, means for analyzing the target person's tone of voice and speaking style using an emotion engine and feeding the results back to the generative AI model, and means for the generative AI model to generate a conversation adapted to the target person's emotions based on the feedback. This makes it possible to check the target person's safety through natural conversation, reduce feelings of loneliness, and understand the target person's physical condition and emotional state from the tone of voice and speaking style, enabling appropriate response.

[1237] "Speech recognition technology" is a technology that converts voice data into text data.

[1238] A "generative AI model" is an artificial intelligence model that analyzes input text data and generates appropriate answers.

[1239] "Speech synthesis technology" is a technology that converts text data into voice data.

[1240] The "emotion engine" is a technology that analyzes the subject's tone of voice and speaking style to assess their emotional state.

[1241] "Feedback" is the process of returning information to the generative AI model based on the analysis results.

[1242] A "target person" is a person who the system calls to check on their safety or to have a conversation with them.

[1243] "Safety check" refers to the act of checking the safety and health of a person.

[1244] "Loneliness reduction" refers to actions aimed at alleviating the sense of loneliness felt by the subject.

[1245] "Vocal timbre" refers to the tone and texture of a speaker's voice.

[1246] "Health management" refers to the act of understanding the subject's health condition and taking appropriate measures.

[1247] This invention is a system that calls specific individuals to check their safety, alleviating feelings of loneliness through conversation and managing their physical condition through tone of voice. This system is composed of a combination of voice recognition technology, generative AI models, voice synthesis technology, and an emotion engine.

[1248] The server first calls the target person. Once the call is connected, the device collects the target person's voice. The collected voice data is converted into text data using speech recognition software (e.g., Google Cloud Speech-to-Text). This text data is then input into a generative AI model (e.g., OpenAI's GPT-4).

[1249] The generative AI model analyzes the input text and generates an appropriate response. This response text is then converted back into audio data using speech synthesis software (e.g., Amazon Polly). The converted audio is then transmitted to the target person via telephone.

[1250] Furthermore, the device uses an emotion engine (e.g., Affectiva's emotion recognition technology) to analyze the subject's tone of voice and speaking style. The analysis results are obtained as data indicating the subject's emotional state. This data is fed back to a generative AI model, which then generates conversation content adapted to the subject's emotions.

[1251] As a concrete example, the following scenario can be considered.

[1252] scenario:

[1253] When the user answers the phone, the server asks, "Hello, how was your day?" If the user replies, "I'm a little tired today," the device collects this speech and uses speech recognition software to convert it into text: "I'm a little tired today." The server inputs this text into a generative AI model, which generates a response: "That's tough. Is there anything I can help you with?" The server inputs this response into speech synthesis software, which the device converts into speech and conveys to the user. The device then uses an emotion engine to analyze the user's level of fatigue from their tone of voice and feeds the results back to the generative AI model. The generative AI model uses this feedback to adjust the content of the next conversation.

[1254] Example prompt sentence:

[1255] "If a subject says, 'I'm a little tired today,' we input the following prompt to the generative AI model: 'If the subject feels tired, how should we respond?'"

[1256] In this way, the system can confirm the safety of the target person and reduce their sense of loneliness through natural conversation.

[1257] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1258] Step 1:

[1259] The server calls the target person.

[1260] The server automatically makes a call using the previously registered telephone number of the target person. Once the call is connected, the process proceeds to the next step. The input is the target person's telephone number, and the output is the connection status of the call.

[1261] Step 2:

[1262] The device collects the subject's voice.

[1263] The device collects the subject's voice in real time via telephone, and temporarily stores the voice data for subsequent processing. The input is the subject's voice, and the output is the collected voice data.

[1264] Step 3:

[1265] The device uses speech recognition software to convert the speech into text.

[1266] The device uses speech recognition software (e.g., Google Cloud Speech-to-Text) to convert the collected voice data into text data. The input is the voice data, and the output is the converted text data.

[1267] Step 4:

[1268] The server inputs text into the generative AI model.

[1269] The server inputs the text data obtained from the speech recognition software into a generative AI model (e.g., OpenAI's GPT-4). The input is the text data, and the output is the analysis result by the generative AI model.

[1270] Step 5:

[1271] A generative AI model generates the appropriate answer.

[1272] A generative AI model generates appropriate answers based on input text. These answers are tailored to the subject's situation and emotions. The input is text data, and the output is the generated answer text.

[1273] Step 6:

[1274] The server inputs the generated response into speech synthesis software.

[1275] The server inputs the answer text obtained from the generative AI model into speech synthesis software (e.g., Amazon Polly). The input is the answer text, and the output is voice data.

[1276] Step 7:

[1277] The device uses speech synthesis software to convert the text to speech.

[1278] The device uses speech synthesis software to convert the response text into audio data, which is then ready to be transmitted to the target person. The input is the response text, and the output is the converted audio data.

[1279] Step 8:

[1280] The device transmits the converted voice to the target person.

[1281] The device transmits the converted voice to the subject via telephone, and the subject can hear the answer generated by the generative AI model. The input is voice data, and the output is voice communication to the subject.

[1282] Step 9:

[1283] The device uses an emotion engine to analyze the subject's tone of voice and speaking style.

[1284] The device uses an emotion engine (e.g., Affectiva's emotion recognition technology) to analyze the subject's tone of voice and speaking style. The analysis results are obtained as data indicating the subject's emotional state. The input is the subject's voice data, and the output is the emotion analysis results.

[1285] Step 10:

[1286] The server feeds back the analysis results of the emotion engine to the generative AI model.

[1287] The server feeds back the analysis results obtained from the emotion engine to the generative AI model. This feedback allows the generative AI model to generate conversation content that takes into account the emotional state of the target person. The input is the emotion analysis results, and the output is the fed-back data.

[1288] Step 11:

[1289] A generative AI model adjusts the conversation based on feedback.

[1290] The generative AI model adjusts the conversation content based on the feedback and generates responses for the next conversation. This process is repeated until the conversation with the target person is completed. The input is the feedback data, and the output is the adjusted conversation content.

[1291] (Application example 1)

[1292] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1293] In modern society, the number of elderly people living alone, physically disabled people, and people who have lost their partners is increasing, and the loneliness and health management of these people are becoming major issues. In particular, regular safety checks and emotional care are necessary, but doing this manually is difficult in terms of cost and effort. In addition, there is a need for a system that can conduct conversations that adapt to the emotions of the target person.

[1294] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server is a system that calls a specific target person to check on their safety and alleviate loneliness through conversation, while also managing their health through tone of voice. The system includes: a means for converting speech to text using a conversion system that mimics a human voice, creating a response using a generation AI, converting the response back to speech, and conducting a conversation over the phone; a means for analyzing the target person's emotions from their tone of voice and speaking style using an emotion analysis engine and feeding the results back to the generation AI; a means for the generation AI to generate a conversation adapted to the target person's emotions based on the emotion analysis results; a means for converting speech to text using a speech recognition API, a generation AI model, and a speech synthesis API, and converting the generated text back to speech; and a means for calling the target person using a smartphone and communicating the generated speech. This reduces the target person's sense of loneliness and enables health management through conversation adapted to their emotions.

[1295] "Specific target groups" refer to people who need regular safety checks and emotional support, such as elderly people living alone, people with disabilities, and those who have lost their partners.

[1296] "Calling to check on safety" refers to the act of contacting the target person by phone to check whether they are safe.

[1297] "Alleviating loneliness through conversation" refers to reducing a subject's sense of loneliness by engaging in a dialogue with them.

[1298] "Health management based on tone of voice" refers to inferring a person's physical condition from their tone of voice and speaking style, and taking the necessary measures.

[1299] "Conversion system" refers to technology that mimics the human voice and converts speech into text.

[1300] "Generative AI" refers to artificial intelligence that analyzes input text and generates appropriate answers.

[1301] An "emotion analysis engine" refers to technology that analyzes emotions from the subject's tone of voice and speaking style.

[1302] "Speech Recognition API" refers to an application programming interface for converting speech to text.

[1303] A "generative AI model" refers to an artificial intelligence model that generates appropriate answers based on input text data.

[1304] "Speech Synthesis API" means an application programming interface for converting text to speech.

[1305] A "smartphone" refers to a mobile information terminal that has telephone functions and can connect to the Internet and run applications.

[1306] The system for carrying out this invention makes telephone calls to specific target individuals to check on their safety, alleviating loneliness through conversation and managing their physical condition through tone of voice. A specific embodiment of this system will be described below.

[1307] Hardware and software used

[1308] Hardware

[1309] Smartphone (iOS or Android)

[1310] software

[1311] Speech Recognition API (Google Cloud Speech-to-Text)

[1312] Generative AI model (OpenAI GPT-4)

[1313] Speech synthesis API (Google Cloud Text-to-Speech)

[1314] Sentiment analysis engine (IBM Watson Tone Analyzer)

[1315] System configuration

[1316] 1. Voice Recognition

[1317] The subject's voice is collected through the smartphone's microphone.

[1318] Use the speech recognition API (Google Cloud Speech-to-Text) to convert the collected audio into text.

[1319] 2. Text Analysis

[1320] Input text into a generative AI model (OpenAI GPT-4) to generate an appropriate answer.

[1321] 3. Emotion analysis

[1322] An emotion analysis engine (IBM Watson Tone Analyzer) is used to analyze emotions from the subject's tone of voice and speaking style.

[1323] The analysis results are fed back to the generation AI.

[1324] 4. Answer generation

[1325] Based on the results of emotion analysis, the generative AI generates conversations that adapt to the emotions of the target person.

[1326] 5. Speech Synthesis

[1327] Use a speech synthesis API (Google Cloud Text-to-Speech) to convert the generated text response into audio.

[1328] 6. Phone Calls

[1329] The subject is called using a smartphone and the generated audio is transmitted.

[1330] Specific examples

[1331] For example, if an elderly person lives alone, the app can automatically call them at a set time every day to check on their safety. Below are some examples of specific prompts:

[1332] Prompt Sentence Examples

[1333] Subject's audio text: "I'm feeling a little under the weather today."

[1334] Prompt to generator AI: "Subject says they are not feeling well. Please offer kind words of encouragement."

[1335] Generator's response: "That's worrying. Please take it easy and get plenty of rest. Let me know if there's anything I can do to help."

[1336] In this way, the system can reduce the subject's sense of loneliness and help them manage their health through emotionally adaptive conversations.

[1337] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1338] Step 1:

[1339] The smartphone device calls the subject and collects the subject's voice through the microphone. The collected voice data is sent to a voice recognition API (Google Cloud Speech-to-Text). The input is the subject's voice data, and the output is text data. The voice recognition API analyzes the voice data and converts it into corresponding text data.

[1340] Step 2:

[1341] The server inputs the text data received from the speech recognition API into a generative AI model (OpenAI GPT-4). The input is the text data from the speech recognition API, and the output is the generated answer text. The generative AI model analyzes the input text data and generates an appropriate answer.

[1342] Step 3:

[1343] The server sends the generated response text to an emotion analysis engine (IBM Watson Tone Analyzer), which analyzes the emotion of the subject based on their tone of voice and speaking style. The input is the generated response text, and the output is the emotion analysis result. The emotion analysis engine analyzes the text data and identifies the subject's emotional state.

[1344] Step 4:

[1345] The server feeds back the emotion analysis results to the generative AI model and generates a conversation that adapts to the emotions. The input is the emotion analysis results and the generated answer text, and the output is the final answer text that adapts to the emotions. The generative AI model takes the emotion analysis results into account and generates an answer that adapts to the target person's emotions.

[1346] Step 5:

[1347] The server sends the final answer text to a speech synthesis API (Google Cloud Text-to-Speech) and converts it into audio data. The input is the final answer text and the output is audio data. The speech synthesis API analyzes the text data and generates the corresponding audio data.

[1348] Step 6:

[1349] The smartphone device plays back the voice data received from the speech synthesis API and transmits it to the target person. The input is the voice data from the speech synthesis API, and the output is the voice that is transmitted to the target person. The smartphone device plays back the generated voice and transmits it to the target person, completing the conversation.

[1350] Example 2

[1351] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1352] In modern society, elderly people living alone, people with physical disabilities, and those who have lost their partners often feel lonely, which can lead to poor mental health. It is also difficult to remotely manage the health of these individuals. Conventional safety confirmation systems simply check the safety of individuals, and are unable to provide dialogue tailored to the individual's emotions and physical condition.

[1353] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1354] In this invention, the server includes: means for communicating with a specific target person to check on their safety and alleviate loneliness through conversation, and also capable of managing their health through voice; means for utilizing a conversion system that mimics a human voice to convert voice to text, creating a response using a generative AI model, converting the response back into voice, and communicating in a conversational format; means for inputting the target person's voice and facial expressions into an emotion engine to detect emotions; means for inputting prompt sentences into the generative AI model based on feedback from the emotion engine to generate dialogue corresponding to the emotions; and means for converting the generated dialogue into mimic voice and providing it to the target person. This allows for the provision of dialogue that corresponds to the target person's emotions and physical condition, reducing loneliness and enabling remote health management.

[1355] "Specific target groups" refer to individuals who are particularly susceptible to loneliness and need to manage their health, such as elderly people living alone, people with physical disabilities, and those who have lost their partners.

[1356] "Communication" refers to the act of sending or receiving information by telephone, internet, or other digital means of communication.

[1357] "Safety check" refers to the act of checking whether a specific person is safe.

[1358] "Voice-based health management" refers to the act of analyzing a person's tone of voice and speaking style to understand their physical and emotional state.

[1359] A "conversion system" refers to a system that analyzes input voice data and generates imitation voices.

[1360] "Generative AI model" refers to an artificial intelligence model that generates natural-looking dialogue based on input text and prompts.

[1361] An "emotion engine" refers to a system that analyzes a subject's voice and facial expressions to detect their emotional state.

[1362] A "prompt sentence" refers to an instruction sentence entered into a generative AI model to generate a specific dialogue.

[1363] "Imitation voice" refers to a voice that is generated based on specific voice data and is similar to the original voice.

[1364] "Conversational style" refers to the style or method of conducting a dialogue, including two-way communication using voice or text.

[1365] MODE FOR CARRYING OUT THE INVENTION

[1366] This invention is a system that communicates with specific target individuals, checks their safety, and alleviates loneliness through conversation, while also managing their physical condition through voice. This system operates in cooperation with three parties: a server, a terminal, and a user.

[1367] Hardware and software used

[1368] Hardware: Servers, devices (PCs, smartphones, etc.)

[1369] Software: Text-to-speech system, emotion engine, generative AI model

[1370] Specific operation of the system

[1371] 1. Uploading audio data

[1372] Users upload audio data of voice actors or loved ones recorded before they passed away from their device to the server.

[1373] Example: A user opens an app on their smartphone, selects audio data, and presses the "Upload" button. The server receives and stores this audio data.

[1374] 2. Analysis of speech data and generation of imitation speech

[1375] The server inputs the uploaded voice data into the voice conversion system.

[1376] Example: The server calls a voice conversion system (e.g., Google Cloud Text-to-Speech API) and analyzes the voice data. Based on the analysis results, it generates an imitation voice and saves it on the server.

[1377] 3. User Emotion Detection

[1378] The terminal inputs the user's voice and facial expressions into an emotion engine to detect emotions.

[1379] Example: The device's camera and microphone capture the user's facial expressions and voice in real time and send them to an emotion engine (e.g., Affectiva SDK). The emotion engine analyzes the user's emotional state and returns the results to the device.

[1380] 4. Emotion-Based Dialogue Generation

[1381] The server inputs prompt sentences into the generative AI model based on feedback from the emotion engine.

[1382] Example: A server sends a prompt to a generative AI model (e.g., OpenAI GPT-3) saying, "The user is sad, so please generate a comforting conversation." The generative AI model generates an appropriate dialogue and returns it to the server.

[1383] 5. Providing dialogue using imitation voice

[1384] The server converts the generated dialogue into imitation voice and transmits it to the terminal.

[1385] Example: The server inputs the generated text into a speech conversion system to generate an imitation voice, which is then sent to the device, which plays it back to the user.

[1386] Prompt Sentence Examples

[1387] "The user is sad, so generate a comforting conversation."

[1388] "Your users are happy, so create conversations that share that joy."

[1389] This system allows users to enjoy conversations in a friendly voice that responds to their emotions, reducing feelings of loneliness and enabling remote health management.

[1390] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1391] Program processing flow

[1392] Step 1: Upload your audio data

[1393] Users upload audio data of voice actors or loved ones recorded before they passed away from their device to the server.

[1394] Input: User-selected audio data file

[1395] Specific operation: The user opens the app on their smartphone, selects the audio data, and presses the "Upload" button. The device then sends the audio data to the server.

[1396] Output: Audio data file stored on the server

[1397] Step 2: Analyzing the audio data and generating imitation voices

[1398] The server inputs the uploaded voice data into the voice conversion system.

[1399] Input: Audio data file stored on the server

[1400] Specific operation: The server calls a voice conversion system (e.g., Google Cloud Text-to-Speech API) and analyzes the voice data. Based on the analysis results, it generates an imitation voice and saves it on the server.

[1401] Output: Imitation voice data stored on the server

[1402] Step 3: Detecting user emotions

[1403] The terminal inputs the user's voice and facial expressions into an emotion engine to detect emotions.

[1404] Input: Real-time user voice and facial expression data

[1405] How it works: The device's camera and microphone capture the user's facial expressions and voice in real time and send them to an emotion engine (e.g., Affectiva SDK). The emotion engine analyzes the user's emotional state and returns the results to the device.

[1406] Output: User's emotional state data

[1407] Step 4: Emotion-based dialogue generation

[1408] The server inputs prompt sentences into the generative AI model based on feedback from the emotion engine.

[1409] Input: User emotional state data

[1410] Specific operation: The server sends a prompt to the generative AI model (e.g., OpenAI GPT-3) saying, "The user is sad, so please generate a comforting conversation." The generative AI model generates an appropriate dialogue and returns it to the server.

[1411] Output: Generated dialogue text

[1412] Step 5: Providing dialogue through imitation voice

[1413] The server converts the generated dialogue into imitation voice and transmits it to the terminal.

[1414] Input: Generated dialogue text

[1415] Specific operation: The server inputs the generated text into the speech conversion system to generate an imitation voice, which is then sent to the terminal, which plays it back to the user.

[1416] Output: The imitation voice played to the user

[1417] In this way, the system provides dialogue that responds to the user's emotions, reducing feelings of loneliness while enabling remote health management.

[1418] (Application example 2)

[1419] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1420] In modern society, elderly people living alone, people with physical disabilities, and those who have lost their partners are prone to feeling lonely and are in need of psychological support. Food delivery services also need systems that allow users to place orders and make inquiries in a friendly voice. However, conventional systems have difficulty providing appropriate dialogue based on emotions, making it difficult to increase user satisfaction.

[1421] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1422] In this invention, the server includes: means for communicating with a specific target person to check on their safety and alleviate loneliness through conversation, and also for managing their health through voice tone; means for utilizing a conversion system that mimics a human voice to convert speech to text, creating a response using a generation AI, converting it back into speech, and communicating in a conversational format; means for detecting the target person's emotions using an emotion engine and generating a conversation according to those emotions; and means for using voice data from a voice actor or a relative recorded before death so that the target person can enjoy a conversation in a familiar voice. This makes it possible to provide appropriate dialogue according to the target person's emotions, alleviating loneliness, and increasing user satisfaction in food delivery services.

[1423] "Specific target groups" refer to individuals who meet certain conditions, such as elderly people living alone, people with physical disabilities, and those who have lost their partners.

[1424] "Communications" refers to the means of sending and receiving voice and data via telephone and the Internet.

[1425] "Safety check" refers to the act of checking the health and safety of the person in question.

[1426] "Conversational" refers to a form of two-way communication using voice.

[1427] "Health management through vocal tone" refers to a method of understanding a person's physical and emotional state by analyzing the tone and quality of their voice.

[1428] "Conversion system" refers to a system that converts speech to text and text to speech.

[1429] "Generative AI" refers to a system that uses artificial intelligence to generate appropriate responses based on input data.

[1430] An "emotion engine" refers to an algorithm or system for detecting emotions from a subject's voice or text.

[1431] A "voice actor" is a professional who performs the voices of others as a profession.

[1432] "Audio data of immediate family members" refers to audio data recorded by the subject's family members or close friends while they were alive.

[1433] "Food delivery service" refers to a service that allows you to order food and drinks and have them delivered to a designated location.

[1434] "User" refers to an individual who uses a system or service.

[1435] The system for carrying out this invention is a system that can communicate with specific target individuals to check their safety, alleviate loneliness through conversation, and also manage their physical condition by tone of voice. A specific embodiment of this system is described below.

[1436] System Configuration

[1437] The system consists of the following main components:

[1438] 1. Communications: Hardware and software for sending and receiving voice and data over the telephone or the Internet.

[1439] 2. Speech conversion system: Software for converting speech to text and text to speech, specifically using the speech_recognition library and gTTS library.

[1440] 3. Generative AI: Artificial intelligence that generates appropriate responses based on input data. Uses the transformers library.

[1441] 4. Emotion Engine: An algorithm for detecting emotions from a subject's speech or text. It uses the emotion analysis model from the transformers library.

[1442] 5. Audio data: Audio data of voice actors or loved ones recorded before they passed away.

[1443] Processing flow

[1444] 1. Voice input: The subject inputs voice through a microphone, which is then transmitted to the server via a communication means.

[1445] 2. Speech Recognition: The server converts the speech to text using the speech_recognition library.

[1446] 3. Sentiment detection: The converted text is fed into the sentiment analysis model in the transformers library to detect emotions.

[1447] 4. Response generation: Based on the detected sentiment, the generative AI generates an appropriate response text.

[1448] 5. Speech generation: The generated response text is converted into speech using the gTTS library.

[1449] 6. Audio output: The generated audio is sent to the subject via a communication means and played back through a speaker.

[1450] Hardware and software used

[1451] Hardware: Microphones, speakers, servers

[1452] Software: speech_recognition library, transformers library, gTTS library, playsound library

[1453] Specific examples

[1454] For example, if a user says, "I'm very tired today," the emotion detection system will determine this as "NEGATIVE," and the generative AI will generate a response like, "Sorry, is there a problem?" This response is converted into audio and played back to the user.

[1455] Prompt Sentence Examples

[1456] User: I'm very tired today

[1457] System: Sorry, is there a problem?

[1458] In this way, it is possible to provide appropriate dialogue that is tailored to the target person's emotions, alleviating feelings of loneliness and increasing user satisfaction in food delivery services.

[1459] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1460] Step 1:

[1461] The user inputs voice through a microphone.

[1462] Input: User's voice

[1463] Output: Audio data

[1464] What it does: When a user says, "I'm so tired today," the audio is captured by a microphone.

[1465] Step 2:

[1466] The terminal transmits the voice data to the server.

[1467] Input: Audio data

[1468] Output: Audio data sent to the server

[1469] What it does: The captured audio data is sent over the internet to a server.

[1470] Step 3:

[1471] The server converts the audio to text using the speech_recognition library.

[1472] Input: Audio data

[1473] Output: Text data

[1474] What happens: The server receives the audio data and uses the speech_recognition library to convert it into text: "I'm very tired today."

[1475] Step 4:

[1476] The server uses the sentiment analysis model from the transformers library to detect sentiment from text data.

[1477] Input: Text data

[1478] Output: Emotion data (e.g., NEGATIVE)

[1479] Specific operation: The server inputs the converted text data into a sentiment analysis model and detects the sentiment "NEGATIVE."

[1480] Step 5:

[1481] The server uses generative AI to generate an appropriate response text based on the detected emotion.

[1482] Input: Emotion data, text data

[1483] Output: Response text (e.g. "Sorry, is there a problem?")

[1484] Specific operation: The server inputs emotion data and text data into the generation AI, which generates a response text saying, "Sorry, is there a problem?"

[1485] Step 6:

[1486] The server converts the response text into speech using the gTTS library.

[1487] Input: Response text

[1488] Output: Response voice data

[1489] Specific operation: The server inputs the generated response text into the gTTS library and converts it into voice data.

[1490] Step 7:

[1491] The server transmits the response voice data to the terminal.

[1492] Input: Response voice data

[1493] Output: Response audio data sent to the terminal

[1494] Specific operation: The server transmits the generated response voice data to the terminal via the Internet.

[1495] Step 8:

[1496] The device will play a response voice through the speaker.

[1497] Input: Response voice data

[1498] Output: The audio played to the user

[1499] Specific operation: The device plays the received response voice data over the speaker, delivering the voice message "Sorry, is there a problem?" to the user.

[1500] Example 3

[1501] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1502] In modern society, elderly people living alone, people with physical disabilities, and those who have lost their partners often feel lonely, which can have a negative impact on their mental health. Furthermore, there is a lack of efficient ways to check on the safety and health of these people. Furthermore, there is the challenge of providing conversations that are sensitive to the feelings of these people.

[1503] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1504] In this invention, the server is a system that can call a specific target person to check on their safety, ease their loneliness by talking to them, and also manage their physical condition by tone of voice,

[1505] Using a conversion system that mimics the human voice, the speech is converted into text, and a generative AI model creates a response, which is then converted into speech and used in a conversation over the phone.

[1506] A way to create a list of contacts and schedule calls based on their lifestyle and preferences.

[1507] An emotion analysis means that analyzes the tone of voice, speaking style, and content of the conversation in real time during a call to detect emotions;

[1508] A means for the generative AI model to generate a response using a prompt sentence based on feedback from the sentiment analysis means;

[1509] a means for communicating the generated response to the subject;

[1510] This will help reduce the subject's sense of loneliness, provide a sense of security, and enable emotionally sensitive dialogue.

[1511] "Specific target groups" refer to individuals who meet certain conditions, such as elderly people living alone, people with physical disabilities, and those who have lost their partners.

[1512] "Safety check" refers to the act of checking whether a person is safe.

[1513] "Health management through voice tone" refers to analyzing the tone of a person's voice and speaking style to understand their physical and emotional state.

[1514] "Conversion system" refers to a system that mimics the human voice and converts speech into text.

[1515] A "generative AI model" refers to an artificial intelligence model that generates appropriate responses based on input text or prompts.

[1516] A "prompt sentence" refers to an instruction or input sentence that is used to generate a response for a generative AI model.

[1517] A "list of subjects" refers to a database or list that compiles information about specific subjects.

[1518] A "calling schedule" refers to a plan that sets the date, time, and frequency of calls to the target person.

[1519] "Emotion analysis means" refers to technology that analyzes the tone of voice, speaking style, and content of the conversation of the person on the phone to detect their emotions.

[1520] "Feedback" refers to the communication of information obtained from sentiment analysis methods to the generative AI model.

[1521] "Means for generating a response" refers to the process by which a generative AI model creates an appropriate response based on a prompt sentence.

[1522] "Means for carrying out a dialogue" refers to the technology used to convey the generated response to the target person and continue the dialogue.

[1523] This invention is a system that makes a call to a specific target person to check on their safety, alleviate feelings of loneliness through conversation, and manage their physical condition by tone of voice. A specific embodiment of this system will be described below.

[1524] System configuration

[1525] Hardware

[1526] The server contains a database, a voice conversion system, and a generative AI model. The terminal has telephone functionality and is equipped with a microphone and speaker for communicating with the target person.

[1527] software

[1528] The server runs the following software:

[1529] Database management system: Manages subject information.

[1530] Scheduling software: Schedule calls to be made.

[1531] Speech-to-text systems: convert speech to text and text to speech.

[1532] Sentiment analysis engine: Analyzes voice data during calls to detect emotions.

[1533] Generative AI model: Generates a response based on the prompt.

[1534] System Operation

[1535] Creating a target list

[1536] The server collects information on elderly people living alone, people with physical disabilities, and people who have lost their partners, and registers it in a database, including the person's name, age, address, phone number, lifestyle, and preferences.

[1537] Setting up a call schedule

[1538] The server schedules calls based on the recipient's lifestyle and preferences. For example, if the recipient is a morning person, the server schedules a call every Monday at 10 a.m.

[1539] Making a call

[1540] The device will call the target person based on the set schedule, and once the call is connected, the device will start the call.

[1541] Emotion analysis

[1542] The device analyzes the tone of voice, speaking style, and content of the conversation in real time during the call. If the person says, "I had a great time today," the emotion analysis engine will detect their joy.

[1543] Response Generation

[1544] The server receives feedback from the sentiment analysis engine and sends a prompt to the generative AI model. For example, it sends a prompt such as, "The subject said that he enjoyed playing with his grandchild. Please generate a response that shares his joy." The generative AI model generates a response based on this prompt.

[1545] Dialogue execution

[1546] The device then communicates the generated response to the subject, for example, "That's great! What did you play?" This keeps the conversation going, reducing the subject's sense of loneliness and providing a sense of security.

[1547] Specific examples

[1548] As a concrete example, the following scenario can be considered.

[1549] Examples:

[1550] Let's say the user is Mr. A, an elderly person living alone. Taking Mr. A's lifestyle into consideration, the server sets a schedule for a call every Monday at 10:00 AM. The device calls Mr. A at 10:00 AM on Monday, and the emotion analysis engine analyzes Mr. A's tone of voice and the content of what he said. When Mr. A says, "My grandchildren came to play with me today and it was fun," the emotion analysis engine detects his joy and sends feedback to the server. The generative AI model generates a response saying, "That's great! What did you play with?"

[1551] Example prompt sentence:

[1552] "The subject said they had fun playing with their grandchild. Generate a response that shares their joy."

[1553] In this way, the system can provide a conversation that is sensitive to the subject's emotions, reduce the subject's sense of loneliness, and provide a sense of security. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1554] Step 1:

[1555] The server creates a list of eligible people. The user enters information about elderly people living alone, people with physical disabilities, and people who have lost their partners into an input form and clicks the submit button. The server saves the entered information (name, age, address, phone number, lifestyle, preferences, etc.) in a database. The input is the target person's information, and the output is a list of eligible people saved in the database.

[1556] Step 2:

[1557] The server sets a call schedule based on the target person's lifestyle and preferences. The server retrieves the target person's information from a database and calculates the optimal time to call. For example, if the target person is a morning person, the server sets 10:00 AM as the optimal time. The input is the target person's lifestyle and preferences, and the output is the set call schedule.

[1558] Step 3:

[1559] The terminal calls the target person based on the set schedule. When the call connects, the terminal greets the target person with a greeting such as "Hello, how are you?" The input is the set call schedule, and the output is the start of a call with the target person.

[1560] Step 4:

[1561] During a call, the device analyzes the target person's tone of voice, speaking style, and content of the conversation in real time. The emotion analysis engine analyzes the target person's voice data and detects their emotions. For example, if the target person says, "I had a great time today," the emotion analysis engine will detect their joy. The input is the target person's voice data, and the output is the detected emotional information.

[1562] Step 5:

[1563] The server receives feedback from the sentiment analysis engine and sends a prompt to the generative AI model. For example, the prompt might read, "The subject said that he enjoyed playing with his grandchild. Please generate a response that shares his joy." The generative AI model generates a response based on this prompt. The input is the feedback from the sentiment analysis engine and the prompt, and the output is the generated response.

[1564] Step 6:

[1565] The device then communicates the generated response to the subject. For example, it might say, "That's great! What did you play?" This allows for a continuation of the dialogue, reducing the subject's sense of loneliness and providing a sense of security. The input is the generated response, and the output is a continuous dialogue with the subject.

[1566] (Application example 3)

[1567] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1568] Elderly people living alone, people with physical disabilities, and those who have lost their partners are prone to feelings of loneliness and anxiety, and require psychological support in their daily lives. However, there are currently no systems in place that regularly check on the safety of these people and provide dialogue that responds to their emotions. Notification functions are also needed to respond quickly to emergencies. Therefore, a system that can solve these issues is needed.

[1569] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes: means for calling a specific target person to check on their safety and alleviate loneliness through conversation, and also for managing their health through tone of voice; means for setting a schedule for regular calls based on the target person's lifestyle and preferences; means including an emotion engine that analyzes emotions from the target person's tone of voice and content of conversation; means for a generation AI to generate conversations based on the results of the emotion analysis; and means for sending a notification to a pre-registered emergency contact in the event of an emergency. This reduces the target person's sense of loneliness, provides a sense of security, and enables a rapid response in the event of an emergency.

[1570] "Specific target groups" refer to individuals who meet certain conditions, such as elderly people living alone, people with physical disabilities, and those who have lost their partners.

[1571] "Making phone calls to check on safety" refers to the act of calling the target person periodically to check on their health and safety.

[1572] "Relieving loneliness through conversation" refers to reducing the sense of loneliness felt by a subject by having a conversation with them.

[1573] "Health management through voice tone" refers to analyzing the subject's tone of voice and speaking style to understand their health and emotional state.

[1574] "Conversion system" refers to a system that mimics the human voice and converts speech into text.

[1575] "Generative AI" refers to artificial intelligence that generates appropriate answers based on input text data.

[1576] An "emotion engine" refers to a system that analyzes emotions from the tone of a person's voice and the content of their speech.

[1577] "Setting a schedule" means deciding on regular times to call based on the person's lifestyle and preferences.

[1578] "Sending a notification to emergency contacts" means that if the target person is in an emergency, a notification will be sent to the contacts registered in advance.

[1579] A system for carrying out the present invention periodically calls a specific target person to check on their safety and provide a dialogue according to their emotions. A specific embodiment of this system will be described below.

[1580] System configuration

[1581] The system consists of the following main components:

[1582] 1. Server: Manages the entire system and processes data.

[1583] 2. Smartphone: A device used by the subject to receive and make phone calls.

[1584] 3. Emotion engine: Analyzes emotions from the subject's tone of voice and content of speech.

[1585] 4. Generative AI: Generates appropriate conversations based on the results of sentiment analysis.

[1586] 5. Emergency notification system: In the event of an emergency, notifications will be sent to pre-registered emergency contacts.

[1587] Program processing

[1588] The server does the following:

[1589] 1. Scheduling regular phone calls: Schedule regular phone calls based on the recipient's lifestyle and preferences. Use Python's Schedule library for this.

[1590] 2. Make a phone call: Use the Twilio API to make a call to the target person.

[1591] 3. Emotion analysis: Analyze the subject's tone of voice and content of speech and use an emotion engine to identify emotions.

[1592] 4. Conversation generation: Based on the results of sentiment analysis, the generative AI generates appropriate conversation.

[1593] 5. Emergency Notification: If the target person experiences an emergency, a notification will be sent to the pre-registered emergency contact.

[1594] Hardware and software used

[1595] Hardware: Smartphone

[1596] software:

[1597] Twilio API: API for making phone calls

[1598] Emotion Analysis Module: A module for analyzing the emotions of the subject

[1599] AI Conversation Module: A module that generates conversations based on the results of sentiment analysis

[1600] Schedule Library: A Python library for scheduling periodic tasks

[1601] Specific examples

[1602] For example, if Tanaka says, "I had fun today," the emotion analysis module will detect the joy, and the generative AI will generate a conversation such as, "That's great! What happened?"

[1603] Prompt Sentence Examples

[1604] If Tanaka says, "I had fun today," the emotion analysis module will detect that joy, and the generative AI will generate a conversation like, "That's great! What happened?"

[1605] In this way, it is possible to provide a dialogue that responds to the subject's emotions, reduce feelings of loneliness, provide a sense of security, and respond quickly in the event of an emergency.

[1606] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1607] Step 1:

[1608] The server schedules regular phone calls based on the target person's lifestyle and preferences. As input, it receives data on the target person's lifestyle and preferences and uses the Python Schedule library to set the schedule. As output, it determines the specific date and time to make the call.

[1609] Step 2:

[1610] The server uses the Twilio API to call the target person according to the set schedule. As input, it receives the scheduled date and time and the target person's phone number, calls the Twilio API to make the call, and as output, the phone call is connected to the target person.

[1611] Step 3:

[1612] The device (smartphone) records the subject's voice and sends it to the server. As input, it receives the subject's voice data and sends it to the server. As output, the voice data is sent to the server.

[1613] Step 4:

[1614] The server inputs the received voice data into the emotion engine and analyzes the emotion of the target person. As input, it receives the voice data and analyzes the emotion using the emotion engine. As output, it obtains the analyzed emotion data.

[1615] Step 5:

[1616] The server inputs the results of the emotion analysis into the generative AI to generate an appropriate conversation. As input, it receives emotion data and generates a conversation using the generative AI. As output, it obtains the generated conversation text.

[1617] Step 6:

[1618] The server converts the generated conversation text into speech and transmits it to the terminal. As input, the server receives the generated conversation text and generates speech data using a speech conversion system. As output, the speech data is transmitted to the terminal.

[1619] Step 7:

[1620] The device plays the received voice data and engages in a dialogue with the subject. As input, the device receives voice data and plays it back through a speaker. As output, a dialogue with the subject takes place.

[1621] Step 8:

[1622] The server sends a notification to pre-registered emergency contacts when a target person is in an emergency. As input, it receives emergency detection data and sends a notification to the emergency contacts. As output, an emergency notification is sent.

[1623] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1624] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1625] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1626] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1627] [Third embodiment]

[1628] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1629] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1630] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32.

[1631] The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1632] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1633] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1634] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1635] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1636] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1637] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1638] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1639] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1640] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1641] "Example 1"

[1642] One aspect of the present invention is a system that calls specific individuals to check on their safety, alleviating loneliness through conversation and enabling health management through voice tone. This system utilizes a conversion system that mimics the human voice, converts speech to text, and then uses a generation AI to create a response, which is then converted back into speech, resulting in a conversation over the phone. Specifically, speech recognition technology is used to convert the target's speech into text, which is then input into the generation AI. The generation AI analyzes the text and generates an appropriate response. The generated response is converted from text to speech and communicated to the target via phone.

[1643] "Example 2"

[1644] In another embodiment of the present invention, the voices of voice actors (male and female) or relatives recorded before they died are used. Specifically, voice data of the voice actor or voice data of a relative recorded before they died is input into a voice conversion system, and a voice that imitates it is generated. In this embodiment, the target person can enjoy conversations with familiar voices, resulting in higher satisfaction.

[1645] "Example 3"

[1646] A further embodiment of the present invention targets elderly people living alone, people with physical disabilities, and those who have lost their partners. Specifically, a list of these people is created and a regular call schedule is set up. The call times can be adjusted to suit the target person's lifestyle and preferences. This embodiment can reduce the amount of time the target person feels lonely and provide a sense of security.

[1647] The processing flow of each embodiment will be described below.

[1648] "Example 1"

[1649] Step 1: The system places calls to specific individuals, scheduled to fit their lifestyles and preferences.

[1650] Step 2: When the subject answers the phone, the system uses voice recognition technology to convert the subject's speech into text.

[1651] Step 3: The text is fed into the generative AI, which analyzes the text and generates an appropriate answer.

[1652] Step 4: The generated responses are converted from text to speech and communicated to the target person over the phone.

[1653] "Example 2"

[1654] Step 1: Input voice data from a voice actor or a recording of a loved one before they pass away into the voice conversion system.

[1655] Step 2: The voice conversion system generates a voice that mimics the input voice data.

[1656] Step 3: The generated audio is transmitted to the subject via telephone and used in a conversation with the subject.

[1657] "Example 3"

[1658] Step 1: Create a list of target groups, including elderly people living alone, people with physical disabilities, and those who have lost their partners.

[1659] Step 2: Schedule regular phone calls for each person. Adjust the call times to suit each person's lifestyle and preferences.

[1660] Step 3: According to the set schedule, the system calls the target person and conducts a conversation.

[1661] Example 1

[1662] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1663] In modern society, elderly people living alone, people with physical disabilities, and those who have lost their partners often feel lonely, which can have a negative impact on their mental health. Furthermore, it is difficult to quickly detect changes in the physical condition of these individuals and take appropriate action. Conventional safety confirmation systems simply make phone calls and lack the functionality to analyze the tone of the individual's voice and manage their physical condition. Therefore, more advanced safety confirmation and health management systems are needed.

[1664] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1665] In this invention, the server includes a speech recognition means for converting speech to text, a means for inputting the converted text into a generative AI model to generate an appropriate response, and a speech synthesis means for converting the generated response from text to speech. This makes it possible to call a specific target person to check on their safety and ease loneliness through conversation, as well as to analyze the target person's tone of voice to detect changes in their physical condition and respond promptly.

[1666] "Specific target groups" refer to individuals who meet certain conditions, such as elderly people living alone, people with physical disabilities, and those who have lost their partners.

[1667] "Making a phone call to check on safety" refers to the act of calling a person and checking their response to check their safety and health.

[1668] "Relieving loneliness through conversation" refers to reducing the sense of loneliness felt by the subject by communicating with them verbally.

[1669] "Health management through voice tone" refers to analyzing the tone and quality of a person's voice and detecting abnormalities in their health based on these changes.

[1670] "Speech recognition means for converting speech into text" refers to technology that collects the speech of a person as digital voice data and converts it into text information.

[1671] A "generative AI model" refers to an artificial intelligence model that analyzes input text data and generates an appropriate response.

[1672] "Text-to-speech synthesis means" refers to a technology that converts generated text data into speech data.

[1673] "Communication means for transmitting voice to a target person via telephone" refers to technology for transmitting generated voice data to a target person via a telephone line.

[1674] "A voice analysis means that analyzes the tone of a subject's voice and detects changes in their physical condition" refers to a technology that analyzes the characteristics of a subject's voice and detects abnormalities in their physical condition based on those changes.

[1675] This invention is a system that can call specific individuals to check on their safety, alleviate loneliness through conversation, and also manage their health by monitoring their tone of voice. This system is realized by combining voice recognition technology, generative AI models, voice synthesis technology, communication technology, and voice analysis technology.

[1676] The server first uses a communication method to call the target person. Specifically, it uses a cloud communication platform to make the call. For example, it uses the Twilio API to call the target person's phone number.

[1677] The device collects the voice after the target person answers the phone, and converts the collected voice into text using speech recognition. Specifically, it uses the Google Cloud Speech-to-Text API to convert voice to text.

[1678] The server inputs the converted text into a generative AI model, which uses OpenAI's GPT-4, and the generative AI model analyzes the input text and generates an appropriate answer.

[1679] The generated answer is then converted from text to speech again using a speech synthesis tool, specifically Amazon Polly, to convert text to speech.

[1680] The server transmits the generated voice to the subject via telephone. The communication means is used to transmit the generated voice data to the subject via a telephone line.

[1681] Furthermore, the device analyzes the subject's tone of voice to detect changes in their physical condition. Using a voice analysis tool, the device analyzes the tone and quality of the subject's voice and detects abnormalities in their physical condition based on these changes. The analysis results are then notified to a medical institution or family member as necessary.

[1682] As a concrete example, the following scenario can be considered.

[1683] When a user wants to check on the safety of an elderly person, the system operates as follows:

[1684] 1. The server uses the Twilio API to call the senior citizen.

[1685] 2. The device collects the elderly person's voice saying "Hello, how are you?" and converts it into text using the Google Cloud Speech-to-Text API.

[1686] 3. The server inputs the converted text "Hello, how are you?" into GPT-4 and generates an appropriate answer.

[1687] 4. The server generates the answer, "Hello, how are you? How was your day?" and converts it into speech using Amazon Polly.

[1688] 5. The server transmits the generated audio to the elderly person via the Twilio API.

[1689] 6. The device analyzes the elderly person's voice tone and notifies medical institutions or family members if any abnormalities are detected.

[1690] An example prompt is, "If an older adult says, 'Hello, how are you?' please generate an appropriate response."

[1691] This system allows users to check the safety of specific individuals, alleviating feelings of loneliness and managing their health.

[1692] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1693] Step 1:

[1694] The server places a call to the target person.

[1695] Input: Subject's phone number

[1696] Specific operation: The server calls the API of the cloud communication platform (e.g., Twilio API) and makes a call to the target person's phone number.

[1697] Output: The call to the target person is successful and the call starts.

[1698] Step 2:

[1699] The terminal collects the subject's voice and converts it into text using speech recognition means.

[1700] Input: Subject's voice

[1701] What it does: The device collects the target person's voice saying "Hello, how are you?" and converts it into text using speech recognition software (e.g., Google Cloud Speech-to-Text API).

[1702] Output: The converted text "Hello, how are you?"

[1703] Step 3:

[1704] The server inputs the converted text into a generative AI model to generate an appropriate answer.

[1705] Input: Transformed text "Hello, how are you?"

[1706] Specific operation: The server inputs the converted text into a generative AI model (e.g., GPT-4), sets a prompt sentence, and generates an appropriate answer.

[1707] Output: Generated answer "Hello, I'm fine. How was your day?"

[1708] Step 4:

[1709] The server converts the generated answers from text to speech.

[1710] Input: Generated answer "Hi, I'm fine. How was your day?"

[1711] What happens: The server uses speech synthesis software (e.g., Amazon Polly) to convert the generated text into speech.

[1712] Output: Generated audio data

[1713] Step 5:

[1714] The server transmits the generated voice to the target person via telephone.

[1715] Input: Generated audio data

[1716] Specific operation: The server uses a communication means (e.g., Twilio API) to transmit the generated voice data to the target person via a telephone line.

[1717] Output: Audio delivered to subject

[1718] Step 6:

[1719] The device analyzes the subject's tone of voice and detects changes in their physical condition.

[1720] Input: Subject's voice data

[1721] Specific operation: The device uses voice analysis software to analyze the tone and quality of the subject's voice, and detects abnormalities in their physical condition based on changes.

[1722] Output: Notification data when abnormalities in health are detected (notification to medical institutions and family members)

[1723] In this way, the system can check the safety of specific individuals, alleviating loneliness and also managing their health.

[1724] (Application example 1)

[1725] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1726] In modern society, the number of elderly people living alone, physically disabled people, and people who have lost their partners is increasing, and confirming the safety of these people and reducing their feelings of loneliness has become an important issue. Managing the health of these people is also important, and a quick response is required, especially in emergencies. However, the current system lacks the means to effectively solve these issues.

[1727] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1728] In this invention, the server includes: a means for calling specific individuals to check on their safety, alleviating loneliness through conversation and enabling health management based on tone of voice; a means for converting speech to text using a conversion system that mimics the human voice, creating a response using a generation AI, converting the response back to speech, and conducting a conversation over the phone; a means for converting the target's speech to text using speech recognition technology; a means for analyzing the text using a generation AI and generating an appropriate response; a means for converting the generated response from text to speech and communicating it to the target over the phone; and a means for inferring the target's health from tone of voice and content of conversation and notifying family or medical institutions as necessary. This makes it possible to check the safety of elderly people living alone, people with disabilities, and those who have lost their partners, reduce loneliness, and manage their health.

[1729] "Specific target groups" refer to people whose safety needs to be confirmed and whose feelings of loneliness need to be alleviated, such as elderly people living alone, people with physical disabilities, and those who have lost their partners.

[1730] "Safety check" refers to the act of checking the current situation and health condition of a person.

[1731] "Conversational" refers to a format in which a conversation is held with the subject over the telephone.

[1732] "Health management based on voice tone" refers to inferring a person's health condition from the tone and quality of their voice and taking the necessary measures.

[1733] "Conversion system" refers to technology that mimics the human voice and converts speech into text.

[1734] "Generative AI" refers to artificial intelligence that analyzes input text and generates appropriate answers.

[1735] "Speech recognition technology" refers to technology that converts speech into text.

[1736] "Text analysis" refers to the process of understanding input text and generating appropriate responses.

[1737] "Speech synthesis" refers to the technology of converting text into speech.

[1738] "Notification" refers to the act of conveying information to family members or medical institutions based on the subject's physical condition and situation.

[1739] The system for implementing this invention makes phone calls to specific individuals to check on their safety, alleviating their feelings of loneliness through conversation and managing their physical condition through voice tone. This system is composed of a combination of voice recognition technology, generation AI, and voice synthesis technology.

[1740] First, the server periodically calls a specific target person. When the call connects, it captures the target person's voice and converts it into text using speech recognition technology. This speech recognition technology uses the speech_recognition library and Google's speech recognition API.

[1741] The server then inputs the converted text into a generation AI, which uses OpenAI's API to analyze the input text and generate an appropriate response. The generated response is then converted from text to speech again, using the gTTS library for speech synthesis.

[1742] The generated voice is transmitted to the target person via telephone. The server then infers the target person's physical condition from the tone of voice and content of the conversation, and notifies family members or medical institutions as necessary. This notification function enables rapid response in emergencies.

[1743] For example, consider the following conversation:

[1744] Elderly person: "I'm feeling a bit tired today."

[1745] Generator: "That's worrying. How are you feeling these days?"

[1746] Example prompt sentence:

[1747] Conversation with an elderly person: I'm feeling a little tired today.

[1748] Generate an appropriate response.

[1749] This system will enable elderly people living alone, people with disabilities, and those who have lost their partners to effectively check on their safety, reduce feelings of loneliness, and manage their health.

[1750] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1751] Step 1:

[1752] The server periodically calls a specific target person. When the call is connected, the target person's voice is acquired. The input is the target person's voice, and the output is voice data. This voice data is collected through a microphone.

[1753] Step 2:

[1754] The server converts the acquired voice data into text using speech recognition technology. Specifically, it uses the speech_recognition library and Google's speech recognition API. The input is voice data, and the output is text data. This conversion allows the subject's speech to be obtained in text format.

[1755] Step 3:

[1756] The server inputs the converted text data into the generation AI. The generation AI uses OpenAI's API to analyze the input text and generate an appropriate answer. The input is text data, and the output is the generated answer text data. This analysis results in an appropriate response to the subject's utterance.

[1757] Step 4:

[1758] The server converts the text data of the generated answer into speech using speech synthesis technology. Specifically, it uses the gTTS library. The input is the text data of the generated answer, and the output is speech data. This conversion results in the generated response in speech format.

[1759] Step 5:

[1760] The server transmits the generated voice data to the subject via telephone. The input is voice data, and the output is the voice transmitted to the subject. This transmission allows the subject to hear the response from the generating AI.

[1761] Step 6:

[1762] The server infers the subject's physical condition from the tone of voice and content of the conversation. If necessary, it notifies family members or medical institutions. The input is the subject's tone of voice and content of the conversation, and the output is notification information. This inference and notification enables a rapid response in the event of an emergency.

[1763] Example 2

[1764] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1765] In modern society, elderly people living alone, people with physical disabilities, and those who have lost their partners often feel lonely, which can lead to poor mental health. Furthermore, there is a lack of efficient ways to check on the safety and health of these people. Furthermore, there is a need for a system that allows these people to enjoy conversations with a friendly voice.

[1766] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1767] In this invention, the server is a system that communicates with specific individuals to check their safety, alleviate loneliness through conversation, and also manage their health through tone of voice, and includes: a means for utilizing a voice-imitating conversion system to convert voice to text, create a response using a generative AI model, convert the response back to voice, and communicate in a conversational format; a means for uploading voice data from a terminal to the server; a means for analyzing the uploaded voice data and extracting features of the voice data; a means for inputting the features of the analyzed voice data into the generative AI model; a means for the generative AI model to generate new voice based on the features of the input voice data; and a means for providing the generated voice data to a user. This allows the individual to enjoy a conversation in a friendly voice, reducing loneliness and enabling efficient safety confirmation and health management.

[1768] "Specific target groups" refer to people who are prone to feeling lonely, such as elderly people living alone, people with physical disabilities, and those who have lost their partners, and who need to be checked on for safety and health management.

[1769] "Communications" refers to the means of sending and receiving voice and data via telephone and the Internet.

[1770] "Safety check" refers to the act of checking the safety and health of a specific person.

[1771] "Conversational" refers to a form of two-way communication using voice.

[1772] "Vocal timbre" refers to characteristics such as the quality, tone, and rhythm of a voice.

[1773] "Health management" refers to the act of monitoring the health status of a specific subject and responding as necessary.

[1774] A "voice imitation conversion system" refers to a system that generates new voices based on input voice data.

[1775] A "generative AI model" refers to an artificial intelligence model that generates new data based on input data.

[1776] "Audio data" refers to data that records audio in digital format.

[1777] "Terminal" refers to a device, such as a PC or smartphone, used to upload audio data to a server.

[1778] "Server" refers to the computer system that analyzes voice data and runs generative AI models.

[1779] "Analysis" refers to the process of extracting features from audio data.

[1780] "Features" refers to characteristics of the audio data such as pitch, rhythm, and timbre.

[1781] A "prompt sentence" refers to an instruction sentence to be input into a generative AI model.

[1782] "New voice" refers to a voice generated by a generative AI model that mimics the original voice data.

[1783] "Providing" refers to the act of transmitting the generated voice data to the user.

[1784] This invention is a system that can communicate with specific individuals to check their safety, alleviate loneliness through conversation, and manage their physical condition through tone of voice. This system utilizes a voice mimicking conversion system to convert speech into text, generate responses using a generative AI model, convert the responses into speech, and communicate in a conversational format.

[1785] Generating a Program

[1786] The server generates a program for the voice conversion system, which includes an algorithm for receiving voice data as input and generating a new voice that imitates it.

[1787] Program processing explanation

[1788] The server causes the speech conversion system to perform the following processes.

[1789] 1. Input audio data:

[1790] Users upload audio data of voice actors or loved ones recorded before they died to the server.

[1791] As specific hardware, users use computers and smartphones.

[1792] As for software, a web browser or dedicated application is used to upload audio data.

[1793] 2. Analysis of audio data:

[1794] The server analyzes the uploaded audio data.

[1795] Analysis is performed using voice recognition software (e.g., Google Cloud Speech-to-Text API).

[1796] Extract features of audio data (pitch, rhythm, timbre, etc.).

[1797] 3. Input to the generative AI model:

[1798] The server inputs the extracted features of the voice data into a generative AI model.

[1799] Generative AI models used include OpenAI's GPT-3 and Google's WaveNet.

[1800] 4. Imitation voice generation:

[1801] The generative AI model generates new speech based on the characteristics of the input speech data.

[1802] This new voice mimics the original voice data.

[1803] 5. Output of generated speech:

[1804] The server provides the generated voice data to the user.

[1805] The user can download the generated audio or play it via streaming.

[1806] Adding specific examples

[1807] As a concrete example, the following prompt sentence is input into the generative AI model:

[1808] Example prompt sentence:

[1809] "This voice data is the voice of my deceased grandfather. Please generate a new voice based on this voice."

[1810] Examples:

[1811] The user uploads audio data of their grandfather recorded before his death to the server. The server analyzes the audio data and inputs it into the generative AI model. The generative AI model generates a new voice that imitates the grandfather's voice, and the server provides that voice to the user. The user can listen to the generated voice and enjoy a conversation with their grandfather.

[1812] In this way, a specific target person can enjoy a conversation in a friendly voice, reducing feelings of loneliness and enabling efficient safety confirmation and health management.

[1813] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1814] Step 1:

[1815] Inputting audio data

[1816] Users upload audio data of voice actors or loved ones recorded before they passed away from their device (PC or smartphone) to the server.

[1817] Specific operation: The user uses a web browser or dedicated application to select an audio file and click the upload button.

[1818] Input: Audio data file (e.g. .wav, .mp3)

[1819] Output: Audio data stored on the server

[1820] Step 2:

[1821] Analysis of audio data

[1822] The server receives the uploaded voice data and analyzes it using voice recognition software (e.g., Google Cloud Speech-to-Text API).

[1823] Specific operation: The server extracts characteristics of the audio data, such as pitch, rhythm, and timbre, and stores them in a database.

[1824] Input: Audio data stored on the server

[1825] Output: Extracted audio features (e.g., pitch, rhythm, timbre)

[1826] Step 3:

[1827] Input to generative AI models

[1828] The server inputs the features of the analyzed voice data into a generative AI model (e.g., OpenAI's GPT-3 or Google's WaveNet).

[1829] Specific operation: The server sends the features of the voice data to the generative AI model as a prompt sentence.

[1830] Input: Extracted audio features

[1831] Output: The prompt input to the generative AI model

[1832] Step 4:

[1833] Imitation voice generation

[1834] The generative AI model generates new speech based on the characteristics of the input speech data.

[1835] How it works: The generative AI model analyzes the characteristics of the voice data and generates imitation voices based on that.

[1836] Input: The prompt given to the generative AI model

[1837] Output: Generated imitation voice data

[1838] Step 5:

[1839] Generated speech output

[1840] The server provides the generated voice data to the user.

[1841] Specific operation: The server sends the generated audio data to the user's device, where the user can download it or play it via streaming.

[1842] Input: Generated imitation voice data

[1843] Output: The audio data provided to the user (e.g., download link, streaming URL)

[1844] (Application example 2)

[1845] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1846] In modern society, elderly people living alone, people with physical disabilities, and those who have lost their partners are prone to feeling lonely and have difficulty managing their health. Furthermore, food delivery services also face the issue of not being able to provide guidance or confirmation in a friendly voice, which hinders the user experience.

[1847] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for utilizing a conversion system that mimics a human voice, converting speech to text, creating a response using a generation AI, converting the response to speech, and communicating in a conversational format, as well as a means for providing guidance and confirmation in a friendly voice. This helps elderly people living alone, people with physical disabilities, and those who have lost their partners to feel less lonely and allows them to manage their health. Furthermore, in food delivery services, users can receive guidance and confirmation in a friendly voice, improving the user experience.

[1848] "Specific eligible individuals" refer to individuals who meet certain conditions, such as elderly people living alone, people with physical disabilities, and those who have lost their partners.

[1849] "Communications" refers to the means of sending and receiving voice and data via telephone and the Internet.

[1850] "Safety check" refers to the act of checking the health and safety of a specific person.

[1851] "Vocal timbre" refers to the characteristics and tone of a voice, which can be used to determine the speaker's emotions and physical condition.

[1852] "Health management" refers to monitoring the health status of a specific subject and taking appropriate measures as necessary.

[1853] "Conversion system" refers to technology that converts speech to text and text to speech.

[1854] "Generative AI" refers to technology that uses artificial intelligence to generate appropriate answers from text data.

[1855] A "familiar voice" refers to a voice that is familiar to or well-liked by the target audience.

[1856] "Guidance and confirmation" refers to actions to provide information to the user or confirm their actions.

[1857] An "audio professional" refers to someone whose job involves working exclusively with audio, such as a voice actor or narrator.

[1858] "Immediate family" refers to family members related by blood.

[1859] The following system configuration will be described as an embodiment of the present invention.

[1860] System Configuration

[1861] This system consists of three main components: a server, a terminal, and a user. The server includes a speech conversion system and a generative AI, and the terminal is a communication device such as a smartphone or tablet. The user is a specific target group, such as elderly people living alone, people with physical disabilities, or those who have lost their partner.

[1862] Program processing

[1863] The server uses a speech-to-text system to convert the user's speech into text and inputs that text into the generation AI. The generation AI generates an appropriate answer based on the input text and converts that answer back into speech. The converted speech is used to provide guidance and confirmation to the user in a friendly voice.

[1864] Hardware and software used

[1865] Hardware: Servers, smartphones, tablets

[1866] Software: Python, pyttsx3 (speech synthesis library), soundfile (audio file reading library), scipy (scientific calculation library)

[1867] Data processing and calculation

[1868] 1. Reading audio data: The server reads the user's audio data using the soundfile library.

[1869] 2. Audio data conversion: The loaded audio data is converted into text using a speech conversion system.

[1870] 3. Answer generation using generative AI: Text data is input into a generative AI model to generate appropriate answers.

[1871] 4. Re-conversion to speech: The generated text response is converted to speech using the pyttsx3 library.

[1872] 5. Play audio: The converted audio is sent to the user's device, providing guidance and confirmation in a friendly voice.

[1873] Specific examples

[1874] For example, if a user is ordering a pizza from a food delivery app, hearing their deceased mother's voice ask, "Are you sure you want to confirm your order?" before confirming the order can create a sense of familiarity and security for the user.

[1875] Prompt Sentence Examples

[1876] "In your food delivery app, implement a friendly voice confirmation feature before the user confirms their order. Use pre-recorded voice data and translate it into the target voice for playback."

[1877] In this way, the embodiments for carrying out the invention have been specifically shown.

[1878] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1879] Step 1:

[1880] The server receives the user's voice data. The input is the user's voice data, and the output is the voice data stored in the server. Specifically, the server reads the voice data sent from the smartphone or tablet using the soundfile library.

[1881] Step 2:

[1882] The server converts the voice data into text using a speech conversion system. The input is voice data and the output is text data. Specifically, the server analyzes the voice data using speech recognition technology and generates corresponding text data.

[1883] Step 3:

[1884] The server inputs text data into the generative AI model and generates an appropriate answer. The input is text data, and the output is the generated answer text. Specifically, the server inputs a prompt sentence into the generative AI model and generates an appropriate answer to the user's question or request.

[1885] Step 4:

[1886] The server converts the generated answer text into speech. The input is the generated answer text, and the output is speech data. Specifically, the server converts text data into speech data using the pyttsx3 library.

[1887] Step 5:

[1888] The server sends the converted voice data to the user's device. The input is the voice data, and the output is the voice data sent to the user's device. Specifically, the server uses a communication protocol to send the voice data to the user's smartphone or tablet.

[1889] Step 6:

[1890] The user's device plays the received voice data. The input is the voice data sent from the server, and the output is the voice the user hears. Specifically, the user's device uses its built-in voice playback function to play the voice data and provides guidance and confirmation in a friendly voice.

[1891] Example 3

[1892] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1893] There is a need to reduce the amount of time that people who are prone to social isolation, such as elderly people living alone, people with physical disabilities, and those who have lost their partners, feel lonely and provide them with a sense of security. It is also important to regularly check on the safety of these people and manage their health. However, there is not yet a sufficient system in place to implement this efficiently and effectively.

[1894] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1895] In this invention, the server includes means for calling specific subjects to check on their safety and alleviate loneliness through conversation, and also for managing their physical condition through tone of voice, means for collecting subject information and storing it in a database, means for setting a call schedule based on the subject's lifestyle and preferences, means for automatically making calls based on the set schedule, and means for adjusting the schedule and updating the database in response to the subject's feedback and changes in lifestyle. This makes it possible to reduce the amount of time the subject feels lonely and provide a sense of security.

[1896] "Specific target groups" refer to people who are prone to social isolation, such as elderly people living alone, people with disabilities, and those who have lost their partners.

[1897] "Safety check" refers to phone calls or conversations to check on the health and safety of the person in question.

[1898] "Health management through vocal tone" refers to analyzing the tone and quality of a person's voice to understand their physical and mental state.

[1899] A "conversion system" refers to a system that mimics a human voice, converts the speech into text, creates an answer using generative AI, and then converts it back into speech.

[1900] "Generative AI" refers to a system that uses artificial intelligence technology to generate appropriate answers based on input text data.

[1901] "Database" refers to an information system for systematically storing and managing information on subjects.

[1902] A "call schedule" refers to a plan for making calls at the optimal time based on the target person's lifestyle and preferences.

[1903] "Feedback" refers to opinions and reactions obtained from subjects, and is information used to improve and adjust the system.

[1904] "Means for making automatic phone calls" refers to the ability of the system to automatically make phone calls based on a set schedule.

[1905] "Schedule adjustment" refers to changing the time and frequency of calls based on the target person's lifestyle and feedback.

[1906] MODE FOR CARRYING OUT THE INVENTION

[1907] This invention is a system that provides a sense of security and alleviates loneliness by making regular phone calls to people who are prone to social isolation, such as elderly people living alone, people with physical disabilities, and those who have lost their partners. A specific embodiment of this system will be described below.

[1908] 1. Program Generation

[1909] The server generates a program that collects information about the target person and stores it in a database. This program sets a call schedule based on the target person's lifestyle and preferences, and automatically makes calls based on the set schedule.

[1910] 2. Program Processing

[1911] The server provides a web form to collect information about the subjects, which users use to enter information such as the subjects' names, phone numbers, lifestyles, preferences, etc. The collected information is stored in a MySQL database.

[1912] The server then uses a Python script to schedule calls for each individual. The script calculates the best time to call based on the individual's lifestyle and preferences. For example, if a person prefers to be called in the morning, the server will schedule the script to call at that time.

[1913] The server uses the Twilio API to automatically make calls based on a set schedule. The time of the call is adjusted to suit the recipient's lifestyle and preferences. For example, if recipient A wants to be called at 10:00 a.m. every day, the server will make the call at that time.

[1914] The server also receives feedback from the subject and adjusts the schedule as needed: for example, if the subject decides they prefer to be called in the afternoon, the information in the database is updated and the schedule is rescheduled.

[1915] 3. Examples and prompts

[1916] As a concrete example, consider the following scenario.

[1917] Subject A (an elderly person living alone) requests a phone call every day at 10:00 a.m.

[1918] Subject B (a physically disabled person) requests phone calls three times a week at 2:00 p.m.

[1919] The server stores this information in a database and uses a Python script to schedule calls, which are then automatically placed at the scheduled time using the Twilio API.

[1920] Example prompts to input to a generative AI model:

[1921] "Please schedule a call to elderly person A who lives alone every day at 10:00 AM."

[1922] "Schedule a call with Person B, who is disabled, three times a week at 2:00 PM."

[1923] This system can reduce the amount of time that a subject feels lonely and provide a sense of security. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1924] Step 1:

[1925] The user uses a web form to enter information about the target person, including their name, phone number, lifestyle, preferences, etc. This information is then sent to the server.

[1926] Input: Subject's name, phone number, lifestyle, preferences

[1927] Output: Information of the target person sent to the server

[1928] Step 2:

[1929] The server stores the received information in a MySQL database by inserting the information into the appropriate table.

[1930] Input: Information about the target person sent by the user

[1931] Output: Subject information stored in a database

[1932] Step 3:

[1933] The server schedules calls for each individual using a Python script that calculates the best time to call based on the individual's lifestyle and preferences.

[1934] Input: Subject information stored in the database

[1935] Output: The configured call schedule

[1936] Step 4:

[1937] The server uses the Twilio API to automatically place calls based on a set schedule, with call times tailored to the recipient's lifestyle and preferences.

[1938] Input: Configured call schedule

[1939] Output: Calls made to the target person

[1940] Step 5:

[1941] The server receives feedback from the subject and adjusts the schedule as needed: for example, if the subject decides they prefer a call in the afternoon, it updates the database and reschedules the call.

[1942] Input: Feedback from subjects

[1943] Output: Adjusted call schedule and updated database

[1944] In this way, the system can reduce the amount of time a subject feels lonely and provide a sense of security.

[1945] (Application example 3)

[1946] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1947] Elderly people living alone, people with physical disabilities, and those who have lost their partners often feel lonely and anxious. Furthermore, these individuals may not receive appropriate support in emergencies. Conventional safety confirmation systems struggle to respond flexibly to individuals' lifestyles and preferences, and are insufficient to provide a sense of security. Furthermore, they lack the functionality to analyze individuals' responses and notify emergency contacts as necessary, potentially delaying emergency response.

[1948] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1949] In this invention, the server includes means for calling specific individuals to check on their safety and alleviate loneliness through conversation, and also for managing their physical condition through tone of voice, means for creating a list of individuals and setting a call schedule based on their lifestyle and preferences, means for automatically making calls based on the set schedule, means for analyzing the individual's responses and notifying an emergency contact as necessary, and means for providing the individual with a sense of security through regular calls. This reduces the individual's sense of loneliness, provides a sense of security, and enables a rapid response in emergencies.

[1950] "Specific target groups" refer to individuals who meet certain conditions, such as elderly people living alone, people with physical disabilities, and those who have lost their partners.

[1951] "Safety check" refers to the act of checking whether a person is safe.

[1952] "Health management through voice tone" refers to a method of analyzing the tone and quality of a person's voice to detect changes or abnormalities in their physical condition.

[1953] "Conversion system" refers to a system that mimics the human voice and converts speech into text.

[1954] "Generative AI" refers to technology that uses artificial intelligence to generate appropriate answers from text.

[1955] "Target list" refers to a list that compiles information about specific targets.

[1956] "Scheduling" refers to the act of setting a time to call based on the target person's lifestyle and preferences.

[1957] "Means for making automatic phone calls" refers to the ability to make automatic phone calls based on a set schedule.

[1958] "Means for analyzing responses" refers to the function of analyzing the subject's telephone responses and notifying emergency contacts as necessary.

[1959] "Means of providing a sense of security" refers to the function of providing a sense of security to the target person through regular phone calls.

[1960] The following system configuration is used as an embodiment of the present invention.

[1961] System Configuration

[1962] 1. Hardware:

[1963] Server: Processes data and runs generative AI models.

[1964] Device: A smartphone or robot that makes phone calls to the subject.

[1965] Communications infrastructure: Internet connection and telephone lines.

[1966] 2. Software:

[1967] Generative AI model: Generates appropriate answers based on the subject's responses.

[1968] Speech-to-text system: imitates the human voice and converts speech to text.

[1969] Scheduling system: Schedule calls to suit your target audience's lifestyle and preferences.

[1970] Response analysis system: Analyzes the subject's response and notifies emergency contacts as necessary.

[1971] Program processing explanation

[1972] The server creates a list of contacts and sets a call schedule based on their lifestyles and preferences. The device automatically makes calls based on the set schedule. During the call, the voice conversion system converts the contact's voice into text, and the generative AI model generates an appropriate response. The generated text is then converted into voice and sent back to the contact. The response analysis system analyzes the contact's response and notifies emergency contacts if necessary.

[1973] Specific examples

[1974] For example, if Tanaka, an elderly person living alone, is registered on the target list, the server sets a schedule to call Tanaka every day at 10:00 to suit his lifestyle. At 10:00, the device automatically calls Tanaka. When Tanaka answers the phone, the speech conversion system converts Tanaka's voice into text, and the generative AI model generates an appropriate response such as "How are you spending your day?" This text is then converted into speech and responded to Tanaka. If an abnormality is detected in Tanaka's response, the response analysis system notifies the emergency contact.

[1975] Prompt Sentence Examples

[1976] "Develop a system for making regular phone calls to elderly people living alone, people with disabilities, and those who have lost their partners. Create a list of people to call and schedule calls based on their lifestyles and preferences. Calls should provide reassurance and include the ability to notify emergency contacts if necessary."

[1977] In this way, it is possible to provide a specific embodiment of the invention.

[1978] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1979] Step 1:

[1980] The server creates a target list.

[1981] Input: Subject's personal information (name, phone number, lifestyle, preferences, etc.)

[1982] Data processing: The subject's information is registered in a database and managed in list format.

[1983] Output: List of target users

[1984] Step 2:

[1985] The server schedules calls based on the recipient's lifestyle and preferences.

[1986] Input: List of subjects, information on their lifestyle and preferences

[1987] Data calculation: Calculate the optimal time to call each target person and set a schedule.

[1988] Output: Call Schedule

[1989] Step 3:

[1990] The device will automatically make calls based on a set schedule.

[1991] Input: Call Schedule

[1992] Specific actions: Call the target person according to the schedule.

[1993] Output: Start of phone call

[1994] Step 4:

[1995] The terminal converts the target person's voice into text using a speech conversion system.

[1996] Input: Subject's voice

[1997] Data processing: Using a voice conversion system, convert the voice into text data.

[1998] Output: Text data

[1999] Step 5:

[2000] The server uses a generative AI model to generate appropriate answers from the text data.

[2001] Input: Text data

[2002] Data Computation: Using generative AI models to generate answers based on text data.

[2003] Output: Generated answer text

[2004] Step 6:

[2005] The terminal converts the generated answer text into speech and responds to the subject.

[2006] Input: Generated answer text

[2007] Data processing: Using a speech conversion system, convert the text into audio data.

[2008] Specific operation: Play audio data to the target person.

[2009] Output: Voice response to the target person

[2010] Step 7:

[2011] The server analyzes the subject's response and notifies emergency contacts if necessary.

[2012] Input: Subject's response text data

[2013] Data calculation: Using the response analysis system, the response content is analyzed, and if an abnormality is detected, an emergency contact is notified.

[2014] Specific behavior: Send a notification to emergency contacts.

[2015] Output:Notify emergency contacts

[2016] In this way, each processing step can be concretely explained.

[2017] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2018] "Example 1"

[2019] One embodiment of the present invention is a system that incorporates an emotion engine. This system makes a phone call to a specific target person and engages in a conversation. The conversation is converted from speech to text using a conversion system that mimics the human voice, and a generation AI creates a response. The created response is converted back into speech and transmitted to the target person via phone. At this time, the emotion engine analyzes the target person's emotions from their tone of voice and speaking style, and feeds the results back to the generation AI. Based on this feedback, the generation AI generates a conversation that is adapted to the target person's emotions.

[2020] "Example 2"

[2021] As a specific example, when the target person is sad, the emotion engine detects that sadness and provides feedback to the generation AI. Based on this information, the generation AI generates conversations that comfort the target person. Similarly, when the target person is happy, it generates conversations that share that joy. This allows the system to provide appropriate dialogue that corresponds to the target person's emotions.

[2022] "Example 3"

[2023] Furthermore, the emotion engine can analyze emotions not only from the subject's tone of voice and speaking style, but also from the content of the conversation. For example, if the subject says, "I had ...

Claims

1. A means for confirming the safety and health status of a specific subject by calling the specific subject and confirming the response of the subject; A means for detecting abnormalities in the physical condition of the subject based on changes in the subject's voice by analyzing the tone and quality of the subject's voice; an emotion engine that recognizes the emotion of the subject by inputting the voice and facial expression of the subject; A means for generating a dialogue appropriate to the emotion by inputting a specific prompt sentence into a generative AI model; a providing means for converting the generated dialogue into an imitation voice and providing the imitation voice to the subject; A system including:

2. The system of claim 1 , wherein the imitation voice comprises a voice that the subject is familiar with or has a favorable impression of.

3. The system of claim 1 , wherein the providing means converts the generated dialogue into an imitation voice using voice data of a voice actor or a relative.

Images