system

Abstract

To provide a system configured so that an elderly people can operate an IT device with a dialect as usual.SOLUTION: A system includes: voice recognition means which recognizes a voice input obtained when an elderly person operates a smartphone; means which identifies a dialect that the elderly person speaks, based on the recognized voice input; means which generates an operation instruction for the smartphone, using a generative AI and a prompt text for generating the operation instruction for the smartphone, based on the recognized voice input and the identified dialect; and means for operating the smartphone following the generated operation instruction for the smartphone.SELECTED DRAWING: Figure 1

Description

Technical Field

[0005] ,

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In Japan's super - aging society, when elderly people operate IT devices by themselves, operation completion only by voice recognition is required. However, there are hundreds to thousands of dialects in Japan, and there is no voice recognition system that can handle all of them. Therefore, not all elderly people can operate IT devices in their usual dialects.

Means for Solving the Problems

[0005] Provide a system that combines voice recognition means and generative AI means. This system has learning means for corresponding to all dialects in Japan, and particularly conducts learning of the Tsugaru dialect, which is said to be the most difficult dialect in Japan. As a result, the practical application of voice recognition becomes possible, and all elderly people can operate IT devices in their usual dialects. [Brief explanation of the drawing]

[0006] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Embodiment 1 of Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1 of Form Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2 of Embodiment 2. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2 of Form Example 2. [Figure 15] This is a sequence diagram showing the processing flow of the data processing system in Embodiment 3 of Example 3. [Figure 16] It is a sequence diagram showing the processing flow of the data processing system in Application Example 3 of Form Example 3. [Figure 17] It is a sequence diagram showing the processing flow of the data processing system in Example 1 of Form Example 1 when combined with an emotion engine. [Figure 18] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1 of Form Example 1 when combined with an emotion engine. [Figure 19] It is a sequence diagram showing the processing flow of the data processing system in Example 2 of Form Example 2 when combined with an emotion engine. [Figure 20] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 of Form Example 2 when combined with an emotion engine. [Figure 21] It is a sequence diagram showing the processing flow of the data processing system in Example 3 of Form Example 3 when combined with an emotion engine. [Figure 22] It is a sequence diagram showing the processing flow of the data processing system in Application Example 3 of Form Example 3 when combined with an emotion engine.

Embodiments for Carrying out the Invention

[0007] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0008] First, the language used in the following description will be explained.

[0009] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)), etc.

[0010] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0011] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0012] In the following embodiments, the numbered communication I / F (Interface) is an interface that includes a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0014] [First Embodiment]

[0015] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0016] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0019] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0022] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0026] Next, the identification process performed by the identification processing unit 290 of the data processing device 12 will be described.

[0027] "Example of form 1"

[0028] The present invention is a system that combines speech recognition means and generative AI means. The speech recognition means recognizes the voice input when an elderly person operates an IT device. The generative AI means generates operations based on the recognized voice input. Furthermore, this system has learning means to support all Japanese dialects. Specifically, by training the system on Tsugaru dialect, which is said to be the most difficult dialect in Japan, the practical application of speech recognition has been achieved.

[0029] "Example of form 2"

[0030] As a specific embodiment, for example, if an elderly person says "Turn on the TV" in the Tsugaru dialect, the voice recognition means recognizes this voice input, and the generative AI means generates an operation to turn on the TV. This operation is realized, for example, by controlling a device that generates infrared signals.

[0031] "Example of form 3"

[0032] Furthermore, this system possesses learning capabilities to handle all Japanese dialects. Specifically, it uses a large amount of Tsugaru dialect audio data to train its speech recognition and generative AI systems. This allows it to handle not only Tsugaru dialect but also other dialects using the same method.

[0033] The following describes the processing flow for each example of the form.

[0034] "Example of form 1"

[0035] Step 1: Enable voice input for elderly individuals to operate IT devices. This voice input includes instructions such as "Turn on the TV."

[0036] Step 2: The speech recognition system recognizes the voice input of the elderly person. This speech recognition is performed using a learning method that can handle all Japanese dialects.

[0037] Step 3: The generative AI system generates an action based on the voice input it recognizes. This action may include a specific action such as turning on the TV.

[0038] "Example of form 2"

[0039] Step 1: The elderly person says "Turn on the TV" in Tsugaru dialect.

[0040] Step 2: The speech recognition system recognizes this voice input. This speech recognition is performed with high accuracy as a result of training on the Tsugaru dialect.

[0041] Step 3: The generative AI system generates the operation to turn on the television. This operation is achieved by controlling a device that generates infrared signals.

[0042] "Example of form 3"

[0043] Step 1: Train speech recognition and generative AI using a large amount of Tsugaru dialect audio data.

[0044] Step 2: Using the trained speech recognition and generative AI systems, the system recognizes the elderly person's voice input and generates commands.

[0045] Step 3: Execute the generated operation. This operation may include specific actions such as turning on the TV.

[0046] (Example 1)

[0047] Next, we will describe Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0048] When elderly people operate information technology devices, there is a need to improve ease of use by using voice input. However, voice recognition systems that can handle Japan's diverse dialects, especially difficult dialects like Tsugaru dialect, have not yet been put into practical use. Therefore, there is a need to develop a voice recognition system that allows elderly people to speak naturally in their own dialect.

[0049] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means. In this invention, the server includes a speech recognition means, a generative artificial intelligence means, and a learning means for supporting all Japanese dialects. This enables elderly people to speak naturally in their own dialect and to operate information technology devices using voice input.

[0050] "Speech recognition means" refers to technology for converting speech data into text data.

[0051] "Generative artificial intelligence means" refers to artificial intelligence technology for generating appropriate operation instructions based on input data.

[0052] "Learning methods to handle all Japanese dialects" refers to a technology that pre-trains a speech recognition system to recognize the diverse dialects used within Japan.

[0053] "Methods for learning Tsugaru dialect" refers to a technology that specially trains a speech recognition system to recognize Tsugaru dialect, which is said to be the most difficult dialect in Japan.

[0054] "Audio data preprocessing means" refers to technologies that improve the accuracy of speech recognition by performing noise reduction and volume normalization on audio data.

[0055] "Means for the practical application of speech recognition" refers to technologies that enable the application of speech recognition technology to actual systems and applications, making it practically usable.

[0056] "Voice input when elderly people operate information technology devices themselves" refers to voice commands that elderly people utter to operate information technology devices themselves.

[0057] This invention is a system for improving the convenience of elderly people when operating information technology devices using voice input. This system includes voice recognition means, generative artificial intelligence means, learning means for handling all Japanese dialects, means for learning Tsugaru dialect, voice data preprocessing means, and means for putting voice recognition into practical use.

[0058] System Configuration

[0059] Speech recognition means

[0060] The server uses speech recognition software to convert speech data into text data. For example, Google's Cloud Speech-to-Text API is used for this purpose. This API is pre-trained to handle a wide variety of Japanese dialects.

[0061] Audio data preprocessing means

[0062] The server uses audio processing libraries such as Librosa to remove noise and normalize the volume of the audio data. This makes the audio data clearer and easier to understand.

[0063] Generative artificial intelligence methods

[0064] The server generates operation instructions based on text data using a generative artificial intelligence model (e.g., OpenAI®'s GPT-4®). This generative AI model is pre-trained on a variety of operation scenarios to understand user intent and generate appropriate operation instructions.

[0065] Execute the operation

[0066] The device executes the operation instructions it receives from the server. For example, it might open an email app and send an email with the specified content.

[0067] Specific example

[0068] Example 1: Sending an email using voice input

[0069] The user uses voice input to say, "Send me an email."

[0070] 1. The server uses speech recognition software to convert speech into text.

[0071] 2. The server uses Librosa to perform noise reduction and volume normalization on the audio data.

[0072] 3. The server uses a generative artificial intelligence model to generate the command "send an email."

[0073] 4. The device opens the email app and sends an email with the specified content.

[0074] Example 2: Obtaining weather information via voice input

[0075] The user voice-inputs, "What's the weather like today?"

[0076] 1. The server uses speech recognition software to convert speech into text.

[0077] 2. The server uses Librosa to perform noise reduction and volume normalization on the audio data.

[0078] 3. The server uses a generative artificial intelligence model to generate the operation instruction "Retrieve weather information".

[0079] 4. The device opens a weather app and displays the current weather information.

[0080] Example of a prompt

[0081] "Design a system that recognizes voice input from elderly people speaking in Tsugaru dialect and generates appropriate actions. Specifically, explain how to combine voice recognition software and a generative artificial intelligence model to generate actions based on voice input."

[0082] In this way, this system can improve the convenience for elderly people when operating information technology devices using voice input.

[0083] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0084] Step 1:

[0085] The user uses voice input to enter information into the terminal. For example, they might say, "Send me an email." This voice input becomes the initial input for the system.

[0086] Step 2:

[0087] The terminal sends the acquired audio data to the server. The server uses an audio processing library such as Librosa to remove noise and normalize the volume of the audio data. This makes the audio data clearer and easier to recognize. The input is raw audio data, and the output is pre-processed audio data.

[0088] Step 3:

[0089] The server sends pre-processed audio data to speech recognition software (e.g., Google Cloud Speech-to-Text API) to convert the audio to text. The input is pre-processed audio data, and the output is text data. For example, the text "Send me an email" is generated.

[0090] Step 4:

[0091] The server uses a generative artificial intelligence model (e.g., OpenAI's GPT-4) to generate operation instructions based on text data. The input is text data obtained through speech recognition, and the output is a specific operation instruction. For example, the operation instruction "send an email" is generated.

[0092] Step 5:

[0093] The terminal executes the operation instructions received from the server. For example, it might open an email application and send an email with specified content. The input is the operation instructions from the server, and the output is the actual result of the operation.

[0094] In this way, the system can improve the convenience for elderly people when operating information technology devices using voice input.

[0095] (Application Example 1)

[0096] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server," and the smart device 14 will be referred to as a "terminal."

[0097] Elderly people face challenges in finding products or asking questions in physical stores because they have difficulty operating IT equipment and obtaining appropriate information. Furthermore, there is a lack of voice recognition systems that support all Japanese dialects, and there is a particular need for systems that can handle complex dialects.

[0098] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0099] In this invention, the server includes: a voice recognition means for recognizing voice input when an elderly person operates a smartphone themselves; a means for identifying the dialect spoken by the elderly person based on the recognized voice input; a means for generating operation instructions for the smartphone using a prompt statement and a generative AI based on the recognized voice input and the identified dialect; and a means for operating the smartphone according to the generated operation instructions. Furthermore, the voice recognition means further includes a means for recognizing voice input when the elderly person asks a question about a product in a physical store, and the server further includes a means for generating an answer to the question using a prompt statement and a generative AI based on the recognized voice input and the identified dialect. This makes it possible to create a system that allows elderly people to easily operate a physical store using voice input when searching for products or asking questions.

[0100] "Speech recognition means" refers to technology for converting speech into text.

[0101] "Generative AI methods" refer to artificial intelligence technologies that generate appropriate responses or operations based on input data.

[0102] "A learning method for handling all Japanese dialects" refers to a technology that learns the data necessary to understand and recognize dialects from various regions of Japan.

[0103] "Methods for learning Tsugaru dialect" refers to techniques for learning the data necessary to understand and recognize Tsugaru dialect, which is said to be the most difficult dialect in Japan.

[0104] "Means for the practical application of speech recognition" refers to technologies for applying speech recognition technology to actual applications and systems.

[0105] "A means of recognizing voice input used by elderly people when searching for products or asking questions in physical stores, and generating appropriate operations or responses" refers to a technology that recognizes the voice input used by elderly people when searching for products or asking questions in physical stores, and generates appropriate operations or responses.

[0106] "Applications installed on smartphones" refers to software installed on a smartphone to provide specific functions.

[0107] A system for carrying out this invention includes a voice recognition means, a generative AI means, a means for identifying all Japanese dialects, a means for learning Tsugaru dialect, a means for putting voice recognition into practical use, a means for recognizing voice input when elderly people search for products or ask questions in a physical store and generating appropriate operations or answers, and an application means installed on a smartphone.

[0108] System program

[0109] This system operates using the following hardware and software:

[0110] Hardware: Smartphone (with microphone)

[0111] Software: Python, SpeechRecognition library, Transformers library (Hugging Face GPT-3® model)

[0112] Processing flow

[0113] Speech recognition

[0114] When a user speaks into their smartphone's microphone, a speech recognition system captures the audio and converts it into text. The SpeechRecognition library is used for this process.

[0115] Generative AI

[0116] The speech data, converted into text by the speech recognition system, is sent to the generative AI. The generative AI uses the Transformers library and a GPT-3 model to generate appropriate operations and responses.

[0117] output

[0118] The generated actions and responses are displayed on the smartphone screen. This allows elderly people to easily use voice input when searching for products or asking questions in physical stores.

[0119] Specific example (smartphone operation)

[0120] When an elderly person says "Send an email" to their smartphone, the application recognizes the voice, and the generative AI identifies that it is in the Tsugaru dialect. It then generates instructions to launch the email sending app, which the application then launches according to those instructions. The generative AI also generates a response such as "Please enter the message you want to send," and the application displays that response.

[0121] Example of a prompt

[0122] "An elderly person is using voice input to say 'send email.' This is in Tsugaru dialect. Please generate instructions for operating a smartphone."

[0123] In this way, a system can be realized that supports elderly people in making it easier for them to operate smartphones.

[0124] Specific example (physical store)

[0125] When an elderly person speaks to their smartphone in a physical store and asks, "Where is product A?", the application recognizes the voice, and a generative AI identifies that it is in the Tsugaru dialect. It then generates and displays a specific answer such as, "Product A is on the left side of aisle 3."

[0126] Example of a prompt

[0127] "An elderly person is voice-inputting, 'Where is product A?' This is in Tsugaru dialect. Please generate an answer to this question."

[0128] Generative AI's answer: The product is located on the left side of aisle 3.

[0129] In this way, a system can be realized that supports elderly people in making shopping at physical stores easier.

[0130] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0131] Step 1:

[0132] The user speaks into the microphone of their smartphone. The input is the user's voice, and the output is audio data. For example, the user might say, "Where can I find this product?"

[0133] Step 2:

[0134] The device uses speech recognition to convert audio data into text. The input is audio data, and the output is text data. Specifically, the SpeechRecognition library is used to convert the audio into the text "Where is product A located?".

[0135] Step 3:

[0136] The device sends text data to the generative AI. The input is text data, and the output is input data to the generative AI. Specifically, the converted text "Where is product A located?" is sent to the generative AI.

[0137] Step 4:

[0138] The server uses generative AI to generate appropriate instructions or responses based on text data. The input is text data, and the output is the generated instructions or responses. Specifically, the Transformers library is used to generate the response "Product A is on the left side of aisle 3" using a GPT-3 model.

[0139] Step 5:

[0140] The server sends the generated instructions and responses to the terminal. The input is the generated instructions and responses, and the output is the data sent to the terminal. Specifically, the generated response "Product A is on the left side of aisle 3" is sent to the terminal.

[0141] Step 6:

[0142] The device operates according to the generated instructions and displays the generated response to the user. Input is the transmitted data, and output is the data displayed to the user. As a specific example, the response "Product A is on the left side of aisle 3" is displayed on the smartphone screen.

[0143] Furthermore, an emotion engine, as described later, may be used to recognize the emotional state of the elderly person, and smartphone operation instructions and answers to questions may be generated taking the elderly person's emotional state into further consideration. In this case, the server further includes means for recognizing the emotional state of the elderly person, and the generating means generates the smartphone operation instructions using a prompt statement for generating smartphone operation instructions based on the recognized voice input, the identified dialect, and the recognized emotional state of the elderly person, and the generation AI.

[0144] (Example 2)

[0145] Next, we will describe Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0146] When elderly people operate information technology devices, there is a challenge in accurately recognizing and generating appropriate commands, especially for voice input using dialects. Furthermore, there is a lack of voice recognition technology capable of handling specific dialects, such as the difficult Tsugaru dialect. Therefore, there is a need for systems that can accurately handle voice input using these dialects.

[0147] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0148] In this invention, the server includes a speech recognition means, a generative artificial intelligence means, a dialect-compatible learning means, a means for learning a specific dialect, a means for putting speech recognition into practical use, a means for converting speech input into text data, a means for transmitting the text data to a generative artificial intelligence model, a means for receiving generated operations, and a means for controlling an infrared signal generator. This makes it possible to accurately recognize speech and generate appropriate operations even when elderly people use dialects to operate information technology devices.

[0149] "Speech recognition means" refers to a device or software for analyzing speech input and converting it into text data.

[0150] "Generative artificial intelligence means" refers to an artificial intelligence model that generates appropriate operations or responses based on input data.

[0151] A "dialect-compatible learning method" is a method for training a speech recognition model to handle dialects from different regions.

[0152] "Methods for learning specific dialects" refer to methods for training speech recognition models specifically on difficult dialects such as Tsugaru dialect.

[0153] "Means for the practical application of speech recognition" refers to methods for integrating speech recognition technology into actual systems and devices to make it usable.

[0154] "Means for converting voice input into text data" refers to means for converting voice input into text format using voice recognition means.

[0155] "Means for sending text data to a generative artificial intelligence model" refers to means for sending converted text data to a generative artificial intelligence model.

[0156] "Means for receiving generated operations" refers to means for receiving operations generated from a generative artificial intelligence model.

[0157] "Means for controlling an infrared signal generating device" refers to means for controlling a device that generates infrared signals and performing a specific operation.

[0158] This invention is a system that accurately recognizes speech and generates appropriate commands when elderly people operate information technology devices using their local dialect. A specific embodiment of this system is described below.

[0159] First, the user speaks in Tsugaru dialect, saying "Turn on the TV." The microphone built into the device captures this voice. Voice recognition software (for example, a voice recognition API) is used as the voice recognition method. This software analyzes the captured voice and converts it into text data.

[0160] Next, the terminal sends the converted text data to a generative artificial intelligence model (for example, a generative AI model). This generative AI model generates appropriate operations based on the input text data. Specifically, prompt statements like the following are generated.

[0161] Example of a prompt:

[0162] The user said "Turn on the TV" in Tsugaru dialect. Generate an operation to create an infrared signal to turn on the TV.

[0163] Based on this prompt, the server uses a generative artificial intelligence model to generate specific commands for turning on the television. These generated commands are then sent to the terminal.

[0164] The terminal controls an infrared signal generator (for example, an infrared signal generator) based on the received operation. This device emits an infrared signal towards the television, and the television receives this signal and turns on.

[0165] As a concrete example, when a user speaks "Turn on the TV" in Tsugaru dialect, the microphone built into the device captures this audio, and a speech recognition API is used to convert the audio into text. This text data is sent to a generative AI model, which generates the following prompt.

[0166] Example of a prompt:

[0167] The user said "Turn on the TV" in Tsugaru dialect. Generate an operation to create an infrared signal to turn on the TV.

[0168] Based on this prompt, the server uses a generative AI model to generate specific instructions for turning on the TV and sends them to the terminal. The terminal controls the infrared signal generator, and the TV receives the infrared signal and turns on.

[0169] In this way, even when users operate information technology devices using dialects, it becomes possible to accurately recognize their speech and generate appropriate commands.

[0170] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0171] Step 1:

[0172] The user performs voice input.

[0173] The user speaks in Tsugaru dialect, saying "Turn on the TV." The input is the user's voice, and the output is audio data captured by the device's microphone.

[0174] Step 2:

[0175] The device recognizes the voice.

[0176] The microphone built into the device captures the user's voice. A speech recognition API is used as the means of speech recognition. The input is the captured audio data, and the output is the parsed text data.

[0177] Step 3:

[0178] The device converts the recognized speech into text.

[0179] The device uses a speech recognition API to convert captured audio into text data. The input is audio data, and the output is text data.

[0180] Step 4:

[0181] The device sends text to an AI model that generates text.

[0182] The terminal sends the converted text data to the generating AI model. The input is text data, and the output is the data sent to the generating AI model.

[0183] Step 5:

[0184] The server generates operations using an AI model.

[0185] The server uses a generative AI model to generate appropriate actions based on the received text data. Specifically, prompt statements like the following are generated:

[0186] Example of a prompt:

[0187] The user said "Turn on the TV" in Tsugaru dialect. Generate an operation to create an infrared signal to turn on the TV.

[0188] The input is text data, and the output is the generated manipulated data.

[0189] Step 6:

[0190] The server sends the generated operation to the terminal.

[0191] The server sends the generated operation to the terminal. The input is the generated operation data, and the output is the operation data sent to the terminal.

[0192] Step 7:

[0193] The terminal controls the infrared signal generator.

[0194] The terminal controls the infrared signal generator based on the received operation. The input is operation data, and the output is an infrared signal.

[0195] Step 8:

[0196] The television operates by receiving infrared signals.

[0197] An infrared signal generator emits an infrared signal towards the television, and the television receives this signal and turns on. The input is an infrared signal, and the output is the television's operation.

[0198] (Application Example 2)

[0199] Next, we will describe application example 2 of form example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0200] In modern brick-and-mortar stores, there is a problem in that customers who speak dialects have difficulty receiving product information via voice. In particular, for the elderly and customers in areas where dialects are spoken, voice recognition systems that only understand standard Japanese cannot provide adequate service. Therefore, there is a need for a system that combines dialect-compatible voice recognition with generative AI.

[0201] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0202] In this invention, the server includes speech recognition means, generative AI means, and learning means for handling all dialects. This makes it possible for customers who speak dialects to receive product information via voice.

[0203] "Speech recognition means" refers to technology for converting speech into text.

[0204] "Generative AI methods" refer to artificial intelligence technologies that generate appropriate responses or operations based on input data.

[0205] "Learning methods for handling all dialects" refers to learning algorithms for understanding and recognizing dialects from different regions.

[0206] "Methods for learning difficult dialects" refers to special learning processes for recognizing dialects that are particularly difficult to understand.

[0207] "Means for the practical application of speech recognition" refers to means that enable speech recognition technology to be used in actual applications.

[0208] "A means of providing information about products within a store" refers to a system used to provide customers with information about the location of products within a store.

[0209] "Applications installed on smart devices" refers to software installed on devices such as smartphones and tablets.

[0210] In order to implement this invention, it is necessary to construct a system that includes a speech recognition means, a generative AI means, a learning means for handling all dialects, a means for learning difficult dialects, a means for putting speech recognition into practical use, a guidance means for providing product information within a store, and an application means to be installed on a smart device.

[0211] System Configuration

[0212] 1. Speech recognition means:

[0213] The speech recognition system is used to convert the user's spoken voice into text. Specifically, it uses the smartphone's microphone to capture the voice and the SpeechRecognition library to convert the voice into text.

[0214] 2. Generative AI means:

[0215] The generative AI system generates appropriate responses and actions based on text acquired by the speech recognition system. Specifically, it utilizes the GPT-3 model with the Hugging Face transformers library.

[0216] 3. Learning methods for handling all dialects:

[0217] This method includes a learning algorithm for understanding and recognizing dialects from different regions. In particular, a special learning process is implemented as a means of learning difficult dialects.

[0218] 4. Means of practical application of speech recognition:

[0219] This is a means to make speech recognition technology usable in actual applications. This includes an interface for a generative AI system to generate an appropriate response based on the speech recognition results and provide it to the user.

[0220] 5. Means of providing information about products within the store:

[0221] This system provides customers with information and location details about products within a store. Specifically, it displays a map showing product locations on a smartphone screen and provides voice guidance.

[0222] 6. Application methods installed on smart devices:

[0223] This is software installed on devices such as smartphones and tablets. This application integrates voice recognition and generative AI to enable users to receive product information via voice.

[0224] Processing flow

[0225] 1. Acquisition of voice input:

[0226] When a user speaks into their smartphone, the voice recognition system captures that voice. For example, the user might say, "Find this product."

[0227] 2. Speech recognition:

[0228] The speech recognition system converts the acquired speech into text. The SpeechRecognition library is used to convert the speech to text.

[0229] 3. Response generation by generative AI:

[0230] A generative AI system uses the converted text as a prompt to generate an appropriate response. Specifically, it uses the Hugging Face GPT-3 model to generate the response.

[0231] 4. Providing a response:

[0232] The generated response is displayed on the smartphone screen or announced via voice. For example, a response such as "This product is on the shelf at the back right of the store" is generated.

[0233] Specific example

[0234] User voice input: "Find this product"

[0235] Generated response: "This item is located on the shelf at the back right of the store."

[0236] Example of a prompt

[0237] "If someone asks me to 'find this product' in Tsugaru dialect, how would I guide them to the product in the store?"

[0238] In this way, combining dialect-compatible speech recognition and generation AI can significantly improve customer service in physical stores.

[0239] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0240] Step 1:

[0241] The user speaks into their smartphone.

[0242] Input: User's voice (e.g., "Find this product")

[0243] Operation: The smartphone's microphone acquires sound.

[0244] Output: Acquired audio data

[0245] Step 2:

[0246] The device uses speech recognition to convert the acquired speech data into text.

[0247] Input: Audio data

[0248] Operation: Converts speech to text using the SpeechRecognition library.

[0249] Output: Converted text (e.g., "Find this product")

[0250] Step 3:

[0251] The terminal uses generative AI means to generate a response based on the converted text.

[0252] Input: Converted text

[0253] Operation: Uses the Hugging Face GPT-3 model to generate prompt sentences and appropriate responses.

[0254] Output: Generated response text (Example: "This item is on the shelf at the back right of the store.")

[0255] Step 4:

[0256] The terminal provides the user with the generated response.

[0257] Input: Generated response text

[0258] Function: Displays the response text on the smartphone screen, or plays the response as audio.

[0259] Output: The user confirms the response.

[0260] Step 5:

[0261] The user searches for the product by following the instructions.

[0262] Input: Response text or voice guidance

[0263] Operation: The user moves around the store and searches for products by following the directions.

[0264] Output: The user finds the desired product.

[0265] In this way, specific actions and data processing / calculations are performed at each step, enabling customer service in physical stores that utilizes dialect-compatible speech recognition and generation AI.

[0266] (Example 3)

[0267] Next, we will describe Embodiment 3 of Embodiment Example 3. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0268] Developing a speech recognition system that can handle all Japanese dialects is particularly challenging, especially for complex dialects. In particular, handling Tsugaru dialect, considered the most difficult dialect in Japan, is difficult with conventional speech recognition technology. Furthermore, accurately recognizing the voice input of elderly users operating information technology devices and generating appropriate responses is also a challenge.

[0269] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means.

[0270] In this invention, the server includes a speech recognition means, a generative artificial intelligence means, and a learning means for handling all Japanese dialects. This enables a system that includes means for the user to input voice into a terminal, means for the terminal to send voice data to the server, means for the server to convert the voice data into text data using speech recognition software, means for the server to input the text data into a generative artificial intelligence model, means for the generative artificial intelligence model to generate an appropriate response, means for the server to send the generated response to the terminal, and means for the terminal to display the response to the user. This enables speech recognition and response generation that can handle all Japanese dialects, and can handle particularly difficult dialects such as Tsugaru dialect. Furthermore, it can accurately recognize voice input when elderly people operate information technology devices themselves and generate appropriate responses.

[0271] "The "voice recognition means" is a technology for converting voice data into text data."

[0272] "The "generative artificial intelligence means" is an artificial intelligence technology for generating an appropriate response based on the input text data."

[0273] "The "learning means" is a method for training a system using specific data to improve the ability to execute a specific task."

[0274] "The "means for performing learning of Tsugaru dialect" is a method for training the voice recognition means and the generative artificial intelligence means using a large amount of voice data of the Tsugaru dialect."

[0275] "The "means for practical application of voice recognition" is a method for applying voice recognition technology to actual applications or systems."

[0276] "The "means for a user to perform voice input to a terminal" is a method for a user to input voice toward a device such as a smartphone or a personal computer."

[0277] "The "means for a terminal to transmit voice data to a server" is a method for a terminal to transmit voice data captured via the Internet to a server."

[0278] "The "means for a server to convert voice data into text data using voice recognition software" is a method for a server to convert received voice data into text data using voice recognition software."

[0279] "The "means for a server to input text data into a generative artificial intelligence model" is a method for a server to input the converted text data into a generative artificial intelligence model."

[0280] "The "means for a generative artificial intelligence model to generate an appropriate response" is a method for a generative artificial intelligence model to generate an appropriate response based on the input text data."

[0281] The means for the server to send the response generated to the terminal is a method for the server to send the response returned from the generative artificial intelligence model to the terminal.

[0282] The means for the terminal to display the response to the user is a method for the terminal to display the response received from the server to the user.

[0283] This invention provides a speech recognition system that supports all dialects in Japan. In particular, it has a learning means for dealing with the Tsugaru dialect, which is said to be the most difficult dialect in Japan. This system includes a speech recognition means, a generative artificial intelligence means, a learning means, a means for practical application of speech recognition, a means for the user to perform voice input on the terminal, a means for the terminal to send voice data to the server, a means for the server to convert the voice data into text data using speech recognition software, a means for the server to input the text data into a generative artificial intelligence model, a means for the generative artificial intelligence model to generate an appropriate response, a means for the server to send the generated response to the terminal, and a means for the terminal to display the response to the user.

[0284] First, the server collects a large amount of voice data in the Tsugaru dialect. This voice data is converted into text data using speech recognition software (e.g., Google Speech-to-Text API). Next, the converted text data is used to train a generative artificial intelligence model (e.g., OpenAI's GPT-4).

[0285] The terminal sends the voice data input by the user to the server in real time. The server converts the received voice data into text data using speech recognition software and inputs the text data into a generative artificial intelligence model. The generative artificial intelligence model generates an appropriate response based on the input text data and returns the response to the terminal as text data.

[0286] The user receives responses generated through the device and provides voice input again as needed. By repeating this process, the user can engage in natural conversations that are compatible with their dialect.

[0287] As a concrete example, consider a scenario where a user speaks in Tsugaru dialect, saying "How's it going today?" The terminal sends this voice data to a server, which uses speech recognition software to convert it into text data: "How's it going today?" Next, this text data is input into a generative artificial intelligence model to generate an appropriate response. For example, the response "It's nice weather today" might be generated. This response is sent back to the terminal as text data and displayed to the user.

[0288] Examples of prompt statements include the following:

[0289] User input: "How's it going today?"

[0290] Prompt to the Generative AI Model: "The user has spoken in Tsugaru dialect, saying 'How are you today?' Please generate an appropriate response."

[0291] In this way, the system provides a means of learning to support all Japanese dialects, enabling users to engage in natural conversations. The flow of specific processing in Example 3 will be explained using Figure 15.

[0292] Step 1:

[0293] The user enters voice input into the device.

[0294] The user speaks into a device such as a smartphone or computer. For example, they might say "How's it going today?" in Tsugaru dialect. This voice is captured by the device's microphone. The input is the user's voice data, and the output is the voice data captured by the device.

[0295] Step 2:

[0296] The terminal sends voice data to the server.

[0297] The terminal compresses the captured voice data and sends it to the server via the Internet. Specifically, the terminal compresses the voice data into the MP3 format and sends it to the server using the HTTPS protocol. The input is the captured voice data, and the output is the voice data sent to the server.

[0298] Step 3:

[0299] The server converts the voice data into text data using voice recognition software.

[0300] The server sends the received voice data to voice recognition software (e.g., Google Speech-to-Text API) and converts it into text data. For example, voice data such as "How are you today?" is converted into text data "How are you today?". The input is the voice data sent to the server, and the output is the converted text data.

[0301] Step 4:

[0302] The server inputs the text data into a generative AI model.

[0303] The server sends the converted text data to a generative AI model (e.g., OpenAI's GPT-4) as an API request. For example, it sends the text data "How are you today?" to the generative AI model. The input is the converted text data, and the output is the API request to the generative AI model.

[0304] Step 5:

[0305] The generative AI model generates an appropriate response.

[0306] Generative artificial intelligence models generate appropriate responses based on input text data. For example, they might generate the response, "It's a nice day today." This response is returned to the server as text data. The input is an API request to the generative artificial intelligence model, and the output is the generated response text data.

[0307] Step 6:

[0308] The server sends the generated response to the terminal.

[0309] The server receives the response text data returned from the generative artificial intelligence model and sends it to the terminal as an API response. For example, it sends the text data "It's a nice day today" to the terminal. The input is the generated response text data, and the output is the response text data sent to the terminal.

[0310] Step 7:

[0311] The terminal displays a response to the user.

[0312] The terminal displays the response text data received from the server to the user. For example, the text "It's a nice day today" is displayed on the smartphone screen. The input is the response text data sent to the terminal, and the output is the response text displayed to the user.

[0313] (Application Example 3)

[0314] Next, we will describe application example 3 of form example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0315] Conventional speech recognition systems are specialized for standard Japanese and have difficulty handling the diverse dialects of Japan. Furthermore, they are particularly unable to handle Tsugaru dialect, considered one of the most difficult dialects in Japan, resulting in low convenience for users who speak the dialect. Additionally, in food delivery services, ordering in dialect is difficult, forcing users to use standard Japanese.

[0316] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes a speech recognition means, a generative AI means, a learning means for supporting all Japanese dialects, a means for learning Tsugaru dialect, which is said to be the most difficult dialect in Japan, a means for putting speech recognition into practical use, a means for recognizing and analyzing speech input in dialect and generating order content, a means for transmitting the generated order content via a communication means, a means for allowing the user to confirm the order content, and a means for notifying the user that the order has been confirmed. As a result, when a user who speaks a dialect uses a food delivery service, they will be able to input speech in their dialect, improving convenience.

[0317] "Speech recognition means" refers to technology for converting speech into text.

[0318] "Generative AI methods" are artificial intelligence technologies that generate new information or operations based on input data.

[0319] "Learning methods to handle all Japanese dialects" refers to technologies that collect and learn the data necessary to recognize and understand dialects from various regions of Japan.

[0320] "Methods for learning Tsugaru dialect" refers to the technology of collecting and learning the data necessary to recognize and understand Tsugaru dialect, which is said to be the most difficult dialect in Japan.

[0321] "Means for the practical application of speech recognition" refers to technologies that make speech recognition technology usable in actual applications and services.

[0322] "A means of recognizing, analyzing, and generating order details from voice input in a dialect" refers to a technology that recognizes voice input in a dialect, analyzes its content, and generates appropriate order details.

[0323] "Means for transmitting generated order details via communication means" refers to technologies for transmitting generated order details using the internet or other communication means.

[0324] "Methods for allowing users to confirm order details" refer to technologies that present the generated order details to the user and ask for their confirmation.

[0325] "Means of notifying that an order has been confirmed" refers to technologies that inform users that their order has been confirmed.

[0326] The following system configuration will be described as an embodiment for carrying out this invention.

[0327] System Configuration

[0328] This system includes a speech recognition means, a generative AI means, a learning means for handling all Japanese dialects, a means for learning Tsugaru dialect, a means for putting speech recognition into practical use, a means for recognizing and analyzing speech input in dialect and generating order details, a means for transmitting the generated order details via a communication means, a means for allowing the user to confirm the order details, and a means for notifying the user that the order has been confirmed.

[0329] Hardware and software to be used

[0330] Hardware: Smartphone (iOS or Android®)

[0331] software:

[0332] Speech recognition API (e.g., Google Cloud Speech-to-Text)

[0333] Generative AI models (e.g., OpenAI GPT-4)

[0334] Food delivery APIs (e.g., Uber Eats API)

[0335] Processing flow

[0336] 1. Voice input: The user speaks their order in their local dialect into the microphone of their smartphone.

[0337] 2. Speech Recognition: The smartphone uses a speech recognition API to convert the input speech into text. This process utilizes a speech recognition model that supports different dialects.

[0338] 3. Text Analysis: The server uses a generative AI model to analyze the recognized text and understand the order details.

[0339] 4. Order Generation: Based on the analysis results, the server uses the food delivery API to generate an appropriate order and sends it to the restaurant.

[0340] 5. Confirmation and Notification: The smartphone allows the user to confirm the order details and notifies them that the order has been confirmed.

[0341] Specific example

[0342] When a user speaks in Tsugaru dialect, saying "I'd like to order one pizza," the smartphone recognizes this, generates an appropriate pizza order, and sends it to the restaurant.

[0343] Example of a prompt

[0344] If a user speaks in Tsugaru dialect and says "I'd like to order one pizza," use a speech recognition API to convert the speech to text, a generative AI model to analyze the order, and a food delivery API to generate a pizza order.

[0345] In this way, a dialect-enabled food delivery assistant can enable users to place orders in their local dialect, providing a convenient service to a wider range of users.

[0346] The flow of the specific processing in Application Example 3 will be explained using Figure 16.

[0347] Step 1:

[0348] The user speaks their order in their local dialect into the smartphone's microphone. The input is the user's voice, and the output is the audio data input to the smartphone's microphone.

[0349] Step 2:

[0350] The device uses a speech recognition API (e.g., Google Cloud Speech-to-Text) to convert the input audio data into text. The input is audio data, and the output is text data. A speech recognition model that supports dialects is used in this process.

[0351] Step 3:

[0352] The server uses a generative AI model (e.g., OpenAI GPT-4) to analyze text data sent from a speech recognition API and understand the order details. The input is text data, and the output is the analyzed order details.

[0353] Step 4:

[0354] Based on the analysis results, the server generates an appropriate order using a food delivery API (e.g., Uber Eats API) and sends it to the restaurant. The input is the analyzed order data, and the output is the order data sent via the food delivery API.

[0355] Step 5:

[0356] The terminal displays the generated order details to allow the user to confirm the order. The input is order confirmation data from the food delivery API, and the output is the order details displayed on the smartphone screen.

[0357] Step 6:

[0358] The user reviews and confirms their order. The input is the user's confirmation action, and the output is the order confirmation signal.

[0359] Step 7:

[0360] The terminal notifies the server that the order has been confirmed, and the server sends this information to the food delivery API. The input is the order confirmation signal, and the output is the order confirmation data sent to the food delivery API.

[0361] Step 8:

[0362] The server notifies the user that the order has been confirmed. The input is order confirmation data from the food delivery API, and the output is an order confirmation notification displayed to the user via the smartphone's notification function.

[0363] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0364] "Example of form 1"

[0365] One embodiment of the present invention provides a system that includes a speech recognition means, a generative AI means, a learning means for handling all Japanese dialects, a means for learning Tsugaru dialect, and a means for putting speech recognition into practical use, as well as an emotion engine that recognizes the user's emotions. This emotion engine recognizes emotions from the user's voice and generates an operation corresponding to that emotion. For example, when a user makes the voice input "Turn on the TV," the emotion engine recognizes the user's emotion from that voice. If it recognizes that the emotion is "joy," the generative AI means generates an operation to turn on the TV.

[0366] "Example of form 2"

[0367] In another embodiment of the present invention, the emotion engine generates an operation in response to the user's emotions. For example, when the user makes a voice input saying "Turn on the radio," the emotion engine recognizes the user's emotions from the voice. If it recognizes that the emotion is "sadness," the generative AI means generates an operation to turn on the radio and provides a result in which the operation is adapted to the user's emotions. Specifically, it generates an operation such as setting the radio channel to one that plays sad songs that the user likes.

[0368] "Example of form 3"

[0369] Furthermore, in another embodiment of the present invention, the emotion engine recognizes the user's emotion, and the generative AI means generates an operation in response to that emotion. For example, when the user makes a voice input saying "Turn on the air conditioner," the emotion engine recognizes the user's emotion from that voice. If it recognizes that the emotion is "anger," the generative AI means generates an operation to turn on the air conditioner and provides a result in which the operation is adapted to the user's emotion. Specifically, it generates an operation such as setting the air conditioner temperature to a low temperature that will calm the user down.

[0370] The following describes the processing flow for each example of the form.

[0371] "Example of form 1"

[0372] Step 1: The user makes a voice input. In this example, the user makes the voice input, "Turn on the TV."

[0373] Step 2: The emotion engine recognizes emotions from the user's voice. In this example, it recognizes "joy."

[0374] Step 3: The generative AI generates an action based on the emotions it recognizes. In this example, it generates the action to turn on the TV.

[0375] "Example of form 2"

[0376] Step 1: The user makes a voice input. In this example, the user makes the voice input, "Turn on the radio."

[0377] Step 2: The emotion engine recognizes emotions from the user's voice. In this example, it recognizes "sadness."

[0378] Step 3: The generative AI generates an action based on the emotions it recognizes and provides a result where the action is adapted to the user's emotions. In this example, it generates an action to turn on the radio and provides a result where the action is adapted to the user's emotions. Specifically, it generates an action such as setting the radio channel to one that plays sad songs that the user likes.

[0379] "Example of form 3"

[0380] Step 1: The user makes a voice input. In this example, the user makes the voice input, "Turn on the air conditioner."

[0381] Step 2: The emotion engine recognizes emotions from the user's voice. In this example, it recognizes "anger."

[0382] Step 3: The generative AI generates an action based on the emotions it recognizes and provides a result where that action is adapted to the user's emotions. In this example, it generates an action to turn on the air conditioner and provides a result where that action is adapted to the user's emotions. Specifically, it generates an action such as setting the air conditioner temperature to a low temperature that will help the user calm down.

[0383] (Example 1)

[0384] Next, we will describe Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0385] When elderly people operate information technology devices, there is a challenge in accurately recognizing voice input and generating appropriate operations. In particular, there is a need to support all Japanese dialects, but the diversity and complexity of dialects are factors that reduce the accuracy of voice recognition. Furthermore, there is a problem in that operations cannot be generated while taking the user's emotions into consideration, resulting in a poor user experience.

[0386] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0387] In this invention, the server includes a speech recognition means, a generative artificial intelligence means, and a learning means for handling all Japanese dialects. This enables accurate recognition of speech input when elderly people operate information technology devices and allows for speech recognition that supports dialects. Furthermore, by including an emotion engine that recognizes the user's emotions, it becomes possible to generate operations that respond to the user's emotions, thereby improving the user experience.

[0388] "Speech recognition means" refers to technology for converting speech input into text data.

[0389] "Generative artificial intelligence means" refers to artificial intelligence technology for generating appropriate operations based on speech recognition results.

[0390] "Learning methods to handle all Japanese dialects" refers to technology for training speech recognition models to recognize the diverse dialects used within Japan.

[0391] "Methods for learning dialects" refer to techniques for specially training speech recognition models to recognize specific dialects, especially complex ones.

[0392] "Means for the practical application of speech recognition" refers to technologies for applying speech recognition technology to actual applications and systems.

[0393] An "emotion engine" is a technology that analyzes emotions from a user's voice and generates appropriate actions based on those emotions.

[0394] Modes for carrying out the invention

[0395] This invention is a system that includes speech recognition means, generative artificial intelligence means, learning means for handling all Japanese dialects, particularly means for learning difficult dialects, means for practical application of speech recognition, and an emotion engine. Specific embodiments of this system are described below.

[0396] Acquiring voice input

[0397] Subject: User

[0398] The user makes voice input to the device. For example, they might say, "Turn on the TV." The device is equipped with a high-performance microphone that captures the user's voice with high accuracy.

[0399] Sending audio data

[0400] Subject: terminal

[0401] The terminal sends the acquired audio data to the server. During this process, the audio data is converted to an appropriate format and sent to the server via the network.

[0402] Speech recognition

[0403] Subject: Server

[0404] The server converts the received audio data into text data using speech recognition. Specifically, it uses speech recognition software (for example, Google Cloud Speech-to-Text API) to convert speech to text.

[0405] Learning for dialect support

[0406] Subject: Server

[0407] The server has the means to learn how to handle all Japanese dialects. In particular, it trains its speech recognition model to recognize complex dialects (for example, Tsugaru dialect).

[0408] emotion recognition

[0409] Subject: terminal

[0410] The device has an emotion engine that recognizes emotions from the user's voice. Specifically, it uses emotion analysis software (for example, Microsoft® Azure®'s Emotion API) to analyze emotions from the user's voice.

[0411] Operation generation

[0412] Subject: Server

[0413] The server uses generative artificial intelligence tools (e.g., OpenAI's GPT-4) to generate an action based on speech recognition and emotion recognition results. The generated action is then sent to the terminal.

[0414] Execute the operation

[0415] Subject: terminal

[0416] The terminal executes the operation received from the server. Specifically, it sends a signal to turn on the television.

[0417] Specific example

[0418] Specific Example 1

[0419] When a user says "Turn on the TV," the device captures the audio and sends it to the server. The server uses speech recognition to convert the audio into text and uses generative artificial intelligence to generate the action "Turn on the TV." The device then executes this action and turns on the TV.

[0420] Specific Example 2

[0421] When a user says "Turn off the air conditioner," the device captures the audio and sends it to the server. The server uses speech recognition to convert the audio into text and uses generative artificial intelligence to generate the action "Turn off the air conditioner." The device then executes this action and turns off the air conditioner.

[0422] Example of a prompt

[0423] "When a user says 'Turn on the TV,' use speech recognition to convert the speech into text, and then use generative artificial intelligence to generate the action to turn on the TV. Also, if the user's emotion is 'joy,' provide that information to the generative artificial intelligence."

[0424] By inputting this prompt into the generating AI model, the system's operation can be simulated.

[0425] The flow of the specific processing in Example 1 will be explained using Figure 17.

[0426] Step 1:

[0427] Acquiring voice input

[0428] Subject: User

[0429] The user makes a voice command, such as "Turn on the TV," into the device. The device is equipped with a high-performance microphone that captures the user's voice with high accuracy.

[0430] Input: User's voice instructions

[0431] Output: Acquired audio data

[0432] Specific action: Instead of the user searching for the remote control in the living room, they speak to the device and say, "Turn on the TV."

[0433] Step 2:

[0434] Sending audio data

[0435] Subject: terminal

[0436] The terminal sends the acquired audio data to the server. During this process, the audio data is converted to an appropriate format and sent to the server via the network.

[0437] Input: Acquired audio data

[0438] Output: Audio data sent to the server

[0439] Specific operation: The device sends audio data to the server via Wi-Fi. The transmission takes only a few seconds.

[0440] Step 3:

[0441] Speech recognition

[0442] Subject: Server

[0443] The server converts the received audio data into text data using speech recognition. Specifically, it uses speech recognition software (for example, Google Cloud Speech-to-Text API) to convert speech to text.

[0444] Input: Audio data sent to the server

[0445] Output: Text data (e.g., "Turn on the TV")

[0446] Specific operation: The server receives the audio data and calls the Google Cloud Speech-to-Text API to generate the text "Turn on the TV".

[0447] Step 4:

[0448] Learning for dialect support

[0449] Subject: Server

[0450] The server has the means to learn how to handle all Japanese dialects. In particular, it trains its speech recognition model to recognize complex dialects (for example, Tsugaru dialect).

[0451] Input: Audio data of dialects

[0452] Output: Speech recognition model that supports dialects

[0453] Specific operation: The server trains a model using Tsugaru dialect audio data to enable it to handle dialects.

[0454] Step 5:

[0455] emotion recognition

[0456] Subject: terminal

[0457] The device has an emotion engine that recognizes emotions from the user's voice. Specifically, it uses emotion analysis software (for example, Microsoft Azure's Emotion API) to analyze emotions from the user's voice.

[0458] Input: User's voice data

[0459] Output: Emotion data (e.g., "joy")

[0460] Specific operation: The device analyzes the user's emotion from the voice command "Turn on the TV" and recognizes that emotion as "joy."

[0461] Step 6:

[0462] Operation generation

[0463] Subject: Server

[0464] The server uses generative artificial intelligence tools (e.g., OpenAI's GPT-4) to generate an action based on speech recognition and emotion recognition results. The generated action is then sent to the terminal.

[0465] Input: Text data (e.g., "Turn on the TV"), emotion data (e.g., "Joy")

[0466] Output: Generated operation (e.g., "Turn on the TV")

[0467] Specific operation: The server inputs the text "Turn on the TV" and the emotion information "joy" into GPT-4, and generates the operation "Turn on the TV".

[0468] Step 7:

[0469] Execute the operation

[0470] Subject: terminal

[0471] The terminal executes the operation received from the server. Specifically, it sends a signal to turn on the television.

[0472] Input: Generated action (e.g., "Turn on the TV")

[0473] Output: The action performed (e.g., the TV is turned on)

[0474] Specific action: The device sends a remote control signal to the TV, and the TV turns on.

[0475] (Application Example 1)

[0476] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server," and the smart device 14 will be referred to as a "terminal."

[0477] There is a problem with the accuracy of voice recognition, making it difficult for elderly customers and those who speak in dialects to operate IT equipment. Furthermore, there is the challenge of appropriately recognizing and responding to customers' emotions in physical stores.

[0478] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes a speech recognition means, a generative AI means, a learning means for handling all Japanese dialects, a means for learning Tsugaru dialect, which is said to be the most difficult dialect in Japan, a means for putting speech recognition into practical use, an emotion recognition means, an application means installed on smart glasses, a means for recognizing customer voice input and for the generative AI means to generate an appropriate response, a means for the emotion recognition means to recognize the customer's emotions and generate a response corresponding to those emotions, and a speech output means. As a result, the accuracy of speech recognition is improved when elderly customers or customers who speak dialects operate IT equipment, and appropriate responses that correspond to customer emotions become possible in physical stores.

[0479] "Speech recognition means" refers to technology that analyzes speech input and converts it into text data.

[0480] "Generative AI methods" refer to artificial intelligence technologies that generate appropriate responses or operations based on input data.

[0481] "Learning methods to handle all Japanese dialects" refers to technology that trains speech recognition models to recognize the diverse dialects used within Japan.

[0482] "Methods for learning Tsugaru dialect" refers to techniques for special learning in order to understand Tsugaru dialect, which is said to be the most difficult dialect in Japan.

[0483] "Means for the practical application of speech recognition" refers to technologies that make speech recognition technology usable in actual systems and applications.

[0484] "Emotion recognition means" refers to technology that analyzes and recognizes a user's emotions from voice and text data.

[0485] "Applications installed on smart glasses" refers to applications that run on smart glasses and are software that utilizes voice recognition and generative AI.

[0486] "A means of recognizing customer voice input and generating an appropriate response using a generative AI means" refers to a technology that recognizes customer voice and generates an appropriate response using a generative AI based on its content.

[0487] "Means for recognizing customer emotions and generating responses in accordance with those emotions" refers to technology that recognizes customer emotions and generates appropriate responses in accordance with those emotions.

[0488] "Speech output means" refers to technology that outputs generated text data or responses as speech.

[0489] A system for carrying out this invention includes a speech recognition means, a generative AI means, a learning means for handling all Japanese dialects, particularly a means for learning Tsugaru dialect, a means for putting speech recognition into practical use, an emotion recognition means, an application means installed in smart glasses, a means for recognizing customer voice input and for the generative AI means to generate an appropriate response, a means for the emotion recognition means to recognize customer emotions and generate an emotion-appropriate response, and a speech output means.

[0490] Hardware and software configuration

[0491] The server uses the speech_recognition library for speech recognition and the transformers library for generative AI. The transformers library's sentiment-analysis pipeline is also used for emotion recognition. The pyttsx3 library is used for speech output. The smart glasses have a built-in microphone and speaker, and this software is installed on them.

[0492] Data processing and data calculation

[0493] 1. Speech Recognition: When the user speaks into the smart glasses, the microphone captures the audio, and the speech_recognition library converts that audio into text data.

[0494] 2. Generative AI: The transformed text data is input into a generative AI model in the transformers library, and an appropriate response is generated.

[0495] 3. Emotion Recognition: Simultaneously, the text data is input into the sentiment-analysis pipeline, where the user's emotions are analyzed. Based on the analysis results, a generative AI model generates an emotion-appropriate response.

[0496] 4. Audio Output: Finally, the generated response is output as audio using the pyttsx3 library.

[0497] Specific example

[0498] For example, if a customer in a physical store says, "Tell me about this product," the system will operate as follows:

[0499] 1. Speech Recognition: The customer's voice is captured by the microphone on the smart glasses, and the speech_recognition library converts it into text data such as "Tell me about this product."

[0500] 2. Generative AI: The converted text data is input into a generative AI model in the transformers library, which generates the response, "This product is the latest smartphone with excellent camera performance. Furthermore, it has good battery life and can be used for extended periods."

[0501] 3. Sentiment Recognition: Simultaneously, text data is input into the sentiment-analysis pipeline and analyzed to determine if the customer is in distress. Generative AI models then generate more detailed explanations.

[0502] 4. Audio Output: Finally, the generated response is output as audio using the pyttsx3 library.

[0503] Example of a prompt

[0504] Customer: Please tell me about this product.

[0505] System: This product is a state-of-the-art smartphone with excellent camera performance. Furthermore, it boasts a long battery life, allowing for extended use.

[0506] In this way, the accuracy of voice recognition improves when elderly customers or customers who speak in dialects operate IT equipment, enabling appropriate responses that respond to customers' emotions in physical stores.

[0507] The flow of a specific process in Application Example 1 will be explained using Figure 18.

[0508] Step 1:

[0509] The user speaks into the smart glasses. The smart glasses' microphone captures the audio. The input is the user's voice, and the output is the captured audio data.

[0510] Step 2:

[0511] The server uses the speech_recognition library as a speech recognition tool to convert captured audio data into text data. The input is audio data, and the output is text data. Specifically, the audio data is passed to the library, and the speech recognition algorithm is applied.

[0512] Step 3:

[0513] The server uses the transformers library as a generative AI tool to generate an appropriate response from transformed text data as input. The input is text data, and the output is the generated response text. Specifically, the text data is input to a generative AI model, and the model generates a response.

[0514] Step 4:

[0515] The server uses the `transformers` library's `sentiment-analysis` pipeline as an emotion recognition tool to analyze the user's emotions from text data. The input is text data, and the output is the emotion analysis result. Specifically, the text data is input into the emotion recognition model, and the emotions are analyzed.

[0516] Step 5:

[0517] The server analyzes emotions, and a generative AI system generates an emotion-appropriate response based on the analysis results. The input is the emotion analysis results and text data, and the output is an emotion-appropriate response text. Specifically, the emotion analysis results are fed back to the generative AI model, and the response is adjusted accordingly.

[0518] Step 6:

[0519] The server uses the pyttsx3 library as its voice output method, outputting the generated response text as speech. The input is the response text, and the output is speech data. Specifically, the response text is input to the speech synthesis engine, and speech data is generated.

[0520] Step 7:

[0521] The smart glasses play audio data and provide responses to the user. The input is audio data, and the output is the audio the user hears. Specifically, the audio data is played through the smart glasses' speaker.

[0522] (Example 2)

[0523] Next, we will describe Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0524] When elderly people operate information technology devices, there is a challenge in accurately recognizing voice input, especially when it uses dialects, and generating appropriate operations. Furthermore, there is a need to provide more user-friendly systems that generate operations that respond to the user's emotions. In addition, a means of actually executing the generated operations is also necessary.

[0525] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0526] In this invention, the server includes a speech recognition means, a generative AI means, a dialect-compatible learning means, an emotion recognition means, a means for putting speech recognition into practical use, and an infrared signal generator control means. This makes it possible to accurately recognize speech even when elderly people use dialects, generate appropriate operations according to the user's emotions, and actually execute those operations.

[0527] "Voice recognition means" refers to a device or software that converts voice input into digital data and analyzes its contents.

[0528] A "generative AI means" is a device or software that uses artificial intelligence technology to generate appropriate operations based on data obtained from a speech recognition means.

[0529] A "dialect-compatible learning tool" is a device or software that trains a speech recognition tool to recognize dialects specific to a particular region or culture.

[0530] "Emotion recognition means" refers to a device or software that analyzes a user's emotions from voice input and recognizes those emotions.

[0531] "Means for the practical application of speech recognition" refers to devices or software that apply speech recognition technology to actual systems and applications, making it practically usable.

[0532] "Infrared signal generator control means" refers to a device or software that generates infrared signals based on instructions from a generation AI means and controls a specific device.

[0533] This invention provides a system that accurately recognizes voice input, particularly that using dialects, when elderly people operate information technology devices, and generates appropriate operations. Furthermore, it can generate operations that respond to the user's emotions and actually execute those operations.

[0534] Hardware and software to be used

[0535] 1. Speech recognition means:

[0536] The device uses its built-in microphone to capture audio.

[0537] For speech recognition software, you can use, for example, the Google Speech-to-Text API.

[0538] 2. Generative AI means:

[0539] The server uses a generated AI model (e.g., OpenAI's GPT-4) to analyze text data obtained from the speech recognition system and generate appropriate operations.

[0540] 3. Dialect-based learning methods:

[0541] The server is trained to recognize dialects specific to a particular region or culture by training its speech recognition system. For example, training data for the Tsugaru dialect of Japan is used.

[0542] 4. Emotion recognition means:

[0543] The server uses emotion recognition software (for example, IBM Watson® Tone Analyzer) to analyze the user's emotions from the voice input and recognize those emotions.

[0544] 5. Means of practical application of speech recognition:

[0545] The device applies voice recognition technology to actual systems and applications, making it practically usable.

[0546] 6. Infrared signal generator control means:

[0547] The terminal sends the generated commands to an infrared signal generator (e.g., BroadLink RM4 Pro) to control a specific device.

[0548] Specific example

[0549] Example 1: Turning on the TV

[0550] 1. The user says "Turn on the TV" in Tsugaru dialect.

[0551] 2. The device uses its built-in microphone to capture audio and the Google Speech-to-Text API to convert the audio to text.

[0552] 3. The server uses a generated AI model to generate the action "Turn on the TV".

[0553] 4. The terminal sends the generated command to the infrared signal generator, turning on the television.

[0554] Example 2: Turning on the radio

[0555] 1. The user says, "Turn on the radio."

[0556] 2. The device uses its built-in microphone to capture audio and the Google Speech-to-Text API to convert the audio to text.

[0557] 3. The server uses emotion recognition software to recognize the user's emotion as "sadness."

[0558] 4. The server uses a generated AI model to generate an action to set the channel to play sad songs.

[0559] 5. The terminal sends a command to the radio and sets it to the specified channel.

[0560] Example of a prompt

[0561] "When a user says 'Turn on the TV' in Tsugaru dialect, a voice recognition system converts the speech into text, and a generation AI system generates an operation to turn on the TV. It then controls an infrared signal generator to turn on the TV."

[0562] "When a user says, 'Turn on the radio,' the emotion engine recognizes the user's emotion, and the generative AI generates radio operations that are adapted to the user's emotion. For example, it might set the radio to a channel playing sad music."

[0563] The flow of the specific processing in Example 2 will be explained using Figure 19.

[0564] Step 1:

[0565] Voice input capture

[0566] The user says "Turn on the TV" in Tsugaru dialect.

[0567] The device uses its built-in microphone to capture audio.

[0568] Input: User's voice

[0569] Output: Audio data

[0570] Specific actions:

[0571] The user is standing in front of the TV in the living room and says, "Turn on the TV."

[0572] The device captures audio through the microphone and saves it as audio data.

[0573] Step 2:

[0574] Text conversion of audio data

[0575] The device uses speech recognition software (for example, Google Speech-to-Text API) to convert the captured audio data into text data.

[0576] Input: Audio data

[0577] Output: Text data

[0578] Specific actions:

[0579] The device sends the captured audio data to the Google Speech-to-Text API.

[0580] The device receives the text data "Turn on the TV" returned from the API.

[0581] Step 3:

[0582] Recognition of emotions (as needed)

[0583] The server uses emotion recognition software (e.g., IBM Watson Tone Analyzer) to analyze the user's emotions from the text data.

[0584] Input: Text data

[0585] Output: Sentiment data

[0586] Specific actions:

[0587] The server sends the text data "Turn on the radio" to IBM Watson Tone Analyzer.

[0588] The server receives "sadness" as the analysis result.

[0589] Step 4:

[0590] Operation generation

[0591] The server uses a generated AI model (e.g., OpenAI's GPT-4) to analyze text data and generate appropriate operations.

[0592] Input: Text data, sentiment data (if necessary)

[0593] Output: Operation command

[0594] Specific actions:

[0595] The server inputs the text data "Turn on the TV" into the AI ​​model that generates the data.

[0596] The server receives the generated command and sends it to a device that generates an infrared signal to "turn on the TV".

[0597] Step 5:

[0598] Execute the operation

[0599] The terminal sends the generated command to the infrared signal generator, executing the operation to turn on the television.

[0600] Input: Operation command

[0601] Output: Action performed (TV turns on)

[0602] Specific actions:

[0603] The terminal sends a "turn on TV" command to the infrared signal generator.

[0604] The infrared signal generator emits an infrared signal towards the television, and the television turns on.

[0605] (Application Example 2)

[0606] Next, we will describe application example 2 of form example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0607] When elderly people operate IT devices, there is a particular problem with voice recognition that supports Japanese dialects. Furthermore, the inability to provide appropriate operation and information in accordance with the user's emotions hinders the improvement of the user experience. Additionally, in physical stores, the inability to suggest products based on the user's emotions prevents the shopping experience from being optimized.

[0608] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0609] In this invention, the server includes a speech recognition means, a generative AI means, a learning means for handling all Japanese dialects, a means for learning Tsugaru dialect in particular, which is said to be the most difficult dialect in Japan, a means for putting speech recognition into practical use, a means for recognizing the user's emotions using an emotion engine, a means for generating operations corresponding to the user's emotions, a means for providing product information in a physical store, and a means for making product suggestions based on the user's emotions. As a result, accurate speech recognition becomes possible even when elderly people use dialects, appropriate operations and information provision corresponding to the user's emotions are realized, and product suggestions based on the user's emotions become possible even in a physical store.

[0610] "Speech recognition means" refers to technology for converting speech input into text data.

[0611] "Generative AI methods" refer to artificial intelligence technologies that generate appropriate responses or operations based on input data.

[0612] A "learning method" is a technique that uses a specific dataset to train a model and improve its ability to perform a specific task.

[0613] "Methods for learning Tsugaru dialect" refers to techniques for providing special training to understand and recognize the dialect used in the Tsugaru region of Japan.

[0614] "Means for the practical application of speech recognition" refers to technologies for integrating speech recognition technology into actual applications and systems.

[0615] An "emotion engine" is a technology that analyzes and recognizes emotions from a user's voice or text.

[0616] "Means for generating operations that respond to user emotions" refers to technologies that generate appropriate operations or responses based on recognized user emotions.

[0617] "Means of providing product information in physical stores" refers to technologies for providing product information to users within a physical store.

[0618] "Methods for suggesting products based on user emotions" refers to techniques for suggesting appropriate products by taking user emotions into consideration.

[0619] This invention is a system that combines speech recognition means, generative AI means, learning means, emotion engine, and means for providing product information in physical stores. Specific embodiments for realizing this system are described below.

[0620] System Configuration

[0621] This system consists of the following main components:

[0622] 1. Speech Recognition Method: This is a technology for converting speech input into text data. Specifically, the speech_recognition library is used.

[0623] 2. Generative AI methods: This is an artificial intelligence technology that generates appropriate responses or operations based on input data. Specifically, it uses the GPT-3 model from the transformers library.

[0624] 3. Learning Methods: This is a technique that trains a model using a specific dataset to improve its ability to perform specific tasks. In particular, it is used for training to handle all Japanese dialects.

[0625] 4. Emotion Engine: This is a technology for analyzing and recognizing emotions from the user's voice and text. Specifically, it uses the emotion analysis model from the transformers library.

[0626] 5. Means of providing product information in physical stores: This refers to technologies for providing product information to users within physical stores.

[0627] Program processing

[0628] The server uses speech recognition to convert the user's voice input into text data. Next, an emotion engine recognizes the user's emotions from the text data. Based on the recognized emotions, a generative AI generates appropriate responses or actions. For example, if a user asks, "Tell me about this product," and the emotion engine recognizes the user's emotion as "excited," the generative AI will generate a response such as, "This product is the latest model and has many added features."

[0629] Hardware and software to be used

[0630] Hardware: Smartphone (microphone, speaker)

[0631] software:

[0632] The speech_recognition library is used to convert speech input to text.

[0633] Transformers library: Generates responses using a generative AI model (GPT-3).

[0634] requests library: Used to communicate with external APIs (as needed).

[0635] Specific example

[0636] When a user speaks to a store and asks, "Tell me about this product," a speech recognition system converts the speech into text, and an emotion engine recognizes the user's emotions. For example, if the user is excited, a generative AI system might generate a response such as, "This product is the latest model and has many added features."

[0637] Example of a prompt

[0638] When a user is sad: "Please suggest products to recommend when a user is sad: Tell me about this product."

[0639] When the user is happy: "Please suggest products to recommend when the user is happy: Tell me about this product."

[0640] In this way, smart shopping assistants can provide appropriate information tailored to the user's emotions, thereby improving the shopping experience.

[0641] The flow of a specific process in Application Example 2 will be explained using Figure 20.

[0642] Step 1:

[0643] The server acquires the user's voice input using speech recognition technology. Specifically, it collects voice data using the smartphone's microphone and converts the voice data into text data using the speech_recognition library.

[0644] Input: User's voice data

[0645] Output: Text data

[0646] Step 2:

[0647] The server uses an emotion engine to recognize the user's emotions from text data. Specifically, it uses the emotion analysis model from the transformers library to analyze the text data and identify the user's emotions.

[0648] Input: Text data

[0649] Output: User's emotions (e.g., joy, sadness, excitement, etc.)

[0650] Step 3:

[0651] The server generates prompt messages based on the recognized user's emotions. Specifically, it creates emotionally appropriate prompt messages and inputs them into a generative AI model. For example, if the user is sad, it generates a prompt message such as, "Suggest products that are suitable for a sad user: Tell me about this product."

[0652] Input: User's sentiment, text data

[0653] Output: Prompt message

[0654] Step 4:

[0655] The server generates an appropriate response based on the prompt using generative AI tools. Specifically, it uses the GPT-3 model from the transformers library to generate the response to the prompt.

[0656] Input: Prompt message

[0657] Output: Generated response (e.g., "This product is the latest model and includes many additional features")

[0658] Step 5:

[0659] The server provides the generated response to the user. Specifically, it either plays the response aloud using the smartphone's speaker or displays it as text on the screen.

[0660] Input: Generated response

[0661] Output: Provides a response to the user (voice or text).

[0662] In this way, the server can recognize the user's voice input, analyze their emotions, and generate and provide an appropriate response.

[0663] (Example 3)

[0664] Next, we will describe Embodiment 3 of Embodiment Example 3. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0665] Conventional speech recognition systems have difficulty accurately recognizing specific dialects and emotions, and have been particularly unable to cope with the diverse dialects of Japan and the emotional range of users. Furthermore, they have struggled to accurately recognize voice input from elderly users operating information technology devices and generate appropriate commands. This has resulted in a reduced user experience and limited the practicality of the systems.

[0666] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means.

[0667] In this invention, the server includes a speech recognition means, a generative artificial intelligence means, and a learning means for handling all Japanese dialects. This makes possible a system that includes means for learning Tsugaru dialect, which is said to be the most difficult dialect in Japan, means for putting speech recognition into practice, means for recognizing the user's emotions using an emotion engine, and means for generating operations according to the recognized emotions.

[0668] "Speech recognition means" refers to technology that analyzes speech data and converts it into corresponding text data.

[0669] "Generative artificial intelligence means" refers to artificial intelligence technology that generates natural dialogue and operations based on input data.

[0670] "A learning method for handling all Japanese dialects" refers to a technology that recognizes and responds to the diverse dialects used within Japan, collects the necessary data, and trains a model accordingly.

[0671] "Methods for learning Tsugaru dialect" refers to the technology of collecting the necessary data to recognize and respond to Tsugaru dialect, a dialect spoken in the Tsugaru region of Japan, and training a model accordingly.

[0672] "Methods for the practical application of speech recognition" refers to technologies that integrate trained speech recognition models into actual systems, making them practically usable.

[0673] An "emotion engine" is a technology that analyzes and identifies a user's emotions from voice data.

[0674] "Means for generating operations in response to recognized emotions" refers to technology that generates and executes appropriate operations based on the user's emotions.

[0675] This invention is a system that includes a speech recognition means, a generative artificial intelligence means, a learning means for handling all Japanese dialects, particularly a means for learning Tsugaru dialect, a means for putting speech recognition into practical use, a means for recognizing the user's emotions using an emotion engine, and a means for generating operations according to the recognized emotions.

[0676] Hardware and software to be used

[0677] Hardware: High-performance servers (e.g., servers with GPUs)

[0678] Software: Speech recognition software (e.g., speech recognition APIs), generative artificial intelligence models (e.g., natural language processing models)

[0679] Explanation of the program's processing

[0680] Audio data collection and preprocessing

[0681] The server collects a large amount of Tsugaru dialect audio data. The collected audio data undergoes preprocessing such as noise reduction and normalization. Specifically, it processes the audio data using an audio processing library.

[0682] Training a speech recognition model

[0683] The server inputs pre-processed audio data into a speech recognition API and retrieves the corresponding text data. The retrieved text data is then used to train the speech recognition model. Specifically, the model is built and trained using a machine learning library.

[0684] Learning of generative artificial intelligence models

[0685] The server inputs text data obtained from the speech recognition model into a natural language processing model to train a generative artificial intelligence model. Once the generative AI model is trained, it is saved and provided as an API. Specifically, the model is managed using a natural language processing library.

[0686] Introducing an emotional engine

[0687] The server implements an emotion engine to recognize emotions from the user's voice. Specifically, it uses voice analysis tools to analyze the tone, speed, and volume of the voice. Based on the analysis results, it identifies the user's emotions and passes that information to a generative artificial intelligence model.

[0688] Emotion-driven manipulation

[0689] The user makes a voice input saying, "Turn on the air conditioner." The device sends this voice to the server. The server converts the voice to text using a speech recognition model. The server uses an emotion engine to recognize the user's emotions. For example, if the voice tone is high and the speed is fast, the server will determine that the user is angry. The server uses generative artificial intelligence means to generate an appropriate action according to the recognized emotion. For example, if the user is angry, it will generate an action to lower the air conditioner temperature.

[0690] Examples of specific cases and prompt statements

[0691] Specific example:

[0692] The user angrily inputs the command, "Turn on the air conditioner."

[0693] The device sends the audio to the server.

[0694] The server uses a speech recognition model to convert speech into text.

[0695] The server uses an emotion engine to recognize that the user's emotion is "anger."

[0696] The server uses generative artificial intelligence means to generate an operation to set the air conditioner temperature low.

[0697] Example of a prompt:

[0698] If a user makes an angry voice input saying "Turn on the air conditioner," the emotion engine recognizes the user's emotion from the voice, and the generative artificial intelligence means generates an operation to set the air conditioner temperature low.

[0699] In this way, the system provides appropriate operations that respond to the user's emotions. The flow of a specific process in Example 3 will be explained using Figure 21.

[0700] Step 1:

[0701] The server collects a large amount of Tsugaru dialect audio data. The collected audio data is used as input and undergoes preprocessing such as noise reduction and normalization. Specifically, it processes the audio data using an audio processing library, removing noise and standardizing the volume. The preprocessed audio data is then output.

[0702] Step 2:

[0703] The server inputs the pre-processed audio data into the speech recognition API and retrieves the corresponding text data. The speech recognition API analyzes the audio data and converts it into text data. As a result, the text data corresponding to the audio data is output.

[0704] Step 3:

[0705] The server uses the acquired text data to train a speech recognition model. Specifically, it builds a model using a machine learning library and trains it using the text data as input. As a result, the trained speech recognition model is output.

[0706] Step 4:

[0707] The server inputs text data obtained from the speech recognition model into a natural language processing model to train a generative artificial intelligence model. Specifically, it manages the model using a natural language processing library and performs training based on the text data. As a result, the trained generative artificial intelligence model is output.

[0708] Step 5:

[0709] The server incorporates an emotion engine to recognize emotions from the user's voice. Specifically, it uses a voice analysis tool to analyze the tone, speed, and volume of the voice, and performs analysis using the voice data as input. This results in the user's emotions being output.

[0710] Step 6:

[0711] The user makes a voice input saying, "Turn on the air conditioner." The terminal sends this voice to the server. The server uses a speech recognition model to convert the voice into text. As a result, text data corresponding to the voice data is output.

[0712] Step 7:

[0713] The server uses an emotion engine to recognize the user's emotions. For example, if the voice tone is high and the speed is fast, it will determine that the user is angry. Based on this, the user's emotion will be output.

[0714] Step 8:

[0715] The server uses generative artificial intelligence to generate appropriate actions in response to recognized emotions. For example, if the user is angry, it will generate an action to lower the air conditioner temperature. This ensures that the appropriate action is output.

[0716] Step 9:

[0717] The terminal executes the operation sent from the server. Specifically, it performs the operation to lower the air conditioner temperature, providing the user with a comfortable environment. This completes the operation requested by the user.

[0718] (Application Example 3)

[0719] Next, we will describe application example 3 of form example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0720] Conventional voice recognition systems are specialized for standard Japanese and have difficulty handling the diverse dialects of Japan. Furthermore, they lacked the ability to recognize user emotions and provide appropriate guidance, resulting in a limited user experience. In particular, in autonomous vehicles, it was difficult to provide appropriate navigation instructions when users spoke in dialect or were emotionally agitated.

[0721] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes a speech recognition means, a generative AI means, and a learning means for supporting all Japanese dialects. This makes it possible to accurately recognize the user's speech even when they speak in a dialect, and further understand the user's emotions to provide appropriate navigation instructions.

[0722] "Speech recognition means" refers to technology for converting speech into text data.

[0723] "Generative AI methods" refer to artificial intelligence technologies that generate appropriate operations or responses based on input data.

[0724] "A learning method for handling all Japanese dialects" refers to a technology that learns the data necessary to recognize the diverse dialects used within Japan.

[0725] "Methods for learning Tsugaru dialect" refers to learning techniques specifically designed to recognize Tsugaru dialect, a dialect spoken in the Tsugaru region of Japan.

[0726] "Means for the practical application of speech recognition" refers to technologies for applying speech recognition technology to actual systems and applications.

[0727] "Emotion recognition means" refers to technology that analyzes and recognizes emotions from a user's voice or text.

[0728] "Means for generating navigation instructions in autonomous vehicles" refers to technologies that generate routes and instructions to a destination in autonomous vehicles.

[0729] "Means of providing navigation instructions that respond to user emotions" refers to technologies that provide optimal navigation instructions by taking into account the user's emotional state.

[0730] The following system configuration will be described as an embodiment for carrying out this invention.

[0731] System Configuration

[0732] hardware

[0733] Microphone: Used to acquire user voice input.

[0734] Speaker: Used to output audio from the system.

[0735] Autonomous vehicle: A vehicle equipped with a navigation system.

[0736] software

[0737] Speech recognition library (speech_recognition): Used to convert user speech into text.

[0738] Emotion recognition library (transformers pipeline): Used to analyze emotions from user text data.

[0739] Speech synthesis library (pyttsx3): Used to convert text into speech.

[0740] Processing flow

[0741] 1. Acquisition of voice input

[0742] When a user speaks into the microphone, a speech recognition library converts the audio into text data.

[0743] For example, if a user asks "Where's the nearest convenience store from here?" in Tsugaru dialect, that voice will be converted into text data.

[0744] 2. Recognition of emotions

[0745] The converted text data is sent to an emotion recognition library, where the user's emotions are analyzed.

[0746] For example, if a user is feeling anxious, the emotion recognition library will recognize that emotion as "anxiety."

[0747] 3. Generating navigation instructions

[0748] Based on the results of emotion recognition, the generative AI generates appropriate navigation instructions.

[0749] For example, if the system detects that the user is in a hurry, it will generate instructions to quickly direct them to the nearest convenience store.

[0750] 4. Audio Output

[0751] The generated navigation instructions are converted into speech using a speech synthesis library and output through the speaker.

[0752] For example, a voice message might say, "Please stay calm. The nearest convenience store is here."

[0753] Specific example

[0754] If a user asks in Tsugaru dialect, "Where's the nearest convenience store from here?", the system will recognize the dialect and, if it determines the user is in a hurry, will quickly guide them to the nearest convenience store.

[0755] Example of a prompt

[0756] User: "Where's the nearest convenience store from here?"

[0757] System: "Please stay calm. The nearest convenience store is here."

[0758] In this way, the dialect-compatible emotion recognition navigation system can understand the user's dialect and emotions and provide appropriate navigation instructions.

[0759] The flow of the specific processing in Application Example 3 will be explained using Figure 22.

[0760] Step 1:

[0761] The user speaks into the microphone. The speech recognition library (speech_recognition) captures the audio and converts it into text data. The input is the user's voice, and the output is text data. Specifically, if the user says "Where's the nearest convenience store from here?" in Tsugaru dialect, that audio will be converted into the text data "Where's the nearest convenience store from here?".

[0762] Step 2:

[0763] The server sends the converted text data to the emotion recognition library (transformers pipeline). The emotion recognition library analyzes the user's emotions from the text data. The input is text data, and the output is emotion data. Specifically, the text data "Where is the nearest convenience store from here?" is converted into emotion data of "anxiety".

[0764] Step 3:

[0765] The server receives the emotion recognition results, and a generative AI generates appropriate navigation instructions. The input is emotion data and text data, and the output is navigation instructions. Specifically, based on the emotion data "anxiety" and the text data "Where is the nearest convenience store from here?", the navigation instruction "Please calm down. The nearest convenience store is here." is generated.

[0766] Step 4:

[0767] The server sends the generated navigation instructions to the speech synthesis library (pyttsx3). The speech synthesis library converts the navigation instructions into speech and outputs it through the speaker. The input is the navigation instructions, and the output is speech data. Specifically, the navigation instruction, "Please stay calm. The nearest convenience store is here," is converted into speech data and output through the speaker.

[0768] In this way, the dialect-compatible emotion recognition navigation system can understand the user's dialect and emotions and provide appropriate navigation instructions.

[0769] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0770] Data generation model 58 is a form of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include the following. Data generation model 58 is

[0771] This is obtained by performing deep learning on a neural network. The data generation model 58 receives prompts containing instructions, as well as inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0772] Other examples of generative AI include Gemini® (registered trademark) (Internet search). <url: https: gemini.google.com ?hl="ja">) are some examples.

[0773] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0774] [Second Embodiment]

[0775] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0776] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0777] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0778] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0779] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0780] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0781] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0782] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0783] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0784] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0785] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0786] Next, the identification process performed by the identification processing unit 290 of the data processing device 12 will be described.

[0787] "Example of form 1"

[0788] The present invention is a system that combines speech recognition means and generative AI means. The speech recognition means recognizes the voice input when an elderly person operates an IT device. The generative AI means generates operations based on the recognized voice input. Furthermore, this system has learning means to support all Japanese dialects. Specifically, by training the system on Tsugaru dialect, which is said to be the most difficult dialect in Japan, the practical application of speech recognition has been achieved.

[0789] "Example of form 2"

[0790] As a specific embodiment, for example, if an elderly person says "Turn on the TV" in the Tsugaru dialect, the voice recognition means recognizes this voice input, and the generative AI means generates an operation to turn on the TV. This operation is realized, for example, by controlling a device that generates infrared signals.

[0791] "Example of form 3"

[0792] Furthermore, this system possesses learning capabilities to handle all Japanese dialects. Specifically, it uses a large amount of Tsugaru dialect audio data to train its speech recognition and generative AI systems. This allows it to handle not only Tsugaru dialect but also other dialects using the same method.

[0793] The following describes the processing flow for each example of the form.

[0794] "Example of form 1"

[0795] Step 1: Enable voice input for elderly individuals to operate IT devices. This voice input includes instructions such as "Turn on the TV."

[0796] Step 2: The speech recognition system recognizes the voice input of the elderly person. This speech recognition is performed using a learning method that can handle all Japanese dialects.

[0797] Step 3: The generative AI system generates an action based on the voice input it recognizes. This action may include a specific action such as turning on the TV.

[0798] "Example of form 2"

[0799] Step 1: The elderly person says "Turn on the TV" in Tsugaru dialect.

[0800] Step 2: The speech recognition system recognizes this voice input. This speech recognition is performed with high accuracy as a result of training on the Tsugaru dialect.

[0801] Step 3: The generative AI system generates the operation to turn on the television. This operation is achieved by controlling a device that generates infrared signals.

[0802] "Example of form 3"

[0803] Step 1: Train speech recognition and generative AI using a large amount of Tsugaru dialect audio data.

[0804] Step 2: Using the trained speech recognition and generative AI systems, the system recognizes the elderly person's voice input and generates commands.

[0805] Step 3: Execute the generated operation. This operation may include specific actions such as turning on the TV.

[0806] (Example 1)

[0807] Next, we will describe Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0808] When elderly people operate information technology devices, there is a need to improve ease of use by using voice input. However, voice recognition systems that can handle Japan's diverse dialects, especially difficult dialects like Tsugaru dialect, have not yet been put into practical use. Therefore, there is a need to develop a voice recognition system that allows elderly people to speak naturally in their own dialect.

[0809] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means. In this invention, the server includes a speech recognition means, a generative artificial intelligence means, and a learning means for supporting all Japanese dialects. This enables elderly people to speak naturally in their own dialect and to operate information technology devices using voice input.

[0810] "Speech recognition means" refers to technology for converting speech data into text data.

[0811] "Generative artificial intelligence means" refers to artificial intelligence technology for generating appropriate operation instructions based on input data.

[0812] "Learning methods to handle all Japanese dialects" refers to a technology that pre-trains a speech recognition system to recognize the diverse dialects used within Japan.

[0813] "Methods for learning Tsugaru dialect" refers to a technology that specially trains a speech recognition system to recognize Tsugaru dialect, which is said to be the most difficult dialect in Japan.

[0814] "Audio data preprocessing means" refers to technologies that improve the accuracy of speech recognition by performing noise reduction and volume normalization on audio data.

[0815] "Means for the practical application of speech recognition" refers to technologies that enable the application of speech recognition technology to actual systems and applications, making it practically usable.

[0816] "Voice input when elderly people operate information technology devices themselves" refers to voice commands that elderly people utter to operate information technology devices themselves.

[0817] This invention is a system for improving the convenience of elderly people when operating information technology devices using voice input. This system includes voice recognition means, generative artificial intelligence means, learning means for handling all Japanese dialects, means for learning Tsugaru dialect, voice data preprocessing means, and means for putting voice recognition into practical use.

[0818] System Configuration

[0819] Speech recognition means

[0820] The server uses speech recognition software to convert speech data into text data. For example, the Google Cloud Speech-to-Text API is used for this purpose. This API is pre-trained to handle a wide variety of Japanese dialects.

[0821] Audio data preprocessing means

[0822] The server uses audio processing libraries such as Librosa to remove noise and normalize the volume of the audio data. This makes the audio data clearer and easier to understand.

[0823] Generative artificial intelligence methods

[0824] The server uses a generative artificial intelligence model (e.g., OpenAI's GPT-4) to generate operation instructions based on text data. This generative AI model is pre-trained on a variety of operation scenarios to understand user intent and generate appropriate operation instructions.

[0825] Execute the operation

[0826] The device executes the operation instructions it receives from the server. For example, it might open an email app and send an email with the specified content.

[0827] Specific example

[0828] Example 1: Sending an email using voice input

[0829] The user uses voice input to say, "Send me an email."

[0830] 1. The server uses speech recognition software to convert speech into text.

[0831] 2. The server uses Librosa to perform noise reduction and volume normalization on the audio data.

[0832] 3. The server uses a generative artificial intelligence model to generate the command "send an email."

[0833] 4. The device opens the email app and sends an email with the specified content.

[0834] Example 2: Obtaining weather information via voice input

[0835] The user voice-inputs, "What's the weather like today?"

[0836] 1. The server uses speech recognition software to convert speech into text.

[0837] 2. The server uses Librosa to perform noise reduction and volume normalization on the audio data.

[0838] 3. The server uses a generative artificial intelligence model to generate the operation instruction "Retrieve weather information".

[0839] 4. The device opens a weather app and displays the current weather information.

[0840] Example of a prompt

[0841] "Design a system that recognizes voice input from elderly people speaking in Tsugaru dialect and generates appropriate actions. Specifically, explain how to combine voice recognition software and a generative artificial intelligence model to generate actions based on voice input."

[0842] In this way, this system can improve the convenience for elderly people when operating information technology devices using voice input.

[0843] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0844] Step 1:

[0845] The user uses voice input to enter information into the terminal. For example, they might say, "Send me an email." This voice input becomes the initial input for the system.

[0846] Step 2:

[0847] The terminal sends the acquired audio data to the server. The server uses an audio processing library such as Librosa to remove noise and normalize the volume of the audio data. This makes the audio data clearer and easier to recognize. The input is raw audio data, and the output is pre-processed audio data.

[0848] Step 3:

[0849] The server sends pre-processed audio data to speech recognition software (e.g., Google Cloud Speech-to-Text API) to convert the audio to text. The input is pre-processed audio data, and the output is text data. For example, the text "Send me an email" is generated.

[0850] Step 4:

[0851] The server uses a generative artificial intelligence model (e.g., OpenAI's GPT-4) to generate operation instructions based on text data. The input is text data obtained through speech recognition, and the output is a specific operation instruction. For example, the operation instruction "send an email" is generated.

[0852] Step 5:

[0853] The terminal executes the operation instructions received from the server. For example, it might open an email application and send an email with specified content. The input is the operation instructions from the server, and the output is the actual result of the operation.

[0854] In this way, the system can improve the convenience for elderly people when operating information technology devices using voice input.

[0855] (Application Example 1)

[0856] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0857] Elderly people face challenges in finding products or asking questions in physical stores because they have difficulty operating IT equipment and obtaining appropriate information. Furthermore, there is a lack of voice recognition systems that support all Japanese dialects, and there is a particular need for systems that can handle complex dialects.

[0858] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0859] In this invention, the server includes: a voice recognition means for recognizing voice input when an elderly person operates a smartphone themselves; a means for identifying the dialect spoken by the elderly person based on the recognized voice input; a means for generating operation instructions for the smartphone using a prompt statement and a generative AI based on the recognized voice input and the identified dialect; and a means for operating the smartphone according to the generated operation instructions. Furthermore, the voice recognition means further includes a means for recognizing voice input when the elderly person asks a question about a product in a physical store, and the server further includes a means for generating an answer to the question using a prompt statement and a generative AI based on the recognized voice input and the identified dialect. This makes it possible to create a system that allows elderly people to easily operate a physical store using voice input when searching for products or asking questions.

[0860] "Speech recognition means" refers to technology for converting speech into text.

[0861] "Generative AI methods" refer to artificial intelligence technologies that generate appropriate responses or operations based on input data.

[0862] "A learning method for handling all Japanese dialects" refers to a technology that learns the data necessary to understand and recognize dialects from various regions of Japan.

[0863] "Methods for learning Tsugaru dialect" refers to techniques for learning the data necessary to understand and recognize Tsugaru dialect, which is said to be the most difficult dialect in Japan.

[0864] "Means for the practical application of speech recognition" refers to technologies for applying speech recognition technology to actual applications and systems.

[0865] "A means of recognizing voice input used by elderly people when searching for products or asking questions in physical stores, and generating appropriate operations or responses" refers to a technology that recognizes the voice input used by elderly people when searching for products or asking questions in physical stores, and generates appropriate operations or responses.

[0866] "Applications installed on smartphones" refers to software installed on a smartphone to provide specific functions.

[0867] A system for carrying out this invention includes a voice recognition means, a generative AI means, a means for identifying all Japanese dialects, a means for learning Tsugaru dialect, a means for putting voice recognition into practical use, a means for recognizing voice input when elderly people search for products or ask questions in a physical store and generating appropriate operations or answers, and an application means installed on a smartphone.

[0868] System program

[0869] This system operates using the following hardware and software:

[0870] Hardware: Smartphone (with microphone)

[0871] Software: Python, SpeechRecognition library, Transformers library (Hugging Face GPT-3® model)

[0872] Processing flow

[0873] Speech recognition

[0874] When a user speaks into their smartphone's microphone, a speech recognition system captures the audio and converts it into text. The SpeechRecognition library is used for this process.

[0875] Generative AI

[0876] The speech data, converted into text by the speech recognition system, is sent to the generative AI. The generative AI uses the Transformers library and a GPT-3 model to generate appropriate operations and responses.

[0877] output

[0878] The generated actions and responses are displayed on the smartphone screen. This allows elderly people to easily use voice input when searching for products or asking questions in physical stores.

[0879] Specific example (smartphone operation)

[0880] When an elderly person says "Send an email" to their smartphone, the application recognizes the voice, and the generative AI identifies that it is in the Tsugaru dialect. It then generates instructions to launch the email sending app, which the application then launches according to those instructions. The generative AI also generates a response such as "Please enter the message you want to send," and the application displays that response.

[0881] Example of a prompt

[0882] "An elderly person is using voice input to say 'send email.' This is in Tsugaru dialect. Please generate instructions for operating a smartphone."

[0883] In this way, a system can be realized that supports elderly people in making it easier for them to operate smartphones.

[0884] Specific example (physical store)

[0885] When an elderly person speaks to their smartphone in a physical store and asks, "Where is product A?", the application recognizes the voice, and a generative AI identifies that it is in the Tsugaru dialect. It then generates and displays a specific answer such as, "Product A is on the left side of aisle 3."

[0886] Example of a prompt

[0887] "An elderly person is voice-inputting, 'Where is product A?' This is in Tsugaru dialect. Please generate an answer to this question."

[0888] Generative AI's answer: The product is located on the left side of aisle 3.

[0889] In this way, a system can be realized that supports elderly people in making shopping at physical stores easier.

[0890] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0891] Step 1:

[0892] The user speaks into the microphone of their smartphone. The input is the user's voice, and the output is audio data. For example, the user might say, "Where can I find this product?"

[0893] Step 2:

[0894] The device uses speech recognition to convert audio data into text. The input is audio data, and the output is text data. Specifically, the SpeechRecognition library is used to convert the audio into the text "Where is product A located?".

[0895] Step 3:

[0896] The device sends text data to the generative AI. The input is text data, and the output is input data to the generative AI. Specifically, the converted text "Where is product A located?" is sent to the generative AI.

[0897] Step 4:

[0898] The server uses generative AI to generate appropriate instructions or responses based on text data. The input is text data, and the output is the generated instructions or responses. Specifically, the Transformers library is used to generate the response "Product A is on the left side of aisle 3" using a GPT-3 model.

[0899] Step 5:

[0900] The server sends the generated instructions and responses to the terminal. The input is the generated instructions and responses, and the output is the data sent to the terminal. Specifically, the generated response "Product A is on the left side of aisle 3" is sent to the terminal.

[0901] Step 6:

[0902] The device operates according to the generated instructions and displays the generated response to the user. Input is the transmitted data, and output is the data displayed to the user. As a specific example, the response "Product A is on the left side of aisle 3" is displayed on the smartphone screen.

[0903] Furthermore, an emotion engine, as described later, may be used to recognize the emotional state of the elderly person, and smartphone operation instructions and answers to questions may be generated taking the elderly person's emotional state into further consideration. In this case, the server further includes means for recognizing the emotional state of the elderly person, and the generating means generates the smartphone operation instructions using a prompt statement for generating smartphone operation instructions based on the recognized voice input, the identified dialect, and the recognized emotional state of the elderly person, and the generation AI.

[0904] (Example 2)

[0905] Next, we will describe Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0906] When elderly people operate information technology devices, there is a challenge in accurately recognizing and generating appropriate commands, especially for voice input using dialects. Furthermore, there is a lack of voice recognition technology capable of handling specific dialects, such as the difficult Tsugaru dialect. Therefore, there is a need for systems that can accurately handle voice input using these dialects.

[0907] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0908] In this invention, the server includes a speech recognition means, a generative artificial intelligence means, a dialect-compatible learning means, a means for learning a specific dialect, a means for putting speech recognition into practical use, a means for converting speech input into text data, a means for transmitting the text data to a generative artificial intelligence model, a means for receiving generated operations, and a means for controlling an infrared signal generator. This makes it possible to accurately recognize speech and generate appropriate operations even when elderly people use dialects to operate information technology devices.

[0909] "Speech recognition means" refers to a device or software for analyzing speech input and converting it into text data.

[0910] "Generative artificial intelligence means" refers to an artificial intelligence model that generates appropriate operations or responses based on input data.

[0911] A "dialect-compatible learning method" is a method for training a speech recognition model to handle dialects from different regions.

[0912] "Methods for learning specific dialects" refer to methods for training speech recognition models specifically on difficult dialects such as Tsugaru dialect.

[0913] "Means for the practical application of speech recognition" refers to methods for integrating speech recognition technology into actual systems and devices to make it usable.

[0914] "Means for converting voice input into text data" refers to means for converting voice input into text format using voice recognition means.

[0915] "Means for sending text data to a generative artificial intelligence model" refers to means for sending converted text data to a generative artificial intelligence model.

[0916] "Means for receiving generated operations" refers to means for receiving operations generated from a generative artificial intelligence model.

[0917] "Means for controlling an infrared signal generating device" refers to means for controlling a device that generates infrared signals and performing a specific operation.

[0918] This invention is a system that accurately recognizes speech and generates appropriate commands when elderly people operate information technology devices using their local dialect. A specific embodiment of this system is described below.

[0919] First, the user speaks in Tsugaru dialect, saying "Turn on the TV." The microphone built into the device captures this voice. Voice recognition software (for example, a voice recognition API) is used as the voice recognition method. This software analyzes the captured voice and converts it into text data.

[0920] Next, the terminal sends the converted text data to a generative artificial intelligence model (for example, a generative AI model). This generative AI model generates appropriate operations based on the input text data. Specifically, prompt statements like the following are generated.

[0921] Example of a prompt:

[0922] The user said "Turn on the TV" in Tsugaru dialect. Generate an operation to create an infrared signal to turn on the TV.

[0923] Based on this prompt, the server uses a generative artificial intelligence model to generate specific commands for turning on the television. These generated commands are then sent to the terminal.

[0924] The terminal controls an infrared signal generator (for example, an infrared signal generator) based on the received operation. This device emits an infrared signal towards the television, and the television receives this signal and turns on.

[0925] As a concrete example, when a user speaks "Turn on the TV" in Tsugaru dialect, the microphone built into the device captures this audio, and a speech recognition API is used to convert the audio into text. This text data is sent to a generative AI model, which generates the following prompt.

[0926] Example of a prompt:

[0927] The user said "Turn on the TV" in Tsugaru dialect. Generate an operation to create an infrared signal to turn on the TV.

[0928] Based on this prompt, the server uses a generative AI model to generate specific instructions for turning on the TV and sends them to the terminal. The terminal controls the infrared signal generator, and the TV receives the infrared signal and turns on.

[0929] In this way, even when users operate information technology devices using dialects, it becomes possible to accurately recognize their speech and generate appropriate commands.

[0930] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0931] Step 1:

[0932] The user performs voice input.

[0933] The user speaks in Tsugaru dialect, saying "Turn on the TV." The input is the user's voice, and the output is audio data captured by the device's microphone.

[0934] Step 2:

[0935] The device recognizes the voice.

[0936] The microphone built into the device captures the user's voice. A speech recognition API is used as the means of speech recognition. The input is the captured audio data, and the output is the parsed text data.

[0937] Step 3:

[0938] The device converts the recognized speech into text.

[0939] The device uses a speech recognition API to convert captured audio into text data. The input is audio data, and the output is text data.

[0940] Step 4:

[0941] The device sends text to an AI model that generates text.

[0942] The terminal sends the converted text data to the generating AI model. The input is text data, and the output is the data sent to the generating AI model.

[0943] Step 5:

[0944] The server generates operations using an AI model.

[0945] The server uses a generative AI model to generate appropriate actions based on the received text data. Specifically, prompt statements like the following are generated:

[0946] Example of a prompt:

[0947] The user said "Turn on the TV" in Tsugaru dialect. Generate an operation to create an infrared signal to turn on the TV.

[0948] The input is text data, and the output is the generated manipulated data.

[0949] Step 6:

[0950] The server sends the generated operation to the terminal.

[0951] The server sends the generated operation to the terminal. The input is the generated operation data, and the output is the operation data sent to the terminal.

[0952] Step 7:

[0953] The terminal controls the infrared signal generator.

[0954] The terminal controls the infrared signal generator based on the received operation. The input is operation data, and the output is an infrared signal.

[0955] Step 8:

[0956] The television operates by receiving infrared signals.

[0957] An infrared signal generator emits an infrared signal towards the television, and the television receives this signal and turns on. The input is an infrared signal, and the output is the television's operation.

[0958] (Application Example 2)

[0959] Next, we will describe application example 2 of form example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0960] In modern brick-and-mortar stores, there is a problem in that customers who speak dialects have difficulty receiving product information via voice. In particular, for the elderly and customers in areas where dialects are spoken, voice recognition systems that only understand standard Japanese cannot provide adequate service. Therefore, there is a need for a system that combines dialect-compatible voice recognition with generative AI.

[0961] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0962] In this invention, the server includes speech recognition means, generative AI means, and learning means for handling all dialects. This makes it possible for customers who speak dialects to receive product information via voice.

[0963] "Speech recognition means" refers to technology for converting speech into text.

[0964] "Generative AI methods" refer to artificial intelligence technologies that generate appropriate responses or operations based on input data.

[0965] "Learning methods for handling all dialects" refers to learning algorithms for understanding and recognizing dialects from different regions.

[0966] "Methods for learning difficult dialects" refers to special learning processes for recognizing dialects that are particularly difficult to understand.

[0967] "Means for the practical application of speech recognition" refers to means that enable speech recognition technology to be used in actual applications.

[0968] "A means of providing information about products within a store" refers to a system used to provide customers with information about the location of products within a store.

[0969] "Applications installed on smart devices" refers to software installed on devices such as smartphones and tablets.

[0970] In order to implement this invention, it is necessary to construct a system that includes a speech recognition means, a generative AI means, a learning means for handling all dialects, a means for learning difficult dialects, a means for putting speech recognition into practical use, a guidance means for providing product information within a store, and an application means to be installed on a smart device.

[0971] System Configuration

[0972] 1. Speech recognition means:

[0973] The speech recognition system is used to convert the user's spoken voice into text. Specifically, it uses the smartphone's microphone to capture the voice and the SpeechRecognition library to convert the voice into text.

[0974] 2. Generative AI means:

[0975] The generative AI system generates appropriate responses and actions based on text acquired by the speech recognition system. Specifically, it utilizes the GPT-3 model with the Hugging Face transformers library.

[0976] 3. Learning methods for handling all dialects:

[0977] This method includes a learning algorithm for understanding and recognizing dialects from different regions. In particular, a special learning process is implemented as a means of learning difficult dialects.

[0978] 4. Means of practical application of speech recognition:

[0979] This is a means to make speech recognition technology usable in actual applications. This includes an interface for a generative AI system to generate an appropriate response based on the speech recognition results and provide it to the user.

[0980] 5. Means of providing information about products within the store:

[0981] This system provides customers with information and location details about products within a store. Specifically, it displays a map showing product locations on a smartphone screen and provides voice guidance.

[0982] 6. Application methods installed on smart devices:

[0983] This is software installed on devices such as smartphones and tablets. This application integrates voice recognition and generative AI to enable users to receive product information via voice.

[0984] Processing flow

[0985] 1. Acquisition of voice input:

[0986] When a user speaks into their smartphone, the voice recognition system captures that voice. For example, the user might say, "Find this product."

[0987] 2. Speech recognition:

[0988] The speech recognition system converts the acquired speech into text. The SpeechRecognition library is used to convert the speech to text.

[0989] 3. Response generation by generative AI:

[0990] A generative AI system uses the converted text as a prompt to generate an appropriate response. Specifically, it uses the Hugging Face GPT-3 model to generate the response.

[0991] 4. Providing a response:

[0992] The generated response is displayed on the smartphone screen or announced via voice. For example, a response such as "This product is on the shelf at the back right of the store" is generated.

[0993] Specific example

[0994] User voice input: "Find this product"

[0995] Generated response: "This item is located on the shelf at the back right of the store."

[0996] Example of a prompt

[0997] "If someone asks me to 'find this product' in Tsugaru dialect, how would I guide them to the product in the store?"

[0998] In this way, combining dialect-compatible speech recognition and generation AI can significantly improve customer service in physical stores.

[0999] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1000] Step 1:

[1001] The user speaks into their smartphone.

[1002] Input: User's voice (e.g., "Find this product")

[1003] Operation: The smartphone's microphone acquires sound.

[1004] Output: Acquired audio data

[1005] Step 2:

[1006] The device uses speech recognition to convert the acquired speech data into text.

[1007] Input: Audio data

[1008] Operation: Converts speech to text using the SpeechRecognition library.

[1009] Output: Converted text (e.g., "Find this product")

[1010] Step 3:

[1011] The terminal uses generative AI means to generate a response based on the converted text.

[1012] Input: Converted text

[1013] Operation: Uses the Hugging Face GPT-3 model to generate prompt sentences and appropriate responses.

[1014] Output: Generated response text (Example: "This item is on the shelf at the back right of the store.")

[1015] Step 4:

[1016] The terminal provides the user with the generated response.

[1017] Input: Generated response text

[1018] Function: Displays the response text on the smartphone screen, or plays the response as audio.

[1019] Output: The user confirms the response.

[1020] Step 5:

[1021] The user searches for the product by following the instructions.

[1022] Input: Response text or voice guidance

[1023] Operation: The user moves around the store and searches for products by following the directions.

[1024] Output: The user finds the desired product.

[1025] In this way, specific actions and data processing / calculations are performed at each step, enabling customer service in physical stores that utilizes dialect-compatible speech recognition and generation AI.

[1026] (Example 3)

[1027] Next, we will describe Embodiment 3 of Embodiment Example 3. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[1028] Developing a speech recognition system that can handle all Japanese dialects is particularly challenging, especially for complex dialects. In particular, handling Tsugaru dialect, considered the most difficult dialect in Japan, is difficult with conventional speech recognition technology. Furthermore, accurately recognizing the voice input of elderly users operating information technology devices and generating appropriate responses is also a challenge.

[1029] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means.

[1030] In this invention, the server includes a speech recognition means, a generative artificial intelligence means, and a learning means for handling all Japanese dialects. This enables a system that includes means for the user to input voice into a terminal, means for the terminal to send voice data to the server, means for the server to convert the voice data into text data using speech recognition software, means for the server to input the text data into a generative artificial intelligence model, means for the generative artificial intelligence model to generate an appropriate response, means for the server to send the generated response to the terminal, and means for the terminal to display the response to the user. This enables speech recognition and response generation that can handle all Japanese dialects, and can handle particularly difficult dialects such as Tsugaru dialect. Furthermore, it can accurately recognize voice input when elderly people operate information technology devices themselves and generate appropriate responses.

[1031] "Speech recognition means" refers to technology for converting speech data into text data.

[1032] "Generative artificial intelligence means" refers to artificial intelligence technology that generates appropriate responses based on input text data.

[1033] A "learning method" is a method of training a system using specific data to improve its ability to perform specific tasks.

[1034] "Methods for learning Tsugaru dialect" refers to a method of training speech recognition and generative artificial intelligence using a large amount of Tsugaru dialect audio data.

[1035] "Means for the practical application of speech recognition" refers to methods for applying speech recognition technology to actual applications and systems.

[1036] "Means by which users input voice into a device" refers to methods by which users input voice into a device such as a smartphone or personal computer.

[1037] "Means of sending audio data from a terminal to a server" refers to a method of sending audio data captured by a terminal to a server via the internet.

[1038] "A means by which a server converts audio data into text data using speech recognition software" refers to a method by which a server converts received audio data into text data using speech recognition software.

[1039] "Means by which a server inputs text data into a generative artificial intelligence model" refers to the method by which a server inputs the converted text data into a generative artificial intelligence model.

[1040] "Means by which a generative artificial intelligence model generates an appropriate response" refers to a method by which a generative artificial intelligence model generates an appropriate response based on input text data.

[1041] "Means for sending the server-generated response to the terminal" refers to a method by which the server sends the response returned from the generative artificial intelligence model to the terminal.

[1042] "Means by which a terminal displays a response to the user" refers to a method by which a terminal displays a response received from a server to the user.

[1043] This invention provides a speech recognition system that supports all Japanese dialects. In particular, it has a learning method for supporting Tsugaru dialect, which is said to be the most difficult dialect in Japan. The system includes a speech recognition means, a generative artificial intelligence means, a learning means, a means for putting speech recognition into practice, a means for the user to input voice into a terminal, a means for the terminal to send voice data to a server, a means for the server to convert the voice data into text data using speech recognition software, a means for the server to input the text data into a generative artificial intelligence model, a means for the generative artificial intelligence model to generate an appropriate response, a means for the server to send the generated response to the terminal, and a means for the terminal to display the response to the user.

[1044] The server first collects a large amount of Tsugaru dialect audio data. This audio data is converted into text data using speech recognition software (e.g., Google Speech-to-Text API). Next, a generative artificial intelligence model (e.g., OpenAI's GPT-4) is trained using the converted text data.

[1045] The terminal transmits the voice data entered by the user to the server in real time. The server converts the received voice data into text data using speech recognition software and inputs that text data into a generative artificial intelligence model. The generative artificial intelligence model generates an appropriate response based on the input text data and sends that response back to the terminal as text data.

[1046] The user receives responses generated through the device and provides voice input again as needed. By repeating this process, the user can engage in natural conversations that are compatible with their dialect.

[1047] As a concrete example, consider a scenario where a user speaks in Tsugaru dialect, saying "How's it going today?" The terminal sends this voice data to a server, which uses speech recognition software to convert it into text data: "How's it going today?" Next, this text data is input into a generative artificial intelligence model to generate an appropriate response. For example, the response "It's nice weather today" might be generated. This response is sent back to the terminal as text data and displayed to the user.

[1048] Examples of prompt statements include the following:

[1049] User input: "How's it going today?"

[1050] Prompt to the Generative AI Model: "The user has spoken in Tsugaru dialect, saying 'How are you today?' Please generate an appropriate response."

[1051] In this way, the system provides a means of learning to support all Japanese dialects, enabling users to engage in natural conversations. The flow of specific processing in Example 3 will be explained using Figure 15.

[1052] Step 1:

[1053] The user enters voice input into the device.

[1054] The user speaks into a device such as a smartphone or computer. For example, they might say "How's it going today?" in Tsugaru dialect. This voice is captured by the device's microphone. The input is the user's voice data, and the output is the voice data captured by the device.

[1055] Step 2:

[1056] The device sends voice data to the server.

[1057] The device compresses the captured audio data and sends it to the server via the internet. Specifically, the device compresses the audio data into MP3 format and sends it to the server using the HTTPS protocol. The input is the captured audio data, and the output is the audio data sent to the server.

[1058] Step 3:

[1059] The server converts the audio data into text data using speech recognition software.

[1060] The server sends the received audio data to speech recognition software (e.g., Google Speech-to-Text API) and converts it into text data. For example, the audio data "How's it going today?" is converted to the text data "How's it going today?". The input is the audio data sent to the server, and the output is the converted text data.

[1061] Step 4:

[1062] The server inputs text data into a generative artificial intelligence model.

[1063] The server sends the converted text data as an API request to a generative artificial intelligence model (e.g., OpenAI's GPT-4). For example, it sends the text data "How's it going today?" to the generative AI model. The input is the converted text data, and the output is an API request to the generative AI model.

[1064] Step 5:

[1065] The generative artificial intelligence model generates an appropriate response.

[1066] Generative artificial intelligence models generate appropriate responses based on input text data. For example, they might generate the response, "It's a nice day today." This response is returned to the server as text data. The input is an API request to the generative artificial intelligence model, and the output is the generated response text data.

[1067] Step 6:

[1068] The server sends the generated response to the terminal.

[1069] The server receives the response text data returned from the generative artificial intelligence model and sends it to the terminal as an API response. For example, it sends the text data "It's a nice day today" to the terminal. The input is the generated response text data, and the output is the response text data sent to the terminal.

[1070] Step 7:

[1071] The terminal displays a response to the user.

[1072] The terminal displays the response text data received from the server to the user. For example, the text "It's a nice day today" is displayed on the smartphone screen. The input is the response text data sent to the terminal, and the output is the response text displayed to the user.

[1073] (Application Example 3)

[1074] Next, we will describe application example 3 of form example 3. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[1075] Conventional speech recognition systems are specialized for standard Japanese and have difficulty handling the diverse dialects of Japan. Furthermore, they are particularly unable to handle Tsugaru dialect, considered one of the most difficult dialects in Japan, resulting in low convenience for users who speak the dialect. Additionally, in food delivery services, ordering in dialect is difficult, forcing users to use standard Japanese.

[1076] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes a speech recognition means, a generative AI means, a learning means for supporting all Japanese dialects, a means for learning Tsugaru dialect, which is said to be the most difficult dialect in Japan, a means for putting speech recognition into practical use, a means for recognizing and analyzing speech input in dialect and generating order content, a means for transmitting the generated order content via a communication means, a means for allowing the user to confirm the order content, and a means for notifying the user that the order has been confirmed. As a result, when a user who speaks a dialect uses a food delivery service, they will be able to input speech in their dialect, improving convenience.

[1077] "Speech recognition means" refers to technology for converting speech into text.

[1078] "Generative AI methods" are artificial intelligence technologies that generate new information or operations based on input data.

[1079] "Learning methods to handle all Japanese dialects" refers to technologies that collect and learn the data necessary to recognize and understand dialects from various regions of Japan.

[1080] "Methods for learning Tsugaru dialect" refers to the technology of collecting and learning the data necessary to recognize and understand Tsugaru dialect, which is said to be the most difficult dialect in Japan.

[1081] "Means for the practical application of speech recognition" refers to technologies that make speech recognition technology usable in actual applications and services.

[1082] "A means of recognizing, analyzing, and generating order details from voice input in a dialect" refers to a technology that recognizes voice input in a dialect, analyzes its content, and generates appropriate order details.

[1083] "Means for transmitting generated order details via communication means" refers to technologies for transmitting generated order details using the internet or other communication means.

[1084] "Methods for allowing users to confirm order details" refer to technologies that present the generated order details to the user and ask for their confirmation.

[1085] "Means of notifying that an order has been confirmed" refers to technologies that inform users that their order has been confirmed.

[1086] The following system configuration will be described as an embodiment for carrying out this invention.

[1087] System Configuration

[1088] This system includes a speech recognition means, a generative AI means, a learning means for handling all Japanese dialects, a means for learning Tsugaru dialect, a means for putting speech recognition into practical use, a means for recognizing and analyzing speech input in dialect and generating order details, a means for transmitting the generated order details via a communication means, a means for allowing the user to confirm the order details, and a means for notifying the user that the order has been confirmed.

[1089] Hardware and software to be used

[1090] Hardware: Smartphone (iOS or Android)

[1091] software:

[1092] Speech recognition API (e.g., Google Cloud Speech-to-Text)

[1093] Generative AI models (e.g., OpenAI GPT-4)

[1094] Food delivery APIs (e.g., Uber Eats API)

[1095] Processing flow

[1096] 1. Voice input: The user speaks their order in their local dialect into the microphone of their smartphone.

[1097] 2. Speech Recognition: The smartphone uses a speech recognition API to convert the input speech into text. This process utilizes a speech recognition model that supports different dialects.

[1098] 3. Text Analysis: The server uses a generative AI model to analyze the recognized text and understand the order details.

[1099] 4. Order Generation: Based on the analysis results, the server uses the food delivery API to generate an appropriate order and sends it to the restaurant.

[1100] 5. Confirmation and Notification: The smartphone allows the user to confirm the order details and notifies them that the order has been confirmed.

[1101] Specific example

[1102] When a user speaks in Tsugaru dialect, saying "I'd like to order one pizza," the smartphone recognizes this, generates an appropriate pizza order, and sends it to the restaurant.

[1103] Example of a prompt

[1104] If a user speaks in Tsugaru dialect and says "I'd like to order one pizza," use a speech recognition API to convert the speech to text, a generative AI model to analyze the order, and a food delivery API to generate a pizza order.

[1105] In this way, a dialect-enabled food delivery assistant can enable users to place orders in their local dialect, providing a convenient service to a wider range of users.

[1106] The flow of the specific processing in Application Example 3 will be explained using Figure 16.

[1107] Step 1:

[1108] The user speaks their order in their local dialect into the smartphone's microphone. The input is the user's voice, and the output is the audio data input to the smartphone's microphone.

[1109] Step 2:

[1110] The device uses a speech recognition API (e.g., Google Cloud Speech-to-Text) to convert the input audio data into text. The input is audio data, and the output is text data. A speech recognition model that supports dialects is used in this process.

[1111] Step 3:

[1112] The server uses a generative AI model (e.g., OpenAI GPT-4) to analyze text data sent from a speech recognition API and understand the order details. The input is text data, and the output is the analyzed order details.

[1113] Step 4:

[1114] Based on the analysis results, the server generates an appropriate order using a food delivery API (e.g., Uber Eats API) and sends it to the restaurant. The input is the analyzed order data, and the output is the order data sent via the food delivery API.

[1115] Step 5:

[1116] The terminal displays the generated order details to allow the user to confirm the order. The input is order confirmation data from the food delivery API, and the output is the order details displayed on the smartphone screen.

[1117] Step 6:

[1118] The user reviews and confirms their order. The input is the user's confirmation action, and the output is the order confirmation signal.

[1119] Step 7:

[1120] The terminal notifies the server that the order has been confirmed, and the server sends this information to the food delivery API. The input is the order confirmation signal, and the output is the order confirmation data sent to the food delivery API.

[1121] Step 8:

[1122] The server notifies the user that the order has been confirmed. The input is order confirmation data from the food delivery API, and the output is an order confirmation notification displayed to the user via the smartphone's notification function.

[1123] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1124] "Example of form 1"

[1125] One embodiment of the present invention provides a system that includes a speech recognition means, a generative AI means, a learning means for handling all Japanese dialects, a means for learning Tsugaru dialect, and a means for putting speech recognition into practical use, as well as an emotion engine that recognizes the user's emotions. This emotion engine recognizes emotions from the user's voice and generates an operation corresponding to that emotion. For example, when a user makes the voice input "Turn on the TV," the emotion engine recognizes the user's emotion from that voice. If it recognizes that the emotion is "joy," the generative AI means generates an operation to turn on the TV.

[1126] "Example of form 2"

[1127] In another embodiment of the present invention, the emotion engine generates an operation in response to the user's emotions. For example, when the user makes a voice input saying "Turn on the radio," the emotion engine recognizes the user's emotions from the voice. If it recognizes that the emotion is "sadness," the generative AI means generates an operation to turn on the radio and provides a result in which the operation is adapted to the user's emotions. Specifically, it generates an operation such as setting the radio channel to one that plays sad songs that the user likes.

[1128] "Example of form 3"

[1129] Furthermore, in another embodiment of the present invention, the emotion engine recognizes the user's emotion, and the generative AI means generates an operation in response to that emotion. For example, when the user makes a voice input saying "Turn on the air conditioner," the emotion engine recognizes the user's emotion from that voice. If it recognizes that the emotion is "anger," the generative AI means generates an operation to turn on the air conditioner and provides a result in which the operation is adapted to the user's emotion. Specifically, it generates an operation such as setting the air conditioner temperature to a low temperature that will calm the user down.

[1130] The following describes the processing flow for each example of the form.

[1131] "Example of form 1"

[1132] Step 1: The user makes a voice input. In this example, the user makes the voice input, "Turn on the TV."

[1133] Step 2: The emotion engine recognizes emotions from the user's voice. In this example, it recognizes "joy."

[1134] Step 3: The generative AI generates an action based on the emotions it recognizes. In this example, it generates the action to turn on the TV.

[1135] "Example of form 2"

[1136] Step 1: The user makes a voice input. In this example, the user makes the voice input, "Turn on the radio."

[1137] Step 2: The emotion engine recognizes emotions from the user's voice. In this example, "sadness."

[1138] Recognize.

[1139] Step 3: The generative AI generates an action based on the emotions it recognizes and provides a result where the action is adapted to the user's emotions. In this example, it generates an action to turn on the radio and provides a result where the action is adapted to the user's emotions. Specifically, it generates an action such as setting the radio channel to one that plays sad songs that the user likes.

[1140] "Example of form 3"

[1141] Step 1: The user makes a voice input. In this example, the user makes the voice input, "Turn on the air conditioner."

[1142] Step 2: The emotion engine recognizes emotions from the user's voice. In this example, it recognizes "anger."

[1143] Step 3: The generative AI generates an action based on the emotions it recognizes and provides a result where that action is adapted to the user's emotions. In this example, it generates an action to turn on the air conditioner and provides a result where that action is adapted to the user's emotions. Specifically, it generates an action such as setting the air conditioner temperature to a low temperature that will help the user calm down.

[1144] (Example 1)

[1145] Next, we will describe Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[1146] When elderly people operate information technology devices, there is a challenge in accurately recognizing voice input and generating appropriate operations. In particular, there is a need to support all Japanese dialects, but the diversity and complexity of dialects are factors that reduce the accuracy of voice recognition. Furthermore, there is a problem in that operations cannot be generated while taking the user's emotions into consideration, resulting in a poor user experience.

[1147] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1148] In this invention, the server includes a speech recognition means, a generative artificial intelligence means, and a learning means for handling all Japanese dialects. This enables accurate recognition of speech input when elderly people operate information technology devices and allows for speech recognition that supports dialects. Furthermore, by including an emotion engine that recognizes the user's emotions, it becomes possible to generate operations that respond to the user's emotions, thereby improving the user experience.

[1149] "Speech recognition means" refers to technology for converting speech input into text data.

[1150] "Generative artificial intelligence means" refers to artificial intelligence technology for generating appropriate operations based on speech recognition results.

[1151] "Learning methods to handle all Japanese dialects" refers to technology for training speech recognition models to recognize the diverse dialects used within Japan.

[1152] "Methods for learning dialects" refer to techniques for specially training speech recognition models to recognize specific dialects, especially complex ones.

[1153] "Means for the practical application of speech recognition" refers to technologies for applying speech recognition technology to actual applications and systems.

[1154] An "emotion engine" is a technology that analyzes emotions from a user's voice and generates appropriate actions based on those emotions.

[1155] Modes for carrying out the invention

[1156] This invention is a system that includes speech recognition means, generative artificial intelligence means, learning means for handling all Japanese dialects, particularly means for learning difficult dialects, means for practical application of speech recognition, and an emotion engine. Specific embodiments of this system are described below.

[1157] Acquiring voice input

[1158] Subject: User

[1159] The user makes voice input to the device. For example, they might say, "Turn on the TV." The device is equipped with a high-performance microphone that captures the user's voice with high accuracy.

[1160] Sending audio data

[1161] Subject: terminal

[1162] The terminal sends the acquired audio data to the server. During this process, the audio data is converted to an appropriate format and sent to the server via the network.

[1163] Speech recognition

[1164] Subject: Server

[1165] The server converts the received audio data into text data using speech recognition. Specifically, it uses speech recognition software (for example, Google Cloud Speech-to-Text API) to convert speech to text.

[1166] Learning for dialect support

[1167] Subject: Server

[1168] The server has the means to learn how to handle all Japanese dialects. In particular, it trains its speech recognition model to recognize complex dialects (for example, Tsugaru dialect).

[1169] emotion recognition

[1170] Subject: terminal

[1171] The device has an emotion engine that recognizes emotions from the user's voice. Specifically, it uses emotion analysis software (for example, Microsoft Azure's Emotion API) to analyze emotions from the user's voice.

[1172] Operation generation

[1173] Subject: Server

[1174] The server uses generative artificial intelligence tools (e.g., OpenAI's GPT-4) to generate an action based on speech recognition and emotion recognition results. The generated action is then sent to the terminal.

[1175] Execute the operation

[1176] Subject: terminal

[1177] The terminal executes the operation received from the server. Specifically, it sends a signal to turn on the television.

[1178] Specific example

[1179] Specific Example 1

[1180] When a user says "Turn on the TV," the device captures the audio and sends it to the server. The server uses speech recognition to convert the audio into text and uses generative artificial intelligence to generate the action "Turn on the TV." The device then executes this action and turns on the TV.

[1181] Specific Example 2

[1182] When a user says "Turn off the air conditioner," the device captures the audio and sends it to the server. The server uses speech recognition to convert the audio into text and uses generative artificial intelligence to generate the action "Turn off the air conditioner." The device then executes this action and turns off the air conditioner.

[1183] Example of a prompt

[1184] "When a user says 'Turn on the TV,' use speech recognition to convert the speech into text, and then use generative artificial intelligence to generate the action to turn on the TV. Also, if the user's emotion is 'joy,' provide that information to the generative artificial intelligence."

[1185] By inputting this prompt into the generating AI model, the system's operation can be simulated.

[1186] The flow of the specific processing in Example 1 will be explained using Figure 17.

[1187] Step 1:

[1188] Acquiring voice input

[1189] Subject: User

[1190] The user makes a voice command, such as "Turn on the TV," into the device. The device is equipped with a high-performance microphone that captures the user's voice with high accuracy.

[1191] Input: User's voice instructions

[1192] Output: Acquired audio data

[1193] Specific action: Instead of the user searching for the remote control in the living room, they speak to the device and say, "Turn on the TV."

[1194] Step 2:

[1195] Sending audio data

[1196] Subject: terminal

[1197] The terminal sends the acquired audio data to the server. During this process, the audio data is converted to an appropriate format and sent to the server via the network.

[1198] Input: Acquired audio data

[1199] Output: Audio data sent to the server

[1200] Specific operation: The device sends audio data to the server via Wi-Fi. The transmission takes only a few seconds.

[1201] Step 3:

[1202] Speech recognition

[1203] Subject: Server

[1204] The server converts the received audio data into text data using speech recognition. Specifically, it uses speech recognition software (for example, Google Cloud Speech-to-Text API) to convert speech to text.

[1205] Input: Audio data sent to the server

[1206] Output: Text data (e.g., "Turn on the TV")

[1207] Specific operation: The server receives the audio data and calls the Google Cloud Speech-to-Text API to generate the text "Turn on the TV".

[1208] Step 4:

[1209] Learning for dialect support

[1210] Subject: Server

[1211] The server has the means to learn how to handle all Japanese dialects. In particular, it trains its speech recognition model to recognize complex dialects (for example, Tsugaru dialect).

[1212] Input: Audio data of dialects

[1213] Output: Speech recognition model that supports dialects

[1214] Specific operation: The server trains a model using Tsugaru dialect audio data to enable it to handle dialects.

[1215] Step 5:

[1216] emotion recognition

[1217] Subject: terminal

[1218] The device has an emotion engine that recognizes emotions from the user's voice. Specifically, it uses emotion analysis software (for example, Microsoft Azure's Emotion API) to analyze emotions from the user's voice.

[1219] Input: User's voice data

[1220] Output: Emotion data (e.g., "joy")

[1221] Specific operation: The device analyzes the user's emotion from the voice command "Turn on the TV" and recognizes that emotion as "joy."

[1222] Step 6:

[1223] Operation generation

[1224] Subject: Server

[1225] The server uses generative artificial intelligence tools (e.g., OpenAI's GPT-4) to generate an action based on speech recognition and emotion recognition results. The generated action is then sent to the terminal.

[1226] Input: Text data (e.g., "Turn on the TV"), emotion data (e.g., "Joy")

[1227] Output: Generated operation (e.g., "Turn on the TV")

[1228] Specific operation: The server inputs the text "Turn on the TV" and the emotion information "joy" into GPT-4, and generates the operation "Turn on the TV".

[1229] Step 7:

[1230] Execute the operation

[1231] Subject: terminal

[1232] The terminal executes the operation received from the server. Specifically, it sends a signal to turn on the television.

[1233] Input: Generated action (e.g., "Turn on the TV")

[1234] Output: The action performed (e.g., the TV is turned on)

[1235] Specific action: The device sends a remote control signal to the TV, and the TV turns on.

[1236] (Application Example 1)

[1237] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[1238] There is a problem with the accuracy of voice recognition, making it difficult for elderly customers and those who speak in dialects to operate IT equipment. Furthermore, there is the challenge of appropriately recognizing and responding to customers' emotions in physical stores.

[1239] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes a speech recognition means, a generative AI means, a learning means for handling all Japanese dialects, a means for learning Tsugaru dialect, which is said to be the most difficult dialect in Japan, a means for putting speech recognition into practical use, an emotion recognition means, an application means installed on smart glasses, a means for recognizing customer voice input and for the generative AI means to generate an appropriate response, a means for the emotion recognition means to recognize the customer's emotions and generate a response corresponding to those emotions, and a speech output means. As a result, the accuracy of speech recognition is improved when elderly customers or customers who speak dialects operate IT equipment, and appropriate responses that correspond to customer emotions become possible in physical stores.

[1240] "Speech recognition means" refers to technology that analyzes speech input and converts it into text data.

[1241] "Generative AI methods" refer to artificial intelligence technologies that generate appropriate responses or operations based on input data.

[1242] "Learning methods to handle all Japanese dialects" refers to technology that trains speech recognition models to recognize the diverse dialects used within Japan.

[1243] "Methods for learning Tsugaru dialect" refers to techniques for special learning in order to understand Tsugaru dialect, which is said to be the most difficult dialect in Japan.

[1244] "Means for the practical application of speech recognition" refers to technologies that make speech recognition technology usable in actual systems and applications.

[1245] "Emotion recognition means" refers to technology that analyzes and recognizes a user's emotions from voice and text data.

[1246] "Applications installed on smart glasses" refers to applications that run on smart glasses and are software that utilizes voice recognition and generative AI.

[1247] "A means of recognizing customer voice input and generating an appropriate response using a generative AI means" refers to a technology that recognizes customer voice and generates an appropriate response using a generative AI based on its content.

[1248] "Means for recognizing customer emotions and generating responses in accordance with those emotions" refers to technology that recognizes customer emotions and generates appropriate responses in accordance with those emotions.

[1249] "Speech output means" refers to technology that outputs generated text data or responses as speech.

[1250] A system for carrying out this invention includes a speech recognition means, a generative AI means, a learning means for handling all Japanese dialects, particularly a means for learning Tsugaru dialect, a means for putting speech recognition into practical use, an emotion recognition means, an application means installed in smart glasses, a means for recognizing customer voice input and for the generative AI means to generate an appropriate response, a means for the emotion recognition means to recognize customer emotions and generate an emotion-appropriate response, and a speech output means.

[1251] Hardware and software configuration

[1252] The server uses the speech_recognition library for speech recognition and the transformers library for generative AI. The transformers library's sentiment-analysis pipeline is also used for emotion recognition. The pyttsx3 library is used for speech output. The smart glasses have a built-in microphone and speaker, and this software is installed on them.

[1253] Data processing and data calculation

[1254] 1. Speech Recognition: When the user speaks into the smart glasses, the microphone captures the audio, and the speech_recognition library converts that audio into text data.

[1255] 2. Generative AI: The transformed text data is input into a generative AI model in the transformers library, and an appropriate response is generated.

[1256] 3. Emotion Recognition: Simultaneously, the text data is input into the sentiment-analysis pipeline, where the user's emotions are analyzed. Based on the analysis results, a generative AI model generates an emotion-appropriate response.

[1257] 4. Audio Output: Finally, the generated response is output as audio using the pyttsx3 library.

[1258] Specific example

[1259] For example, if a customer in a physical store says, "Tell me about this product," the system will operate as follows:

[1260] 1. Speech Recognition: The customer's voice is captured by the microphone on the smart glasses, and the speech_recognition library converts it into text data such as "Tell me about this product."

[1261] 2. Generative AI: The converted text data is input into a generative AI model in the transformers library, which generates the response, "This product is the latest smartphone with excellent camera performance. Furthermore, it has good battery life and can be used for extended periods."

[1262] 3. Sentiment Recognition: Simultaneously, text data is input into the sentiment-analysis pipeline and analyzed to determine if the customer is in distress. Generative AI models then generate more detailed explanations.

[1263] 4. Audio Output: Finally, the generated response is output as audio using the pyttsx3 library.

[1264] Example of a prompt

[1265] Customer: Please tell me about this product.

[1266] System: This product is a state-of-the-art smartphone with excellent camera performance. Furthermore, it boasts a long battery life, allowing for extended use.

[1267] In this way, the accuracy of voice recognition improves when elderly customers or customers who speak in dialects operate IT equipment, enabling appropriate responses that respond to customers' emotions in physical stores.

[1268] The flow of a specific process in Application Example 1 will be explained using Figure 18.

[1269] Step 1:

[1270] The user speaks into the smart glasses. The smart glasses' microphone captures the audio. The input is the user's voice, and the output is the captured audio data.

[1271] Step 2:

[1272] The server uses the speech_recognition library as a speech recognition tool to convert captured audio data into text data. The input is audio data, and the output is text data. Specifically, the audio data is passed to the library, and the speech recognition algorithm is applied.

[1273] Step 3:

[1274] The server uses the transformers library as a generative AI tool to generate an appropriate response from transformed text data as input. The input is text data, and the output is the generated response text. Specifically, the text data is input to a generative AI model, and the model generates a response.

[1275] Step 4:

[1276] The server uses the `transformers` library's `sentiment-analysis` pipeline as an emotion recognition tool to analyze the user's emotions from text data. The input is text data, and the output is the emotion analysis result. Specifically, the text data is input into the emotion recognition model, and the emotions are analyzed.

[1277] Step 5:

[1278] The server analyzes emotions, and a generative AI system generates an emotion-appropriate response based on the analysis results. The input is the emotion analysis results and text data, and the output is an emotion-appropriate response text. Specifically, the emotion analysis results are fed back to the generative AI model, and the response is adjusted accordingly.

[1279] Step 6:

[1280] The server uses the pyttsx3 library as its voice output method, outputting the generated response text as speech. The input is the response text, and the output is speech data. Specifically, the response text is input to the speech synthesis engine, and speech data is generated.

[1281] Step 7:

[1282] The smart glasses play audio data and provide responses to the user. The input is audio data, and the output is the audio the user hears. Specifically, the audio data is played through the smart glasses' speaker.

[1283] (Example 2)

[1284] Next, we will describe Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[1285] When elderly people operate information technology devices, there is a challenge in accurately recognizing voice input, especially when it uses dialects, and generating appropriate operations. Furthermore, there is a need to provide more user-friendly systems that generate operations that respond to the user's emotions. In addition, a means of actually executing the generated operations is also necessary.

[1286] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1287] In this invention, the server includes a speech recognition means, a generative AI means, a dialect-compatible learning means, an emotion recognition means, a means for putting speech recognition into practical use, and an infrared signal generator control means. This makes it possible to accurately recognize speech even when elderly people use dialects, generate appropriate operations according to the user's emotions, and actually execute those operations.

[1288] "Voice recognition means" refers to a device or software that converts voice input into digital data and analyzes its contents.

[1289] A "generative AI means" is a device or software that uses artificial intelligence technology to generate appropriate operations based on data obtained from a speech recognition means.

[1290] A "dialect-compatible learning tool" is a device or software that trains a speech recognition tool to recognize dialects specific to a particular region or culture.

[1291] "Emotion recognition means" refers to a device or software that analyzes a user's emotions from voice input and recognizes those emotions.

[1292] "Means for the practical application of speech recognition" refers to devices or software that apply speech recognition technology to actual systems and applications, making it practically usable.

[1293] "Infrared signal generator control means" refers to a device or software that generates infrared signals based on instructions from a generation AI means and controls a specific device.

[1294] This invention provides a system that accurately recognizes voice input, particularly that using dialects, when elderly people operate information technology devices, and generates appropriate operations. Furthermore, it can generate operations that respond to the user's emotions and actually execute those operations.

[1295] Hardware and software to be used

[1296] 1. Speech recognition means:

[1297] The device uses its built-in microphone to capture audio.

[1298] For speech recognition software, you can use, for example, the Google Speech-to-Text API.

[1299] 2. Generative AI means:

[1300] The server uses a generated AI model (e.g., OpenAI's GPT-4) to analyze text data obtained from the speech recognition system and generate appropriate operations.

[1301] 3. Dialect-based learning methods:

[1302] The server is trained to recognize dialects specific to a particular region or culture by training its speech recognition system. For example, training data for the Tsugaru dialect of Japan is used.

[1303] 4. Emotion recognition means:

[1304] The server uses emotion recognition software (for example, IBM Watson Tone Analyzer) to analyze the user's emotions from the voice input and recognize those emotions.

[1305] 5. Means of practical application of speech recognition:

[1306] The device applies voice recognition technology to actual systems and applications, making it practically usable.

[1307] 6. Infrared signal generator control means:

[1308] The terminal sends the generated commands to an infrared signal generator (e.g., BroadLink RM4 Pro) to control a specific device.

[1309] Specific example

[1310] Example 1: Turning on the TV

[1311] 1. The user says "Turn on the TV" in Tsugaru dialect.

[1312] 2. The device uses its built-in microphone to capture audio and the Google Speech-to-Text API to convert the audio to text.

[1313] 3. The server uses a generated AI model to generate the action "Turn on the TV".

[1314] 4. The terminal sends the generated command to the infrared signal generator, turning on the television.

[1315] Example 2: Turning on the radio

[1316] 1. The user says, "Turn on the radio."

[1317] 2. The device uses its built-in microphone to capture audio and the Google Speech-to-Text API to convert the audio to text.

[1318] 3. The server uses emotion recognition software to recognize the user's emotion as "sadness."

[1319] 4. The server uses a generated AI model to generate an action to set the channel to play sad songs.

[1320] 5. The terminal sends a command to the radio and sets it to the specified channel.

[1321] Example of a prompt

[1322] "When a user says 'Turn on the TV' in Tsugaru dialect, a voice recognition system converts the speech into text, and a generation AI system generates an operation to turn on the TV. It then controls an infrared signal generator to turn on the TV."

[1323] "When a user says, 'Turn on the radio,' the emotion engine recognizes the user's emotion, and the generative AI generates radio operations that are adapted to the user's emotion. For example, it might set the radio to a channel playing sad music."

[1324] The flow of the specific processing in Example 2 will be explained using Figure 19.

[1325] Step 1:

[1326] Voice input capture

[1327] The user says "Turn on the TV" in Tsugaru dialect.

[1328] The device uses its built-in microphone to capture audio.

[1329] Input: User's voice

[1330] Output: Audio data

[1331] Specific actions:

[1332] The user is standing in front of the TV in the living room and says, "Turn on the TV."

[1333] The device captures audio through the microphone and saves it as audio data.

[1334] Step 2:

[1335] Text conversion of audio data

[1336] The device uses speech recognition software (for example, Google Speech-to-Text API) to convert the captured audio data into text data.

[1337] Input: Audio data

[1338] Output: Text data

[1339] Specific actions:

[1340] The device sends the captured audio data to the Google Speech-to-Text API.

[1341] The device receives the text data "Turn on the TV" returned from the API.

[1342] Step 3:

[1343] Recognition of emotions (as needed)

[1344] The server uses emotion recognition software (e.g., IBM Watson Tone Analyzer) to analyze the user's emotions from the text data.

[1345] Input: Text data

[1346] Output: Sentiment data

[1347] Specific actions:

[1348] The server sends the text data "Turn on the radio" to IBM Watson Tone Analyzer.

[1349] The server receives "sadness" as the analysis result.

[1350] Step 4:

[1351] Operation generation

[1352] The server uses a generated AI model (e.g., OpenAI's GPT-4) to analyze text data and generate appropriate operations.

[1353] Input: Text data, sentiment data (if necessary)

[1354] Output: Operation command

[1355] Specific actions:

[1356] The server inputs the text data "Turn on the TV" into the AI ​​model that generates the data.

[1357] The server receives the generated command and sends it to a device that generates an infrared signal to "turn on the TV".

[1358] Step 5:

[1359] Execute the operation

[1360] The terminal sends the generated command to the infrared signal generator, executing the operation to turn on the television.

[1361] Input: Operation command

[1362] Output: Action performed (TV turns on)

[1363] Specific actions:

[1364] The terminal sends a "turn on TV" command to the infrared signal generator.

[1365] The infrared signal generator emits an infrared signal towards the television, and the television turns on.

[1366] (Application Example 2)

[1367] Next, we will describe application example 2 of form example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[1368] When elderly people operate IT devices, there is a particular problem with voice recognition that supports Japanese dialects. Furthermore, the inability to provide appropriate operation and information in accordance with the user's emotions hinders the improvement of the user experience. Additionally, in physical stores, the inability to suggest products based on the user's emotions prevents the shopping experience from being optimized.

[1369] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1370] In this invention, the server includes a speech recognition means, a generative AI means, a learning means for handling all Japanese dialects, a means for learning Tsugaru dialect in particular, which is said to be the most difficult dialect in Japan, a means for putting speech recognition into practical use, a means for recognizing the user's emotions using an emotion engine, a means for generating operations corresponding to the user's emotions, a means for providing product information in a physical store, and a means for making product suggestions based on the user's emotions. As a result, accurate speech recognition becomes possible even when elderly people use dialects, appropriate operations and information provision corresponding to the user's emotions are realized, and product suggestions based on the user's emotions become possible even in a physical store.

[1371] "Speech recognition means" refers to technology for converting speech input into text data.

[1372] "Generative AI methods" refer to artificial intelligence technologies that generate appropriate responses or operations based on input data.

[1373] A "learning method" is a technique that uses a specific dataset to train a model and improve its ability to perform a specific task.

[1374] "Methods for learning Tsugaru dialect" refers to techniques for providing special training to understand and recognize the dialect used in the Tsugaru region of Japan.

[1375] "Means for the practical application of speech recognition" refers to technologies for integrating speech recognition technology into actual applications and systems.

[1376] An "emotion engine" is a technology that analyzes and recognizes emotions from a user's voice or text.

[1377] "Means for generating operations that respond to user emotions" refers to technologies that generate appropriate operations or responses based on recognized user emotions.

[1378] "Means of providing product information in physical stores" refers to technologies for providing product information to users within a physical store.

[1379] "Methods for suggesting products based on user emotions" refers to techniques for suggesting appropriate products by taking user emotions into consideration.

[1380] This invention is a system that combines speech recognition means, generative AI means, learning means, emotion engine, and means for providing product information in physical stores. Specific embodiments for realizing this system are described below.

[1381] System Configuration

[1382] This system consists of the following main components:

[1383] 1. Speech Recognition Method: This is a technology for converting speech input into text data. Specifically, the speech_recognition library is used.

[1384] 2. Generative AI methods: This is an artificial intelligence technology that generates appropriate responses or operations based on input data. Specifically, it uses the GPT-3 model from the transformers library.

[1385] 3. Learning Methods: This is a technique that trains a model using a specific dataset to improve its ability to perform specific tasks. In particular, it is used for training to handle all Japanese dialects.

[1386] 4. Emotion Engine: This is a technology for analyzing and recognizing emotions from the user's voice and text. Specifically, it uses the emotion analysis model from the transformers library.

[1387] 5. Means of providing product information in physical stores: This refers to technologies for providing product information to users within physical stores.

[1388] Program processing

[1389] The server uses speech recognition to convert the user's voice input into text data. Next, an emotion engine recognizes the user's emotions from the text data. Based on the recognized emotions, a generative AI generates appropriate responses or actions. For example, if a user asks, "Tell me about this product," and the emotion engine recognizes the user's emotion as "excited," the generative AI will generate a response such as, "This product is the latest model and has many added features."

[1390] Hardware and software to be used

[1391] Hardware: Smartphone (microphone, speaker)

[1392] software:

[1393] The speech_recognition library is used to convert speech input to text.

[1394] Transformers library: Generates responses using a generative AI model (GPT-3).

[1395] requests library: Used to communicate with external APIs (as needed).

[1396] Specific example

[1397] When a user speaks to a store and asks, "Tell me about this product," a speech recognition system converts the speech into text, and an emotion engine recognizes the user's emotions. For example, if the user is excited, a generative AI system might generate a response such as, "This product is the latest model and has many added features."

[1398] Example of a prompt

[1399] When a user is sad: "Please suggest products to recommend when a user is sad: Tell me about this product."

[1400] When the user is happy: "Please suggest products to recommend when the user is happy: Tell me about this product."

[1401] In this way, smart shopping assistants can provide appropriate information tailored to the user's emotions, thereby improving the shopping experience.

[1402] The flow of a specific process in Application Example 2 will be explained using Figure 20.

[1403] Step 1:

[1404] The server acquires the user's voice input using speech recognition technology. Specifically, it collects voice data using the smartphone's microphone and converts the voice data into text data using the speech_recognition library.

[1405] Input: User's voice data

[1406] Output: Text data

[1407] Step 2:

[1408] The server uses an emotion engine to recognize the user's emotions from text data. Specifically, it uses the emotion analysis model from the transformers library to analyze the text data and identify the user's emotions.

[1409] Input: Text data

[1410] Output: User's emotions (e.g., joy, sadness, excitement, etc.)

[1411] Step 3:

[1412] The server generates prompt messages based on the recognized user's emotions. Specifically, it creates emotionally appropriate prompt messages and inputs them into a generative AI model. For example, if the user is sad, it generates a prompt message such as, "Suggest products that are suitable for a sad user: Tell me about this product."

[1413] Input: User's sentiment, text data

[1414] Output: Prompt message

[1415] Step 4:

[1416] The server generates an appropriate response based on the prompt using generative AI tools. Specifically, it uses the GPT-3 model from the transformers library to generate the response to the prompt.

[1417] Input: Prompt message

[1418] Output: Generated response (e.g., "This product is the latest model and includes many additional features")

[1419] Step 5:

[1420] The server provides the generated response to the user. Specifically, it either plays the response aloud using the smartphone's speaker or displays it as text on the screen.

[1421] Input: Generated response

[1422] Output: Provides a response to the user (voice or text).

[1423] In this way, the server can recognize the user's voice input, analyze their emotions, and generate and provide an appropriate response.

[1424] (Example 3)

[1425] Next, we will describe Embodiment 3 of Embodiment Example 3. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[1426] Conventional speech recognition systems have difficulty accurately recognizing specific dialects and emotions, and have been particularly unable to cope with the diverse dialects of Japan and the emotional range of users. Furthermore, they have struggled to accurately recognize voice input from elderly users operating information technology devices and generate appropriate commands. This has resulted in a reduced user experience and limited the practicality of the systems.

[1427] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means.

[1428] In this invention, the server includes a speech recognition means, a generative artificial intelligence means, and a learning means for handling all Japanese dialects. This makes possible a system that includes means for learning Tsugaru dialect, which is said to be the most difficult dialect in Japan, means for putting speech recognition into practice, means for recognizing the user's emotions using an emotion engine, and means for generating operations according to the recognized emotions.

[1429] "Speech recognition means" refers to technology that analyzes speech data and converts it into corresponding text data.

[1430] "Generative artificial intelligence means" refers to artificial intelligence technology that generates natural dialogue and operations based on input data.

[1431] "A learning method for handling all Japanese dialects" refers to a technology that recognizes and responds to the diverse dialects used within Japan, collects the necessary data, and trains a model accordingly.

[1432] "Methods for learning Tsugaru dialect" refers to the technology of collecting the necessary data to recognize and respond to Tsugaru dialect, a dialect spoken in the Tsugaru region of Japan, and training a model accordingly.

[1433] "Methods for the practical application of speech recognition" refers to technologies that integrate trained speech recognition models into actual systems, making them practically usable.

[1434] An "emotion engine" is a technology that analyzes and identifies a user's emotions from voice data.

[1435] "Means for generating operations in response to recognized emotions" refers to technology that generates and executes appropriate operations based on the user's emotions.

[1436] This invention is a system that includes a speech recognition means, a generative artificial intelligence means, a learning means for handling all Japanese dialects, particularly a means for learning Tsugaru dialect, a means for putting speech recognition into practical use, a means for recognizing the user's emotions using an emotion engine, and a means for generating operations according to the recognized emotions.

[1437] Hardware and software to be used

[1438] Hardware: High-performance servers (e.g., servers with GPUs)

[1439] Software: Speech recognition software (e.g., speech recognition APIs), generative artificial intelligence models (e.g., natural language processing models)

[1440] Explanation of the program's processing

[1441] Audio data collection and preprocessing

[1442] The server collects a large amount of Tsugaru dialect audio data. The collected audio data undergoes preprocessing such as noise reduction and normalization. Specifically, it processes the audio data using an audio processing library.

[1443] Training a speech recognition model

[1444] The server inputs pre-processed audio data into a speech recognition API and retrieves the corresponding text data. The retrieved text data is then used to train the speech recognition model. Specifically, the model is built and trained using a machine learning library.

[1445] Learning of generative artificial intelligence models

[1446] The server inputs text data obtained from the speech recognition model into a natural language processing model to train a generative artificial intelligence model. Once the generative AI model is trained, it is saved and provided as an API. Specifically, the model is managed using a natural language processing library.

[1447] Introducing an emotional engine

[1448] The server implements an emotion engine to recognize emotions from the user's voice. Specifically, it uses voice analysis tools to analyze the tone, speed, and volume of the voice. Based on the analysis results, it identifies the user's emotions and passes that information to a generative artificial intelligence model.

[1449] Emotion-driven manipulation

[1450] The user makes a voice input saying, "Turn on the air conditioner." The device sends this voice to the server. The server converts the voice to text using a speech recognition model. The server uses an emotion engine to recognize the user's emotions. For example, if the voice tone is high and the speed is fast, the server will determine that the user is angry. The server uses generative artificial intelligence means to generate an appropriate action according to the recognized emotion. For example, if the user is angry, it will generate an action to lower the air conditioner temperature.

[1451] Examples of specific cases and prompt statements

[1452] Specific example:

[1453] The user angrily inputs the command, "Turn on the air conditioner."

[1454] The device sends the audio to the server.

[1455] The server uses a speech recognition model to convert speech into text.

[1456] The server uses an emotion engine to recognize that the user's emotion is "anger."

[1457] The server uses generative artificial intelligence means to generate an operation to set the air conditioner temperature low.

[1458] Example of a prompt:

[1459] If a user makes an angry voice input saying "Turn on the air conditioner," the emotion engine recognizes the user's emotion from the voice, and the generative artificial intelligence means generates an operation to set the air conditioner temperature low.

[1460] In this way, the system provides appropriate operations that respond to the user's emotions. The flow of a specific process in Example 3 will be explained using Figure 21.

[1461] Step 1:

[1462] The server collects a large amount of Tsugaru dialect audio data. The collected audio data is used as input and undergoes preprocessing such as noise reduction and normalization. Specifically, it processes the audio data using an audio processing library, removing noise and standardizing the volume. The preprocessed audio data is then output.

[1463] Step 2:

[1464] The server inputs the pre-processed audio data into the speech recognition API and retrieves the corresponding text data. The speech recognition API analyzes the audio data and converts it into text data. As a result, the text data corresponding to the audio data is output.

[1465] Step 3:

[1466] The server uses the acquired text data to train a speech recognition model. Specifically, it builds a model using a machine learning library and trains it using the text data as input. As a result, the trained speech recognition model is output.

[1467] Step 4:

[1468] The server inputs text data obtained from the speech recognition model into a natural language processing model to train a generative artificial intelligence model. Specifically, it manages the model using a natural language processing library and performs training based on the text data. As a result, the trained generative artificial intelligence model is output.

[1469] Step 5:

[1470] The server incorporates an emotion engine to recognize emotions from the user's voice. Specifically, it uses a voice analysis tool to analyze the tone, speed, and volume of the voice, and performs analysis using the voice data as input. This results in the user's emotions being output.

[1471] Step 6:

[1472] The user makes a voice input saying, "Turn on the air conditioner." The terminal sends this voice to the server. The server uses a speech recognition model to convert the voice into text. As a result, text data corresponding to the voice data is output.

[1473] Step 7:

[1474] The server uses an emotion engine to recognize the user's emotions. For example, if the voice tone is high and the speed is fast, it will determine that the user is angry. Based on this, the user's emotion will be output.

[1475] Step 8:

[1476] The server uses generative artificial intelligence to generate appropriate actions in response to recognized emotions. For example, if the user is angry, it will generate an action to lower the air conditioner temperature. This ensures that the appropriate action is output.

[1477] Step 9:

[1478] The terminal executes the operation sent from the server. Specifically, it performs the operation to lower the air conditioner temperature, providing the user with a comfortable environment. This completes the operation requested by the user.

[1479] (Application Example 3)

[1480] Next, we will describe application example 3 of form example 3. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[1481] Conventional voice recognition systems are specialized for standard Japanese and have difficulty handling the diverse dialects of Japan. Furthermore, they lacked the ability to recognize user emotions and provide appropriate guidance, resulting in a limited user experience. In particular, in autonomous vehicles, it was difficult to provide appropriate navigation instructions when users spoke in dialect or were emotionally agitated.

[1482] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes a speech recognition means, a generative AI means, and a learning means for supporting all Japanese dialects. This makes it possible to accurately recognize the user's speech even when they speak in a dialect, and further understand the user's emotions to provide appropriate navigation instructions.

[1483] "Speech recognition means" refers to technology for converting speech into text data.

[1484] "Generative AI methods" refer to artificial intelligence technologies that generate appropriate operations or responses based on input data.

[1485] "A learning method for handling all Japanese dialects" refers to a technology that learns the data necessary to recognize the diverse dialects used within Japan.

[1486] "Methods for learning Tsugaru dialect" refers to learning techniques specifically designed to recognize Tsugaru dialect, a dialect spoken in the Tsugaru region of Japan.

[1487] "Means for the practical application of speech recognition" refers to technologies for applying speech recognition technology to actual systems and applications.

[1488] "Emotion recognition means" refers to technology that analyzes and recognizes emotions from a user's voice or text.

[1489] "Means for generating navigation instructions in autonomous vehicles" refers to technologies that generate routes and instructions to a destination in autonomous vehicles.

[1490] "Means of providing navigation instructions that respond to user emotions" refers to technologies that provide optimal navigation instructions by taking into account the user's emotional state.

[1491] The following system configuration will be described as an embodiment for carrying out this invention.

[1492] System Configuration

[1493] hardware

[1494] Microphone: Used to acquire user voice input.

[1495] Speaker: Used to output audio from the system.

[1496] Autonomous vehicle: A vehicle equipped with a navigation system.

[1497] software

[1498] Speech recognition library (speech_recognition): Used to convert user speech into text.

[1499] Emotion recognition library (transformers pipeline): Used to analyze emotions from user text data.

[1500] Speech synthesis library (pyttsx3): Used to convert text into speech.

[1501] Processing flow

[1502] 1. Acquisition of voice input

[1503] When a user speaks into the microphone, a speech recognition library converts the audio into text data.

[1504] For example, if a user asks "Where's the nearest convenience store from here?" in Tsugaru dialect, that voice will be converted into text data.

[1505] 2. Recognition of emotions

[1506] The converted text data is sent to an emotion recognition library, where the user's emotions are analyzed.

[1507] For example, if a user is feeling anxious, the emotion recognition library will recognize that emotion as "anxiety."

[1508] 3. Generating navigation instructions

[1509] Based on the results of emotion recognition, the generative AI generates appropriate navigation instructions.

[1510] For example, if the system detects that the user is in a hurry, it will generate instructions to quickly direct them to the nearest convenience store.

[1511] 4. Audio Output

[1512] The generated navigation instructions are converted into speech using a speech synthesis library and output through the speaker.

[1513] For example, a voice message might say, "Please stay calm. The nearest convenience store is here."

[1514] Specific example

[1515] If a user asks in Tsugaru dialect, "Where's the nearest convenience store from here?", the system will recognize the dialect and, if it determines the user is in a hurry, will quickly guide them to the nearest convenience store.

[1516] Example of a prompt

[1517] User: "Where's the nearest convenience store from here?"

[1518] System: "Please stay calm. The nearest convenience store is here."

[1519] In this way, the dialect-compatible emotion recognition navigation system can understand the user's dialect and emotions and provide appropriate navigation instructions.

[1520] The flow of the specific processing in Application Example 3 will be explained using Figure 22.

[1521] Step 1:

[1522] The user speaks into the microphone. The speech recognition library (speech_recognition) captures the audio and converts it into text data. The input is the user's voice, and the output is text data. Specifically, if the user says "Where's the nearest convenience store from here?" in Tsugaru dialect, that audio will be converted into the text data "Where's the nearest convenience store from here?".

[1523] Step 2:

[1524] The server sends the converted text data to the emotion recognition library (transformers pipeline). The emotion recognition library analyzes the user's emotions from the text data. The input is text data, and the output is emotion data. Specifically, the text data "Where is the nearest convenience store from here?" is converted into emotion data of "anxiety".

[1525] Step 3:

[1526] The server receives the emotion recognition results, and a generative AI generates appropriate navigation instructions. The input is emotion data and text data, and the output is navigation instructions. Specifically, based on the emotion data "anxiety" and the text data "Where is the nearest convenience store from here?", the navigation instruction "Please calm down. The nearest convenience store is here." is generated.

[1527] Step 4:

[1528] The server sends the generated navigation instructions to the speech synthesis library (pyttsx3). The speech synthesis library converts the navigation instructions into speech and outputs it through the speaker. The input is the navigation instructions, and the output is speech data. Specifically, the navigation instruction, "Please stay calm. The nearest convenience store is here," is converted into speech data and output through the speaker.

[1529] In this way, the dialect-compatible emotion recognition navigation system can understand the user's dialect and emotions and provide appropriate navigation instructions.

[1530] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1531] The data generation model 58 is a form of so-called generative AI (Artificial Intelligence). One example of the data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1532] Other examples of generative AI include Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are some examples.

[1533] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1534] [Third Embodiment]

[1535] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1536] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1537] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1538] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1539] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1540] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1541] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1542] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1543] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1544] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1545] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1546] Next, the identification process performed by the identification processing unit 290 of the data processing device 12 will be described.

[1547] "Example of form 1"

[1548] The present invention is a system that combines speech recognition means and generative AI means. The speech recognition means recognizes the voice input when an elderly person operates an IT device. The generative AI means generates operations based on the recognized voice input. Furthermore, this system has learning means to support all Japanese dialects. Specifically, by training the system on Tsugaru dialect, which is said to be the most difficult dialect in Japan, the practical application of speech recognition has been achieved.

[1549] "Example of form 2"

[1550] As a specific embodiment, for example, if an elderly person says "Turn on the TV" in the Tsugaru dialect, the voice recognition means recognizes this voice input, and the generative AI means generates an operation to turn on the TV. This operation is realized, for example, by controlling a device that generates infrared signals.

[1551] "Example of form 3"

[1552] Furthermore, this system possesses learning capabilities to handle all Japanese dialects. Specifically, it uses a large amount of Tsugaru dialect audio data to train its speech recognition and generative AI systems. This allows it to handle not only Tsugaru dialect but also other dialects using the same method.

[1553] The following describes the processing flow for each example of the form.

[1554] "Example of form 1"

[1555] Step 1: Enable voice input for elderly individuals to operate IT devices. This voice input includes instructions such as "Turn on the TV."

[1556] Step 2: The speech recognition system recognizes the voice input of the elderly person. This speech recognition is performed using a learning method that can handle all Japanese dialects.

[1557] Step 3: The generative AI system generates an action based on the voice input it recognizes. This action may include a specific action such as turning on the TV.

[1558] "Example of form 2"

[1559] Step 1: The elderly person says "Turn on the TV" in Tsugaru dialect.

[1560] Step 2: The speech recognition system recognizes this voice input. This speech recognition is performed with high accuracy as a result of training on the Tsugaru dialect.

[1561] Step 3: The generative AI system generates the operation to turn on the television. This operation is achieved by controlling a device that generates infrared signals.

[1562] "Example of form 3"

[1563] Step 1: Train speech recognition and generative AI using a large amount of Tsugaru dialect audio data.

[1564] Step 2: Using the trained speech recognition and generative AI systems, the system recognizes the elderly person's voice input and generates commands.

[1565] Step 3: Execute the generated operation. This operation may include specific actions such as turning on the TV.

[1566] (Example 1)

[1567] Next, we will describe Embodiment 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1568] When elderly people operate information technology devices, there is a need to improve ease of use by using voice input. However, voice recognition systems that can handle Japan's diverse dialects, especially difficult dialects like Tsugaru dialect, have not yet been put into practical use. Therefore, there is a need to develop a voice recognition system that allows elderly people to speak naturally in their own dialect.

[1569] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means. In this invention, the server includes a speech recognition means, a generative artificial intelligence means, and a learning means for supporting all Japanese dialects. This enables elderly people to speak naturally in their own dialect and to operate information technology devices using voice input.

[1570] "Speech recognition means" refers to technology for converting speech data into text data.

[1571] "Generative artificial intelligence means" refers to artificial intelligence technology for generating appropriate operation instructions based on input data.

[1572] "Learning methods to handle all Japanese dialects" refers to a technology that pre-trains a speech recognition system to recognize the diverse dialects used within Japan.

[1573] "Methods for learning Tsugaru dialect" refers to a technology that specially trains a speech recognition system to recognize Tsugaru dialect, which is said to be the most difficult dialect in Japan.

[1574] "Audio data preprocessing means" refers to technologies that improve the accuracy of speech recognition by performing noise reduction and volume normalization on audio data.

[1575] "Means for the practical application of speech recognition" refers to technologies that enable the application of speech recognition technology to actual systems and applications, making it practically usable.

[1576] "Voice input when elderly people operate information technology devices themselves" refers to voice commands that elderly people utter to operate information technology devices themselves.

[1577] This invention is a system for improving the convenience of elderly people when operating information technology devices using voice input. This system includes voice recognition means, generative artificial intelligence means, learning means for handling all Japanese dialects, means for learning Tsugaru dialect, voice data preprocessing means, and means for putting voice recognition into practical use.

[1578] System Configuration

[1579] Speech recognition means

[1580] The server uses speech recognition software to convert speech data into text data. For example, the Google Cloud Speech-to-Text API is used for this purpose. This API is pre-trained to handle a wide variety of Japanese dialects.

[1581] Audio data preprocessing means

[1582] The server uses audio processing libraries such as Librosa to remove noise and normalize the volume of the audio data. This makes the audio data clearer and easier to understand.

[1583] Generative artificial intelligence methods

[1584] The server uses a generative artificial intelligence model (e.g., OpenAI's GPT-4) to generate operation instructions based on text data. This generative AI model is pre-trained on a variety of operation scenarios to understand user intent and generate appropriate operation instructions.

[1585] Execute the operation

[1586] The device executes the operation instructions it receives from the server. For example, it might open an email app and send an email with the specified content.

[1587] Specific example

[1588] Example 1: Sending an email using voice input

[1589] The user uses voice input to say, "Send me an email."

[1590] 1. The server uses speech recognition software to convert speech into text.

[1591] 2. The server uses Librosa to perform noise reduction and volume normalization on the audio data.

[1592] 3. The server uses a generative artificial intelligence model to generate the command "send an email."

[1593] 4. The device opens the email app and sends an email with the specified content.

[1594] Example 2: Obtaining weather information via voice input

[1595] The user voice-inputs, "What's the weather like today?"

[1596] 1. The server uses speech recognition software to convert speech into text.

[1597] 2. The server uses Librosa to perform noise reduction and volume normalization on the audio data.

[1598] 3. The server uses a generative artificial intelligence model to generate the operation instruction "Retrieve weather information".

[1599] 4. The device opens a weather app and displays the current weather information.

[1600] Example of a prompt

[1601] "Design a system that recognizes voice input from elderly people speaking in Tsugaru dialect and generates appropriate actions. Specifically, explain how to combine voice recognition software and a generative artificial intelligence model to generate actions based on voice input."

[1602] In this way, this system can improve the convenience for elderly people when operating information technology devices using voice input.

[1603] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1604] Step 1:

[1605] The user uses voice input to enter information into the terminal. For example, they might say, "Send me an email." This voice input becomes the initial input for the system.

[1606] Step 2:

[1607] The terminal sends the acquired audio data to the server. The server uses an audio processing library such as Librosa to remove noise and normalize the volume of the audio data. This makes the audio data clearer and easier to recognize. The input is raw audio data, and the output is pre-processed audio data.

[1608] Step 3:

[1609] The server sends pre-processed audio data to speech recognition software (e.g., Google Cloud Speech-to-Text API) to convert the audio to text. The input is pre-processed audio data, and the output is text data. For example, the text "Send me an email" is generated.

[1610] Step 4:

[1611] The server uses a generative artificial intelligence model (e.g., OpenAI's GPT-4) to generate operation instructions based on text data. The input is text data obtained through speech recognition, and the output is a specific operation instruction. For example, the operation instruction "send an email" is generated.

[1612] Step 5:

[1613] The terminal executes the operation instructions received from the server. For example, it might open an email application and send an email with specified content. The input is the operation instructions from the server, and the output is the actual result of the operation.

[1614] In this way, the system can improve the convenience for elderly people when operating information technology devices using voice input.

[1615] (Application Example 1)

[1616] Next, we will describe Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1617] Elderly people face challenges in finding products or asking questions in physical stores because they have difficulty operating IT equipment and obtaining appropriate information. Furthermore, there is a lack of voice recognition systems that support all Japanese dialects, and there is a particular need for systems that can handle complex dialects.

[1618] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1619] In this invention, the server includes: a voice recognition means for recognizing voice input when an elderly person operates a smartphone themselves; a means for identifying the dialect spoken by the elderly person based on the recognized voice input; a means for generating operation instructions for the smartphone using a prompt statement and a generative AI based on the recognized voice input and the identified dialect; and a means for operating the smartphone according to the generated operation instructions. Furthermore, the voice recognition means further includes a means for recognizing voice input when the elderly person asks a question about a product in a physical store, and the server further includes a means for generating an answer to the question using a prompt statement and a generative AI based on the recognized voice input and the identified dialect. This makes it possible to create a system that allows elderly people to easily operate a physical store using voice input when searching for products or asking questions.

[1620] "Speech recognition means" refers to technology for converting speech into text.

[1621] "Generative AI methods" refer to artificial intelligence technologies that generate appropriate responses or operations based on input data.

[1622] "A learning method for handling all Japanese dialects" refers to a technology that learns the data necessary to understand and recognize dialects from various regions of Japan.

[1623] "Methods for learning Tsugaru dialect" refers to techniques for learning the data necessary to understand and recognize Tsugaru dialect, which is said to be the most difficult dialect in Japan.

[1624] "Means for the practical application of speech recognition" refers to technologies for applying speech recognition technology to actual applications and systems.

[1625] "A means of recognizing voice input used by elderly people when searching for products or asking questions in physical stores, and generating appropriate operations or responses" refers to a technology that recognizes the voice input used by elderly people when searching for products or asking questions in physical stores, and generates appropriate operations or responses.

[1626] "Applications installed on smartphones" refers to software installed on a smartphone to provide specific functions.

[1627] A system for carrying out this invention includes a voice recognition means, a generative AI means, a means for identifying all Japanese dialects, a means for learning Tsugaru dialect, a means for putting voice recognition into practical use, a means for recognizing voice input when elderly people search for products or ask questions in a physical store and generating appropriate operations or answers, and an application means installed on a smartphone.

[1628] System program

[1629] This system operates using the following hardware and software:

[1630] Hardware: Smartphone (with microphone)

[1631] Software: Python, SpeechRecognition library, Transformers library (Hugging Face GPT-3® model)

[1632] Processing flow

[1633] Speech recognition

[1634] When a user speaks into their smartphone's microphone, a speech recognition system captures the audio and converts it into text. The SpeechRecognition library is used for this process.

[1635] Generative AI

[1636] The speech data, converted into text by the speech recognition system, is sent to the generative AI. The generative AI uses the Transformers library and a GPT-3 model to generate appropriate operations and responses.

[1637] output

[1638] The generated actions and responses are displayed on the smartphone screen. This allows elderly people to easily use voice input when searching for products or asking questions in physical stores.

[1639] Specific example (smartphone operation)

[1640] When an elderly person says "Send an email" to their smartphone, the application recognizes the voice, and the generative AI identifies that it is in the Tsugaru dialect. It then generates instructions to launch the email sending app, which the application then launches according to those instructions. The generative AI also generates a response such as "Please enter the message you want to send," and the application displays that response.

[1641] Example of a prompt

[1642] "An elderly person is using voice input to say 'send email.' This is in Tsugaru dialect. Please generate instructions for operating a smartphone."

[1643] In this way, a system can be realized that supports elderly people in making it easier for them to operate smartphones.

[1644] Specific example (physical store)

[1645] When an elderly person speaks to their smartphone in a physical store and asks, "Where is product A?", the application recognizes the voice, and a generative AI identifies that it is in the Tsugaru dialect. It then generates and displays a specific answer such as, "Product A is on the left side of aisle 3."

[1646] Example of a prompt

[1647] "An elderly person is voice-inputting, 'Where is product A?' This is in Tsugaru dialect. Please generate an answer to this question."

[1648] Generative AI's answer: The product is located on the left side of aisle 3.

[1649] In this way, a system can be realized that supports elderly people in making shopping at physical stores easier.

[1650] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1651] Step 1:

[1652] The user speaks into the microphone of their smartphone. The input is the user's voice, and the output is audio data. For example, the user might say, "Where can I find this product?"

[1653] Step 2:

[1654] The device uses speech recognition to convert audio data into text. The input is audio data, and the output is text data. Specifically, the SpeechRecognition library is used to convert the audio into the text "Where is product A located?".

[1655] Step 3:

[1656] The device sends text data to the generative AI. The input is text data, and the output is input data to the generative AI. Specifically, the converted text "Where is product A located?" is sent to the generative AI.

[1657] Step 4:

[1658] The server uses generative AI to generate appropriate instructions or responses based on text data. The input is text data, and the output is the generated instructions or responses. Specifically, the Transformers library is used to generate the response "Product A is on the left side of aisle 3" using a GPT-3 model.

[1659] Step 5:

[1660] The server sends the generated instructions and responses to the terminal. The input is the generated instructions and responses, and the output is the data sent to the terminal. Specifically, the generated response "Product A is on the left side of aisle 3" is sent to the terminal.

[1661] Step 6:

[1662] The device operates according to the generated instructions and displays the generated response to the user. Input is the transmitted data, and output is the data displayed to the user. As a specific example, the response "Product A is on the left side of aisle 3" is displayed on the smartphone screen.

[1663] Furthermore, an emotion engine, as described later, may be used to recognize the emotional state of the elderly person, and smartphone operation instructions and answers to questions may be generated taking the elderly person's emotional state into further consideration. In this case, the server further includes means for recognizing the emotional state of the elderly person, and the generating means generates the smartphone operation instructions using a prompt statement for generating smartphone operation instructions based on the recognized voice input, the identified dialect, and the recognized emotional state of the elderly person, and the generation AI.

[1664] (Example 2)

[1665] Next, we will describe Example 2 of the morphological example. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1666] When elderly people operate information technology devices, there is a challenge in accurately recognizing and generating appropriate commands, especially for voice input using dialects. Furthermore, there is a lack of voice recognition technology capable of handling specific dialects, such as the difficult Tsugaru dialect. Therefore, there is a need for systems that can accurately handle voice input using these dialects.

[1667] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1668] In this invention, the server includes a speech recognition means, a generative artificial intelligence means, a dialect-compatible learning means, a means for learning a specific dialect, a means for putting speech recognition into practical use, a means for converting speech input into text data, a means for transmitting the text data to a generative artificial intelligence model, a means for receiving generated operations, and a means for controlling an infrared signal generator. This makes it possible to accurately recognize speech and generate appropriate operations even when elderly people use dialects to operate information technology devices.

[1669] "Speech recognition means" refers to a device or software for analyzing speech input and converting it into text data.

[1670] "Generative artificial intelligence means" refers to an artificial intelligence model that generates appropriate operations or responses based on input data.

[1671] A "dialect-compatible learning method" is a method for training a speech recognition model to handle dialects from different regions.

[1672] "Methods for learning specific dialects" refer to methods for training speech recognition models specifically on difficult dialects such as Tsugaru dialect.

[1673] "Means for the practical application of speech recognition" refers to methods for integrating speech recognition technology into actual systems and devices to make it usable.

[1674] "Means for converting voice input into text data" refers to means for converting voice input into text format using voice recognition means.

[1675] "Means for sending text data to a generative artificial intelligence model" refers to means for sending converted text data to a generative artificial intelligence model.

[1676] "Means for receiving generated operations" refers to means for receiving operations generated from a generative artificial intelligence model.

[1677] "Means for controlling an infrared signal generating device" refers to means for controlling a device that generates infrared signals and performing a specific operation.

[1678] This invention is a system that accurately recognizes speech and generates appropriate commands when elderly people operate information technology devices using their local dialect. A specific embodiment of this system is described below.

[1679] First, the user speaks in Tsugaru dialect, saying "Turn on the TV." The microphone built into the device captures this voice. Voice recognition software (for example, a voice recognition API) is used as the voice recognition method. This software analyzes the captured voice and converts it into text data.

[1680] Next, the terminal sends the converted text data to a generative artificial intelligence model (for example, a generative AI model). This generative AI model generates appropriate operations based on the input text data. Specifically, prompt statements like the following are generated.

[1681] Example of a prompt:

[1682] The user said "Turn on the TV" in Tsugaru dialect. Generate an operation to create an infrared signal to turn on the TV.

[1683] Based on this prompt, the server uses a generative artificial intelligence model to generate specific commands for turning on the television. These generated commands are then sent to the terminal.

[1684] The terminal controls an infrared signal generator (for example, an infrared signal generator) based on the received operation. This device emits an infrared signal towards the television, and the television receives this signal and turns on.

[1685] As a concrete example, when a user speaks "Turn on the TV" in Tsugaru dialect, the microphone built into the device captures this audio, and a speech recognition API is used to convert the audio into text. This text data is sent to a generative AI model, which generates the following prompt.

[1686] Example of a prompt:

[1687] The user said "Turn on the TV" in Tsugaru dialect. Generate an operation to create an infrared signal to turn on the TV.

[1688] Based on this prompt, the server uses a generative AI model to generate specific instructions for turning on the TV and sends them to the terminal. The terminal controls the infrared signal generator, and the TV receives the infrared signal and turns on.

[1689] In this way, even when users operate information technology devices using dialects, it becomes possible to accurately recognize their speech and generate appropriate commands.

[1690] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1691] Step 1:

[1692] The user performs voice input.

[1693] The user speaks in Tsugaru dialect, saying "Turn on the TV." The input is the user's voice, and the output is audio data captured by the device's microphone.

[1694] Step 2:

[1695] The device recognizes the voice.

[1696] The microphone built into the device captures the user's voice. A speech recognition API is used as the means of speech recognition. The input is the captured audio data, and the output is the parsed text data.

[1697] Step 3:

[1698] The device converts the recognized speech into text.

[1699] The device uses a speech recognition API to convert captured audio into text data. The input is audio data, and the output is text data.

[1700] Step 4:

[1701] The device sends text to an AI model that generates text.

[1702] The terminal sends the converted text data to the generating AI model. The input is text data, and the output is the data sent to the generating AI model.

[1703] Step 5:

[1704] The server generates operations using an AI model.

[1705] The server uses a generative AI model to generate appropriate actions based on the received text data. Specifically, prompt statements like the following are generated:

[1706] Example of a prompt:

[1707] The user said "Turn on the TV" in Tsugaru dialect. Generate an operation to create an infrared signal to turn on the TV.

[1708] The input is text data, and the output is the generated manipulated data.

[1709] Step 6:

[1710] The server sends the generated operation to the terminal.

[1711] The server sends the generated operation to the terminal. The input is the generated operation data, and the output is the operation data sent to the terminal.

[1712] Step 7:

[1713] The terminal controls the infrared signal generator.

[1714] The terminal controls the infrared signal generator based on the received operation. The input is operation data, and the output is an infrared signal.

[1715] Step 8:

[1716] The television operates by receiving infrared signals.

[1717] An infrared signal generator emits an infrared signal towards the television, and the television receives this signal and turns on. The input is an infrared signal, and the output is the television's operation.

[1718] (Application Example 2)

[1719] Next, we will describe application example 2 of form example 2. In the following description, the data processing device 12 will be referred to as a "server," and the headset-type terminal 314 will be referred to as a "terminal."

[1720] In modern brick-and-mortar stores, there is a problem in that customers who speak dialects have difficulty receiving product information via voice. In particular, for the elderly and customers in areas where dialects are spoken, voice recognition systems that only understand standard Japanese cannot provide adequate service. Therefore, there is a need for a system that combines dialect-compatible voice recognition with generative AI.

[1721] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1722] In this invention, the server includes speech recognition means, generative AI means, and learning means for handling all dialects. This makes it possible for customers who speak dialects to receive product information via voice.

[1723] "Speech recognition means" refers to technology for converting speech into text.

[1724] "Generative AI methods" refer to artificial intelligence technologies that generate appropriate responses or operations based on input data.

[1725] "Learning methods for handling all dialects" refers to learning algorithms for understanding and recognizing dialects from different regions.

[1726] "Methods for learning difficult dialects" refers to special learning processes for recognizing dialects that are particularly difficult to understand.

[1727] "Means for the practical application of speech recognition" refers to means that enable speech recognition technology to be used in actual applications.

[1728] "A means of providing information about products within a store" refers to a system used to provide customers with information about the location of products within a store.

[1729] "Applications installed on smart devices" refers to software installed on devices such as smartphones and tablets.

[1730] In order to implement this invention, it is necessary to construct a system that includes a speech recognition means, a generative AI means, a learning means for handling all dialects, a means for learning difficult dialects, a means for putting speech recognition into practical use, a guidance means for providing product information within a store, and an application means to be installed on a smart device.

[1731] System Configuration

[1732] 1. Speech recognition means:

[1733] The speech recognition system is used to convert the user's spoken voice into text. Specifically, it uses the smartphone's microphone to capture the voice and the SpeechRecognition library to convert the voice into text.

[1734] 2. Generative AI means:

[1735] The generative AI system generates appropriate responses and actions based on text acquired by the speech recognition system. Specifically, it utilizes the GPT-3 model with the Hugging Face transformers library.

[1736] 3. Learning methods for handling all dialects:

[1737] This method includes a learning algorithm for understanding and recognizing dialects from different regions. In particular, a special learning process is implemented as a means of learning difficult dialects.

[1738] 4. Means of practical application of speech recognition:

[1739] This is a means to make speech recognition technology usable in actual applications. This includes an interface for a generative AI system to generate an appropriate response based on the speech recognition results and provide it to the user.

[1740] 5. Means of providing information about products within the store:

[1741] This system provides customers with information and location details about products within a store. Specifically, it displays a map showing product locations on a smartphone screen and provides voice guidance.

[1742] 6. Application methods installed on smart devices:

[1743] This is software installed on devices such as smartphones and tablets. This application integrates voice recognition and generative AI to enable users to receive product information via voice.

[1744] Processing flow

[1745] 1. Acquisition of voice input:

[1746] When a user speaks into their smartphone, the voice recognition system captures that voice. For example, the user might say, "Find this product."

[1747] 2. Speech recognition:

[1748] The speech recognition system converts the acquired speech into text. The SpeechRecognition library is used to convert the speech to text.

[1749] 3. Response generation by generative AI:

[1750] A generative AI system uses the converted text as a prompt to generate an appropriate response. Specifically, it uses the Hugging Face GPT-3 model to generate the response.

[1751] 4. Providing a response:

[1752] The generated response is displayed on the smartphone screen or announced via voice. For example, a response such as "This product is on the shelf at the back right of the store" is generated.

[1753] Specific example

[1754] User voice input: "Find this product"

[1755] Generated response: "This item is located on the shelf at the back right of the store."

[1756] Example of a prompt

[1757] "If someone asks me to 'find this product' in Tsugaru dialect, how would I guide them to the product in the store?"

[1758] In this way, combining dialect-compatible speech recognition and generation AI can significantly improve customer service in physical stores.

[1759] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1760] Step 1:

[1761] The user speaks into their smartphone.

[1762] Input: User's voice (e.g., "Find this product")

[1763] Operation: The smartphone's microphone acquires sound.

[1764] Output: Acquired audio data

[1765] Step 2:

[1766] The device uses speech recognition to convert the acquired speech data into text.

[1767] Input: Audio data

[1768] Operation: Converts speech to text using the SpeechRecognition library.

[1769] Output: Converted text (e.g., "Find this product")

[1770] Step 3:

[1771] The terminal uses generative AI means to generate a response based on the converted text.

[1772] Input: Converted text

[1773] Operation: Uses the Hugging Face GPT-3 model to generate prompt sentences and appropriate responses.

[1774] Output: Generated response text (Example: "This item is on the shelf at the back right of the store.")

[1775] Step 4:

[1776] The terminal provides the user with the generated response.

[1777] Input: Generated response text

[1778] Function: Displays the response text on the smartphone screen, or plays the response as audio.

[1779] Output: The user confirms the response.

[1780] Step 5:

[1781] The user searches for the product by following the instructions.

[1782] Input: Response text or voice guidance

[1783] Operation: The user moves around the store and searches for products by following the directions.

[1784] Output: The user finds the desired product.

[1785] In this way, specific actions and data processing / calculations are performed at each step, enabling customer service in physical stores that utilizes dialect-compatible speech recognition and generation AI.

[1786] (Example 3)

[1787] Next, we will describe Embodiment 3 of Embodiment Example 3. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1788] Developing a speech recognition system that can handle all Japanese dialects is particularly challenging, especially for complex dialects. In particular, handling Tsugaru dialect, considered the most difficult dialect in Japan, is difficult with conventional speech recognition technology. Furthermore, accurately recognizing the voice input of elderly users operating information technology devices and generating appropriate responses is also a challenge.

[1789] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means.

[1790] In this invention, the server includes a speech recognition means, a generative artificial intelligence means, and a learning means for handling all Japanese dialects. This enables a system that includes means for the user to input voice into a terminal, means for the terminal to send voice data to the server, means for the server to convert the voice data into text data using speech recognition software, means for the server to input the text data into a generative artificial intelligence model, means for the generative artificial intelligence model to generate an appropriate response, means for the server to send the generated response to the terminal, and means for the terminal to display the response to the user. This enables speech recognition and response generation that can handle all Japanese dialects, and can handle particularly difficult dialects such as Tsugaru dialect. Furthermore, it can accurately recognize voice input when elderly people operate information technology devices themselves and generate appropriate responses.

[1791] "Speech recognition means" refers to technology for converting speech data into text data.

[1792] "Generative artificial intelligence means" refers to artificial intelligence technology that generates appropriate responses based on input text data.

[1793] A "learning method" is a method of training a system using specific data to improve its ability to perform specific tasks.

[1794] "Methods for learning Tsugaru dialect" refers to a method of training speech recognition and generative artificial intelligence using a large amount of Tsugaru dialect audio data.

[1795] "Means for the practical application of speech recognition" refers to methods for applying speech recognition technology to actual applications and systems.

[1796] "Means by which users input voice into a device" refers to methods by which users input voice into a device such as a smartphone or personal computer.

[1797] "Means of sending audio data from a terminal to a server" refers to a method of sending audio data captured by a terminal to a server via the internet.

[1798] "A means by which a server converts audio data into text data using speech recognition software" refers to a method by which a server converts received audio data into text data using speech recognition software.

[1799] "Means by which a server inputs text data into a generative artificial intelligence model" refers to the method by which a server inputs the converted text data into a generative artificial intelligence model.

[1800] "Means by which a generative artificial intelligence model generates an appropriate response" refers to a method by which a generative artificial intelligence model generates an appropriate response based on input text data.

[1801] "Means for sending the server-generated response to the terminal" refers to a method by which the server sends the response returned from the generative artificial intelligence model to the terminal.

[1802] "Means by which a terminal displays a response to the user" refers to a method by which a terminal displays a response received from a server to the user.

[1803] This invention provides a speech recognition system that supports all Japanese dialects. In particular, it has a learning method for supporting Tsugaru dialect, which is said to be the most difficult dialect in Japan. The system includes a speech recognition means, a generative artificial intelligence means, a learning means, a means for putting speech recognition into practice, a means for the user to input voice into a terminal, a means for the terminal to send voice data to a server, a means for the server to convert the voice data into text data using speech recognition software, a means for the server to input the text data into a generative artificial intelligence model, a means for the generative artificial intelligence model to generate an appropriate response, a means for the server to send the generated response to the terminal, and a means for the terminal to display the response to the user.

[1804] The server first collects a large amount of Tsugaru dialect audio data. This audio data is converted into text data using speech recognition software (e.g., Google Speech-to-Text API). Next, a generative artificial intelligence model (e.g., OpenAI's GPT-4) is trained using the converted text data.

[1805] The terminal transmits the voice data entered by the user to the server in real time. The server converts the received voice data into text data using speech recognition software and inputs that text data into a generative artificial intelligence model. The generative artificial intelligence model generates an appropriate response based on the input text data and sends that response back to the terminal as text data.

[1806] The user receives responses generated through the device and provides voice input again as needed. By repeating this process, the user can engage in natural conversations that are compatible with their dialect.

[1807] As a concrete example, consider a scenario where a user speaks in Tsugaru dialect, saying "How's it going today?" The terminal sends this voice data to a server, which uses speech recognition software to convert it into text data: "How's it going today?" Next, this text data is input into a generative artificial intelligence model to generate an appropriate response. For example, the response "It's nice weather today" might be generated. This response is sent back to the terminal as text data and displayed to the user.

[1808] Examples of prompt statements include the following:

[1809] User input: "How's it going today?"

[1810] Prompt to the Generative AI Model: "The user has spoken in Tsugaru dialect, saying 'How are you today?' Please generate an appropriate response."

[1811] In this way, the system provides a means of learning to support all Japanese dialects, enabling users to engage in natural conversations. The flow of specific processing in Example 3 will be explained using Figure 15.

[1812] Step 1:

[1813] The user enters voice input into the device.

[1814] The user speaks into a device such as a smartphone or computer. For example, they might say "How's it going today?" in Tsugaru dialect. This voice is captured by the device's microphone. The input is the user's voice data, and the output is the voice data captured by the device.

[1815] Step 2:

[1816] The device sends voice data to the server.

[1817] The device compresses the captured audio data and sends it to the server via the internet. Specifically, the device compresses the audio data into MP3 format and sends it to the server using the HTTPS protocol. The input is the captured audio data, and the output is the audio data sent to the server.

[1818] Step 3:

[1819] The server converts the audio data into text data using speech recognition software.

[1820] The server sends the received audio data to speech recognition software (e.g., Google Speech-to-Text API) and converts it into text data. For example, the audio data "How's it going today?" is converted to the text data "How's it going today?". The input is the audio data sent to the server, and the output is the converted text data.

[1821] Step 4:

[1822] The server inputs text data into a generative artificial intelligence model.

[1823] The server sends the converted text data as an API request to a generative artificial intelligence model (e.g., OpenAI's GPT-4). For example, it sends the text data "How's it going today?" to the generative AI model. The input is the converted text data, and the output is an API request to the generative AI model.

[1824] Step 5:

[1825] The generative artificial intelligence model generates an appropriate response.

[1826] Generative artificial intelligence models generate appropriate responses based on input text data. For example, they might generate the response, "It's a nice day today." This response is returned to the server as text data. The input is an API request to the generative artificial intelligence model, and the output is the generated response text data.

[1827] Step 6:

[1828] The server sends the generated response to the terminal.

[1829] The server receives the response text data returned from the generative artificial intelligence model and sends it to the terminal as an API response. For example, it sends the text data "It's a nice day today" to the terminal. The input is the generated response text data, and the output is the response text data sent to the terminal.

[1830] Step 7:

[1831] The terminal displays a response to the user.

[1832] The terminal displays the response text data received from the server to the user. For example, the text "It's a nice day today" is displayed on the smartphone screen. The input is the response text data sent to the terminal, and the output is the response text displayed to the user.

[1833] (Application Example 3)

[1834] Next, we will describe application example 3 of form example 3. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1835] Conventional speech recognition systems are specialized for standard Japanese and have difficulty handling the diverse dialects of Japan. Furthermore, they are particularly unable to handle Tsugaru dialect, considered one of the most difficult dialects in Japan, resulting in low convenience for users who speak the dialect. Additionally, in food delivery services, ordering in dialect is difficult, forcing users to use standard Japanese.

[1836] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes a speech recognition means, a generative AI means, a learning means for supporting all Japanese dialects, a means for learning Tsugaru dialect, which is said to be the most difficult dialect in Japan, a means for putting speech recognition into practical use, a means for recognizing and analyzing speech input in dialect and generating order content, a means for transmitting the generated order content via a communication means, a means for allowing the user to confirm the order content, and a means for notifying the user that the order has been confirmed. As a result, when a user who speaks a dialect uses a food delivery service, they will be able to input speech in their dialect, improving convenience.

[1837] "Speech recognition means" refers to technology for converting speech into text.

[1838] "Generative AI methods" are artificial intelligence technologies that generate new information or operations based on input data.

[1839] "Learning methods to handle all Japanese dialects" refers to technologies that collect and learn the data necessary to recognize and understand dialects from various regions of Japan.

[1840] "Methods for learning Tsugaru dialect" refers to the technology of collecting and learning the data necessary to recognize and understand Tsugaru dialect, which is said to be the most difficult dialect in Japan.

[1841] "Means for the practical application of speech recognition" refers to technologies that make speech recognition technology usable in actual applications and services.

[1842] "A means of recognizing, analyzing, and generating order details from voice input in a dialect" refers to a technology that recognizes voice input in a dialect, analyzes its content, and generates appropriate order details.

[1843] "Means for transmitting generated order details via communication means" refers to technologies for transmitting generated order details using the internet or other communication means.

[1844] "Methods for allowing users to confirm order details" refer to technologies that present the generated order details to the user and ask for their confirmation.

[1845] "Means of notifying that an order has been confirmed" refers to technologies that inform users that their order has been confirmed.

[1846] The following system configuration will be described as an embodiment for carrying out this invention.

[1847] System Configuration

[1848] This system includes a speech recognition means, a generative AI means, a learning means for handling all Japanese dialects, a means for learning Tsugaru dialect, a means for putting speech recognition into practical use, a means for recognizing and analyzing speech input in dialect and generating order details, a means for transmitting the generated order details via a communication means, a means for allowing the user to confirm the order details, and a means for notifying the user that the order has been confirmed.

[1849] Hardware and software to be used

[1850] Hardware: Smartphone (iOS or Android)

[1851] software:

[1852] Speech recognition API (e.g., Google Cloud Speech-to-Text)

[1853] Generative AI models (e.g., OpenAI GPT-4)

[1854] Food delivery APIs (e.g., Uber Eats API)

[1855] Processing flow

[1856] 1. Voice input: The user speaks their order in their local dialect into the microphone of their smartphone.

[1857] 2. Speech Recognition: The smartphone uses a speech recognition API to convert the input speech into text. This process utilizes a speech recognition model that supports different dialects.

[1858] 3. Text Analysis: The server uses a generative AI model to analyze the recognized text and understand the order details.

[1859] 4. Order Generation: Based on the analysis results, the server uses the food delivery API to generate an appropriate order and sends it to the restaurant.

[1860] 5. Confirmation and Notification: The smartphone allows the user to confirm the order details and notifies them that the order has been confirmed.

[1861] Specific example

[1862] When a user speaks in Tsugaru dialect, saying "I'd like to order one pizza," the smartphone recognizes this, generates an appropriate pizza order, and sends it to the restaurant.

[1863] Example of a prompt

[1864] If a user speaks in Tsugaru dialect and says "I'd like to order one pizza," use a speech recognition API to convert the speech to text, a generative AI model to analyze the order, and a food delivery API to generate a pizza order.

[1865] In this way, a dialect-enabled food delivery assistant can enable users to place orders in their local dialect, providing a convenient service to a wider range of users.

[1866] The flow of the specific processing in Application Example 3 will be explained using Figure 16.

[1867] Step 1:

[1868] The user speaks their order in their local dialect into the smartphone's microphone. The input is the user's voice, and the output is the audio data input to the smartphone's microphone.

[1869] Step 2:

[1870] The device uses a speech recognition API (e.g., Google Cloud Speech-to-Text) to convert the input audio data into text. The input is audio data, and the output is text data. A speech recognition model that supports dialects is used in this process.

[1871] Step 3:

[1872] The server uses a generative AI model (e.g., OpenAI GPT-4) to analyze text data sent from a speech recognition API and understand the order details. The input is text data, and the output is the analyzed order details.

[1873] Step 4:

[1874] Based on the analysis results, the server generates an appropriate order using a food delivery API (e.g., Uber Eats API) and sends it to the restaurant. The input is the analyzed order data, and the output is the order data sent via the food delivery API.

[1875] Step 5:

[1876] The terminal displays the generated order details to allow the user to confirm the order. The input is order confirmation data from the food delivery API, and the output is the order details displayed on the smartphone screen.

[1877] Step 6:

[1878] The user reviews and confirms their order. The input is the user's confirmation action, and the output is the order confirmation signal.

[1879] Step 7:

[1880] The terminal notifies the server that the order has been confirmed, and the server sends this information to the food delivery API. The input is the order confirmation signal, and the output is the order confirmation data sent to the food delivery API.

[1881] Step 8:

[1882] The server notifies the user that the order has been confirmed. The input is order confirmation data from the food delivery API, and the output is an order confirmation notification displayed to the user via the smartphone's notification function.

[1883] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1884] "Example of form 1"

[1885] One embodiment of the present invention provides a system that includes a speech recognition means, a generative AI means, a learning means for handling all Japanese dialects, a means for learning Tsugaru dialect, and a means for putting speech recognition into practical use, as well as an emotion engine that recognizes the user's emotions. This emotion engine recognizes emotions from the user's voice and generates an operation corresponding to that emotion. For example, when a user makes the voice input "Turn on the TV," the emotion engine recognizes the user's emotion from that voice. If it recognizes that the emotion is "joy," the generative AI means generates an operation to turn on the TV.

[1886] "Example of form 2"

[1887] In another embodiment of the present invention, the emotion engine generates an operation in response to the user's emotions. For example, when the user makes a voice input saying "Turn on the radio," the emotion engine recognizes the user's emotions from the voice. If it recognizes that the emotion is "sadness," the generative AI means generates an operation to turn on the radio and provides a result in which the operation is adapted to the user's emotions. Specifically, it generates an operation such as setting the radio channel to one that plays sad songs that the user likes.

[1888] "Example of form 3"

[1889] Furthermore, in another embodiment of the present invention, the emotion engine recognizes the user's emotion, and the generative AI means generates an operation in response to that emotion. For example, when the user makes a voice input saying "Turn on the air conditioner," the emotion engine recognizes the user's emotion from that voice. If it recognizes that the emotion is "anger," the generative AI means generates an operation to turn on the air conditioner and provides a result in which the operation is adapted to the user's emotion. Specifically, it generates an operation such as setting the air conditioner temperature to a low temperature that will calm the user down.

[1890] The following describes the processing flow for each example of the form.

[1891] "Example of form 1"

[1892] Step 1: The user makes a voice input. In this example, the user makes the voice input, "Turn on the TV."

[1893] Step 2: The emotion engine recognizes emotions from the user's voice. In this example, it recognizes "joy."

[1894] Step 3: The generative AI generates an action based on the emotions it recognizes. In this example, it generates the action to turn on the TV.

[1895] "Example of form 2"

[1896] Step 1: The user makes a voice input. In this example, the user makes the voice input, "Turn on the radio."

[1897] Step 2: The emotion engine recognizes emotions from the user's voice. In this example, it recognizes "sadness."

[1898] Step 3: The generative AI generates an action based on the emotions it recognizes and provides a result where the action is adapted to the user's emotions. In this example, it generates an action to turn on the radio and provides a result where the action is adapted to the user's emotions. Specifically, it generates an action such as setting the radio channel to one that plays sad songs that the user likes.

[1899] "Example of form 3"

[1900] Step 1: The user makes a voice input. In this example, the user makes the voice input, "Turn on the air conditioner."

[1901] Step 2: The emotion engine recognizes emotions from the user's voice. In this example, it recognizes "anger."

[1902] Step 3: The generative AI generates an action based on the emotions it recognizes and provides a result where that action is adapted to the user's emotions. In this example, it generates an action to turn on the air conditioner and provides a result where that action is adapted to the user's emotions. Specifically, it generates an action such as setting the air conditioner temperature to a low temperature that will help the user calm down.

[1903] (Example 1)

[1904] Next, we will describe Embodiment 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1905] When elderly people operate information technology devices, there is a challenge in accurately recognizing voice input and generating appropriate operations. In particular, there is a need to support all Japanese dialects, but the diversity and complexity of dialects are factors that reduce the accuracy of voice recognition. Furthermore, there is a ...

Claims

[Claim 1] A voice recognition means that recognizes voice input when elderly people operate their smartphones themselves and converts the voice data into text data, A means for identifying the dialect spoken by the elderly person based on the recognized voice input, A means for generating a prompt sentence that includes the text data, information identifying the identified dialect, and an instruction sentence indicating that the operation instructions for the smartphone should be generated, and for inputting the prompt sentence into a generation AI, thereby causing the generation AI to generate the operation instructions and responses for the smartphone, A means for operating the smartphone according to the operation instructions of the smartphone and displaying the answer on the smartphone, Includes, The aforementioned smartphone operation instructions include instructions to launch an email sending app. The aforementioned response is a system that includes wording prompting the user to enter the message they want to send.

Images