System

Abstract

To provide a system configured so that an elderly people can operate an IT device with a dialect as usual.SOLUTION: A system includes: voice recognition means which recognizes a voice input obtained when an elderly person operates a smartphone; means which identifies a dialect that the elderly person speaks, based on the recognized voice input; means which generates an operation instruction for the smartphone, using a generative AI and a prompt text for generating the operation instruction for the smartphone, based on the recognized voice input and the identified dialect; and means for operating the smartphone following the generated operation instruction for the smartphone.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In Japan's super-aging society, elderly people are required to be able to operate IT devices by themselves using voice recognition alone. However, there are hundreds to thousands of dialects in Japan, and no voice recognition system that can handle all of them does not allow all elderly people to operate IT devices in their natural dialect. [Means for solving the problem]

[0005] We provide a system that combines speech recognition and generative AI. This system has a learning method that can handle all Japanese dialects, and is particularly trained to handle Tsugaru dialect, which is said to be the most difficult dialect in Japan. This will enable the practical application of speech recognition, and enable all elderly people to operate IT devices in their natural dialect. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] The present invention is a system that combines a speech recognition means and a generative AI means. The speech recognition means recognizes the voice input when an elderly person operates an IT device. The generative AI means generates operations based on the recognized voice input. The system also has a learning means to support all Japanese dialects. Specifically, the practical application of speech recognition has been realized by training the system in Tsugaru dialect, which is said to be the most difficult dialect in Japan.

[0029] "Example 2"

[0030] In a specific embodiment, for example, if an elderly person says "Turn on the TV" in Tsugaru dialect, the speech recognition means recognizes this speech input, and the generative AI means generates an operation to turn on the TV. This operation is realized, for example, by controlling a device that generates an infrared signal.

[0031] "Example 3"

[0032] This system also has a learning method to support all dialects in Japan. Specifically, the speech recognition and generative AI methods are trained using a large amount of Tsugaru dialect speech data. This makes it possible to use the same method to support not only Tsugaru dialect but also other dialects.

[0033] The processing flow of each embodiment will be described below.

[0034] "Example 1"

[0035] Step 1: The elderly person performs voice input to operate an IT device. This voice input includes instructions such as "Turn on the TV."

[0036] Step 2: The voice recognition means recognizes the voice input of the elderly person. This voice recognition is performed using a training means to support all Japanese dialects.

[0037] Step 3: The generative AI means generates an action based on the recognized voice input, which may include a specific action such as turning on the TV.

[0038] "Example 2"

[0039] Step 1: An elderly person speaks in Tsugaru dialect and says, "Turn on the TV."

[0040] Step 2: The speech recognition means recognizes this speech input. This speech recognition is highly accurate as a result of having been trained in Tsugaru dialect.

[0041] Step 3: The generative AI means generates the operation to turn on the TV. This operation is realized by controlling a device that generates an infrared signal.

[0042] "Example 3"

[0043] Step 1: Train the speech recognition and generative AI methods using a large amount of Tsugaru dialect speech data.

[0044] Step 2: Using the trained voice recognition means and generative AI means, recognize the elderly person's voice input and generate operations.

[0045] Step 3: Execute the generated operation, which may include a specific operation such as turning on the television.

[0046] Example 1

[0047] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0048] When elderly people operate IT devices, there is a demand for improved ease of operation through voice input. However, a voice recognition system that can handle Japan's diverse dialects, especially difficult dialects such as Tsugaru dialect, has not yet been put into practical use. Therefore, it is necessary to develop a voice recognition system that allows elderly people to speak naturally in their own dialect.

[0049] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a voice recognition means, a generative artificial intelligence means, and a learning means for handling all Japanese dialects. This allows elderly people to speak naturally in their own dialect and operate information technology devices using voice input.

[0050] "Speech recognition means" is a technology for converting voice data into text data.

[0051] "Generative artificial intelligence means" refers to an artificial intelligence technology for generating appropriate operational instructions based on input data.

[0052] "Learning method to support all Japanese dialects" is a technology that pre-trains a voice recognition system to recognize the various dialects used in Japan.

[0053] "Means for learning Tsugaru dialect" refers to a technology that specially trains a voice recognition system to recognize Tsugaru dialect, which is said to be the most difficult dialect in Japan.

[0054] "Voice data preprocessing means" refers to technology for improving the accuracy of voice recognition by removing noise from the voice data and normalizing the volume.

[0055] "Means for practical application of speech recognition" refers to the technology for applying speech recognition technology to actual systems and applications, making it practically usable.

[0056] "Voice input when an elderly person operates an information technology device by themselves" refers to voice commands that an elderly person issues to operate an information technology device by themselves.

[0057] This invention is a system for improving the convenience of elderly people when operating information technology devices by voice input. This system includes a speech recognition means, a generative artificial intelligence means, a learning means for supporting all Japanese dialects, a means for training the Tsugaru dialect, a speech data preprocessing means, and a means for implementing speech recognition.

[0058] System configuration

[0059] Voice recognition means

[0060] The server converts the audio data into text using speech recognition software, such as the Google® Cloud Speech-to-Text API, which is pre-trained to accommodate the various dialects of Japan.

[0061] Audio data preprocessing method

[0062] The server uses audio processing libraries such as Librosa to denoise and normalize the volume of the audio data, making it clearer and easier to understand.

[0063] Generative AI methods

[0064] The server generates operational instructions based on the text data using a generative artificial intelligence model (e.g., GPT-4 (registered trademark) by OpenAI (registered trademark)). This generative artificial intelligence model is trained in advance on various operational scenarios to understand the user's intentions and generate appropriate operational instructions.

[0065] Executing an operation

[0066] The device executes the operation instructions received from the server, for example, opening an email application and sending an email with the specified content.

[0067] Specific examples

[0068] Example 1: Sending email by voice input

[0069] The user speaks "send email."

[0070] 1. The server uses speech recognition software to convert speech to text.

[0071] 2. The server uses Librosa to denoise and normalize the volume of the audio data.

[0072] 3. The server uses a generative artificial intelligence model to generate the operation instruction "send an email."

[0073] 4. The device will open the email app and send an email with the specified content.

[0074] Example 2: Getting weather information by voice input

[0075] The user speaks, "What's the weather like today?"

[0076] 1. The server uses speech recognition software to convert speech to text.

[0077] 2. The server uses Librosa to denoise and normalize the volume of the audio data.

[0078] 3. The server uses a generative artificial intelligence model to generate an operational instruction to "obtain weather information."

[0079] 4. The device will open the weather app and display the current weather information.

[0080] Prompt Sentence Examples

[0081] "Design a system that recognizes voice input spoken by an elderly person in Tsugaru dialect and generates appropriate actions. Specifically, explain how you combine speech recognition software with a generative artificial intelligence model to generate actions based on the voice input."

[0082] In this way, the system can improve the convenience of elderly people when operating information technology devices using voice input.

[0083] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0084] Step 1:

[0085] The user speaks to the terminal, for example, "send an email." This voice input becomes the initial input for the system.

[0086] Step 2:

[0087] The device sends the captured audio data to the server, which uses an audio processing library such as Librosa to remove noise and normalize the volume of the audio data, making the audio clearer and easier to understand. The input is raw audio data, and the output is preprocessed audio data.

[0088] Step 3:

[0089] The server sends the preprocessed speech data to speech recognition software (e.g., Google Cloud Speech-to-Text API) to convert the speech to text. The input is the preprocessed speech data, and the output is text data. For example, the generated text is "send me an email."

[0090] Step 4:

[0091] The server uses a generative artificial intelligence model (e.g., OpenAI's GPT-4) to generate operational instructions based on the text data. The input is text data obtained by speech recognition, and the output is a specific operational instruction. For example, the generated operational instruction is "send an email."

[0092] Step 5:

[0093] The device executes the operation instructions received from the server. For example, it opens an email application and sends an email with the specified content. The input is the operation instruction from the server, and the output is the actual operation result.

[0094] In this way, the system can improve the convenience of seniors when operating information technology devices using voice input.

[0095] (Application example 1)

[0096] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0097] When elderly people search for products or ask questions in physical stores, they face challenges such as difficulty in operating IT devices and difficulty in obtaining appropriate information. Furthermore, there is a lack of voice recognition systems that can handle all dialects in Japan, and there is a particular need to support difficult dialects.

[0098] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0099] In this invention, the server includes a voice recognition unit that recognizes voice inputs made by an elderly person when operating a smartphone, a unit that identifies the dialect spoken by the elderly person based on the recognized voice input, a prompt sentence that generates operation instructions for the smartphone based on the recognized voice input and the identified dialect, a unit that generates operation instructions for the smartphone using a generative AI, and a unit that operates the smartphone in accordance with the generated operation instructions for the smartphone. The voice recognition unit further recognizes voice inputs made by the elderly person when asking a question about a product in a physical store, and the server further includes a prompt sentence that generates an answer to the question based on the recognized voice input and the identified dialect, and a unit that generates an answer to the question using the generative AI. This enables a system that allows elderly people to easily operate a physical store using voice input when searching for products or asking questions.

[0100] "Speech recognition means" is a technology for converting speech into text.

[0101] "Generative AI means" is an artificial intelligence technology that generates appropriate responses and operations based on input data.

[0102] "Learning methods to support all dialects of Japan" refers to technology that learns the data necessary to understand and recognize dialects from all over Japan.

[0103] "Means for learning Tsugaru dialect" refers to technology that learns the data necessary to understand and recognize Tsugaru dialect, which is said to be the most difficult dialect in Japan.

[0104] "Means for practical application of speech recognition" refers to the technology for applying speech recognition technology to actual applications and systems.

[0105] "A means for recognizing voice input when elderly people search for products or ask questions in physical stores, and generating appropriate operations or responses" is a technology that recognizes the voices of elderly people when they search for products or ask questions using voice input in physical stores, and generates appropriate operations or responses.

[0106] The "application means installed on the smartphone" is software that is installed on the smartphone and provides a specific function.

[0107] The system for implementing this invention includes a speech recognition means, a generative AI means, a means for identifying all Japanese dialects, a means for learning Tsugaru dialect, a means for putting speech recognition into practical use, a means for recognizing voice input when an elderly person searches for a product or asks a question in a physical store and generating appropriate operations or responses, and an application means to be installed on a smartphone.

[0108] System Program

[0109] This system operates using the following hardware and software:

[0110] Hardware: Smartphone with microphone

[0111] Software: Python, SpeechRecognition library, Transformers library (Hugging Face's GPT-3 (registered trademark) model)

[0112] Processing flow

[0113] Voice Recognition

[0114] When a user speaks into the smartphone's microphone, the speech recognizer picks up the speech and converts it into text, a process that uses the SpeechRecognition library.

[0115] Generative AI

[0116] The speech data converted to text by the speech recognition tool is sent to the generative AI, which uses the Transformers library to generate appropriate actions and responses using the GPT-3 model.

[0117] output

[0118] The generated operations and answers are displayed on the smartphone screen, allowing seniors to easily use voice input when searching for products or asking questions in physical stores.

[0119] Specific example (smartphone operation)

[0120] When an elderly person says "send email" to their smartphone, the application recognizes the voice, and the generative AI identifies that it is in the Tsugaru dialect. It then generates instructions to launch the email sending app, which then launches the app according to those instructions. The generative AI also generates a response such as "Please enter the message you want to send," and the application displays the response.

[0121] Prompt Sentence Examples

[0122] "An elderly person is saying 'Send email' by voice. This is in Tsugaru dialect. Please generate instructions for smartphone operation."

[0123] In this way, a system can be created that helps elderly people operate smartphones more easily.

[0124] Specific example (physical store)

[0125] When an elderly person speaks to their smartphone in a physical store and asks, "Where is product A?", the application recognizes the speech, and the generative AI identifies that it is in Tsugaru dialect. It then generates and displays a specific answer, such as, "Product A is on the left side of aisle 3."

[0126] Prompt Sentence Examples

[0127] "An elderly person is typing, 'Where is product A?' This is in Tsugaru dialect. Please generate an answer to this question."

[0128] Generative AI answer: The item is on the left side of aisle 3.

[0129] In this way, a system can be created that helps seniors to shop more easily in brick-and-mortar stores.

[0130] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0131] Step 1:

[0132] The user speaks into the smartphone's microphone. The input is the user's voice, and the output is voice data. A specific example of this behavior is when the user says, "Where is this item?"

[0133] Step 2:

[0134] The device uses a speech recognition means to convert the voice data into text. The input is voice data and the output is text data. Specifically, the SpeechRecognition library is used to convert the voice into the text "Where is product A?"

[0135] Step 3:

[0136] The terminal sends text data to the generative AI. The input is text data, and the output is the input data to the generative AI. The specific operation is to send the converted text "Where is product A?" to the generative AI.

[0137] Step 4:

[0138] The server uses generative AI to generate appropriate operation instructions and answers based on the text data. The input is text data, and the output is the generated operation instructions and answers. Specifically, the GPT-3 model uses the Transformers library to generate the answer "Product A is located on the left side of aisle 3."

[0139] Step 5:

[0140] The server sends the generated operation instructions and answers to the terminal. The input is the generated operation instructions and answers, and the output is the data sent to the terminal. A specific operation is to send the generated answer "Product A is on the left side of aisle 3" to the terminal.

[0141] Step 6:

[0142] The device operates according to the generated operation instructions and displays the generated answer to the user. The input is the transmitted data, and the output is the data displayed to the user. A specific operation is to display the answer "Product A is located on the left side of aisle 3" on the smartphone screen.

[0143] It is also possible to use an emotion engine, which will be described later, to recognize the emotional state of the elderly person and generate smartphone operation instructions and answers to questions by further taking the elderly person's emotional state into consideration. In this case, the server further includes means for recognizing the elderly person's emotional state, and the generating means generates the smartphone operation instructions using a prompt sentence that generates the smartphone operation instructions based on the recognized voice input, the identified dialect, and the recognized emotional state of the elderly person, and the generative AI.

[0144] Example 2

[0145] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0146] When elderly people operate IT devices, they face the challenge of accurately recognizing speech input, especially when it is in a dialect, and generating appropriate operations. Furthermore, due to a lack of speech recognition technology capable of handling certain dialects, such as the Tsugaru dialect, there is a need for a system that can accurately handle speech input in these dialects.

[0147] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0148] In this invention, the server includes a speech recognition unit, a generative artificial intelligence unit, a dialect adaptation learning unit, a specific dialect learning unit, a speech recognition implementation unit, a unit for converting speech input into text data, a unit for transmitting the text data to the generative artificial intelligence model, a unit for receiving the generated operation, and a unit for controlling the infrared signal generator. This makes it possible to accurately recognize speech and generate appropriate operations even when an elderly person operates information technology equipment using a dialect.

[0149] "Speech recognition means" refers to a device or software that analyzes voice input and converts it into text data.

[0150] "Generative artificial intelligence means" refers to an artificial intelligence model that generates appropriate operations and responses based on input data.

[0151] The "dialect-compatible learning means" is a means for training a speech recognition model to accommodate dialects of different regions.

[0152] A "specific dialect learning means" is a means for training a speech recognition model specializing in a specific dialect, for example, a difficult dialect such as Tsugaru dialect.

[0153] "Means for practical application of speech recognition" refers to the means for incorporating speech recognition technology into actual systems and devices and making it usable.

[0154] The "means for converting speech input into text data" refers to a means for converting speech input into text format using a speech recognition means.

[0155] The "means for transmitting text data to a generative artificial intelligence model" refers to a means for transmitting converted text data to a generative artificial intelligence model.

[0156] The "means for receiving a generated operation" is a means for receiving an operation generated from a generative artificial intelligence model.

[0157] "Means for controlling an infrared signal generating device" refers to means for controlling a device that generates an infrared signal and for performing a specific operation.

[0158] The present invention provides a system for accurately recognizing speech and generating appropriate operations when an elderly person operates information technology equipment using a dialect. Specific embodiments of this system are described below.

[0159] First, the user says "Turn on the TV" in Tsugaru dialect. The microphone on the device captures this voice. Speech recognition software (for example, a speech recognition API) is used as the speech recognition method. This software analyzes the captured voice and converts it into text data.

[0160] The device then sends the converted text data to a generative artificial intelligence model (e.g., a generative AI model). This generative AI model generates appropriate actions based on the input text data. Specifically, the following prompt sentence is generated:

[0161] Example prompt sentence:

[0162] The user says "Turn on the TV" in Tsugaru dialect. Generate an operation that generates an infrared signal to turn on the TV.

[0163] Based on this prompt, the server uses a generative artificial intelligence model to generate a specific command to turn on the TV, and the generated command is sent to the device.

[0164] Based on the received operation, the terminal controls an infrared signal generator (e.g., an infrared signal generator), which emits an infrared signal toward the television, which receives the signal and turns on.

[0165] For example, if a user says "Turn on the TV" in Tsugaru dialect, the device's microphone will capture this speech and convert it into text using a speech recognition API. This text data will be sent to the generative AI model, which will generate the following prompt:

[0166] Example prompt sentence:

[0167] The user says "Turn on the TV" in Tsugaru dialect. Generate an operation that generates an infrared signal to turn on the TV.

[0168] Based on this prompt, the server uses a generative AI model to generate specific commands to turn on the TV and sends them to the device, which then controls an infrared signal generator, which turns on the TV when it receives the infrared signal.

[0169] In this way, even when a user operates information technology equipment using a dialect, it is possible to accurately recognize speech and generate appropriate operations.

[0170] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0171] Step 1:

[0172] The user provides voice input.

[0173] A user says "Turn on the TV" in Tsugaru dialect. The input is the user's voice, and the output is the voice data captured by the device's microphone.

[0174] Step 2:

[0175] The device recognizes the voice.

[0176] The microphone on the device captures the user's voice. A speech recognition API is used as the speech recognition method. The input is the captured voice data, and the output is analyzed text data.

[0177] Step 3:

[0178] The device converts the recognized speech into text.

[0179] The device uses a speech recognition API to convert the captured voice into text data. The input is voice data and the output is text data.

[0180] Step 4:

[0181] The device sends the text to a generative AI model.

[0182] The terminal sends the converted text data to the generative AI model. The input is the text data, and the output is the data sent to the generative AI model.

[0183] Step 5:

[0184] The server generates operations using a generative AI model.

[0185] The server uses a generative AI model to generate appropriate actions based on the received text data. Specifically, the following prompt sentence is generated:

[0186] Example prompt sentence:

[0187] The user says "Turn on the TV" in Tsugaru dialect. Generate an operation that generates an infrared signal to turn on the TV.

[0188] The input is text data and the output is the generated operational data.

[0189] Step 6:

[0190] The server sends the generated operation to the terminal.

[0191] The server sends the generated operation to the terminal. The input is the generated operation data, and the output is the operation data sent to the terminal.

[0192] Step 7:

[0193] The terminal controls the infrared signal generator.

[0194] The terminal controls the infrared signal generator based on the received operation. The input is the operation data, and the output is the infrared signal.

[0195] Step 8:

[0196] The TV operates by receiving infrared signals.

[0197] The infrared signal generator emits an infrared signal toward the television, which receives the signal and turns on. The input is the infrared signal, and the output is the operation of the television.

[0198] (Application example 2)

[0199] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0200] In modern brick-and-mortar stores, customers who speak dialects have difficulty receiving product information through voice. Particularly for elderly customers and customers in areas where dialects are spoken, speech recognition systems that only understand standard Japanese cannot provide sufficient service. For this reason, there is a demand for a system that combines dialect-compatible speech recognition with generative AI.

[0201] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0202] In this invention, the server includes a speech recognition unit, a generative AI unit, and a learning unit for supporting various dialects, which allows customers who speak different dialects to receive product information by voice.

[0203] "Speech recognition means" is a technology for converting speech into text.

[0204] "Generative AI means" is an artificial intelligence technology that generates appropriate responses and operations based on input data.

[0205] A "dialect-specific learning method" is a learning algorithm for understanding and recognizing dialects from different regions.

[0206] A "means of learning difficult dialects" is a special learning process for recognizing dialects that are particularly difficult to understand.

[0207] "Means for practical application of speech recognition" refers to means for making speech recognition technology usable in actual applications.

[0208] The "guide means for providing product information within the store" is a system for providing product locations and information to customers within the store.

[0209] "Application means installed on a smart device" refers to software installed on a device such as a smartphone or tablet.

[0210] In order to implement this invention, it is necessary to construct a system that includes a speech recognition means, a generative AI means, a learning means for handling all dialects, a means for learning difficult dialects, a means for putting speech recognition into practical use, a guidance means for providing product information in a store, and an application means to be installed on a smart device.

[0211] System configuration

[0212] 1. Voice recognition means:

[0213] The speech recognition means is used to convert the user's speech into text. Specifically, the smartphone's microphone is used to capture the speech, and the SpeechRecognition library is used to convert the speech into text.

[0214] 2. Generative AI means:

[0215] The generative AI mechanism generates appropriate responses and actions based on the text captured by the speech recognition mechanism, specifically leveraging the GPT-3 model using the Hugging Face transformers library.

[0216] 3. Learning tools for all dialects:

[0217] The means includes a learning algorithm for understanding and recognizing different regional dialects, and a special learning process is implemented to provide a means for learning particularly difficult dialects.

[0218] 4. Practical application of speech recognition:

[0219] It is a means to make speech recognition technology usable in actual applications, including an interface that allows generative AI means to generate appropriate responses based on speech recognition results and provide them to users.

[0220] 5. Guidance methods for in-store product guidance:

[0221] This is a system that provides customers with product locations and information in stores. Specifically, it displays a map showing product locations on the smartphone screen and provides audio guidance.

[0222] 6. Application means installed on smart devices:

[0223] This software is installed on devices such as smartphones and tablets. This application integrates speech recognition and generative AI, allowing users to receive product information by voice.

[0224] Processing flow

[0225] 1. Acquiring voice input:

[0226] When a user speaks into the smartphone, the voice recognition unit picks up the voice. For example, the user might say, "Look for this product."

[0227] 2. Speech Recognition:

[0228] The speech recognition means converts the speech captured into text. The SpeechRecognition library is used to convert speech into text.

[0229] 3. Response generation by generative AI:

[0230] A generative AI method uses the converted text as a prompt to generate an appropriate response, specifically using the Hugging Face GPT-3 model to generate the response.

[0231] 4. Providing a response:

[0232] The generated response is displayed on the smartphone screen or given as a voice prompt, for example, "This product is on the shelf at the back right of the store."

[0233] Specific examples

[0234] User says: "Find this product"

[0235] Generated response: "This item is on the back right shelf of the store."

[0236] Prompt Sentence Examples

[0237] "If someone says 'Look for this product' in Tsugaru dialect, how can I guide them to the product in the store?"

[0238] In this way, combining dialect-aware speech recognition with generative AI can significantly improve customer service in brick-and-mortar stores.

[0239] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0240] Step 1:

[0241] The user speaks into the smartphone.

[0242] Input: User speech (e.g., "Look for this product")

[0243] How it works: The smartphone microphone picks up the audio.

[0244] Output: Captured audio data

[0245] Step 2:

[0246] The terminal uses a voice recognition means to convert the captured voice data into text.

[0247] Input: Audio data

[0248] What it does: Converts speech to text using the SpeechRecognition library.

[0249] Output: The converted text (e.g. "Look for this product")

[0250] Step 3:

[0251] The device uses generative AI means to generate a response based on the converted text.

[0252] Input: Translated text

[0253] How it works: It uses Hugging Face's GPT-3 model to generate prompt sentences and generate appropriate responses.

[0254] Output: Generated response text (e.g., "This item is on the back right shelf in the store.")

[0255] Step 4:

[0256] The terminal provides the generated response to the user.

[0257] Input: Generated response text

[0258] What it does: Displays a text response on the phone screen or plays an audio response.

[0259] Output: The user confirms the response.

[0260] Step 5:

[0261] The user follows the instructions to find the product.

[0262] Input: Response text or voice prompts

[0263] Action: The user navigates the store and follows the instructions to find a product.

[0264] Output: The user finds the product they are looking for.

[0265] In this way, specific actions and data processing and calculations are performed at each step, enabling customer service in physical stores to be realized using dialect-compatible voice recognition and generative AI.

[0266] Example 3

[0267] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0268] It is difficult to realize a speech recognition system that can handle all dialects in Japan, especially difficult dialects. In particular, it is difficult for conventional speech recognition technology to handle Tsugaru dialect, which is said to be the most difficult dialect in Japan. Another challenge is accurately recognizing voice input and generating appropriate responses when elderly people operate information technology devices themselves.

[0269] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0270] In this invention, the server includes a speech recognition unit, a generative artificial intelligence unit, and a learning unit for supporting all Japanese dialects. This enables a system including a means for a user to input speech into a terminal, a means for the terminal to transmit speech data to the server, a means for the server to convert the speech data into text data using speech recognition software, a means for the server to input the text data into a generative artificial intelligence model, a means for the generative artificial intelligence model to generate an appropriate response, a means for the server to transmit the generated response to the terminal, and a means for the terminal to display the response to the user. This enables speech recognition and response generation compatible with all Japanese dialects, particularly difficult dialects such as Tsugaru dialect. It also enables accurate recognition of speech input and generation of appropriate responses when elderly people operate information technology devices themselves.

[0271] "Speech recognition means" is a technology for converting voice data into text data.

[0272] "Generative AI means" refers to an AI technology for generating appropriate responses based on input text data.

[0273] A "learning tool" is a method for training a system with specific data to improve its ability to perform a specific task.

[0274] The "means for learning Tsugaru dialect" is a method for training speech recognition means and generative artificial intelligence means using a large amount of Tsugaru dialect speech data.

[0275] "Means for practical application of speech recognition" refers to methods for applying speech recognition technology to actual applications and systems.

[0276] "Means for users to input voice into a terminal" refers to a method by which a user inputs voice into a device such as a smartphone or PC.

[0277] The "means for the terminal to transmit audio data to the server" refers to a method for transmitting audio data captured by the terminal to the server via the Internet.

[0278] "Means for the server to convert voice data into text data using voice recognition software" refers to a method in which the server converts received voice data into text data using voice recognition software.

[0279] "Means for the server to input text data into the generative artificial intelligence model" refers to a method in which the server inputs converted text data into the generative artificial intelligence model.

[0280] The "means by which a generative artificial intelligence model generates an appropriate response" is a method by which a generative artificial intelligence model generates an appropriate response based on input text data.

[0281] "Means for the server to transmit the generated response to the terminal" refers to a method by which the server transmits the response returned from the generative artificial intelligence model to the terminal.

[0282] The "means by which the terminal displays the response to the user" refers to the method by which the terminal displays the response received from the server to the user.

[0283] This invention provides a speech recognition system that supports all Japanese dialects. In particular, it has a learning means for supporting Tsugaru dialect, which is said to be the most difficult dialect in Japan. This system includes a speech recognition means, a generative artificial intelligence means, a learning means, a speech recognition implementation means, a means for a user to input speech into a terminal, a means by the terminal transmitting speech data to a server, a means by the server converting the speech data into text data using speech recognition software, a means by the server inputting the text data into a generative artificial intelligence model, a means by the generative artificial intelligence model generating an appropriate response, a means by the server transmitting the generated response to a terminal, and a means by the terminal displaying the response to a user.

[0284] The server first collects a large amount of Tsugaru dialect speech data. This speech data is converted into text data using speech recognition software (e.g., Google Speech-to-Text API). The converted text data is then used to train a generative artificial intelligence model (e.g., OpenAI's GPT-4).

[0285] The device transmits the voice data entered by the user to the server in real time. The server converts the received voice data into text data using speech recognition software, and inputs the text data into a generative AI model. The generative AI model generates an appropriate response based on the input text data and returns the response as text data to the device.

[0286] The user receives the generated response through the terminal and, if necessary, re-enters the voice input. By repeating this process, the user can have a natural conversation that corresponds to the dialect.

[0287] As a concrete example, consider the case where a user says "How is it today?" in Tsugaru dialect. The device sends this voice data to the server, which uses speech recognition software to convert it into text data such as "How is it today?". This text data is then input into a generative artificial intelligence model, which generates an appropriate response. For example, the response generated might be "It's a nice day today." This response is then sent back to the device as text data and displayed to the user.

[0288] An example of a prompt might be:

[0289] User input: "How's your day going?"

[0290] Prompt to generative AI model: "The user says, 'How is today?' in Tsugaru dialect. Please generate an appropriate response."

[0291] In this way, the system provides a learning means for dealing with all dialects in Japan, enabling the user to have natural conversations. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0292] Step 1:

[0293] The user inputs voice into the terminal.

[0294] The user speaks into a device such as a smartphone or PC. For example, they say, "How is it today?" in Tsugaru dialect. This speech is captured by the device's microphone. The input is the user's voice data, and the output is the voice data captured by the device.

[0295] Step 2:

[0296] The terminal transmits the voice data to the server.

[0297] The device compresses the captured audio data and sends it to a server via the Internet. Specifically, the device compresses the audio data into MP3 format and sends it to the server using the HTTPS protocol. The input is the captured audio data, and the output is the audio data sent to the server.

[0298] Step 3:

[0299] The server converts the voice data into text data using voice recognition software.

[0300] The server sends the received voice data to speech recognition software (for example, Google Speech-to-Text API) and converts it into text data. For example, voice data such as "How is it today?" is converted into text data such as "How is it today?". The input is the voice data sent to the server, and the output is the converted text data.

[0301] Step 4:

[0302] The server inputs the text data into a generative artificial intelligence model.

[0303] The server sends the converted text data to a generative AI model (e.g., OpenAI's GPT-4) as an API request. For example, send the text data "How is it today?" to the generative AI model. The input is the converted text data, and the output is an API request to the generative AI model.

[0304] Step 5:

[0305] A generative artificial intelligence model generates appropriate responses.

[0306] A generative AI model generates an appropriate response based on input text data. For example, a response such as "It's nice weather today" may be generated. This response is returned to the server as text data. The input is an API request to the generative AI model, and the output is the generated response text data.

[0307] Step 6:

[0308] The server sends the generated response to the terminal.

[0309] The server receives the response text data returned from the generative AI model and sends it to the device as an API response. For example, the text data "It's nice weather today" is sent to the device. The input is the generated response text data, and the output is the response text data sent to the device.

[0310] Step 7:

[0311] The terminal displays the response to the user.

[0312] The terminal displays the response text data received from the server to the user. For example, the text "It's a nice day today" is displayed on the screen of a smartphone. The input is the response text data sent to the terminal, and the output is the response text displayed to the user.

[0313] (Application example 3)

[0314] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0315] Conventional voice recognition systems are specialized in standard Japanese, making it difficult to handle the various dialects of Japan. Furthermore, they are unable to handle Tsugaru dialect, which is said to be the most difficult dialect in Japan, making them inconvenient for users who speak dialects. Furthermore, food delivery services have the problem of making it difficult to order in dialect, forcing users to use standard Japanese.

[0316] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes a voice recognition means, a generative AI means, a learning means for supporting all Japanese dialects, a means for training Tsugaru dialect, which is said to be the most difficult dialect in Japan, a practical voice recognition means, a means for recognizing and analyzing voice input in a dialect and generating order details, a means for transmitting the generated order details via a communication means, a means for allowing the user to confirm the order details, and a means for notifying the user that the order has been confirmed. This enables users who speak a dialect to input voice in their dialect when using a food delivery service, improving convenience.

[0317] "Speech recognition means" is a technology for converting speech into text.

[0318] "Generative AI means" is an artificial intelligence technology that generates new information and operations based on input data.

[0319] "Learning tools to support all dialects of Japan" refers to technology that collects and learns the data necessary to recognize and understand dialects from all over Japan.

[0320] "Means for learning Tsugaru dialect" refers to a technology for collecting and learning the data necessary to recognize and understand Tsugaru dialect, which is said to be the most difficult dialect in Japan.

[0321] "Means for practical application of speech recognition" refers to technologies that make speech recognition technology usable in actual applications and services.

[0322] "Means for recognizing and analyzing speech input in a dialect and generating order details" refers to technology that recognizes speech input in a dialect, analyzes the content, and generates appropriate order details.

[0323] The "means for transmitting the generated order details via a communication means" refers to a technique for transmitting the generated order details using the Internet or other communication means.

[0324] The "means for having the user confirm the order details" is a technique for presenting the generated order details to the user and requesting their confirmation.

[0325] "Means for notifying that an order has been confirmed" refers to a technique for notifying a user that an order has been confirmed.

[0326] The following system configuration will be described as an embodiment of the present invention.

[0327] System Configuration

[0328] This system includes a speech recognition means, a generative AI means, a learning means to support all Japanese dialects, a means for learning Tsugaru dialect, a means for putting speech recognition into practical use, a means for recognizing and analyzing speech input in a dialect and generating order details, a means for transmitting the generated order details via a communication means, a means for allowing the user to confirm the order details, and a means for notifying that the order has been confirmed.

[0329] Hardware and software used

[0330] Hardware: Smartphone (iOS or ANDROID (registered trademark))

[0331] software:

[0332] Speech recognition API (e.g. Google Cloud Speech-to-Text)

[0333] Generative AI models (e.g., OpenAI GPT-4)

[0334] Food delivery API (e.g. Uber Eats API)

[0335] Processing flow

[0336] 1. Voice input: The user speaks their order in their local dialect into the microphone on their smartphone.

[0337] 2. Speech recognition: The smartphone uses a speech recognition API to convert the input speech into text, using a dialect-aware speech recognition model.

[0338] 3. Text analysis: The server uses a generative AI model to analyze the recognized text and understand the order.

[0339] 4. Order Generation: Based on the analysis results, the server generates an appropriate order using the food delivery API and sends it to the restaurant.

[0340] 5. Confirmation and Notification: The smartphone will prompt the user to confirm the order and notify them that the order has been placed.

[0341] Specific examples

[0342] When a user says "I'd like to order one pizza" in Tsugaru dialect, the smartphone recognizes this and generates an appropriate pizza order and sends it to the restaurant.

[0343] Prompt Sentence Examples

[0344] If a user says "I'd like a pizza" in Tsugaru dialect, use a speech recognition API to convert the speech to text, use a generative AI model to analyze the order, and use a food delivery API to generate a pizza order.

[0345] In this way, the dialect-enabled food delivery assistant allows users to place orders in their dialect, providing convenient services to more users.

[0346] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0347] Step 1:

[0348] The user speaks their order in their local dialect into the smartphone microphone. The input is the user's voice, and the output is the voice data input into the smartphone microphone.

[0349] Step 2:

[0350] The device uses a speech recognition API (e.g., Google Cloud Speech-to-Text) to convert the input voice data into text. The input is voice data, and the output is text data. In this case, a dialect-compatible speech recognition model is used.

[0351] Step 3:

[0352] The server uses a generative AI model (e.g., OpenAI GPT-4) to analyze the text data sent from the speech recognition API and understand the order details. The input is text data, and the output is the analyzed order details data.

[0353] Step 4:

[0354] Based on the analysis results, the server generates an appropriate order using a food delivery API (e.g., Uber Eats API) and sends it to the restaurant. The input is the parsed order data, and the output is the order data sent through the food delivery API.

[0355] Step 5:

[0356] The terminal displays the generated order details to the user to confirm the order details. The input is the order confirmation data from the food delivery API, and the output is the order details displayed on the smartphone screen.

[0357] Step 6:

[0358] The user confirms the order details and confirms it. The input is the user's confirmation operation, and the output is a signal that the order has been confirmed.

[0359] Step 7:

[0360] The terminal notifies the server that the order has been confirmed, and the server sends that information to the food delivery API. The input is the order confirmation signal, and the output is the order confirmation data sent to the food delivery API.

[0361] Step 8:

[0362] The server notifies the user that the order has been placed. The input is the order confirmation data from the food delivery API, and the output is a confirmation of the order that is displayed to the user via the smartphone's notification function.

[0363] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0364] "Example 1"

[0365] In one embodiment of the present invention, a system is provided that includes a speech recognition means, a generative AI means, a learning means for supporting all Japanese dialects, a means for learning Tsugaru dialect, a means for implementing speech recognition, and an emotion engine that recognizes a user's emotion. This emotion engine recognizes the emotion from the user's voice and generates an operation corresponding to that emotion. For example, when a user vocally inputs "Turn on the TV," the emotion engine recognizes the user's emotion from the voice. If the emotion is recognized as "joy," the generative AI means generates an operation to turn on the TV.

[0366] "Example 2"

[0367] In another embodiment of the present invention, the emotion engine generates an operation according to the user's emotion. For example, when the user makes a voice input such as "Turn on the radio," the emotion engine recognizes the user's emotion from the voice. If the emotion is recognized as "sadness," the generative AI means generates an operation to turn on the radio, and provides a result in which the operation is adapted to the user's emotion. Specifically, the generative AI means generates an operation such as setting the radio to a channel that plays sad songs that the user likes.

[0368] "Example 3"

[0369] Furthermore, in another embodiment of the present invention, the emotion engine recognizes the user's emotion, and the generative AI means generates an operation according to that emotion. For example, when the user makes a voice input such as "Turn on the air conditioner," the emotion engine recognizes the user's emotion from the voice. If the emotion is recognized as "anger," the generative AI means generates an operation to turn on the air conditioner, providing a result where the operation is adapted to the user's emotion. Specifically, the generative AI means generates an operation such as setting the air conditioner temperature to a low temperature that will make the user feel calm.

[0370] The processing flow of each embodiment will be described below.

[0371] "Example 1"

[0372] Step 1: The user gives a voice input. In this example, the voice input is "Turn on the TV."

[0373] Step 2: The emotion engine recognizes the emotion from the user's voice. In this example, it recognizes "joy."

[0374] Step 3: The generative AI generates an action based on the recognized emotion. In this example, it generates the action to turn on the TV.

[0375] "Example 2"

[0376] Step 1: The user gives a voice input. In this example, the voice input is "Turn on the radio."

[0377] Step 2: The emotion engine recognizes the emotion from the user's voice. In this example, it recognizes "sadness."

[0378] Step 3: The generative AI generates an operation based on the recognized emotion, and provides a result that adapts to the user's emotion. In this example, it generates an operation to turn on the radio, and provides a result that adapts to the user's emotion. Specifically, it generates an operation such as setting the radio channel to a channel that plays sad songs that the user likes.

[0379] "Example 3"

[0380] Step 1: The user gives a voice input. In this example, the user gives the voice input "Turn on the air conditioner."

[0381] Step 2: The emotion engine recognizes the emotion from the user's voice. In this example, it recognizes "anger."

[0382] Step 3: The generative AI generates an operation based on the recognized emotion, and provides a result that adapts the operation to the user's emotion. In this example, it generates an operation to turn on the air conditioner, and provides a result that adapts the operation to the user's emotion. Specifically, it generates an operation such as setting the air conditioner temperature to a low temperature that will make the user feel calm.

[0383] Example 1

[0384] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0385] When elderly people operate IT devices, it is difficult to accurately recognize voice input and generate appropriate operations. In particular, there is a need to support all Japanese dialects, but the diversity and difficulty of dialects reduces the accuracy of voice recognition. In addition, there is also the problem that it is not possible to generate operations that take the user's emotions into account, which results in a poor user experience.

[0386] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0387] In this invention, the server includes a speech recognition unit, a generative artificial intelligence unit, and a learning unit for supporting all Japanese dialects. This allows for accurate recognition of speech input when elderly people operate information technology devices, and enables dialect-compatible speech recognition. Furthermore, by including an emotion engine that recognizes the user's emotions, it becomes possible to generate operations according to the user's emotions, improving the user experience.

[0388] "Speech recognition means" is a technology for converting voice input into text data.

[0389] "Generative artificial intelligence means" refers to artificial intelligence technology for generating appropriate operations based on the results of speech recognition.

[0390] "Learning method to support all Japanese dialects" is a technology that trains a speech recognition model to recognize the various dialects used in Japan.

[0391] "Dialect training" refers to the technique of specifically training a speech recognition model to recognize a particular dialect, especially a difficult dialect.

[0392] "Means for practical application of speech recognition" refers to the technology for applying speech recognition technology to actual applications and systems.

[0393] An "emotion engine" is a technology that analyzes emotions from a user's voice and generates appropriate operations based on those emotions.

[0394] MODE FOR CARRYING OUT THE INVENTION

[0395] This invention is a system that includes a speech recognition means, a generative artificial intelligence means, a learning means for dealing with all dialects of Japan, a means for learning particularly difficult dialects, a means for putting speech recognition into practical use, and an emotion engine. A specific embodiment of this system will be described below.

[0396] Acquiring voice input

[0397] Subject: User

[0398] The user speaks into the device, for example, to give instructions such as "Turn on the TV." The device is equipped with a high-performance microphone that picks up the user's voice with high accuracy.

[0399] Sending audio data

[0400] Subject: Terminal

[0401] The device transmits the acquired voice data to the server, where it is converted into an appropriate format and sent to the server via the network.

[0402] Voice Recognition

[0403] Subject: Server

[0404] The server converts the received voice data into text data using a voice recognition means. Specifically, it converts the voice into text using voice recognition software (for example, Google Cloud Speech-to-Text API).

[0405] Dialect-aware learning

[0406] Subject: Server

[0407] The server has training tools to handle all dialects in Japan, and in particular trains speech recognition models to recognize difficult dialects (e.g., Tsugaru dialect).

[0408] emotion recognition

[0409] Subject: Terminal

[0410] The device has an emotion engine that recognizes emotions from the user's voice. Specifically, it uses emotion analysis software (for example, Microsoft® Azure® Emotion API) to analyze emotions from the user's voice.

[0411] Generate Operations

[0412] Subject: Server

[0413] The server uses a generative artificial intelligence (e.g., OpenAI's GPT-4) to generate actions based on the speech recognition and emotion recognition results. The generated actions are sent to the device.

[0414] Executing an operation

[0415] Subject: Terminal

[0416] The device executes the operation received from the server, specifically, sending a signal to turn on the TV.

[0417] Specific examples

[0418] Example 1

[0419] When a user says "Turn on the TV," the device captures the speech and sends it to the server. The server converts the speech into text using a speech recognition means and generates the operation "Turn on the TV" using a generative artificial intelligence means. The device executes the operation and turns on the TV.

[0420] Example 2

[0421] When a user says "Turn off the air conditioner," the device captures the speech and sends it to the server. The server converts the speech into text using a speech recognition means and generates the operation "Turn off the air conditioner" using a generative artificial intelligence means. The device then executes the operation and turns off the air conditioner.

[0422] Prompt Sentence Examples

[0423] "When a user says 'Turn on the TV,' convert the speech to text using the speech recognition means, and generate the operation to turn on the TV using the generative artificial intelligence means. Also, if the user's emotion is 'joy,' provide that information to the generative artificial intelligence means."

[0424] By inputting this prompt into a generative AI model, the system's behavior can be simulated.

[0425] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0426] Step 1:

[0427] Acquiring voice input

[0428] Subject: User

[0429] The user speaks to the device, saying, "Turn on the TV." The device is equipped with a high-performance microphone, which picks up the user's voice with high accuracy.

[0430] Input: User's voice commands

[0431] Output: Captured audio data

[0432] Specific action: Instead of searching for the remote control in the living room, the user speaks to the device and says, "Turn on the TV."

[0433] Step 2:

[0434] Sending audio data

[0435] Subject: Terminal

[0436] The device transmits the acquired voice data to the server, where it is converted into an appropriate format and sent to the server via the network.

[0437] Input: Captured audio data

[0438] Output: Audio data sent to the server

[0439] How it works: Your device sends audio data to the server over Wi-Fi, which takes just a few seconds.

[0440] Step 3:

[0441] Voice Recognition

[0442] Subject: Server

[0443] The server converts the received voice data into text data using a voice recognition means. Specifically, it converts the voice into text using voice recognition software (for example, Google Cloud Speech-to-Text API).

[0444] Input: Audio data sent to the server

[0445] Output: Text data (e.g. "Turn on the TV")

[0446] Specific operation: The server receives the voice data and calls the Google Cloud Speech-to-Text API to generate the text "Turn on the TV."

[0447] Step 4:

[0448] Dialect-aware learning

[0449] Subject: Server

[0450] The server has training tools to handle all dialects in Japan, and in particular trains speech recognition models to recognize difficult dialects (e.g., Tsugaru dialect).

[0451] Input: Dialect speech data

[0452] Output: dialect-aware speech recognition model

[0453] Specific operation: The server uses Tsugaru dialect speech data to train the model so that it can handle dialects.

[0454] Step 5:

[0455] emotion recognition

[0456] Subject: Terminal

[0457] The device has an emotion engine that recognizes emotions from the user's voice. Specifically, it uses emotion analysis software (for example, Microsoft Azure's Emotion API) to analyze emotions from the user's voice.

[0458] Input: User's voice data

[0459] Output: Emotion data (e.g., "joy")

[0460] Specific operation: The device analyzes the user's emotion from the voice saying "Turn on the TV" and recognizes that emotion as "joy."

[0461] Step 6:

[0462] Generate Operations

[0463] Subject: Server

[0464] The server uses a generative artificial intelligence (e.g., OpenAI's GPT-4) to generate actions based on the speech recognition and emotion recognition results. The generated actions are sent to the device.

[0465] Input: Text data (e.g., "Turn on the TV"), emotion data (e.g., "Joy")

[0466] Output: The generated action (e.g. "Turn on the TV")

[0467] Specific operation: The server inputs the text "Turn on the TV" and the emotion information "joy" into GPT-4, and generates the operation "Turn on the TV."

[0468] Step 7:

[0469] Executing an operation

[0470] Subject: Terminal

[0471] The device executes the operation received from the server, specifically, sending a signal to turn on the TV.

[0472] Input: The generated action (e.g. "Turn on the TV")

[0473] Output: The action taken (e.g. the TV is turned on)

[0474] Specific operation: The device sends a TV remote control signal, and the TV turns on.

[0475] (Application example 1)

[0476] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0477] When elderly people or customers who speak dialects operate IT devices, the accuracy of voice recognition is low, making it difficult to operate them. There is also the issue of it being difficult to properly recognize customer emotions in physical stores and respond accordingly.

[0478] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes a voice recognition means, a generative AI means, a learning means for supporting all Japanese dialects, a learning means for Tsugaru dialect, which is said to be the most difficult dialect in Japan, a voice recognition practical use means, an emotion recognition means, an application means installed on the smart glasses, a means for recognizing a customer's voice input and generating an appropriate response by the generative AI means, a means for recognizing the customer's emotion and generating a response according to the emotion by the emotion recognition means, and a voice output means. This improves the accuracy of voice recognition when elderly people or customers who speak dialects operate IT devices, enabling appropriate responses according to the customer's emotion in physical stores.

[0479] "Speech recognition means" is a technology that analyzes voice input and converts it into text data.

[0480] "Generative AI means" is an artificial intelligence technology that generates appropriate responses and operations based on input data.

[0481] "Learning method to support all Japanese dialects" is a technology that trains a speech recognition model to recognize the various dialects used in Japan.

[0482] "Means for learning Tsugaru dialect" refers to special learning techniques to understand Tsugaru dialect, which is said to be the most difficult dialect in Japan.

[0483] "Means for practical application of speech recognition" refers to technologies that make speech recognition technology usable in actual systems and applications.

[0484] "Emotion recognition means" is a technology that analyzes and recognizes a user's emotions from voice and text data.

[0485] "Application means installed on smart glasses" refers to applications that run on smart glasses, and is software for utilizing voice recognition and generative AI.

[0486] "Means for recognizing customer voice input and for a generative AI means to generate an appropriate response" refers to technology that recognizes customer voice and for a generative AI means to generate an appropriate response based on the content of that voice.

[0487] "Means for recognizing a customer's emotion and generating a response according to the emotion" refers to a technology that recognizes a customer's emotion and generates an appropriate response according to the emotion.

[0488] "Audio output means" refers to a technology for outputting generated text data and responses as audio.

[0489] A system for implementing this invention includes a voice recognition means, a generative AI means, a learning means for supporting all dialects of Japan, in particular a means for learning Tsugaru dialect, a means for putting voice recognition into practical use, an emotion recognition means, an application means to be installed in the smart glasses, a means for recognizing a customer's voice input and for the generative AI means to generate an appropriate response, a means for the emotion recognition means to recognize the customer's emotion and for generating a response according to the emotion, and a voice output means.

[0490] Hardware and Software Configuration

[0491] The server uses the speech_recognition library for speech recognition and the transformers library for generative AI. It also uses the sentiment-analysis pipeline of the transformers library for emotion recognition. It uses the pyttsx3 library for audio output. The smart glasses have a built-in microphone and speaker, and these software are installed.

[0492] Data processing and calculation

[0493] 1. Speech recognition: When a user speaks into the smart glasses, the microphone captures the voice and the speech_recognition library converts the voice into text data.

[0494] 2. Generative AI: The converted text data is fed into a generative AI model from the transformers library to generate an appropriate response.

[0495] 3. Emotion Recognition: At the same time, the text data is input into a sentiment-analysis pipeline to analyze the user's emotions. Based on the analysis results, a generative AI model generates a response according to the emotion.

[0496] 4. Audio Output: Finally, the generated response is output as audio using the pyttsx3 library.

[0497] Specific examples

[0498] For example, if a customer in a physical store says, "Tell me about this product," the system operates as follows:

[0499] 1. Speech recognition: The customer's voice is captured by the microphone in the smart glasses and converted into text data using the speech_recognition library: "Tell me about this product."

[0500] 2. Generative AI: The converted text data is fed into a generative AI model from the Transformers library, which generates a response such as, "This product is a state-of-the-art smartphone with an excellent camera. It also has a long battery life, allowing for extended use."

[0501] 3. Emotion recognition: At the same time, the text data is fed into a sentiment-analysis pipeline to identify customer concerns. A generative AI model then generates a more detailed explanation.

[0502] 4. Audio Output: Finally, the generated response is output as audio using the pyttsx3 library.

[0503] Prompt Sentence Examples

[0504] Customer: Tell me about this product.

[0505] System: This is a state-of-the-art smartphone with an excellent camera. It also has a long battery life, allowing it to be used for a long time.

[0506] In this way, the accuracy of voice recognition will be improved when elderly people or customers who speak dialects operate IT devices, making it possible to respond appropriately to customers' emotions in physical stores.

[0507] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0508] Step 1:

[0509] A user speaks into the smart glasses. The microphone in the smart glasses captures the audio. The input is the user's voice and the output is the captured audio data.

[0510] Step 2:

[0511] The server uses the speech_recognition library as a speech recognition tool to convert the captured audio data into text data. The input is audio data and the output is text data. Specifically, the audio data is passed to the library, and the speech recognition algorithm is applied.

[0512] Step 3:

[0513] The server uses the transformers library as a generative AI tool, and generates an appropriate response using the converted text data as input. The input is text data, and the output is the generated response text. Specifically, the text data is input into the generative AI model, and the model generates a response.

[0514] Step 4:

[0515] The server uses the sentiment-analysis pipeline of the Transformers library as an emotion recognition method to analyze user emotions from text data. The input is text data, and the output is the emotion analysis result. Specifically, the text data is input into the emotion recognition model, and the emotion is analyzed.

[0516] Step 5:

[0517] Based on the emotion analysis results, the generative AI means generates a response according to the emotion. The input is the emotion analysis results and text data, and the output is a response text according to the emotion. Specifically, the emotion analysis results are fed back to the generative AI model, which then adjusts the response.

[0518] Step 6:

[0519] The server uses the pyttsx3 library as a speech output means to output the generated response text as speech. The input is the response text and the output is audio data. Specifically, the response text is input into a speech synthesis engine, and audio data is generated.

[0520] Step 7:

[0521] The smart glasses play the audio data and provide a response to the user. The input is the audio data, and the output is the audio heard by the user. In particular, the audio data is played through the speakers of the smart glasses.

[0522] Example 2

[0523] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0524] When elderly people operate information technology devices, it is difficult to accurately recognize voice input, especially when using dialects, and generate appropriate operations. There is also a need for a more user-friendly system that generates operations according to the user's emotions. Furthermore, a means for actually executing the generated operations is also required.

[0525] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0526] In this invention, the server includes a speech recognition means, a generative AI means, a dialect-based learning means, an emotion recognition means, a speech recognition practical use means, and an infrared signal generator control means, which enables accurate speech recognition even when an elderly person uses a dialect, generates appropriate operations according to the user's emotions, and actually executes the operations.

[0527] "Speech recognition means" refers to a device or software that converts voice input into digital data and analyzes its content.

[0528] "Generative AI means" refers to devices or software that use artificial intelligence technology to generate appropriate operations based on data obtained from speech recognition means.

[0529] A "dialect-aware training means" is a device or software that trains a speech recognition means to recognize dialects specific to a particular region or culture.

[0530] The "emotion recognition means" is a device or software for analyzing the user's emotion from voice input and recognizing that emotion.

[0531] "Means for practical use of speech recognition" refers to devices or software that apply speech recognition technology to actual systems and applications, making it practically usable.

[0532] "Infrared signal generator control means" refers to a device or software for generating an infrared signal based on instructions from the generating AI means and controlling a specific device.

[0533] This invention is a system that can accurately recognize voice input, particularly in dialects, when elderly people operate information technology devices, and generate appropriate operations. Furthermore, it can generate operations according to the user's emotions and actually execute those operations.

[0534] Hardware and software used

[0535] 1. Voice recognition means:

[0536] Your device uses the built-in microphone to capture audio.

[0537] For example, Google Speech-to-Text API is used as speech recognition software.

[0538] 2. Generative AI means:

[0539] The server uses a generative AI model (e.g., OpenAI's GPT-4) to analyze the text data obtained from the speech recognition means and generate appropriate actions.

[0540] 3. Dialect-compatible learning tools:

[0541] The server trains its speech recognition tools to recognize dialects specific to a particular region or culture, for example, using training data for the Japanese Tsugaru dialect.

[0542] 4. Emotion recognition means:

[0543] The server uses emotion recognition software (e.g., IBM Watson® Tone Analyzer) to analyze the user's emotion from the voice input and recognize the emotion.

[0544] 5. Practical application of speech recognition:

[0545] The terminal applies voice recognition technology to actual systems and applications, making it practical to use.

[0546] 6. Infrared signal generator control means:

[0547] The terminal sends the generated command to an infrared signal generator (e.g., BroadLink RM4 Pro) to control a specific device.

[0548] Specific examples

[0549] Example 1: Turning on the TV

[0550] 1. The user says "Turn on the TV" in Tsugaru dialect.

[0551] 2. The device captures audio using its built-in microphone and converts it to text using the Google Speech-to-Text API.

[0552] 3. The server uses the generative AI model to generate the "turn on TV" command.

[0553] 4. The device sends the generated command to the infrared signal generator to turn on the TV.

[0554] Example 2: Turning on the radio

[0555] 1. The user says, "Turn on the radio."

[0556] 2. The device captures audio using its built-in microphone and converts it to text using the Google Speech-to-Text API.

[0557] 3. The server uses emotion recognition software to recognize the user's emotion as "sadness."

[0558] 4. The server uses the generative AI model to generate an action to set the channel to a sad song.

[0559] 5. The device sends a command to the radio to set it to the specified channel.

[0560] Prompt Sentence Examples

[0561] "When a user says 'Turn on the TV' in Tsugaru dialect, the speech recognition means converts the speech into text, and the generation AI means generates an operation to turn on the TV. It controls an infrared signal generator to turn on the TV."

[0562] "When a user says 'turn on the radio,' the emotion engine recognizes the user's emotion, and the generative AI means generates radio controls adapted to the user's emotion. For example, it sets the radio to a channel playing sad songs."

[0563] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0564] Step 1:

[0565] Capturing voice input

[0566] The user says "Turn on the TV" in Tsugaru dialect.

[0567] Your device uses the built-in microphone to capture audio.

[0568] Input: User's voice

[0569] Output: Audio data

[0570] Specific behavior:

[0571] A user stands in front of the TV in the living room and says, "Turn on the TV."

[0572] The device captures audio through the microphone and saves it as audio data.

[0573] Step 2:

[0574] Converting audio data to text

[0575] The device uses speech recognition software (e.g., Google Speech-to-Text API) to convert the captured voice data into text data.

[0576] Input: Audio data

[0577] Output: Text data

[0578] Specific behavior:

[0579] The device sends the captured audio data to the Google Speech-to-Text API.

[0580] The device receives the text data "Turn on the TV" returned from the API.

[0581] Step 3:

[0582] Emotion recognition (if needed)

[0583] The server uses emotion recognition software (e.g., IBM Watson Tone Analyzer) to analyze the user's emotions from the text data.

[0584] Input: Text data

[0585] Output: Emotion data

[0586] Specific behavior:

[0587] The server sends the text data "Turn on the radio" to the IBM Watson Tone Analyzer.

[0588] The server receives "sadness" as the analysis result.

[0589] Step 4:

[0590] Generate Operations

[0591] The server uses a generative AI model (e.g., OpenAI's GPT-4) to analyze the text data and generate appropriate actions.

[0592] Input: Text data, emotion data (if necessary)

[0593] Output: Operation command

[0594] Specific behavior:

[0595] The server inputs the text data "Turn on the TV" into the generative AI model.

[0596] The server receives the generated command and sends it to a device that generates an infrared signal to "turn on the TV."

[0597] Step 5:

[0598] Executing an operation

[0599] The terminal transmits the generated command to the infrared signal generator to perform the operation of turning on the television.

[0600] Input: Operation command

[0601] Output: Action performed (TV turns on)

[0602] Specific behavior:

[0603] The terminal sends a "Turn on the TV" command to the infrared signal generator.

[0604] The infrared signal generator sends an infrared signal to the television, which turns on the television.

[0605] (Application example 2)

[0606] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0607] When elderly people operate IT devices, they face difficulties with voice recognition, especially in Japanese dialects. Another issue is that the user experience is not improved because appropriate operations and information cannot be provided based on the user's emotions. Furthermore, there is also the issue that the shopping experience is not optimized because product suggestions cannot be made in physical stores based on the user's emotions.

[0608] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0609] In this invention, the server includes a speech recognition means, a generative AI means, a learning means for handling all dialects in Japan, particularly a means for learning Tsugaru dialect, which is said to be the most difficult dialect in Japan, a practical speech recognition means, a means for recognizing a user's emotions using an emotion engine, a means for generating operations according to the user's emotions, a means for providing product information in a physical store, and a means for suggesting products based on the user's emotions. This enables accurate speech recognition even when an elderly person speaks in a dialect, realizes appropriate operations and information provision according to the user's emotions, and makes it possible to suggest products based on the user's emotions even in a physical store.

[0610] "Speech recognition means" is a technology for converting voice input into text data.

[0611] "Generative AI means" is an artificial intelligence technology that generates appropriate responses and operations based on input data.

[0612] A "learner" is a technique that uses a specific dataset to train a model and improve its ability to perform a specific task.

[0613] "Means for learning Tsugaru dialect" refers to a special training technique for understanding and recognizing the dialect used in the Tsugaru region of Japan.

[0614] "Means for practical application of speech recognition" refers to the technology for incorporating speech recognition technology into actual applications and systems.

[0615] An "emotion engine" is a technology for analyzing and recognizing emotions from a user's voice or text.

[0616] "Means for generating operations according to the user's emotions" refers to a technology that generates appropriate operations and responses based on the recognized user's emotions.

[0617] "Means for providing product information in a physical store" refers to technology for providing information about products to users in a physical store.

[0618] The "means for suggesting products based on the user's emotions" is a technology for suggesting appropriate products by taking into consideration the user's emotions.

[0619] This invention is a system that combines a speech recognition unit, a generative AI unit, a learning unit, an emotion engine, and a product information provision unit in a physical store. Specific embodiments for realizing this system are described below.

[0620] System Configuration

[0621] The system consists of the following major components:

[0622] 1. Speech recognition means: A technology for converting voice input into text data. Specifically, it uses the speech_recognition library.

[0623] 2. Generative AI: This is an artificial intelligence technique that generates appropriate responses and actions based on input data. Specifically, we use the GPT-3 model from the transformers library.

[0624] 3. Training: This is a technique that uses specific datasets to train a model and improve its ability to perform a specific task, especially to train it to handle all Japanese dialects.

[0625] 4. Emotion Engine: This technology analyzes and recognizes emotions from user speech and text. Specifically, it uses the emotion analysis model from the transformers library.

[0626] 5. Means of providing product information in physical stores: This is a technology for providing product information to users in physical stores.

[0627] Program processing

[0628] The server uses a speech recognition means to convert the user's voice input into text data. Next, it uses an emotion engine to recognize the user's emotion from the text data. Based on the recognized emotion, the generative AI means generates an appropriate response or operation. For example, if a user says, "Tell me about this product," and the emotion engine recognizes the user's emotion as "excited," the generative AI means will generate a response such as, "This product is the latest model, with many new features."

[0629] Hardware and software used

[0630] Hardware: Smartphone (microphone, speaker)

[0631] software:

[0632] speech_recognition library: Used to convert voice input to text.

[0633] Transformers library: Generates responses using a generative AI model (GPT-3).

[0634] The requests library: Used to communicate with external APIs (if necessary).

[0635] Specific examples

[0636] When a user says, "Tell me about this product" in a physical store, the speech recognition means converts the speech into text, and the emotion engine recognizes the user's emotion. For example, if the user is excited, the generative AI means generates a response such as, "This product is the latest model and has many new features."

[0637] Prompt Sentence Examples

[0638] When the user is sad: "Suggest a product when the user is sad: Tell me about this product"

[0639] When the user is happy: "Suggest a product when the user is happy: Tell me about this product"

[0640] In this way, the smart shopping assistant can provide relevant information according to the user's emotions and improve the shopping experience.

[0641] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0642] Step 1:

[0643] The server acquires the user's voice input using a speech recognition means, specifically by collecting voice data using the smartphone's microphone and converting the voice data into text data using the speech_recognition library.

[0644] Input: User's voice data

[0645] Output: Text data

[0646] Step 2:

[0647] The server uses an emotion engine to recognize user emotions from text data. Specifically, it uses the emotion analysis model from the transformers library to analyze the text data and identify the user's emotions.

[0648] Input: Text data

[0649] Output: User emotion (e.g., happy, sad, excited, etc.)

[0650] Step 3:

[0651] The server generates a prompt based on the user's recognized emotion. Specifically, it creates a prompt based on the emotion and inputs it into the generative AI model. For example, if the user is sad, it generates a prompt such as "Please suggest products that you would recommend when the user is sad: Tell me about this product."

[0652] Input: User sentiment, text data

[0653] Output: prompt statement

[0654] Step 4:

[0655] The server generates an appropriate response based on the prompt sentence using generative AI means, specifically, the GPT-3 model from the transformers library to generate a response to the prompt sentence.

[0656] Input: prompt statement

[0657] Output: The generated response (e.g., "This product is the latest model and has many new features")

[0658] Step 5:

[0659] The server then provides the generated response to the user, either by playing it aloud using the smartphone's speaker or by displaying it as text on the screen.

[0660] Input: Generated response

[0661] Output: Providing a response to the user (audio or text)

[0662] In this way, the server can recognize the user's voice input, analyze emotions, and generate and provide an appropriate response.

[0663] Example 3

[0664] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0665] Conventional voice recognition systems have difficulty accurately recognizing specific dialects and emotions, and are unable to accommodate the diverse dialects and emotions of users, particularly in Japan. It is also difficult to accurately recognize voice input and generate appropriate operations when elderly people operate IT devices. This has resulted in a poor user experience and limited the practicality of the system.

[0666] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0667] In this invention, the server includes a speech recognition means, a generative artificial intelligence means, and a learning means for supporting all dialects of Japan. This enables a system including a means for learning Tsugaru dialect, which is said to be the most difficult dialect in Japan, a practical speech recognition means, a means for recognizing a user's emotion using an emotion engine, and a means for generating an operation in accordance with the recognized emotion.

[0668] "Speech recognition means" is a technology that analyzes voice data and converts it into corresponding text data.

[0669] "Generative AI means" refers to AI technology that generates natural dialogue and operations based on input data.

[0670] "Learning method to support all Japanese dialects" is a technology that recognizes the various dialects used in Japan, collects the data necessary to support them, and trains a model.

[0671] "Means for learning Tsugaru dialect" refers to technology that collects the data necessary to recognize and respond to the Tsugaru dialect, a dialect used in the Tsugaru region of Japan, and trains a model.

[0672] "Means for practical application of speech recognition" refers to technology that incorporates trained speech recognition models into actual systems, enabling them to be used in practical applications.

[0673] An "emotion engine" is a technology that analyzes and identifies a user's emotions from voice data.

[0674] "Means for generating an operation according to the recognized emotion" is a technology that generates and executes an appropriate operation based on the user's emotion.

[0675] This invention is a system including a speech recognition means, a generative artificial intelligence means, a learning means for dealing with all dialects of Japan, in particular a means for learning Tsugaru dialect, a means for putting speech recognition into practical use, a means for recognizing a user's emotions using an emotion engine, and a means for generating operations in accordance with the recognized emotions.

[0676] Hardware and software used

[0677] Hardware: High-performance server (e.g., server with GPU)

[0678] Software: speech recognition software (e.g., speech recognition APIs), generative artificial intelligence models (e.g., natural language processing models)

[0679] Explanation of program processing

[0680] Acquisition and preprocessing of audio data

[0681] The server collects a large amount of Tsugaru dialect speech data. It then performs preprocessing such as noise removal and normalization on the collected speech data. Specifically, it processes the speech data using a speech processing library.

[0682] Training a speech recognition model

[0683] The server inputs the preprocessed voice data into the voice recognition API and retrieves the corresponding text data. The retrieved text data is used to train a voice recognition model. Specifically, a machine learning library is used to build and train the model.

[0684] Learning generative artificial intelligence models

[0685] The server inputs the text data obtained from the speech recognition model into the natural language processing model and trains the generative AI model. Once trained, the generative AI model is saved and provided as an API. Specifically, the model is managed using a natural language processing library.

[0686] Introducing the Emotion Engine

[0687] The server uses an emotion engine to recognize emotions from the user's voice. Specifically, it uses a voice analysis tool to analyze the tone, speed, and strength of the voice. Based on the analysis results, it identifies the user's emotion and passes that information to a generative AI model.

[0688] Generating operations according to emotions

[0689] The user makes a voice input such as "Turn on the air conditioner." The device sends the voice to the server. The server converts the voice into text using a voice recognition model. The server recognizes the user's emotions using an emotion engine. For example, if the voice is high-pitched and fast-paced, it determines that the user is angry. The server uses generative artificial intelligence means to generate an appropriate operation according to the recognized emotion. For example, if the user is angry, it generates an operation to set the air conditioner temperature low.

[0690] Examples of specific examples and prompts

[0691] Examples:

[0692] The user angrily voice-inputs "Turn on the air conditioner."

[0693] The terminal transmits the audio to the server.

[0694] The server converts the speech to text using a speech recognition model.

[0695] The server uses an emotion engine to recognize that the user's emotion is "anger."

[0696] The server uses a generative artificial intelligence means to generate an operation to set the temperature of the air conditioner low.

[0697] Example prompt sentence:

[0698] When a user makes a voice input saying "Turn on the air conditioner" with an angry emotion, the emotion engine recognizes the user's emotion from the voice, and the generative artificial intelligence means generates an operation to set the air conditioner temperature low.

[0699] In this way, the system provides an appropriate operation according to the user's emotion. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0700] Step 1:

[0701] The server collects a large amount of Tsugaru dialect speech data. It uses the collected speech data as input and performs preprocessing such as noise removal and normalization. Specifically, it processes the speech data using a speech processing library to remove noise and normalize the volume. The preprocessed speech data is then output.

[0702] Step 2:

[0703] The server inputs the preprocessed voice data into a voice recognition API to obtain the corresponding text data. The voice recognition API analyzes the voice data and converts it into text data. This outputs the text data corresponding to the voice data.

[0704] Step 3:

[0705] The server uses the acquired text data to train a speech recognition model. Specifically, it builds a model using a machine learning library and trains it using the text data as input. The trained speech recognition model is then output.

[0706] Step 4:

[0707] The server inputs the text data obtained from the speech recognition model into the natural language processing model and trains the generative AI model. Specifically, it manages the model using a natural language processing library and trains it based on the text data. This outputs the trained generative AI model.

[0708] Step 5:

[0709] The server uses an emotion engine to recognize emotions from the user's voice. Specifically, it uses a voice analysis tool to analyze the tone, speed, and strength of the voice, and analyzes the voice data as input. This results in the output of the user's emotion.

[0710] Step 6:

[0711] The user makes a voice input such as "Turn on the air conditioner." The device sends the voice to the server. The server converts the voice into text using a speech recognition model. This outputs text data corresponding to the voice data.

[0712] Step 7:

[0713] The server uses an emotion engine to recognize the user's emotion. For example, if the voice tone is high and the speed is fast, it determines that the user is angry. Based on this, the server outputs the user's emotion.

[0714] Step 8:

[0715] The server uses a generative artificial intelligence means to generate an appropriate operation according to the recognized emotion. For example, if the user is angry, the server generates an operation to set the air conditioner temperature low. This results in an appropriate operation being output.

[0716] Step 9:

[0717] The terminal executes the operation sent from the server. Specifically, it executes the operation to set the air conditioner temperature low, providing a comfortable environment for the user. This completes the operation requested by the user.

[0718] (Application example 3)

[0719] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0720] Conventional voice recognition systems are specialized in standard Japanese, making it difficult to handle Japan's diverse dialects. Furthermore, they lacked the ability to recognize user emotions and provide appropriate operations, limiting the user experience. In particular, in autonomous vehicles, it was difficult to provide appropriate navigation instructions when the user spoke in a dialect or was emotional.

[0721] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes a voice recognition means, a generative AI means, and a learning means for handling all Japanese dialects. This makes it possible to accurately recognize speech even when the user speaks in a dialect, and further to understand the user's emotions and provide appropriate navigation instructions.

[0722] "Speech recognition means" is a technology for converting speech into text data.

[0723] "Generative AI means" is an artificial intelligence technology that generates appropriate operations and responses based on input data.

[0724] "Learning method to support all Japanese dialects" is a technology that learns the data necessary to recognize the various dialects used in Japan.

[0725] The "means of learning Tsugaru dialect" is a specialized learning technique for recognizing Tsugaru dialect, a dialect used in the Tsugaru region of Japan.

[0726] "Means for practical application of speech recognition" refers to the technology for applying speech recognition technology to actual systems and applications.

[0727] "Emotion recognition means" is a technology that analyzes and recognizes emotions from a user's voice or text.

[0728] "Means for generating navigation instructions in an autonomous vehicle" refers to technology that generates a route and instructions to a destination in an autonomous vehicle.

[0729] The "means for providing navigation instructions according to the user's emotions" is a technology for providing optimal navigation instructions taking into account the user's emotional state.

[0730] The following system configuration will be described as an embodiment of the present invention.

[0731] System Configuration

[0732] Hardware

[0733] Microphone: Used to capture the user's voice input.

[0734] Speaker: Used to provide audio output from the system.

[0735] Self-driving vehicle: A vehicle equipped with a navigation system.

[0736] software

[0737] Speech Recognition Library (speech_recognition): Used to convert the user's speech into text.

[0738] Emotion recognition library (transformers pipeline): Used to analyze emotions from user text data.

[0739] Speech synthesis library (pyttsx3): Used to convert text to speech.

[0740] Processing flow

[0741] 1. Acquiring voice input

[0742] When a user speaks into the microphone, a speech recognition library converts the speech into text data.

[0743] For example, if a user asks in Tsugaru dialect, "Where is the nearest convenience store from here?", the speech is converted into text data.

[0744] 2. Emotional Recognition

[0745] The converted text data is sent to an emotion recognition library to analyze the user's emotions.

[0746] For example, if a user is anxious, the emotion recognition library will recognize the emotion as "anxiety."

[0747] 3. Generate navigation instructions

[0748] Based on the results of emotion recognition, generative AI generates appropriate navigation instructions.

[0749] For example, if the user is recognized as being impatient, instructions are generated to quickly guide the user to the nearest convenience store.

[0750] 4. Audio Output

[0751] The generated navigation instructions are converted into voice using a speech synthesis library and output from a speaker.

[0752] For example, a voice saying "Please stay calm. The nearest convenience store is here" is output.

[0753] Specific examples

[0754] If a user asks in Tsugaru dialect, "Where is the nearest convenience store from here?", the system will recognize the dialect and, if it determines that the user is in a hurry, will quickly guide them to the nearest convenience store.

[0755] Prompt Sentence Examples

[0756] User: "Where's the nearest convenience store from here?"

[0757] System: "Please stay calm. The nearest convenience store is here."

[0758] In this way, the dialect-aware emotion-aware navigation system can understand the dialect and emotions of the user and provide appropriate navigation instructions.

[0759] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0760] Step 1:

[0761] The user speaks into the microphone. The speech recognition library (speech_recognition) picks up the speech and converts it into text data. The input is the user's voice, and the output is text data. For example, if the user says "Where is the nearest convenience store from here?" in Tsugaru dialect, the speech is converted into text data: "Where is the nearest convenience store from here?"

[0762] Step 2:

[0763] The server sends the converted text data to the emotion recognition library (transformers pipeline). The emotion recognition library analyzes the user's emotions from the text data. The input is text data and the output is emotion data. Specifically, the text data "Where is the nearest convenience store from here?" is converted into the emotion data "impatience."

[0764] Step 3:

[0765] The server receives the emotion recognition results, and the generative AI generates appropriate navigation instructions. The input is emotion data and text data, and the output is navigation instructions. Specifically, based on the emotion data of "impatience" and the text data of "Where is the nearest convenience store from here?", the navigation instruction "Please stay calm. The nearest convenience store is here" is generated.

[0766] Step 4:

[0767] The server sends the generated navigation instructions to a speech synthesis library (pyttsx3). The speech synthesis library converts the navigation instructions into speech and outputs it from the speaker. The input is the navigation instructions, and the output is speech data. Specifically, the navigation instruction "Please stay calm. The nearest convenience store is here" is converted into speech data and output from the speaker.

[0768] In this way, the dialect-aware emotion-aware navigation system can understand the dialect and emotions of the user and provide appropriate navigation instructions.

[0769] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0770] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generative AIs. Data generation models 58

[0771] , obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in the data format of voice data, text data, etc. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0772] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0773] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0774] [Second embodiment]

[0775] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0776] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0777] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0778] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0779] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0780] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0781] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0782] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0783] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0784] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0785] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0786] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0787] "Example 1"

[0788] The present invention is a system that combines a speech recognition means and a generative AI means. The speech recognition means recognizes the voice input when an elderly person operates an IT device. The generative AI means generates operations based on the recognized voice input. The system also has a learning means to support all Japanese dialects. Specifically, the practical application of speech recognition has been realized by training the system in Tsugaru dialect, which is said to be the most difficult dialect in Japan.

[0789] "Example 2"

[0790] In a specific embodiment, for example, if an elderly person says "Turn on the TV" in Tsugaru dialect, the speech recognition means recognizes this speech input, and the generative AI means generates an operation to turn on the TV. This operation is realized, for example, by controlling a device that generates an infrared signal.

[0791] "Example 3"

[0792] This system also has a learning method to support all dialects in Japan. Specifically, the speech recognition and generative AI methods are trained using a large amount of Tsugaru dialect speech data. This makes it possible to use the same method to support not only Tsugaru dialect but also other dialects.

[0793] The processing flow of each embodiment will be described below.

[0794] "Example 1"

[0795] Step 1: The elderly person performs voice input to operate an IT device. This voice input includes instructions such as "Turn on the TV."

[0796] Step 2: The voice recognition means recognizes the voice input of the elderly person. This voice recognition is performed using a training means to support all Japanese dialects.

[0797] Step 3: The generative AI means generates an action based on the recognized voice input, which may include a specific action such as turning on the TV.

[0798] "Example 2"

[0799] Step 1: An elderly person speaks in Tsugaru dialect and says, "Turn on the TV."

[0800] Step 2: The speech recognition means recognizes this speech input. This speech recognition is highly accurate as a result of having been trained in Tsugaru dialect.

[0801] Step 3: The generative AI means generates the operation to turn on the TV. This operation is realized by controlling a device that generates an infrared signal.

[0802] "Example 3"

[0803] Step 1: Train the speech recognition and generative AI methods using a large amount of Tsugaru dialect speech data.

[0804] Step 2: Using the trained voice recognition means and generative AI means, recognize the elderly person's voice input and generate operations.

[0805] Step 3: Execute the generated operation, which may include a specific operation such as turning on the television.

[0806] Example 1

[0807] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0808] When elderly people operate IT devices, there is a demand for improved ease of operation through voice input. However, a voice recognition system that can handle Japan's diverse dialects, especially difficult dialects such as Tsugaru dialect, has not yet been put into practical use. Therefore, it is necessary to develop a voice recognition system that allows elderly people to speak naturally in their own dialect.

[0809] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a voice recognition means, a generative artificial intelligence means, and a learning means for handling all Japanese dialects. This allows elderly people to speak naturally in their own dialect and operate information technology devices using voice input.

[0810] "Speech recognition means" is a technology for converting voice data into text data.

[0811] "Generative artificial intelligence means" refers to an artificial intelligence technology for generating appropriate operational instructions based on input data.

[0812] "Learning method to support all Japanese dialects" is a technology that pre-trains a voice recognition system to recognize the various dialects used in Japan.

[0813] "Means for learning Tsugaru dialect" refers to a technology that specially trains a voice recognition system to recognize Tsugaru dialect, which is said to be the most difficult dialect in Japan.

[0814] "Voice data preprocessing means" refers to technology for improving the accuracy of voice recognition by removing noise from the voice data and normalizing the volume.

[0815] "Means for practical application of speech recognition" refers to the technology for applying speech recognition technology to actual systems and applications, making it practically usable.

[0816] "Voice input when an elderly person operates an information technology device by themselves" refers to voice commands that an elderly person issues to operate an information technology device by themselves.

[0817] This invention is a system for improving the convenience of elderly people when operating information technology devices by voice input. This system includes a speech recognition means, a generative artificial intelligence means, a learning means for supporting all Japanese dialects, a means for training the Tsugaru dialect, a speech data preprocessing means, and a means for implementing speech recognition.

[0818] System configuration

[0819] Voice recognition means

[0820] The server converts the audio data into text using speech recognition software, such as the Google Cloud Speech-to-Text API, which is pre-trained to accommodate Japan's diverse dialects.

[0821] Audio data preprocessing method

[0822] The server uses audio processing libraries such as Librosa to denoise and normalize the volume of the audio data, making it clearer and easier to understand.

[0823] Generative AI methods

[0824] The server generates operation instructions based on the text data using a generative artificial intelligence model (e.g., OpenAI's GPT-4). This generative artificial intelligence model has been trained in advance on a variety of operation scenarios to understand the user's intentions and generate appropriate operation instructions.

[0825] Executing an operation

[0826] The device executes the operation instructions received from the server, for example, opening an email application and sending an email with the specified content.

[0827] Specific examples

[0828] Example 1: Sending email by voice input

[0829] The user speaks "send email."

[0830] 1. The server uses speech recognition software to convert speech to text.

[0831] 2. The server uses Librosa to denoise and normalize the volume of the audio data.

[0832] 3. The server uses a generative artificial intelligence model to generate the operation instruction "send an email."

[0833] 4. The device will open the email app and send an email with the specified content.

[0834] Example 2: Getting weather information by voice input

[0835] The user speaks, "What's the weather like today?"

[0836] 1. The server uses speech recognition software to convert speech to text.

[0837] 2. The server uses Librosa to denoise and normalize the volume of the audio data.

[0838] 3. The server uses a generative artificial intelligence model to generate an operational instruction to "obtain weather information."

[0839] 4. The device will open the weather app and display the current weather information.

[0840] Prompt Sentence Examples

[0841] "Design a system that recognizes voice input spoken by an elderly person in Tsugaru dialect and generates appropriate actions. Specifically, explain how you combine speech recognition software with a generative artificial intelligence model to generate actions based on the voice input."

[0842] In this way, the system can improve the convenience of elderly people when operating information technology devices using voice input.

[0843] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0844] Step 1:

[0845] The user speaks to the terminal, for example, "send an email." This voice input becomes the initial input for the system.

[0846] Step 2:

[0847] The device sends the captured audio data to the server, which uses an audio processing library such as Librosa to remove noise and normalize the volume of the audio data, making the audio clearer and easier to understand. The input is raw audio data, and the output is preprocessed audio data.

[0848] Step 3:

[0849] The server sends the preprocessed speech data to speech recognition software (e.g., Google Cloud Speech-to-Text API) to convert the speech to text. The input is the preprocessed speech data, and the output is text data. For example, the generated text is "send me an email."

[0850] Step 4:

[0851] The server uses a generative artificial intelligence model (e.g., OpenAI's GPT-4) to generate operational instructions based on the text data. The input is text data obtained by speech recognition, and the output is a specific operational instruction. For example, the generated operational instruction is "send an email."

[0852] Step 5:

[0853] The device executes the operation instructions received from the server. For example, it opens an email application and sends an email with the specified content. The input is the operation instruction from the server, and the output is the actual operation result.

[0854] In this way, the system can improve the convenience of seniors when operating information technology devices using voice input.

[0855] (Application example 1)

[0856] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0857] When elderly people search for products or ask questions in physical stores, they face challenges such as difficulty in operating IT devices and difficulty in obtaining appropriate information. Furthermore, there is a lack of voice recognition systems that can handle all dialects in Japan, and there is a particular need to support difficult dialects.

[0858] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0859] In this invention, the server includes a voice recognition unit that recognizes voice inputs made by an elderly person when operating a smartphone, a unit that identifies the dialect spoken by the elderly person based on the recognized voice input, a prompt sentence that generates operation instructions for the smartphone based on the recognized voice input and the identified dialect, a unit that generates operation instructions for the smartphone using a generative AI, and a unit that operates the smartphone in accordance with the generated operation instructions for the smartphone. The voice recognition unit further recognizes voice inputs made by the elderly person when asking a question about a product in a physical store, and the server further includes a prompt sentence that generates an answer to the question based on the recognized voice input and the identified dialect, and a unit that generates an answer to the question using the generative AI. This enables a system that allows elderly people to easily operate a physical store using voice input when searching for products or asking questions.

[0860] "Speech recognition means" is a technology for converting speech into text.

[0861] "Generative AI means" is an artificial intelligence technology that generates appropriate responses and operations based on input data.

[0862] "Learning methods to support all dialects of Japan" refers to technology that learns the data necessary to understand and recognize dialects from all over Japan.

[0863] "Means for learning Tsugaru dialect" refers to technology that learns the data necessary to understand and recognize Tsugaru dialect, which is said to be the most difficult dialect in Japan.

[0864] "Means for practical application of speech recognition" refers to the technology for applying speech recognition technology to actual applications and systems.

[0865] "A means for recognizing voice input when elderly people search for products or ask questions in physical stores, and generating appropriate operations or responses" is a technology that recognizes the voices of elderly people when they search for products or ask questions using voice input in physical stores, and generates appropriate operations or responses.

[0866] The "application means installed on the smartphone" is software that is installed on the smartphone and provides a specific function.

[0867] The system for implementing this invention includes a speech recognition means, a generative AI means, a means for identifying all Japanese dialects, a means for learning Tsugaru dialect, a means for putting speech recognition into practical use, a means for recognizing voice input when an elderly person searches for a product or asks a question in a physical store and generating appropriate operations or responses, and an application means to be installed on a smartphone.

[0868] System Program

[0869] This system operates using the following hardware and software:

[0870] Hardware: Smartphone with microphone

[0871] Software: Python, SpeechRecognition library, Transformers library (Hugging Face's GPT-3 (registered trademark) model)

[0872] Processing flow

[0873] Voice Recognition

[0874] When a user speaks into the smartphone's microphone, the speech recognizer picks up the speech and converts it into text, a process that uses the SpeechRecognition library.

[0875] Generative AI

[0876] The speech data converted to text by the speech recognition tool is sent to the generative AI, which uses the Transformers library to generate appropriate actions and responses using the GPT-3 model.

[0877] output

[0878] The generated operations and answers are displayed on the smartphone screen, allowing seniors to easily use voice input when searching for products or asking questions in physical stores.

[0879] Specific example (smartphone operation)

[0880] When an elderly person says "send email" to their smartphone, the application recognizes the voice, and the generative AI identifies that it is in the Tsugaru dialect. It then generates instructions to launch the email sending app, which then launches the app according to those instructions. The generative AI also generates a response such as "Please enter the message you want to send," and the application displays the response.

[0881] Prompt Sentence Examples

[0882] "An elderly person is saying 'Send email' by voice. This is in Tsugaru dialect. Please generate instructions for smartphone operation."

[0883] In this way, a system can be created that helps elderly people operate smartphones more easily.

[0884] Specific example (physical store)

[0885] When an elderly person speaks to their smartphone in a physical store and asks, "Where is product A?", the application recognizes the speech, and the generative AI identifies that it is in Tsugaru dialect. It then generates and displays a specific answer, such as, "Product A is on the left side of aisle 3."

[0886] Prompt Sentence Examples

[0887] "An elderly person is typing, 'Where is product A?' This is in Tsugaru dialect. Please generate an answer to this question."

[0888] Generative AI answer: The item is on the left side of aisle 3.

[0889] In this way, a system can be created that helps seniors to shop more easily in brick-and-mortar stores.

[0890] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0891] Step 1:

[0892] The user speaks into the smartphone's microphone. The input is the user's voice, and the output is voice data. A specific example of this behavior is when the user says, "Where is this item?"

[0893] Step 2:

[0894] The device uses a speech recognition means to convert the voice data into text. The input is voice data and the output is text data. Specifically, the SpeechRecognition library is used to convert the voice into the text "Where is product A?"

[0895] Step 3:

[0896] The terminal sends text data to the generative AI. The input is text data, and the output is the input data to the generative AI. The specific operation is to send the converted text "Where is product A?" to the generative AI.

[0897] Step 4:

[0898] The server uses generative AI to generate appropriate operation instructions and answers based on the text data. The input is text data, and the output is the generated operation instructions and answers. Specifically, the GPT-3 model uses the Transformers library to generate the answer "Product A is located on the left side of aisle 3."

[0899] Step 5:

[0900] The server sends the generated operation instructions and answers to the terminal. The input is the generated operation instructions and answers, and the output is the data sent to the terminal. A specific operation is to send the generated answer "Product A is on the left side of aisle 3" to the terminal.

[0901] Step 6:

[0902] The device operates according to the generated operation instructions and displays the generated answer to the user. The input is the transmitted data, and the output is the data displayed to the user. A specific operation is to display the answer "Product A is located on the left side of aisle 3" on the smartphone screen.

[0903] It is also possible to use an emotion engine, which will be described later, to recognize the emotional state of the elderly person and generate smartphone operation instructions and answers to questions by further taking the elderly person's emotional state into consideration. In this case, the server further includes means for recognizing the elderly person's emotional state, and the generating means generates the smartphone operation instructions using a prompt sentence that generates the smartphone operation instructions based on the recognized voice input, the identified dialect, and the recognized emotional state of the elderly person, and the generative AI.

[0904] Example 2

[0905] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0906] When elderly people operate IT devices, they face the challenge of accurately recognizing speech input, especially when it is in a dialect, and generating appropriate operations. Furthermore, due to a lack of speech recognition technology capable of handling certain dialects, such as the Tsugaru dialect, there is a need for a system that can accurately handle speech input in these dialects.

[0907] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0908] In this invention, the server includes a speech recognition unit, a generative artificial intelligence unit, a dialect adaptation learning unit, a specific dialect learning unit, a speech recognition implementation unit, a unit for converting speech input into text data, a unit for transmitting the text data to the generative artificial intelligence model, a unit for receiving the generated operation, and a unit for controlling the infrared signal generator. This makes it possible to accurately recognize speech and generate appropriate operations even when an elderly person operates information technology equipment using a dialect.

[0909] "Speech recognition means" refers to a device or software that analyzes voice input and converts it into text data.

[0910] "Generative artificial intelligence means" refers to an artificial intelligence model that generates appropriate operations and responses based on input data.

[0911] The "dialect-compatible learning means" is a means for training a speech recognition model to accommodate dialects of different regions.

[0912] A "specific dialect learning means" is a means for training a speech recognition model specializing in a specific dialect, for example, a difficult dialect such as Tsugaru dialect.

[0913] "Means for practical application of speech recognition" refers to the means for incorporating speech recognition technology into actual systems and devices and making it usable.

[0914] The "means for converting speech input into text data" refers to a means for converting speech input into text format using a speech recognition means.

[0915] The "means for transmitting text data to a generative artificial intelligence model" refers to a means for transmitting converted text data to a generative artificial intelligence model.

[0916] The "means for receiving a generated operation" is a means for receiving an operation generated from a generative artificial intelligence model.

[0917] "Means for controlling an infrared signal generating device" refers to means for controlling a device that generates an infrared signal and for performing a specific operation.

[0918] The present invention provides a system for accurately recognizing speech and generating appropriate operations when an elderly person operates information technology equipment using a dialect. Specific embodiments of this system are described below.

[0919] First, the user says "Turn on the TV" in Tsugaru dialect. The microphone on the device captures this voice. Speech recognition software (for example, a speech recognition API) is used as the speech recognition method. This software analyzes the captured voice and converts it into text data.

[0920] The device then sends the converted text data to a generative artificial intelligence model (e.g., a generative AI model). This generative AI model generates appropriate actions based on the input text data. Specifically, the following prompt sentence is generated:

[0921] Example prompt sentence:

[0922] The user says "Turn on the TV" in Tsugaru dialect. Generate an operation that generates an infrared signal to turn on the TV.

[0923] Based on this prompt, the server uses a generative artificial intelligence model to generate a specific command to turn on the TV, and the generated command is sent to the device.

[0924] Based on the received operation, the terminal controls an infrared signal generator (e.g., an infrared signal generator), which emits an infrared signal toward the television, which receives the signal and turns on.

[0925] For example, if a user says "Turn on the TV" in Tsugaru dialect, the device's microphone will capture this speech and convert it into text using a speech recognition API. This text data will be sent to the generative AI model, which will generate the following prompt:

[0926] Example prompt sentence:

[0927] The user says "Turn on the TV" in Tsugaru dialect. Generate an operation that generates an infrared signal to turn on the TV.

[0928] Based on this prompt, the server uses a generative AI model to generate specific commands to turn on the TV and sends them to the device, which then controls an infrared signal generator, which turns on the TV when it receives the infrared signal.

[0929] In this way, even when a user operates information technology equipment using a dialect, it is possible to accurately recognize speech and generate appropriate operations.

[0930] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0931] Step 1:

[0932] The user provides voice input.

[0933] A user says "Turn on the TV" in Tsugaru dialect. The input is the user's voice, and the output is the voice data captured by the device's microphone.

[0934] Step 2:

[0935] The device recognizes the voice.

[0936] The microphone on the device captures the user's voice. A speech recognition API is used as the speech recognition method. The input is the captured voice data, and the output is analyzed text data.

[0937] Step 3:

[0938] The device converts the recognized speech into text.

[0939] The device uses a speech recognition API to convert the captured voice into text data. The input is voice data and the output is text data.

[0940] Step 4:

[0941] The device sends the text to a generative AI model.

[0942] The terminal sends the converted text data to the generative AI model. The input is the text data, and the output is the data sent to the generative AI model.

[0943] Step 5:

[0944] The server generates operations using a generative AI model.

[0945] The server uses a generative AI model to generate appropriate actions based on the received text data. Specifically, the following prompt sentence is generated:

[0946] Example prompt sentence:

[0947] The user says "Turn on the TV" in Tsugaru dialect. Generate an operation that generates an infrared signal to turn on the TV.

[0948] The input is text data and the output is the generated operational data.

[0949] Step 6:

[0950] The server sends the generated operation to the terminal.

[0951] The server sends the generated operation to the terminal. The input is the generated operation data, and the output is the operation data sent to the terminal.

[0952] Step 7:

[0953] The terminal controls the infrared signal generator.

[0954] The terminal controls the infrared signal generator based on the received operation. The input is the operation data, and the output is the infrared signal.

[0955] Step 8:

[0956] The TV operates by receiving infrared signals.

[0957] The infrared signal generator emits an infrared signal toward the television, which receives the signal and turns on. The input is the infrared signal, and the output is the operation of the television.

[0958] (Application example 2)

[0959] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0960] In modern brick-and-mortar stores, customers who speak dialects have difficulty receiving product information through voice. Particularly for elderly customers and customers in areas where dialects are spoken, speech recognition systems that only understand standard Japanese cannot provide sufficient service. For this reason, there is a demand for a system that combines dialect-compatible speech recognition with generative AI.

[0961] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0962] In this invention, the server includes a speech recognition unit, a generative AI unit, and a learning unit for supporting various dialects, which allows customers who speak different dialects to receive product information by voice.

[0963] "Speech recognition means" is a technology for converting speech into text.

[0964] "Generative AI means" is an artificial intelligence technology that generates appropriate responses and operations based on input data.

[0965] A "dialect-specific learning method" is a learning algorithm for understanding and recognizing dialects from different regions.

[0966] A "means of learning difficult dialects" is a special learning process for recognizing dialects that are particularly difficult to understand.

[0967] "Means for practical application of speech recognition" refers to means for making speech recognition technology usable in actual applications.

[0968] The "guide means for providing product information within the store" is a system for providing product locations and information to customers within the store.

[0969] "Application means installed on a smart device" refers to software installed on a device such as a smartphone or tablet.

[0970] In order to implement this invention, it is necessary to construct a system that includes a speech recognition means, a generative AI means, a learning means for handling all dialects, a means for learning difficult dialects, a means for putting speech recognition into practical use, a guidance means for providing product information in a store, and an application means to be installed on a smart device.

[0971] System configuration

[0972] 1. Voice recognition means:

[0973] The speech recognition means is used to convert the user's speech into text. Specifically, the smartphone's microphone is used to capture the speech, and the SpeechRecognition library is used to convert the speech into text.

[0974] 2. Generative AI means:

[0975] The generative AI mechanism generates appropriate responses and actions based on the text captured by the speech recognition mechanism, specifically leveraging the GPT-3 model using the Hugging Face transformers library.

[0976] 3. Learning tools for all dialects:

[0977] The means includes a learning algorithm for understanding and recognizing different regional dialects, and a special learning process is implemented to provide a means for learning particularly difficult dialects.

[0978] 4. Practical application of speech recognition:

[0979] It is a means to make speech recognition technology usable in actual applications, including an interface that allows generative AI means to generate appropriate responses based on speech recognition results and provide them to users.

[0980] 5. Guidance methods for in-store product guidance:

[0981] This is a system that provides customers with product locations and information in stores. Specifically, it displays a map showing product locations on the smartphone screen and provides audio guidance.

[0982] 6. Application means installed on smart devices:

[0983] This software is installed on devices such as smartphones and tablets. This application integrates speech recognition and generative AI, allowing users to receive product information by voice.

[0984] Processing flow

[0985] 1. Acquiring voice input:

[0986] When a user speaks into the smartphone, the voice recognition unit picks up the voice. For example, the user might say, "Look for this product."

[0987] 2. Speech Recognition:

[0988] The speech recognition means converts the speech captured into text. The SpeechRecognition library is used to convert speech into text.

[0989] 3. Response generation by generative AI:

[0990] A generative AI method uses the converted text as a prompt to generate an appropriate response, specifically using the Hugging Face GPT-3 model to generate the response.

[0991] 4. Providing a response:

[0992] The generated response is displayed on the smartphone screen or given as a voice prompt, for example, "This product is on the shelf at the back right of the store."

[0993] Specific examples

[0994] User says: "Find this product"

[0995] Generated response: "This item is on the back right shelf of the store."

[0996] Prompt Sentence Examples

[0997] "If someone says 'Look for this product' in Tsugaru dialect, how can I guide them to the product in the store?"

[0998] In this way, combining dialect-aware speech recognition with generative AI can significantly improve customer service in brick-and-mortar stores.

[0999] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1000] Step 1:

[1001] The user speaks into the smartphone.

[1002] Input: User speech (e.g., "Look for this product")

[1003] How it works: The smartphone microphone picks up the audio.

[1004] Output: Captured audio data

[1005] Step 2:

[1006] The terminal uses a voice recognition means to convert the captured voice data into text.

[1007] Input: Audio data

[1008] What it does: Converts speech to text using the SpeechRecognition library.

[1009] Output: The converted text (e.g. "Look for this product")

[1010] Step 3:

[1011] The device uses generative AI means to generate a response based on the converted text.

[1012] Input: Translated text

[1013] How it works: It uses Hugging Face's GPT-3 model to generate prompt sentences and generate appropriate responses.

[1014] Output: Generated response text (e.g., "This item is on the back right shelf in the store.")

[1015] Step 4:

[1016] The terminal provides the generated response to the user.

[1017] Input: Generated response text

[1018] What it does: Displays a text response on the phone screen or plays an audio response.

[1019] Output: The user confirms the response.

[1020] Step 5:

[1021] The user follows the instructions to find the product.

[1022] Input: Response text or voice prompts

[1023] Action: The user navigates the store and follows the instructions to find a product.

[1024] Output: The user finds the product they are looking for.

[1025] In this way, specific actions and data processing and calculations are performed at each step, enabling customer service in physical stores to be realized using dialect-compatible voice recognition and generative AI.

[1026] Example 3

[1027] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1028] It is difficult to realize a speech recognition system that can handle all dialects in Japan, especially difficult dialects. In particular, it is difficult for conventional speech recognition technology to handle Tsugaru dialect, which is said to be the most difficult dialect in Japan. Another challenge is accurately recognizing voice input and generating appropriate responses when elderly people operate information technology devices themselves.

[1029] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1030] In this invention, the server includes a speech recognition unit, a generative artificial intelligence unit, and a learning unit for supporting all Japanese dialects. This enables a system including a means for a user to input speech into a terminal, a means for the terminal to transmit speech data to the server, a means for the server to convert the speech data into text data using speech recognition software, a means for the server to input the text data into a generative artificial intelligence model, a means for the generative artificial intelligence model to generate an appropriate response, a means for the server to transmit the generated response to the terminal, and a means for the terminal to display the response to the user. This enables speech recognition and response generation compatible with all Japanese dialects, particularly difficult dialects such as Tsugaru dialect. It also enables accurate recognition of speech input and generation of appropriate responses when elderly people operate information technology devices themselves.

[1031] "Speech recognition means" is a technology for converting voice data into text data.

[1032] "Generative AI means" refers to an AI technology for generating appropriate responses based on input text data.

[1033] A "learning tool" is a method for training a system with specific data to improve its ability to perform a specific task.

[1034] The "means for learning Tsugaru dialect" is a method for training speech recognition means and generative artificial intelligence means using a large amount of Tsugaru dialect speech data.

[1035] "Means for practical application of speech recognition" refers to methods for applying speech recognition technology to actual applications and systems.

[1036] "Means for users to input voice into a terminal" refers to a method by which a user inputs voice into a device such as a smartphone or PC.

[1037] The "means for the terminal to transmit audio data to the server" refers to a method for transmitting audio data captured by the terminal to the server via the Internet.

[1038] "Means for the server to convert voice data into text data using voice recognition software" refers to a method in which the server converts received voice data into text data using voice recognition software.

[1039] "Means for the server to input text data into the generative artificial intelligence model" refers to a method in which the server inputs converted text data into the generative artificial intelligence model.

[1040] The "means by which a generative artificial intelligence model generates an appropriate response" is a method by which a generative artificial intelligence model generates an appropriate response based on input text data.

[1041] "Means for the server to transmit the generated response to the terminal" refers to a method by which the server transmits the response returned from the generative artificial intelligence model to the terminal.

[1042] The "means by which the terminal displays the response to the user" refers to the method by which the terminal displays the response received from the server to the user.

[1043] This invention provides a speech recognition system that supports all Japanese dialects. In particular, it has a learning means for supporting Tsugaru dialect, which is said to be the most difficult dialect in Japan. This system includes a speech recognition means, a generative artificial intelligence means, a learning means, a speech recognition implementation means, a means for a user to input speech into a terminal, a means by the terminal transmitting speech data to a server, a means by the server converting the speech data into text data using speech recognition software, a means by the server inputting the text data into a generative artificial intelligence model, a means by the generative artificial intelligence model generating an appropriate response, a means by the server transmitting the generated response to a terminal, and a means by the terminal displaying the response to a user.

[1044] The server first collects a large amount of Tsugaru dialect speech data. This speech data is converted into text data using speech recognition software (e.g., Google Speech-to-Text API). The converted text data is then used to train a generative artificial intelligence model (e.g., OpenAI's GPT-4).

[1045] The device transmits the voice data entered by the user to the server in real time. The server converts the received voice data into text data using speech recognition software, and inputs the text data into a generative AI model. The generative AI model generates an appropriate response based on the input text data and returns the response as text data to the device.

[1046] The user receives the generated response through the terminal and, if necessary, re-enters the voice input. By repeating this process, the user can have a natural conversation that corresponds to the dialect.

[1047] As a concrete example, consider the case where a user says "How is it today?" in Tsugaru dialect. The device sends this voice data to the server, which uses speech recognition software to convert it into text data such as "How is it today?". This text data is then input into a generative artificial intelligence model, which generates an appropriate response. For example, the response generated might be "It's a nice day today." This response is then sent back to the device as text data and displayed to the user.

[1048] An example of a prompt might be:

[1049] User input: "How's your day going?"

[1050] Prompt to generative AI model: "The user says, 'How is today?' in Tsugaru dialect. Please generate an appropriate response."

[1051] In this way, the system provides a learning means for dealing with all dialects in Japan, enabling the user to have natural conversations. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1052] Step 1:

[1053] The user inputs voice into the terminal.

[1054] The user speaks into a device such as a smartphone or PC. For example, they say, "How is it today?" in Tsugaru dialect. This speech is captured by the device's microphone. The input is the user's voice data, and the output is the voice data captured by the device.

[1055] Step 2:

[1056] The terminal transmits the voice data to the server.

[1057] The device compresses the captured audio data and sends it to a server via the Internet. Specifically, the device compresses the audio data into MP3 format and sends it to the server using the HTTPS protocol. The input is the captured audio data, and the output is the audio data sent to the server.

[1058] Step 3:

[1059] The server converts the voice data into text data using voice recognition software.

[1060] The server sends the received voice data to speech recognition software (for example, Google Speech-to-Text API) and converts it into text data. For example, voice data such as "How is it today?" is converted into text data such as "How is it today?". The input is the voice data sent to the server, and the output is the converted text data.

[1061] Step 4:

[1062] The server inputs the text data into a generative artificial intelligence model.

[1063] The server sends the converted text data to a generative AI model (e.g., OpenAI's GPT-4) as an API request. For example, send the text data "How is it today?" to the generative AI model. The input is the converted text data, and the output is an API request to the generative AI model.

[1064] Step 5:

[1065] A generative artificial intelligence model generates appropriate responses.

[1066] A generative AI model generates an appropriate response based on input text data. For example, a response such as "It's nice weather today" may be generated. This response is returned to the server as text data. The input is an API request to the generative AI model, and the output is the generated response text data.

[1067] Step 6:

[1068] The server sends the generated response to the terminal.

[1069] The server receives the response text data returned from the generative AI model and sends it to the device as an API response. For example, the text data "It's nice weather today" is sent to the device. The input is the generated response text data, and the output is the response text data sent to the device.

[1070] Step 7:

[1071] The terminal displays the response to the user.

[1072] The terminal displays the response text data received from the server to the user. For example, the text "It's a nice day today" is displayed on the screen of a smartphone. The input is the response text data sent to the terminal, and the output is the response text displayed to the user.

[1073] (Application example 3)

[1074] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1075] Conventional voice recognition systems are specialized in standard Japanese, making it difficult to handle the various dialects of Japan. Furthermore, they are unable to handle Tsugaru dialect, which is said to be the most difficult dialect in Japan, making them inconvenient for users who speak dialects. Furthermore, food delivery services have the problem of making it difficult to order in dialect, forcing users to use standard Japanese.

[1076] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes a voice recognition means, a generative AI means, a learning means for supporting all Japanese dialects, a means for training Tsugaru dialect, which is said to be the most difficult dialect in Japan, a practical voice recognition means, a means for recognizing and analyzing voice input in a dialect and generating order details, a means for transmitting the generated order details via a communication means, a means for allowing the user to confirm the order details, and a means for notifying the user that the order has been confirmed. This enables users who speak a dialect to input voice in their dialect when using a food delivery service, improving convenience.

[1077] "Speech recognition means" is a technology for converting speech into text.

[1078] "Generative AI means" is an artificial intelligence technology that generates new information and operations based on input data.

[1079] "Learning tools to support all dialects of Japan" refers to technology that collects and learns the data necessary to recognize and understand dialects from all over Japan.

[1080] "Means for learning Tsugaru dialect" refers to a technology for collecting and learning the data necessary to recognize and understand Tsugaru dialect, which is said to be the most difficult dialect in Japan.

[1081] "Means for practical application of speech recognition" refers to technologies that make speech recognition technology usable in actual applications and services.

[1082] "Means for recognizing and analyzing speech input in a dialect and generating order details" refers to technology that recognizes speech input in a dialect, analyzes the content, and generates appropriate order details.

[1083] The "means for transmitting the generated order details via a communication means" refers to a technique for transmitting the generated order details using the Internet or other communication means.

[1084] The "means for having the user confirm the order details" is a technique for presenting the generated order details to the user and requesting their confirmation.

[1085] "Means for notifying that an order has been confirmed" refers to a technique for notifying a user that an order has been confirmed.

[1086] The following system configuration will be described as an embodiment of the present invention.

[1087] System Configuration

[1088] This system includes a speech recognition means, a generative AI means, a learning means to support all Japanese dialects, a means for learning Tsugaru dialect, a means for putting speech recognition into practical use, a means for recognizing and analyzing speech input in a dialect and generating order details, a means for transmitting the generated order details via a communication means, a means for allowing the user to confirm the order details, and a means for notifying that the order has been confirmed.

[1089] Hardware and software used

[1090] Hardware: Smartphone (iOS or Android)

[1091] software:

[1092] Speech recognition API (e.g. Google Cloud Speech-to-Text)

[1093] Generative AI models (e.g., OpenAI GPT-4)

[1094] Food delivery API (e.g. Uber Eats API)

[1095] Processing flow

[1096] 1. Voice input: The user speaks their order in their local dialect into the microphone on their smartphone.

[1097] 2. Speech recognition: The smartphone uses a speech recognition API to convert the input speech into text, using a dialect-aware speech recognition model.

[1098] 3. Text analysis: The server uses a generative AI model to analyze the recognized text and understand the order.

[1099] 4. Order Generation: Based on the analysis results, the server generates an appropriate order using the food delivery API and sends it to the restaurant.

[1100] 5. Confirmation and Notification: The smartphone will prompt the user to confirm the order and notify them that the order has been placed.

[1101] Specific examples

[1102] When a user says "I'd like to order one pizza" in Tsugaru dialect, the smartphone recognizes this and generates an appropriate pizza order and sends it to the restaurant.

[1103] Prompt Sentence Examples

[1104] If a user says "I'd like a pizza" in Tsugaru dialect, use a speech recognition API to convert the speech to text, use a generative AI model to analyze the order, and use a food delivery API to generate a pizza order.

[1105] In this way, the dialect-enabled food delivery assistant allows users to place orders in their dialect, providing convenient services to more users.

[1106] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1107] Step 1:

[1108] The user speaks their order in their local dialect into the smartphone microphone. The input is the user's voice, and the output is the voice data input into the smartphone microphone.

[1109] Step 2:

[1110] The device uses a speech recognition API (e.g., Google Cloud Speech-to-Text) to convert the input voice data into text. The input is voice data, and the output is text data. In this case, a dialect-compatible speech recognition model is used.

[1111] Step 3:

[1112] The server uses a generative AI model (e.g., OpenAI GPT-4) to analyze the text data sent from the speech recognition API and understand the order details. The input is text data, and the output is the analyzed order details data.

[1113] Step 4:

[1114] Based on the analysis results, the server generates an appropriate order using a food delivery API (e.g., Uber Eats API) and sends it to the restaurant. The input is the parsed order data, and the output is the order data sent through the food delivery API.

[1115] Step 5:

[1116] The terminal displays the generated order details to the user to confirm the order details. The input is the order confirmation data from the food delivery API, and the output is the order details displayed on the smartphone screen.

[1117] Step 6:

[1118] The user confirms the order details and confirms it. The input is the user's confirmation operation, and the output is a signal that the order has been confirmed.

[1119] Step 7:

[1120] The terminal notifies the server that the order has been confirmed, and the server sends that information to the food delivery API. The input is the order confirmation signal, and the output is the order confirmation data sent to the food delivery API.

[1121] Step 8:

[1122] The server notifies the user that the order has been placed. The input is the order confirmation data from the food delivery API, and the output is a confirmation of the order that is displayed to the user via the smartphone's notification function.

[1123] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1124] "Example 1"

[1125] In one embodiment of the present invention, a system is provided that includes a speech recognition means, a generative AI means, a learning means for supporting all Japanese dialects, a means for learning Tsugaru dialect, a means for implementing speech recognition, and an emotion engine that recognizes a user's emotion. This emotion engine recognizes the emotion from the user's voice and generates an operation corresponding to that emotion. For example, when a user vocally inputs "Turn on the TV," the emotion engine recognizes the user's emotion from the voice. If the emotion is recognized as "joy," the generative AI means generates an operation to turn on the TV.

[1126] "Example 2"

[1127] In another embodiment of the present invention, the emotion engine generates an operation according to the user's emotion. For example, when the user makes a voice input such as "Turn on the radio," the emotion engine recognizes the user's emotion from the voice. If the emotion is recognized as "sadness," the generative AI means generates an operation to turn on the radio, and provides a result in which the operation is adapted to the user's emotion. Specifically, the generative AI means generates an operation such as setting the radio to a channel that plays sad songs that the user likes.

[1128] "Example 3"

[1129] Furthermore, in another embodiment of the present invention, the emotion engine recognizes the user's emotion, and the generative AI means generates an operation according to that emotion. For example, when the user makes a voice input such as "Turn on the air conditioner," the emotion engine recognizes the user's emotion from the voice. If the emotion is recognized as "anger," the generative AI means generates an operation to turn on the air conditioner, providing a result where the operation is adapted to the user's emotion. Specifically, the generative AI means generates an operation such as setting the air conditioner temperature to a low temperature that will make the user feel calm.

[1130] The processing flow of each embodiment will be described below.

[1131] "Example 1"

[1132] Step 1: The user gives a voice input. In this example, the voice input is "Turn on the TV."

[1133] Step 2: The emotion engine recognizes the emotion from the user's voice. In this example, it recognizes "joy."

[1134] Step 3: The generative AI generates an action based on the recognized emotion. In this example, it generates the action to turn on the TV.

[1135] "Example 2"

[1136] Step 1: The user gives a voice input. In this example, the voice input is "Turn on the radio."

[1137] Step 2: The emotion engine recognizes the emotion from the user's voice. In this example, "sadness"

[1138] Recognize.

[1139] Step 3: The generative AI generates an operation based on the recognized emotion, and provides a result that adapts to the user's emotion. In this example, it generates an operation to turn on the radio, and provides a result that adapts to the user's emotion. Specifically, it generates an operation such as setting the radio channel to a channel that plays sad songs that the user likes.

[1140] "Example 3"

[1141] Step 1: The user gives a voice input. In this example, the user gives the voice input "Turn on the air conditioner."

[1142] Step 2: The emotion engine recognizes the emotion from the user's voice. In this example, it recognizes "anger."

[1143] Step 3: The generative AI generates an operation based on the recognized emotion, and provides a result that adapts the operation to the user's emotion. In this example, it generates an operation to turn on the air conditioner, and provides a result that adapts the operation to the user's emotion. Specifically, it generates an operation such as setting the air conditioner temperature to a low temperature that will make the user feel calm.

[1144] Example 1

[1145] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1146] When elderly people operate IT devices, it is difficult to accurately recognize voice input and generate appropriate operations. In particular, there is a need to support all Japanese dialects, but the diversity and difficulty of dialects reduces the accuracy of voice recognition. In addition, there is also the problem that it is not possible to generate operations that take the user's emotions into account, which results in a poor user experience.

[1147] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1148] In this invention, the server includes a speech recognition unit, a generative artificial intelligence unit, and a learning unit for supporting all Japanese dialects. This allows for accurate recognition of speech input when elderly people operate information technology devices, and enables dialect-compatible speech recognition. Furthermore, by including an emotion engine that recognizes the user's emotions, it becomes possible to generate operations according to the user's emotions, improving the user experience.

[1149] "Speech recognition means" is a technology for converting voice input into text data.

[1150] "Generative artificial intelligence means" refers to artificial intelligence technology for generating appropriate operations based on the results of speech recognition.

[1151] "Learning method to support all Japanese dialects" is a technology that trains a speech recognition model to recognize the various dialects used in Japan.

[1152] "Dialect training" refers to the technique of specifically training a speech recognition model to recognize a particular dialect, especially a difficult dialect.

[1153] "Means for practical application of speech recognition" refers to the technology for applying speech recognition technology to actual applications and systems.

[1154] An "emotion engine" is a technology that analyzes emotions from a user's voice and generates appropriate operations based on those emotions.

[1155] MODE FOR CARRYING OUT THE INVENTION

[1156] This invention is a system that includes a speech recognition means, a generative artificial intelligence means, a learning means for dealing with all dialects of Japan, a means for learning particularly difficult dialects, a means for putting speech recognition into practical use, and an emotion engine. A specific embodiment of this system will be described below.

[1157] Acquiring voice input

[1158] Subject: User

[1159] The user speaks into the device, for example, to give instructions such as "Turn on the TV." The device is equipped with a high-performance microphone that picks up the user's voice with high accuracy.

[1160] Sending audio data

[1161] Subject: Terminal

[1162] The device transmits the acquired voice data to the server, where it is converted into an appropriate format and sent to the server via the network.

[1163] Voice Recognition

[1164] Subject: Server

[1165] The server converts the received voice data into text data using a voice recognition means. Specifically, it converts the voice into text using voice recognition software (for example, Google Cloud Speech-to-Text API).

[1166] Dialect-aware learning

[1167] Subject: Server

[1168] The server has training tools to handle all dialects in Japan, and in particular trains speech recognition models to recognize difficult dialects (e.g., Tsugaru dialect).

[1169] emotion recognition

[1170] Subject: Terminal

[1171] The device has an emotion engine that recognizes emotions from the user's voice. Specifically, it uses emotion analysis software (for example, Microsoft Azure's Emotion API) to analyze emotions from the user's voice.

[1172] Generate Operations

[1173] Subject: Server

[1174] The server uses a generative artificial intelligence (e.g., OpenAI's GPT-4) to generate actions based on the speech recognition and emotion recognition results. The generated actions are sent to the device.

[1175] Executing an operation

[1176] Subject: Terminal

[1177] The device executes the operation received from the server, specifically, sending a signal to turn on the TV.

[1178] Specific examples

[1179] Example 1

[1180] When a user says "Turn on the TV," the device captures the speech and sends it to the server. The server converts the speech into text using a speech recognition means and generates the operation "Turn on the TV" using a generative artificial intelligence means. The device executes the operation and turns on the TV.

[1181] Example 2

[1182] When a user says "Turn off the air conditioner," the device captures the speech and sends it to the server. The server converts the speech into text using a speech recognition means and generates the operation "Turn off the air conditioner" using a generative artificial intelligence means. The device then executes the operation and turns off the air conditioner.

[1183] Prompt Sentence Examples

[1184] "When a user says 'Turn on the TV,' convert the speech to text using the speech recognition means, and generate the operation to turn on the TV using the generative artificial intelligence means. Also, if the user's emotion is 'joy,' provide that information to the generative artificial intelligence means."

[1185] By inputting this prompt into a generative AI model, the system's behavior can be simulated.

[1186] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1187] Step 1:

[1188] Acquiring voice input

[1189] Subject: User

[1190] The user speaks to the device, saying, "Turn on the TV." The device is equipped with a high-performance microphone, which picks up the user's voice with high accuracy.

[1191] Input: User's voice commands

[1192] Output: Captured audio data

[1193] Specific action: Instead of searching for the remote control in the living room, the user speaks to the device and says, "Turn on the TV."

[1194] Step 2:

[1195] Sending audio data

[1196] Subject: Terminal

[1197] The device transmits the acquired voice data to the server, where it is converted into an appropriate format and sent to the server via the network.

[1198] Input: Captured audio data

[1199] Output: Audio data sent to the server

[1200] How it works: Your device sends audio data to the server over Wi-Fi, which takes just a few seconds.

[1201] Step 3:

[1202] Voice Recognition

[1203] Subject: Server

[1204] The server converts the received voice data into text data using a voice recognition means. Specifically, it converts the voice into text using voice recognition software (for example, Google Cloud Speech-to-Text API).

[1205] Input: Audio data sent to the server

[1206] Output: Text data (e.g. "Turn on the TV")

[1207] Specific operation: The server receives the voice data and calls the Google Cloud Speech-to-Text API to generate the text "Turn on the TV."

[1208] Step 4:

[1209] Dialect-aware learning

[1210] Subject: Server

[1211] The server has training tools to handle all dialects in Japan, and in particular trains speech recognition models to recognize difficult dialects (e.g., Tsugaru dialect).

[1212] Input: Dialect speech data

[1213] Output: dialect-aware speech recognition model

[1214] Specific operation: The server uses Tsugaru dialect speech data to train the model so that it can handle dialects.

[1215] Step 5:

[1216] emotion recognition

[1217] Subject: Terminal

[1218] The device has an emotion engine that recognizes emotions from the user's voice. Specifically, it uses emotion analysis software (for example, Microsoft Azure's Emotion API) to analyze emotions from the user's voice.

[1219] Input: User's voice data

[1220] Output: Emotion data (e.g., "joy")

[1221] Specific operation: The device analyzes the user's emotion from the voice saying "Turn on the TV" and recognizes that emotion as "joy."

[1222] Step 6:

[1223] Generate Operations

[1224] Subject: Server

[1225] The server uses a generative artificial intelligence (e.g., OpenAI's GPT-4) to generate actions based on the speech recognition and emotion recognition results. The generated actions are sent to the device.

[1226] Input: Text data (e.g., "Turn on the TV"), emotion data (e.g., "Joy")

[1227] Output: The generated action (e.g. "Turn on the TV")

[1228] Specific operation: The server inputs the text "Turn on the TV" and the emotion information "joy" into GPT-4, and generates the operation "Turn on the TV."

[1229] Step 7:

[1230] Executing an operation

[1231] Subject: Terminal

[1232] The device executes the operation received from the server, specifically, sending a signal to turn on the TV.

[1233] Input: The generated action (e.g. "Turn on the TV")

[1234] Output: The action taken (e.g. the TV is turned on)

[1235] Specific operation: The device sends a TV remote control signal, and the TV turns on.

[1236] (Application example 1)

[1237] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1238] When elderly people or customers who speak dialects operate IT devices, the accuracy of voice recognition is low, making it difficult to operate them. There is also the issue of it being difficult to properly recognize customer emotions in physical stores and respond accordingly.

[1239] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes a voice recognition means, a generative AI means, a learning means for supporting all Japanese dialects, a learning means for Tsugaru dialect, which is said to be the most difficult dialect in Japan, a voice recognition practical use means, an emotion recognition means, an application means installed on the smart glasses, a means for recognizing a customer's voice input and generating an appropriate response by the generative AI means, a means for recognizing the customer's emotion and generating a response according to the emotion by the emotion recognition means, and a voice output means. This improves the accuracy of voice recognition when elderly people or customers who speak dialects operate IT devices, enabling appropriate responses according to the customer's emotion in physical stores.

[1240] "Speech recognition means" is a technology that analyzes voice input and converts it into text data.

[1241] "Generative AI means" is an artificial intelligence technology that generates appropriate responses and operations based on input data.

[1242] "Learning method to support all Japanese dialects" is a technology that trains a speech recognition model to recognize the various dialects used in Japan.

[1243] "Means for learning Tsugaru dialect" refers to special learning techniques to understand Tsugaru dialect, which is said to be the most difficult dialect in Japan.

[1244] "Means for practical application of speech recognition" refers to technologies that make speech recognition technology usable in actual systems and applications.

[1245] "Emotion recognition means" is a technology that analyzes and recognizes a user's emotions from voice and text data.

[1246] "Application means installed on smart glasses" refers to applications that run on smart glasses, and is software for utilizing voice recognition and generative AI.

[1247] "Means for recognizing customer voice input and for a generative AI means to generate an appropriate response" refers to technology that recognizes customer voice and for a generative AI means to generate an appropriate response based on the content of that voice.

[1248] "Means for recognizing a customer's emotion and generating a response according to the emotion" refers to a technology that recognizes a customer's emotion and generates an appropriate response according to the emotion.

[1249] "Audio output means" refers to a technology for outputting generated text data and responses as audio.

[1250] A system for implementing this invention includes a voice recognition means, a generative AI means, a learning means for supporting all dialects of Japan, in particular a means for learning Tsugaru dialect, a means for putting voice recognition into practical use, an emotion recognition means, an application means to be installed in the smart glasses, a means for recognizing a customer's voice input and for the generative AI means to generate an appropriate response, a means for the emotion recognition means to recognize the customer's emotion and for generating a response according to the emotion, and a voice output means.

[1251] Hardware and Software Configuration

[1252] The server uses the speech_recognition library for speech recognition and the transformers library for generative AI. It also uses the sentiment-analysis pipeline of the transformers library for emotion recognition. It uses the pyttsx3 library for audio output. The smart glasses have a built-in microphone and speaker, and these software are installed.

[1253] Data processing and calculation

[1254] 1. Speech recognition: When a user speaks into the smart glasses, the microphone captures the voice and the speech_recognition library converts the voice into text data.

[1255] 2. Generative AI: The converted text data is fed into a generative AI model from the transformers library to generate an appropriate response.

[1256] 3. Emotion Recognition: At the same time, the text data is input into a sentiment-analysis pipeline to analyze the user's emotions. Based on the analysis results, a generative AI model generates a response according to the emotion.

[1257] 4. Audio Output: Finally, the generated response is output as audio using the pyttsx3 library.

[1258] Specific examples

[1259] For example, if a customer in a physical store says, "Tell me about this product," the system operates as follows:

[1260] 1. Speech recognition: The customer's voice is captured by the microphone in the smart glasses and converted into text data using the speech_recognition library: "Tell me about this product."

[1261] 2. Generative AI: The converted text data is fed into a generative AI model from the Transformers library, which generates a response such as, "This product is a state-of-the-art smartphone with an excellent camera. It also has a long battery life, allowing for extended use."

[1262] 3. Emotion recognition: At the same time, the text data is fed into a sentiment-analysis pipeline to identify customer concerns. A generative AI model then generates a more detailed explanation.

[1263] 4. Audio Output: Finally, the generated response is output as audio using the pyttsx3 library.

[1264] Prompt Sentence Examples

[1265] Customer: Tell me about this product.

[1266] System: This is a state-of-the-art smartphone with an excellent camera. It also has a long battery life, allowing it to be used for a long time.

[1267] In this way, the accuracy of voice recognition will be improved when elderly people or customers who speak dialects operate IT devices, making it possible to respond appropriately to customers' emotions in physical stores.

[1268] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1269] Step 1:

[1270] A user speaks into the smart glasses. The microphone in the smart glasses captures the audio. The input is the user's voice and the output is the captured audio data.

[1271] Step 2:

[1272] The server uses the speech_recognition library as a speech recognition tool to convert the captured audio data into text data. The input is audio data and the output is text data. Specifically, the audio data is passed to the library, and the speech recognition algorithm is applied.

[1273] Step 3:

[1274] The server uses the transformers library as a generative AI tool, and generates an appropriate response using the converted text data as input. The input is text data, and the output is the generated response text. Specifically, the text data is input into the generative AI model, and the model generates a response.

[1275] Step 4:

[1276] The server uses the sentiment-analysis pipeline of the Transformers library as an emotion recognition method to analyze user emotions from text data. The input is text data, and the output is the emotion analysis result. Specifically, the text data is input into the emotion recognition model, and the emotion is analyzed.

[1277] Step 5:

[1278] Based on the emotion analysis results, the generative AI means generates a response according to the emotion. The input is the emotion analysis results and text data, and the output is a response text according to the emotion. Specifically, the emotion analysis results are fed back to the generative AI model, which then adjusts the response.

[1279] Step 6:

[1280] The server uses the pyttsx3 library as a speech output means to output the generated response text as speech. The input is the response text and the output is audio data. Specifically, the response text is input into a speech synthesis engine, and audio data is generated.

[1281] Step 7:

[1282] The smart glasses play the audio data and provide a response to the user. The input is the audio data, and the output is the audio heard by the user. In particular, the audio data is played through the speakers of the smart glasses.

[1283] Example 2

[1284] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1285] When elderly people operate information technology devices, it is difficult to accurately recognize voice input, especially when using dialects, and generate appropriate operations. There is also a need for a more user-friendly system that generates operations according to the user's emotions. Furthermore, a means for actually executing the generated operations is also required.

[1286] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1287] In this invention, the server includes a speech recognition means, a generative AI means, a dialect-based learning means, an emotion recognition means, a speech recognition practical use means, and an infrared signal generator control means, which enables accurate speech recognition even when an elderly person uses a dialect, generates appropriate operations according to the user's emotions, and actually executes the operations.

[1288] "Speech recognition means" refers to a device or software that converts voice input into digital data and analyzes its content.

[1289] "Generative AI means" refers to devices or software that use artificial intelligence technology to generate appropriate operations based on data obtained from speech recognition means.

[1290] A "dialect-aware training means" is a device or software that trains a speech recognition means to recognize dialects specific to a particular region or culture.

[1291] The "emotion recognition means" is a device or software for analyzing the user's emotion from voice input and recognizing that emotion.

[1292] "Means for practical use of speech recognition" refers to devices or software that apply speech recognition technology to actual systems and applications, making it practically usable.

[1293] "Infrared signal generator control means" refers to a device or software for generating an infrared signal based on instructions from the generating AI means and controlling a specific device.

[1294] This invention is a system that can accurately recognize voice input, particularly in dialects, when elderly people operate information technology devices, and generate appropriate operations. Furthermore, it can generate operations according to the user's emotions and actually execute those operations.

[1295] Hardware and software used

[1296] 1. Voice recognition means:

[1297] Your device uses the built-in microphone to capture audio.

[1298] For example, Google Speech-to-Text API is used as speech recognition software.

[1299] 2. Generative AI means:

[1300] The server uses a generative AI model (e.g., OpenAI's GPT-4) to analyze the text data obtained from the speech recognition means and generate appropriate actions.

[1301] 3. Dialect-compatible learning tools:

[1302] The server trains its speech recognition tools to recognize dialects specific to a particular region or culture, for example, using training data for the Japanese Tsugaru dialect.

[1303] 4. Emotion recognition means:

[1304] The server uses emotion recognition software (e.g., IBM Watson Tone Analyzer) to analyze the user's emotion from the voice input and recognize that emotion.

[1305] 5. Practical application of speech recognition:

[1306] The terminal applies voice recognition technology to actual systems and applications, making it practical to use.

[1307] 6. Infrared signal generator control means:

[1308] The terminal sends the generated command to an infrared signal generator (e.g., BroadLink RM4 Pro) to control a specific device.

[1309] Specific examples

[1310] Example 1: Turning on the TV

[1311] 1. The user says "Turn on the TV" in Tsugaru dialect.

[1312] 2. The device captures audio using its built-in microphone and converts it to text using the Google Speech-to-Text API.

[1313] 3. The server uses the generative AI model to generate the "turn on TV" command.

[1314] 4. The device sends the generated command to the infrared signal generator to turn on the TV.

[1315] Example 2: Turning on the radio

[1316] 1. The user says, "Turn on the radio."

[1317] 2. The device captures audio using its built-in microphone and converts it to text using the Google Speech-to-Text API.

[1318] 3. The server uses emotion recognition software to recognize the user's emotion as "sadness."

[1319] 4. The server uses the generative AI model to generate an action to set the channel to a sad song.

[1320] 5. The device sends a command to the radio to set it to the specified channel.

[1321] Prompt Sentence Examples

[1322] "When a user says 'Turn on the TV' in Tsugaru dialect, the speech recognition means converts the speech into text, and the generation AI means generates an operation to turn on the TV. It controls an infrared signal generator to turn on the TV."

[1323] "When a user says 'turn on the radio,' the emotion engine recognizes the user's emotion, and the generative AI means generates radio controls adapted to the user's emotion. For example, it sets the radio to a channel playing sad songs."

[1324] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1325] Step 1:

[1326] Capturing voice input

[1327] The user says "Turn on the TV" in Tsugaru dialect.

[1328] Your device uses the built-in microphone to capture audio.

[1329] Input: User's voice

[1330] Output: Audio data

[1331] Specific behavior:

[1332] A user stands in front of the TV in the living room and says, "Turn on the TV."

[1333] The device captures audio through the microphone and saves it as audio data.

[1334] Step 2:

[1335] Converting audio data to text

[1336] The device uses speech recognition software (e.g., Google Speech-to-Text API) to convert the captured voice data into text data.

[1337] Input: Audio data

[1338] Output: Text data

[1339] Specific behavior:

[1340] The device sends the captured audio data to the Google Speech-to-Text API.

[1341] The device receives the text data "Turn on the TV" returned from the API.

[1342] Step 3:

[1343] Emotion recognition (if needed)

[1344] The server uses emotion recognition software (e.g., IBM Watson Tone Analyzer) to analyze the user's emotions from the text data.

[1345] Input: Text data

[1346] Output: Emotion data

[1347] Specific behavior:

[1348] The server sends the text data "Turn on the radio" to the IBM Watson Tone Analyzer.

[1349] The server receives "sadness" as the analysis result.

[1350] Step 4:

[1351] Generate Operations

[1352] The server uses a generative AI model (e.g., OpenAI's GPT-4) to analyze the text data and generate appropriate actions.

[1353] Input: Text data, emotion data (if necessary)

[1354] Output: Operation command

[1355] Specific behavior:

[1356] The server inputs the text data "Turn on the TV" into the generative AI model.

[1357] The server receives the generated command and sends it to a device that generates an infrared signal to "turn on the TV."

[1358] Step 5:

[1359] Executing an operation

[1360] The terminal transmits the generated command to the infrared signal generator to perform the operation of turning on the television.

[1361] Input: Operation command

[1362] Output: Action performed (TV turns on)

[1363] Specific behavior:

[1364] The terminal sends a "Turn on the TV" command to the infrared signal generator.

[1365] The infrared signal generator sends an infrared signal to the television, which turns on the television.

[1366] (Application example 2)

[1367] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1368] When elderly people operate IT devices, they face difficulties with voice recognition, especially in Japanese dialects. Another issue is that the user experience is not improved because appropriate operations and information cannot be provided based on the user's emotions. Furthermore, there is also the issue that the shopping experience is not optimized because product suggestions cannot be made in physical stores based on the user's emotions.

[1369] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1370] In this invention, the server includes a speech recognition means, a generative AI means, a learning means for handling all dialects in Japan, particularly a means for learning Tsugaru dialect, which is said to be the most difficult dialect in Japan, a practical speech recognition means, a means for recognizing a user's emotions using an emotion engine, a means for generating operations according to the user's emotions, a means for providing product information in a physical store, and a means for suggesting products based on the user's emotions. This enables accurate speech recognition even when an elderly person speaks in a dialect, realizes appropriate operations and information provision according to the user's emotions, and makes it possible to suggest products based on the user's emotions even in a physical store.

[1371] "Speech recognition means" is a technology for converting voice input into text data.

[1372] "Generative AI means" is an artificial intelligence technology that generates appropriate responses and operations based on input data.

[1373] A "learner" is a technique that uses a specific dataset to train a model and improve its ability to perform a specific task.

[1374] "Means for learning Tsugaru dialect" refers to a special training technique for understanding and recognizing the dialect used in the Tsugaru region of Japan.

[1375] "Means for practical application of speech recognition" refers to the technology for incorporating speech recognition technology into actual applications and systems.

[1376] An "emotion engine" is a technology for analyzing and recognizing emotions from a user's voice or text.

[1377] "Means for generating operations according to the user's emotions" refers to a technology that generates appropriate operations and responses based on the recognized user's emotions.

[1378] "Means for providing product information in a physical store" refers to technology for providing information about products to users in a physical store.

[1379] The "means for suggesting products based on the user's emotions" is a technology for suggesting appropriate products by taking into consideration the user's emotions.

[1380] This invention is a system that combines a speech recognition unit, a generative AI unit, a learning unit, an emotion engine, and a product information provision unit in a physical store. Specific embodiments for realizing this system are described below.

[1381] System Configuration

[1382] The system consists of the following major components:

[1383] 1. Speech recognition means: A technology for converting voice input into text data. Specifically, it uses the speech_recognition library.

[1384] 2. Generative AI: This is an artificial intelligence technique that generates appropriate responses and actions based on input data. Specifically, we use the GPT-3 model from the transformers library.

[1385] 3. Training: This is a technique that uses specific datasets to train a model and improve its ability to perform a specific task, especially to train it to handle all Japanese dialects.

[1386] 4. Emotion Engine: This technology analyzes and recognizes emotions from user speech and text. Specifically, it uses the emotion analysis model from the transformers library.

[1387] 5. Means of providing product information in physical stores: This is a technology for providing product information to users in physical stores.

[1388] Program processing

[1389] The server uses a speech recognition means to convert the user's voice input into text data. Next, it uses an emotion engine to recognize the user's emotion from the text data. Based on the recognized emotion, the generative AI means generates an appropriate response or operation. For example, if a user says, "Tell me about this product," and the emotion engine recognizes the user's emotion as "excited," the generative AI means will generate a response such as, "This product is the latest model, with many new features."

[1390] Hardware and software used

[1391] Hardware: Smartphone (microphone, speaker)

[1392] software:

[1393] speech_recognition library: Used to convert voice input to text.

[1394] Transformers library: Generates responses using a generative AI model (GPT-3).

[1395] The requests library: Used to communicate with external APIs (if necessary).

[1396] Specific examples

[1397] When a user says, "Tell me about this product" in a physical store, the speech recognition means converts the speech into text, and the emotion engine recognizes the user's emotion. For example, if the user is excited, the generative AI means generates a response such as, "This product is the latest model and has many new features."

[1398] Prompt Sentence Examples

[1399] When the user is sad: "Suggest a product when the user is sad: Tell me about this product"

[1400] When the user is happy: "Suggest a product when the user is happy: Tell me about this product"

[1401] In this way, the smart shopping assistant can provide relevant information according to the user's emotions and improve the shopping experience.

[1402] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1403] Step 1:

[1404] The server acquires the user's voice input using a speech recognition means, specifically by collecting voice data using the smartphone's microphone and converting the voice data into text data using the speech_recognition library.

[1405] Input: User's voice data

[1406] Output: Text data

[1407] Step 2:

[1408] The server uses an emotion engine to recognize user emotions from text data. Specifically, it uses the emotion analysis model from the transformers library to analyze the text data and identify the user's emotions.

[1409] Input: Text data

[1410] Output: User emotion (e.g., happy, sad, excited, etc.)

[1411] Step 3:

[1412] The server generates a prompt based on the user's recognized emotion. Specifically, it creates a prompt based on the emotion and inputs it into the generative AI model. For example, if the user is sad, it generates a prompt such as "Please suggest products that you would recommend when the user is sad: Tell me about this product."

[1413] Input: User sentiment, text data

[1414] Output: prompt statement

[1415] Step 4:

[1416] The server generates an appropriate response based on the prompt sentence using generative AI means, specifically, the GPT-3 model from the transformers library to generate a response to the prompt sentence.

[1417] Input: prompt statement

[1418] Output: The generated response (e.g., "This product is the latest model and has many new features")

[1419] Step 5:

[1420] The server then provides the generated response to the user, either by playing it aloud using the smartphone's speaker or by displaying it as text on the screen.

[1421] Input: Generated response

[1422] Output: Providing a response to the user (audio or text)

[1423] In this way, the server can recognize the user's voice input, analyze emotions, and generate and provide an appropriate response.

[1424] Example 3

[1425] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1426] Conventional voice recognition systems have difficulty accurately recognizing specific dialects and emotions, and are unable to accommodate the diverse dialects and emotions of users, particularly in Japan. It is also difficult to accurately recognize voice input and generate appropriate operations when elderly people operate IT devices. This has resulted in a poor user experience and limited the practicality of the system.

[1427] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1428] In this invention, the server includes a speech recognition means, a generative artificial intelligence means, and a learning means for supporting all dialects of Japan. This enables a system including a means for learning Tsugaru dialect, which is said to be the most difficult dialect in Japan, a practical speech recognition means, a means for recognizing a user's emotion using an emotion engine, and a means for generating an operation in accordance with the recognized emotion.

[1429] "Speech recognition means" is a technology that analyzes voice data and converts it into corresponding text data.

[1430] "Generative AI means" refers to AI technology that generates natural dialogue and operations based on input data.

[1431] "Learning method to support all Japanese dialects" is a technology that recognizes the various dialects used in Japan, collects the data necessary to support them, and trains a model.

[1432] "Means for learning Tsugaru dialect" refers to technology that collects the data necessary to recognize and respond to the Tsugaru dialect, a dialect used in the Tsugaru region of Japan, and trains a model.

[1433] "Means for practical application of speech recognition" refers to technology that incorporates trained speech recognition models into actual systems, enabling them to be used in practical applications.

[1434] An "emotion engine" is a technology that analyzes and identifies a user's emotions from voice data.

[1435] "Means for generating an operation according to the recognized emotion" is a technology that generates and executes an appropriate operation based on the user's emotion.

[1436] This invention is a system including a speech recognition means, a generative artificial intelligence means, a learning means for dealing with all dialects of Japan, in particular a means for learning Tsugaru dialect, a means for putting speech recognition into practical use, a means for recognizing a user's emotions using an emotion engine, and a means for generating operations in accordance with the recognized emotions.

[1437] Hardware and software used

[1438] Hardware: High-performance server (e.g., server with GPU)

[1439] Software: speech recognition software (e.g., speech recognition APIs), generative artificial intelligence models (e.g., natural language processing models)

[1440] Explanation of program processing

[1441] Acquisition and preprocessing of audio data

[1442] The server collects a large amount of Tsugaru dialect speech data. It then performs preprocessing such as noise removal and normalization on the collected speech data. Specifically, it processes the speech data using a speech processing library.

[1443] Training a speech recognition model

[1444] The server inputs the preprocessed voice data into the voice recognition API and retrieves the corresponding text data. The retrieved text data is used to train a voice recognition model. Specifically, a machine learning library is used to build and train the model.

[1445] Learning generative artificial intelligence models

[1446] The server inputs the text data obtained from the speech recognition model into the natural language processing model and trains the generative AI model. Once trained, the generative AI model is saved and provided as an API. Specifically, the model is managed using a natural language processing library.

[1447] Introducing the Emotion Engine

[1448] The server uses an emotion engine to recognize emotions from the user's voice. Specifically, it uses a voice analysis tool to analyze the tone, speed, and strength of the voice. Based on the analysis results, it identifies the user's emotion and passes that information to a generative AI model.

[1449] Generating operations according to emotions

[1450] The user makes a voice input such as "Turn on the air conditioner." The device sends the voice to the server. The server converts the voice into text using a voice recognition model. The server recognizes the user's emotions using an emotion engine. For example, if the voice is high-pitched and fast-paced, it determines that the user is angry. The server uses generative artificial intelligence means to generate an appropriate operation according to the recognized emotion. For example, if the user is angry, it generates an operation to set the air conditioner temperature low.

[1451] Examples of specific examples and prompts

[1452] Examples:

[1453] The user angrily voice-inputs "Turn on the air conditioner."

[1454] The terminal transmits the audio to the server.

[1455] The server converts the speech to text using a speech recognition model.

[1456] The server uses an emotion engine to recognize that the user's emotion is "anger."

[1457] The server uses a generative artificial intelligence means to generate an operation to set the temperature of the air conditioner low.

[1458] Example prompt sentence:

[1459] When a user makes a voice input saying "Turn on the air conditioner" with an angry emotion, the emotion engine recognizes the user's emotion from the voice, and the generative artificial intelligence means generates an operation to set the air conditioner temperature low.

[1460] In this way, the system provides an appropriate operation according to the user's emotion. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1461] Step 1:

[1462] The server collects a large amount of Tsugaru dialect speech data. It uses the collected speech data as input and performs preprocessing such as noise removal and normalization. Specifically, it processes the speech data using a speech processing library to remove noise and normalize the volume. The preprocessed speech data is then output.

[1463] Step 2:

[1464] The server inputs the preprocessed voice data into a voice recognition API to obtain the corresponding text data. The voice recognition API analyzes the voice data and converts it into text data. This outputs the text data corresponding to the voice data.

[1465] Step 3:

[1466] The server uses the acquired text data to train a speech recognition model. Specifically, it builds a model using a machine learning library and trains it using the text data as input. The trained speech recognition model is then output.

[1467] Step 4:

[1468] The server inputs the text data obtained from the speech recognition model into the natural language processing model and trains the generative AI model. Specifically, it manages the model using a natural language processing library and trains it based on the text data. This outputs the trained generative AI model.

[1469] Step 5:

[1470] The server uses an emotion engine to recognize emotions from the user's voice. Specifically, it uses a voice analysis tool to analyze the tone, speed, and strength of the voice, and analyzes the voice data as input. This results in the output of the user's emotion.

[1471] Step 6:

[1472] The user makes a voice input such as "Turn on the air conditioner." The device sends the voice to the server. The server converts the voice into text using a speech recognition model. This outputs text data corresponding to the voice data.

[1473] Step 7:

[1474] The server uses an emotion engine to recognize the user's emotion. For example, if the voice tone is high and the speed is fast, it determines that the user is angry. Based on this, the server outputs the user's emotion.

[1475] Step 8:

[1476] The server uses a generative artificial intelligence means to generate an appropriate operation according to the recognized emotion. For example, if the user is angry, the server generates an operation to set the air conditioner temperature low. This results in an appropriate operation being output.

[1477] Step 9:

[1478] The terminal executes the operation sent from the server. Specifically, it executes the operation to set the air conditioner temperature low, providing a comfortable environment for the user. This completes the operation requested by the user.

[1479] (Application example 3)

[1480] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1481] Conventional voice recognition systems are specialized in standard Japanese, making it difficult to handle Japan's diverse dialects. Furthermore, they lacked the ability to recognize user emotions and provide appropriate operations, limiting the user experience. In particular, in autonomous vehicles, it was difficult to provide appropriate navigation instructions when the user spoke in a dialect or was emotional.

[1482] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes a voice recognition means, a generative AI means, and a learning means for handling all Japanese dialects. This makes it possible to accurately recognize speech even when the user speaks in a dialect, and further to understand the user's emotions and provide appropriate navigation instructions.

[1483] "Speech recognition means" is a technology for converting speech into text data.

[1484] "Generative AI means" is an artificial intelligence technology that generates appropriate operations and responses based on input data.

[1485] "Learning method to support all Japanese dialects" is a technology that learns the data necessary to recognize the various dialects used in Japan.

[1486] The "means of learning Tsugaru dialect" is a specialized learning technique for recognizing Tsugaru dialect, a dialect used in the Tsugaru region of Japan.

[1487] "Means for practical application of speech recognition" refers to the technology for applying speech recognition technology to actual systems and applications.

[1488] "Emotion recognition means" is a technology that analyzes and recognizes emotions from a user's voice or text.

[1489] "Means for generating navigation instructions in an autonomous vehicle" refers to technology that generates a route and instructions to a destination in an autonomous vehicle.

[1490] The "means for providing navigation instructions according to the user's emotions" is a technology for providing optimal navigation instructions taking into account the user's emotional state.

[1491] The following system configuration will be described as an embodiment of the present invention.

[1492] System Configuration

[1493] Hardware

[1494] Microphone: Used to capture the user's voice input.

[1495] Speaker: Used to provide audio output from the system.

[1496] Self-driving vehicle: A vehicle equipped with a navigation system.

[1497] software

[1498] Speech Recognition Library (speech_recognition): Used to convert the user's speech into text.

[1499] Emotion recognition library (transformers pipeline): Used to analyze emotions from user text data.

[1500] Speech synthesis library (pyttsx3): Used to convert text to speech.

[1501] Processing flow

[1502] 1. Acquiring voice input

[1503] When a user speaks into the microphone, a speech recognition library converts the speech into text data.

[1504] For example, if a user asks in Tsugaru dialect, "Where is the nearest convenience store from here?", the speech is converted into text data.

[1505] 2. Emotional Recognition

[1506] The converted text data is sent to an emotion recognition library to analyze the user's emotions.

[1507] For example, if a user is anxious, the emotion recognition library will recognize the emotion as "anxiety."

[1508] 3. Generate navigation instructions

[1509] Based on the results of emotion recognition, generative AI generates appropriate navigation instructions.

[1510] For example, if the user is recognized as being impatient, instructions are generated to quickly guide the user to the nearest convenience store.

[1511] 4. Audio Output

[1512] The generated navigation instructions are converted into voice using a speech synthesis library and output from a speaker.

[1513] For example, a voice saying "Please stay calm. The nearest convenience store is here" is output.

[1514] Specific examples

[1515] If a user asks in Tsugaru dialect, "Where is the nearest convenience store from here?", the system will recognize the dialect and, if it determines that the user is in a hurry, will quickly guide them to the nearest convenience store.

[1516] Prompt Sentence Examples

[1517] User: "Where's the nearest convenience store from here?"

[1518] System: "Please stay calm. The nearest convenience store is here."

[1519] In this way, the dialect-aware emotion-aware navigation system can understand the dialect and emotions of the user and provide appropriate navigation instructions.

[1520] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1521] Step 1:

[1522] The user speaks into the microphone. The speech recognition library (speech_recognition) picks up the speech and converts it into text data. The input is the user's voice, and the output is text data. For example, if the user says "Where is the nearest convenience store from here?" in Tsugaru dialect, the speech is converted into text data: "Where is the nearest convenience store from here?"

[1523] Step 2:

[1524] The server sends the converted text data to the emotion recognition library (transformers pipeline). The emotion recognition library analyzes the user's emotions from the text data. The input is text data and the output is emotion data. Specifically, the text data "Where is the nearest convenience store from here?" is converted into the emotion data "impatience."

[1525] Step 3:

[1526] The server receives the emotion recognition results, and the generative AI generates appropriate navigation instructions. The input is emotion data and text data, and the output is navigation instructions. Specifically, based on the emotion data of "impatience" and the text data of "Where is the nearest convenience store from here?", the navigation instruction "Please stay calm. The nearest convenience store is here" is generated.

[1527] Step 4:

[1528] The server sends the generated navigation instructions to a speech synthesis library (pyttsx3). The speech synthesis library converts the navigation instructions into speech and outputs it from the speaker. The input is the navigation instructions, and the output is speech data. Specifically, the navigation instruction "Please stay calm. The nearest convenience store is here" is converted into speech data and output from the speaker.

[1529] In this way, the dialect-aware emotion-aware navigation system can understand the dialect and emotions of the user and provide appropriate navigation instructions.

[1530] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1531] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1532] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1533] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1534] [Third embodiment]

[1535] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1536] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1537] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1538] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1539] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1540] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1541] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1542] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1543] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1544] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1545] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1546] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1547] "Example 1"

[1548] The present invention is a system that combines a speech recognition means and a generative AI means. The speech recognition means recognizes the voice input when an elderly person operates an IT device. The generative AI means generates operations based on the recognized voice input. The system also has a learning means to support all Japanese dialects. Specifically, the practical application of speech recognition has been realized by training the system in Tsugaru dialect, which is said to be the most difficult dialect in Japan.

[1549] "Example 2"

[1550] In a specific embodiment, for example, if an elderly person says "Turn on the TV" in Tsugaru dialect, the speech recognition means recognizes this speech input, and the generative AI means generates an operation to turn on the TV. This operation is realized, for example, by controlling a device that generates an infrared signal.

[1551] "Example 3"

[1552] This system also has a learning method to support all dialects in Japan. Specifically, the speech recognition and generative AI methods are trained using a large amount of Tsugaru dialect speech data. This makes it possible to use the same method to support not only Tsugaru dialect but also other dialects.

[1553] The processing flow of each embodiment will be described below.

[1554] "Example 1"

[1555] Step 1: The elderly person performs voice input to operate an IT device. This voice input includes instructions such as "Turn on the TV."

[1556] Step 2: The voice recognition means recognizes the voice input of the elderly person. This voice recognition is performed using a training means to support all Japanese dialects.

[1557] Step 3: The generative AI means generates an action based on the recognized voice input, which may include a specific action such as turning on the TV.

[1558] "Example 2"

[1559] Step 1: An elderly person speaks in Tsugaru dialect and says, "Turn on the TV."

[1560] Step 2: The speech recognition means recognizes this speech input. This speech recognition is highly accurate as a result of having been trained in Tsugaru dialect.

[1561] Step 3: The generative AI means generates the operation to turn on the TV. This operation is realized by controlling a device that generates an infrared signal.

[1562] "Example 3"

[1563] Step 1: Train the speech recognition and generative AI methods using a large amount of Tsugaru dialect speech data.

[1564] Step 2: Using the trained voice recognition means and generative AI means, recognize the elderly person's voice input and generate operations.

[1565] Step 3: Execute the generated operation, which may include a specific operation such as turning on the television.

[1566] Example 1

[1567] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1568] When elderly people operate IT devices, there is a demand for improved ease of operation through voice input. However, a voice recognition system that can handle Japan's diverse dialects, especially difficult dialects such as Tsugaru dialect, has not yet been put into practical use. Therefore, it is necessary to develop a voice recognition system that allows elderly people to speak naturally in their own dialect.

[1569] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a voice recognition means, a generative artificial intelligence means, and a learning means for handling all Japanese dialects. This allows elderly people to speak naturally in their own dialect and operate information technology devices using voice input.

[1570] "Speech recognition means" is a technology for converting voice data into text data.

[1571] "Generative artificial intelligence means" refers to an artificial intelligence technology for generating appropriate operational instructions based on input data.

[1572] "Learning method to support all Japanese dialects" is a technology that pre-trains a voice recognition system to recognize the various dialects used in Japan.

[1573] "Means for learning Tsugaru dialect" refers to a technology that specially trains a voice recognition system to recognize Tsugaru dialect, which is said to be the most difficult dialect in Japan.

[1574] "Voice data preprocessing means" refers to technology for improving the accuracy of voice recognition by removing noise from the voice data and normalizing the volume.

[1575] "Means for practical application of speech recognition" refers to the technology for applying speech recognition technology to actual systems and applications, making it practically usable.

[1576] "Voice input when an elderly person operates an information technology device by themselves" refers to voice commands that an elderly person issues to operate an information technology device by themselves.

[1577] This invention is a system for improving the convenience of elderly people when operating information technology devices by voice input. This system includes a speech recognition means, a generative artificial intelligence means, a learning means for supporting all Japanese dialects, a means for training the Tsugaru dialect, a speech data preprocessing means, and a means for implementing speech recognition.

[1578] System configuration

[1579] Voice recognition means

[1580] The server converts the audio data into text using speech recognition software, such as the Google Cloud Speech-to-Text API, which is pre-trained to accommodate Japan's diverse dialects.

[1581] Audio data preprocessing method

[1582] The server uses audio processing libraries such as Librosa to denoise and normalize the volume of the audio data, making it clearer and easier to understand.

[1583] Generative AI methods

[1584] The server generates operation instructions based on the text data using a generative artificial intelligence model (e.g., OpenAI's GPT-4). This generative artificial intelligence model has been trained in advance on a variety of operation scenarios to understand the user's intentions and generate appropriate operation instructions.

[1585] Executing an operation

[1586] The device executes the operation instructions received from the server, for example, opening an email application and sending an email with the specified content.

[1587] Specific examples

[1588] Example 1: Sending email by voice input

[1589] The user speaks "send email."

[1590] 1. The server uses speech recognition software to convert speech to text.

[1591] 2. The server uses Librosa to denoise and normalize the volume of the audio data.

[1592] 3. The server uses a generative artificial intelligence model to generate the operation instruction "send an email."

[1593] 4. The device will open the email app and send an email with the specified content.

[1594] Example 2: Getting weather information by voice input

[1595] The user speaks, "What's the weather like today?"

[1596] 1. The server uses speech recognition software to convert speech to text.

[1597] 2. The server uses Librosa to denoise and normalize the volume of the audio data.

[1598] 3. The server uses a generative artificial intelligence model to generate an operational instruction to "obtain weather information."

[1599] 4. The device will open the weather app and display the current weather information.

[1600] Prompt Sentence Examples

[1601] "Design a system that recognizes voice input spoken by an elderly person in Tsugaru dialect and generates appropriate actions. Specifically, explain how you combine speech recognition software with a generative artificial intelligence model to generate actions based on the voice input."

[1602] In this way, the system can improve the convenience of elderly people when operating information technology devices using voice input.

[1603] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1604] Step 1:

[1605] The user speaks to the terminal, for example, "send an email." This voice input becomes the initial input for the system.

[1606] Step 2:

[1607] The device sends the captured audio data to the server, which uses an audio processing library such as Librosa to remove noise and normalize the volume of the audio data, making the audio clearer and easier to understand. The input is raw audio data, and the output is preprocessed audio data.

[1608] Step 3:

[1609] The server sends the preprocessed speech data to speech recognition software (e.g., Google Cloud Speech-to-Text API) to convert the speech to text. The input is the preprocessed speech data, and the output is text data. For example, the generated text is "send me an email."

[1610] Step 4:

[1611] The server uses a generative artificial intelligence model (e.g., OpenAI's GPT-4) to generate operational instructions based on the text data. The input is text data obtained by speech recognition, and the output is a specific operational instruction. For example, the generated operational instruction is "send an email."

[1612] Step 5:

[1613] The device executes the operation instructions received from the server. For example, it opens an email application and sends an email with the specified content. The input is the operation instruction from the server, and the output is the actual operation result.

[1614] In this way, the system can improve the convenience of seniors when operating information technology devices using voice input.

[1615] (Application example 1)

[1616] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1617] When elderly people search for products or ask questions in physical stores, they face challenges such as difficulty in operating IT devices and difficulty in obtaining appropriate information. Furthermore, there is a lack of voice recognition systems that can handle all dialects in Japan, and there is a particular need to support difficult dialects.

[1618] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1619] In this invention, the server includes a voice recognition unit that recognizes voice inputs made by an elderly person when operating a smartphone, a unit that identifies the dialect spoken by the elderly person based on the recognized voice input, a prompt sentence that generates operation instructions for the smartphone based on the recognized voice input and the identified dialect, a unit that generates operation instructions for the smartphone using a generative AI, and a unit that operates the smartphone in accordance with the generated operation instructions for the smartphone. The voice recognition unit further recognizes voice inputs made by the elderly person when asking a question about a product in a physical store, and the server further includes a prompt sentence that generates an answer to the question based on the recognized voice input and the identified dialect, and a unit that generates an answer to the question using the generative AI. This enables a system that allows elderly people to easily operate a physical store using voice input when searching for products or asking questions.

[1620] "Speech recognition means" is a technology for converting speech into text.

[1621] "Generative AI means" is an artificial intelligence technology that generates appropriate responses and operations based on input data.

[1622] "Learning methods to support all dialects of Japan" refers to technology that learns the data necessary to understand and recognize dialects from all over Japan.

[1623] "Means for learning Tsugaru dialect" refers to technology that learns the data necessary to understand and recognize Tsugaru dialect, which is said to be the most difficult dialect in Japan.

[1624] "Means for practical application of speech recognition" refers to the technology for applying speech recognition technology to actual applications and systems.

[1625] "A means for recognizing voice input when elderly people search for products or ask questions in physical stores, and generating appropriate operations or responses" is a technology that recognizes the voices of elderly people when they search for products or ask questions using voice input in physical stores, and generates appropriate operations or responses.

[1626] The "application means installed on the smartphone" is software that is installed on the smartphone and provides a specific function.

[1627] The system for implementing this invention includes a speech recognition means, a generative AI means, a means for identifying all Japanese dialects, a means for learning Tsugaru dialect, a means for putting speech recognition into practical use, a means for recognizing voice input when an elderly person searches for a product or asks a question in a physical store and generating appropriate operations or responses, and an application means to be installed on a smartphone.

[1628] System Program

[1629] This system operates using the following hardware and software:

[1630] Hardware: Smartphone with microphone

[1631] Software: Python, SpeechRecognition library, Transformers library (Hugging Face's GPT-3 (registered trademark) model)

[1632] Processing flow

[1633] Voice Recognition

[1634] When a user speaks into the smartphone's microphone, the speech recognizer picks up the speech and converts it into text, a process that uses the SpeechRecognition library.

[1635] Generative AI

[1636] The speech data converted to text by the speech recognition tool is sent to the generative AI, which uses the Transformers library to generate appropriate actions and responses using the GPT-3 model.

[1637] output

[1638] The generated operations and answers are displayed on the smartphone screen, allowing seniors to easily use voice input when searching for products or asking questions in physical stores.

[1639] Specific example (smartphone operation)

[1640] When an elderly person says "send email" to their smartphone, the application recognizes the voice, and the generative AI identifies that it is in the Tsugaru dialect. It then generates instructions to launch the email sending app, which then launches the app according to those instructions. The generative AI also generates a response such as "Please enter the message you want to send," and the application displays the response.

[1641] Prompt Sentence Examples

[1642] "An elderly person is saying 'Send email' by voice. This is in Tsugaru dialect. Please generate instructions for smartphone operation."

[1643] In this way, a system can be created that helps elderly people operate smartphones more easily.

[1644] Specific example (physical store)

[1645] When an elderly person speaks to their smartphone in a physical store and asks, "Where is product A?", the application recognizes the speech, and the generative AI identifies that it is in Tsugaru dialect. It then generates and displays a specific answer, such as, "Product A is on the left side of aisle 3."

[1646] Prompt Sentence Examples

[1647] "An elderly person is typing, 'Where is product A?' This is in Tsugaru dialect. Please generate an answer to this question."

[1648] Generative AI answer: The item is on the left side of aisle 3.

[1649] In this way, a system can be created that helps seniors to shop more easily in brick-and-mortar stores.

[1650] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1651] Step 1:

[1652] The user speaks into the smartphone's microphone. The input is the user's voice, and the output is voice data. A specific example of this behavior is when the user says, "Where is this item?"

[1653] Step 2:

[1654] The device uses a speech recognition means to convert the voice data into text. The input is voice data and the output is text data. Specifically, the SpeechRecognition library is used to convert the voice into the text "Where is product A?"

[1655] Step 3:

[1656] The terminal sends text data to the generative AI. The input is text data, and the output is the input data to the generative AI. The specific operation is to send the converted text "Where is product A?" to the generative AI.

[1657] Step 4:

[1658] The server uses generative AI to generate appropriate operation instructions and answers based on the text data. The input is text data, and the output is the generated operation instructions and answers. Specifically, the GPT-3 model uses the Transformers library to generate the answer "Product A is located on the left side of aisle 3."

[1659] Step 5:

[1660] The server sends the generated operation instructions and answers to the terminal. The input is the generated operation instructions and answers, and the output is the data sent to the terminal. A specific operation is to send the generated answer "Product A is on the left side of aisle 3" to the terminal.

[1661] Step 6:

[1662] The device operates according to the generated operation instructions and displays the generated answer to the user. The input is the transmitted data, and the output is the data displayed to the user. A specific operation is to display the answer "Product A is located on the left side of aisle 3" on the smartphone screen.

[1663] It is also possible to use an emotion engine, which will be described later, to recognize the emotional state of the elderly person and generate smartphone operation instructions and answers to questions by further taking the elderly person's emotional state into consideration. In this case, the server further includes means for recognizing the elderly person's emotional state, and the generating means generates the smartphone operation instructions using a prompt sentence that generates the smartphone operation instructions based on the recognized voice input, the identified dialect, and the recognized emotional state of the elderly person, and the generative AI.

[1664] Example 2

[1665] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1666] When elderly people operate IT devices, they face the challenge of accurately recognizing speech input, especially when it is in a dialect, and generating appropriate operations. Furthermore, due to a lack of speech recognition technology capable of handling certain dialects, such as the Tsugaru dialect, there is a need for a system that can accurately handle speech input in these dialects.

[1667] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1668] In this invention, the server includes a speech recognition unit, a generative artificial intelligence unit, a dialect adaptation learning unit, a specific dialect learning unit, a speech recognition implementation unit, a unit for converting speech input into text data, a unit for transmitting the text data to the generative artificial intelligence model, a unit for receiving the generated operation, and a unit for controlling the infrared signal generator. This makes it possible to accurately recognize speech and generate appropriate operations even when an elderly person operates information technology equipment using a dialect.

[1669] "Speech recognition means" refers to a device or software that analyzes voice input and converts it into text data.

[1670] "Generative artificial intelligence means" refers to an artificial intelligence model that generates appropriate operations and responses based on input data.

[1671] The "dialect-compatible learning means" is a means for training a speech recognition model to accommodate dialects of different regions.

[1672] A "specific dialect learning means" is a means for training a speech recognition model specializing in a specific dialect, for example, a difficult dialect such as Tsugaru dialect.

[1673] "Means for practical application of speech recognition" refers to the means for incorporating speech recognition technology into actual systems and devices and making it usable.

[1674] The "means for converting speech input into text data" refers to a means for converting speech input into text format using a speech recognition means.

[1675] The "means for transmitting text data to a generative artificial intelligence model" refers to a means for transmitting converted text data to a generative artificial intelligence model.

[1676] The "means for receiving a generated operation" is a means for receiving an operation generated from a generative artificial intelligence model.

[1677] "Means for controlling an infrared signal generating device" refers to means for controlling a device that generates an infrared signal and for performing a specific operation.

[1678] The present invention provides a system for accurately recognizing speech and generating appropriate operations when an elderly person operates information technology equipment using a dialect. Specific embodiments of this system are described below.

[1679] First, the user says "Turn on the TV" in Tsugaru dialect. The microphone on the device captures this voice. Speech recognition software (for example, a speech recognition API) is used as the speech recognition method. This software analyzes the captured voice and converts it into text data.

[1680] The device then sends the converted text data to a generative artificial intelligence model (e.g., a generative AI model). This generative AI model generates appropriate actions based on the input text data. Specifically, the following prompt sentence is generated:

[1681] Example prompt sentence:

[1682] The user says "Turn on the TV" in Tsugaru dialect. Generate an operation that generates an infrared signal to turn on the TV.

[1683] Based on this prompt, the server uses a generative artificial intelligence model to generate a specific command to turn on the TV, and the generated command is sent to the device.

[1684] Based on the received operation, the terminal controls an infrared signal generator (e.g., an infrared signal generator), which emits an infrared signal toward the television, which receives the signal and turns on.

[1685] For example, if a user says "Turn on the TV" in Tsugaru dialect, the device's microphone will capture this speech and convert it into text using a speech recognition API. This text data will be sent to the generative AI model, which will generate the following prompt:

[1686] Example prompt sentence:

[1687] The user says "Turn on the TV" in Tsugaru dialect. Generate an operation that generates an infrared signal to turn on the TV.

[1688] Based on this prompt, the server uses a generative AI model to generate specific commands to turn on the TV and sends them to the device, which then controls an infrared signal generator, which turns on the TV when it receives the infrared signal.

[1689] In this way, even when a user operates information technology equipment using a dialect, it is possible to accurately recognize speech and generate appropriate operations.

[1690] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1691] Step 1:

[1692] The user provides voice input.

[1693] A user says "Turn on the TV" in Tsugaru dialect. The input is the user's voice, and the output is the voice data captured by the device's microphone.

[1694] Step 2:

[1695] The device recognizes the voice.

[1696] The microphone on the device captures the user's voice. A speech recognition API is used as the speech recognition method. The input is the captured voice data, and the output is analyzed text data.

[1697] Step 3:

[1698] The device converts the recognized speech into text.

[1699] The device uses a speech recognition API to convert the captured voice into text data. The input is voice data and the output is text data.

[1700] Step 4:

[1701] The device sends the text to a generative AI model.

[1702] The terminal sends the converted text data to the generative AI model. The input is the text data, and the output is the data sent to the generative AI model.

[1703] Step 5:

[1704] The server generates operations using a generative AI model.

[1705] The server uses a generative AI model to generate appropriate actions based on the received text data. Specifically, the following prompt sentence is generated:

[1706] Example prompt sentence:

[1707] The user says "Turn on the TV" in Tsugaru dialect. Generate an operation that generates an infrared signal to turn on the TV.

[1708] The input is text data and the output is the generated operational data.

[1709] Step 6:

[1710] The server sends the generated operation to the terminal.

[1711] The server sends the generated operation to the terminal. The input is the generated operation data, and the output is the operation data sent to the terminal.

[1712] Step 7:

[1713] The terminal controls the infrared signal generator.

[1714] The terminal controls the infrared signal generator based on the received operation. The input is the operation data, and the output is the infrared signal.

[1715] Step 8:

[1716] The TV operates by receiving infrared signals.

[1717] The infrared signal generator emits an infrared signal toward the television, which receives the signal and turns on. The input is the infrared signal, and the output is the operation of the television.

[1718] (Application example 2)

[1719] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1720] In modern brick-and-mortar stores, customers who speak dialects have difficulty receiving product information through voice. Particularly for elderly customers and customers in areas where dialects are spoken, speech recognition systems that only understand standard Japanese cannot provide sufficient service. For this reason, there is a demand for a system that combines dialect-compatible speech recognition with generative AI.

[1721] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1722] In this invention, the server includes a speech recognition unit, a generative AI unit, and a learning unit for supporting various dialects, which allows customers who speak different dialects to receive product information by voice.

[1723] "Speech recognition means" is a technology for converting speech into text.

[1724] "Generative AI means" is an artificial intelligence technology that generates appropriate responses and operations based on input data.

[1725] A "dialect-specific learning method" is a learning algorithm for understanding and recognizing dialects from different regions.

[1726] A "means of learning difficult dialects" is a special learning process for recognizing dialects that are particularly difficult to understand.

[1727] "Means for practical application of speech recognition" refers to means for making speech recognition technology usable in actual applications.

[1728] The "guide means for providing product information within the store" is a system for providing product locations and information to customers within the store.

[1729] "Application means installed on a smart device" refers to software installed on a device such as a smartphone or tablet.

[1730] In order to implement this invention, it is necessary to construct a system that includes a speech recognition means, a generative AI means, a learning means for handling all dialects, a means for learning difficult dialects, a means for putting speech recognition into practical use, a guidance means for providing product information in a store, and an application means to be installed on a smart device.

[1731] System configuration

[1732] 1. Voice recognition means:

[1733] The speech recognition means is used to convert the user's speech into text. Specifically, the smartphone's microphone is used to capture the speech, and the SpeechRecognition library is used to convert the speech into text.

[1734] 2. Generative AI means:

[1735] The generative AI mechanism generates appropriate responses and actions based on the text captured by the speech recognition mechanism, specifically leveraging the GPT-3 model using the Hugging Face transformers library.

[1736] 3. Learning tools for all dialects:

[1737] The means includes a learning algorithm for understanding and recognizing different regional dialects, and a special learning process is implemented to provide a means for learning particularly difficult dialects.

[1738] 4. Practical application of speech recognition:

[1739] It is a means to make speech recognition technology usable in actual applications, including an interface that allows generative AI means to generate appropriate responses based on speech recognition results and provide them to users.

[1740] 5. Guidance methods for in-store product guidance:

[1741] This is a system that provides customers with product locations and information in stores. Specifically, it displays a map showing product locations on the smartphone screen and provides audio guidance.

[1742] 6. Application means installed on smart devices:

[1743] This software is installed on devices such as smartphones and tablets. This application integrates speech recognition and generative AI, allowing users to receive product information by voice.

[1744] Processing flow

[1745] 1. Acquiring voice input:

[1746] When a user speaks into the smartphone, the voice recognition unit picks up the voice. For example, the user might say, "Look for this product."

[1747] 2. Speech Recognition:

[1748] The speech recognition means converts the speech captured into text. The SpeechRecognition library is used to convert speech into text.

[1749] 3. Response generation by generative AI:

[1750] A generative AI method uses the converted text as a prompt to generate an appropriate response, specifically using the Hugging Face GPT-3 model to generate the response.

[1751] 4. Providing a response:

[1752] The generated response is displayed on the smartphone screen or given as a voice prompt, for example, "This product is on the shelf at the back right of the store."

[1753] Specific examples

[1754] User says: "Find this product"

[1755] Generated response: "This item is on the back right shelf of the store."

[1756] Prompt Sentence Examples

[1757] "If someone says 'Look for this product' in Tsugaru dialect, how can I guide them to the product in the store?"

[1758] In this way, combining dialect-aware speech recognition with generative AI can significantly improve customer service in brick-and-mortar stores.

[1759] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1760] Step 1:

[1761] The user speaks into the smartphone.

[1762] Input: User speech (e.g., "Look for this product")

[1763] How it works: The smartphone microphone picks up the audio.

[1764] Output: Captured audio data

[1765] Step 2:

[1766] The terminal uses a voice recognition means to convert the captured voice data into text.

[1767] Input: Audio data

[1768] What it does: Converts speech to text using the SpeechRecognition library.

[1769] Output: The converted text (e.g. "Look for this product")

[1770] Step 3:

[1771] The device uses generative AI means to generate a response based on the converted text.

[1772] Input: Translated text

[1773] How it works: It uses Hugging Face's GPT-3 model to generate prompt sentences and generate appropriate responses.

[1774] Output: Generated response text (e.g., "This item is on the back right shelf in the store.")

[1775] Step 4:

[1776] The terminal provides the generated response to the user.

[1777] Input: Generated response text

[1778] What it does: Displays a text response on the phone screen or plays an audio response.

[1779] Output: The user confirms the response.

[1780] Step 5:

[1781] The user follows the instructions to find the product.

[1782] Input: Response text or voice prompts

[1783] Action: The user navigates the store and follows the instructions to find a product.

[1784] Output: The user finds the product they are looking for.

[1785] In this way, specific actions and data processing and calculations are performed at each step, enabling customer service in physical stores to be realized using dialect-compatible voice recognition and generative AI.

[1786] Example 3

[1787] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1788] It is difficult to realize a speech recognition system that can handle all dialects in Japan, especially difficult dialects. In particular, it is difficult for conventional speech recognition technology to handle Tsugaru dialect, which is said to be the most difficult dialect in Japan. Another challenge is accurately recognizing voice input and generating appropriate responses when elderly people operate information technology devices themselves.

[1789] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1790] In this invention, the server includes a speech recognition unit, a generative artificial intelligence unit, and a learning unit for supporting all Japanese dialects. This enables a system including a means for a user to input speech into a terminal, a means for the terminal to transmit speech data to the server, a means for the server to convert the speech data into text data using speech recognition software, a means for the server to input the text data into a generative artificial intelligence model, a means for the generative artificial intelligence model to generate an appropriate response, a means for the server to transmit the generated response to the terminal, and a means for the terminal to display the response to the user. This enables speech recognition and response generation compatible with all Japanese dialects, particularly difficult dialects such as Tsugaru dialect. It also enables accurate recognition of speech input and generation of appropriate responses when elderly people operate information technology devices themselves.

[1791] "Speech recognition means" is a technology for converting voice data into text data.

[1792] "Generative AI means" refers to an AI technology for generating appropriate responses based on input text data.

[1793] A "learning tool" is a method for training a system with specific data to improve its ability to perform a specific task.

[1794] The "means for learning Tsugaru dialect" is a method for training speech recognition means and generative artificial intelligence means using a large amount of Tsugaru dialect speech data.

[1795] "Means for practical application of speech recognition" refers to methods for applying speech recognition technology to actual applications and systems.

[1796] "Means for users to input voice into a terminal" refers to a method by which a user inputs voice into a device such as a smartphone or PC.

[1797] The "means for the terminal to transmit audio data to the server" refers to a method for transmitting audio data captured by the terminal to the server via the Internet.

[1798] "Means for the server to convert voice data into text data using voice recognition software" refers to a method in which the server converts received voice data into text data using voice recognition software.

[1799] "Means for the server to input text data into the generative artificial intelligence model" refers to a method in which the server inputs converted text data into the generative artificial intelligence model.

[1800] The "means by which a generative artificial intelligence model generates an appropriate response" is a method by which a generative artificial intelligence model generates an appropriate response based on input text data.

[1801] "Means for the server to transmit the generated response to the terminal" refers to a method by which the server transmits the response returned from the generative artificial intelligence model to the terminal.

[1802] The "means by which the terminal displays the response to the user" refers to the method by which the terminal displays the response received from the server to the user.

[1803] This invention provides a speech recognition system that supports all Japanese dialects. In particular, it has a learning means for supporting Tsugaru dialect, which is said to be the most difficult dialect in Japan. This system includes a speech recognition means, a generative artificial intelligence means, a learning means, a speech recognition implementation means, a means for a user to input speech into a terminal, a means by the terminal transmitting speech data to a server, a means by the server converting the speech data into text data using speech recognition software, a means by the server inputting the text data into a generative artificial intelligence model, a means by the generative artificial intelligence model generating an appropriate response, a means by the server transmitting the generated response to a terminal, and a means by the terminal displaying the response to a user.

[1804] The server first collects a large amount of Tsugaru dialect speech data. This speech data is converted into text data using speech recognition software (e.g., Google Speech-to-Text API). The converted text data is then used to train a generative artificial intelligence model (e.g., OpenAI's GPT-4).

[1805] The device transmits the voice data entered by the user to the server in real time. The server converts the received voice data into text data using speech recognition software, and inputs the text data into a generative AI model. The generative AI model generates an appropriate response based on the input text data and returns the response as text data to the device.

[1806] The user receives the generated response through the terminal and, if necessary, re-enters the voice input. By repeating this process, the user can have a natural conversation that corresponds to the dialect.

[1807] As a concrete example, consider the case where a user says "How is it today?" in Tsugaru dialect. The device sends this voice data to the server, which uses speech recognition software to convert it into text data such as "How is it today?". This text data is then input into a generative artificial intelligence model, which generates an appropriate response. For example, the response generated might be "It's a nice day today." This response is then sent back to the device as text data and displayed to the user.

[1808] An example of a prompt might be:

[1809] User input: "How's your day going?"

[1810] Prompt to generative AI model: "The user says, 'How is today?' in Tsugaru dialect. Please generate an appropriate response."

[1811] In this way, the system provides a learning means for dealing with all dialects in Japan, enabling the user to have natural conversations. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1812] Step 1:

[1813] The user inputs voice into the terminal.

[1814] The user speaks into a device such as a smartphone or PC. For example, they say, "How is it today?" in Tsugaru dialect. This speech is captured by the device's microphone. The input is the user's voice data, and the output is the voice data captured by the device.

[1815] Step 2:

[1816] The terminal transmits the voice data to the server.

[1817] The device compresses the captured audio data and sends it to a server via the Internet. Specifically, the device compresses the audio data into MP3 format and sends it to the server using the HTTPS protocol. The input is the captured audio data, and the output is the audio data sent to the server.

[1818] Step 3:

[1819] The server converts the voice data into text data using voice recognition software.

[1820] The server sends the received voice data to speech recognition software (for example, Google Speech-to-Text API) and converts it into text data. For example, voice data such as "How is it today?" is converted into text data such as "How is it today?". The input is the voice data sent to the server, and the output is the converted text data.

[1821] Step 4:

[1822] The server inputs the text data into a generative artificial intelligence model.

[1823] The server sends the converted text data to a generative AI model (e.g., OpenAI's GPT-4) as an API request. For example, send the text data "How is it today?" to the generative AI model. The input is the converted text data, and the output is an API request to the generative AI model.

[1824] Step 5:

[1825] A generative artificial intelligence model generates appropriate responses.

[1826] A generative AI model generates an appropriate response based on input text data. For example, a response such as "It's nice weather today" may be generated. This response is returned to the server as text data. The input is an API request to the generative AI model, and the output is the generated response text data.

[1827] Step 6:

[1828] The server sends the generated response to the terminal.

[1829] The server receives the response text data returned from the generative AI model and sends it to the device as an API response. For example, the text data "It's nice weather today" is sent to the device. The input is the generated response text data, and the output is the response text data sent to the device.

[1830] Step 7:

[1831] The terminal displays the response to the user.

[1832] The terminal displays the response text data received from the server to the user. For example, the text "It's a nice day today" is displayed on the screen of a smartphone. The input is the response text data sent to the terminal, and the output is the response text displayed to the user.

[1833] (Application example 3)

[1834] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1835] Conventional voice recognition systems are specialized in standard Japanese, making it difficult to handle the various dialects of Japan. Furthermore, they are unable to handle Tsugaru dialect, which is said to be the most difficult dialect in Japan, making them inconvenient for users who speak dialects. Furthermore, food delivery services have the problem of making it difficult to order in dialect, forcing users to use standard Japanese.

[1836] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes a voice recognition means, a generative AI means, a learning means for supporting all Japanese dialects, a means for training Tsugaru dialect, which is said to be the most difficult dialect in Japan, a practical voice recognition means, a means for recognizing and analyzing voice input in a dialect and generating order details, a means for transmitting the generated order details via a communication means, a means for allowing the user to confirm the order details, and a means for notifying the user that the order has been confirmed. This enables users who speak a dialect to input voice in their dialect when using a food delivery service, improving convenience.

[1837] "Speech recognition means" is a technology for converting speech into text.

[1838] "Generative AI means" is an artificial intelligence technology that generates new information and operations based on input data.

[1839] "Learning tools to support all dialects of Japan" refers to technology that collects and learns the data necessary to recognize and understand dialects from all over Japan.

[1840] "Means for learning Tsugaru dialect" refers to a technology for collecting and learning the data necessary to recognize and understand Tsugaru dialect, which is said to be the most difficult dialect in Japan.

[1841] "Means for practical application of speech recognition" refers to technologies that make speech recognition technology usable in actual applications and services.

[1842] "Means for recognizing and analyzing speech input in a dialect and generating order details" refers to technology that recognizes speech input in a dialect, analyzes the content, and generates appropriate order details.

[1843] The "means for transmitting the generated order details via a communication means" refers to a technique for transmitting the generated order details using the Internet or other communication means.

[1844] The "means for having the user confirm the order details" is a technique for presenting the generated order details to the user and requesting their confirmation.

[1845] "Means for notifying that an order has been confirmed" refers to a technique for notifying a user that an order has been confirmed.

[1846] The following system configuration will be described as an embodiment of the present invention.

[1847] System Configuration

[1848] This system includes a speech recognition means, a generative AI means, a learning means to support all Japanese dialects, a means for learning Tsugaru dialect, a means for putting speech recognition into practical use, a means for recognizing and analyzing speech input in a dialect and generating order details, a means for transmitting the generated order details via a communication means, a means for allowing the user to confirm the order details, and a means for notifying that the order has been confirmed.

[1849] Hardware and software used

[1850] Hardware: Smartphone (iOS or Android)

[1851] software:

[1852] Speech recognition API (e.g. Google Cloud Speech-to-Text)

[1853] Generative AI models (e.g., OpenAI GPT-4)

[1854] Food delivery API (e.g. Uber Eats API)

[1855] Processing flow

[1856] 1. Voice input: The user speaks their order in their local dialect into the microphone on their smartphone.

[1857] 2. Speech recognition: The smartphone uses a speech recognition API to convert the input speech into text, using a dialect-aware speech recognition model.

[1858] 3. Text analysis: The server uses a generative AI model to analyze the recognized text and understand the order.

[1859] 4. Order Generation: Based on the analysis results, the server generates an appropriate order using the food delivery API and sends it to the restaurant.

[1860] 5. Confirmation and Notification: The smartphone will prompt the user to confirm the order and notify them that the order has been placed.

[1861] Specific examples

[1862] When a user says "I'd like to order one pizza" in Tsugaru dialect, the smartphone recognizes this and generates an appropriate pizza order and sends it to the restaurant.

[1863] Prompt Sentence Examples

[1864] If a user says "I'd like a pizza" in Tsugaru dialect, use a speech recognition API to convert the speech to text, use a generative AI model to analyze the order, and use a food delivery API to generate a pizza order.

[1865] In this way, the dialect-enabled food delivery assistant allows users to place orders in their dialect, providing convenient services to more users.

[1866] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1867] Step 1:

[1868] The user speaks their order in their local dialect into the smartphone microphone. The input is the user's voice, and the output is the voice data input into the smartphone microphone.

[1869] Step 2:

[1870] The device uses a speech recognition API (e.g., Google Cloud Speech-to-Text) to convert the input voice data into text. The input is voice data, and the output is text data. In this case, a dialect-compatible speech recognition model is used.

[1871] Step 3:

[1872] The server uses a generative AI model (e.g., OpenAI GPT-4) to analyze the text data sent from the speech recognition API and understand the order details. The input is text data, and the output is the analyzed order details data.

[1873] Step 4:

[1874] Based on the analysis results, the server generates an appropriate order using a food delivery API (e.g., Uber Eats API) and sends it to the restaurant. The input is the parsed order data, and the output is the order data sent through the food delivery API.

[1875] Step 5:

[1876] The terminal displays the generated order details to the user to confirm the order details. The input is the order confirmation data from the food delivery API, and the output is the order details displayed on the smartphone screen.

[1877] Step 6:

[1878] The user confirms the order details and confirms it. The input is the user's confirmation operation, and the output is a signal that the order has been confirmed.

[1879] Step 7:

[1880] The terminal notifies the server that the order has been confirmed, and the server sends that information to the food delivery API. The input is the order confirmation signal, and the output is the order confirmation data sent to the food delivery API.

[1881] Step 8:

[1882] The server notifies the user that the order has been placed. The input is the order confirmation data from the food delivery API, and the output is a confirmation of the order that is displayed to the user via the smartphone's notification function.

[1883] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1884] "Example 1"

[1885] In one embodiment of the present invention, a system is provided that includes a speech recognition means, a generative AI means, a learning means for supporting all Japanese dialects, a means for learning Tsugaru dialect, a means for implementing speech recognition, and an emotion engine that recognizes a user's emotion. This emotion engine recognizes the emotion from the user's voice and generates an operation corresponding to that emotion. For example, when a user vocally inputs "Turn on the TV," the emotion engine recognizes the user's emotion from the voice. If the emotion is recognized as "joy," the generative AI means generates an operation to turn on the TV.

[1886] "Example 2"

[1887] In another embodiment of the present invention, the emotion engine generates an operation according to the user's emotion. For example, when the user makes a voice input such as "Turn on the radio," the emotion engine recognizes the user's emotion from the voice. If the emotion is recognized as "sadness," the generative AI means generates an operation to turn on the radio, and provides a result in which the operation is adapted to the user's emotion. Specifically, the generative AI means generates an operation such as setting the radio to a channel that plays sad songs that the user likes.

[1888] "Example 3"

[1889] Furthermore, in another embodiment of the present invention, the emotion engine recognizes the user's emotion, and the generative AI means generates an operation according to that emotion. For example, when the user makes a voice input such as "Turn on the air conditioner," the emotion engine recognizes the user's emotion from the voice. If the emotion is recognized as "anger," the generative AI means generates an operation to turn on the air conditioner, providing a result where the operation is adapted to the user's emotion. Specifically, the generative AI means generates an operation such as setting the air conditioner temperature to a low temperature that will make the user feel calm.

[1890] The processing flow of each embodiment will be described below.

[1891] "Example 1"

[1892] Step 1: The user gives a voice input. In this example, the voice input is "Turn on the TV."

[1893] Step 2: The emotion engine recognizes the emotion from the user's voice. In this example, it recognizes "joy."

[1894] Step 3: The generative AI generates an action based on the recognized emotion. In this example, it generates the action to turn on the TV.

[1895] "Example 2"

[1896] Step 1: The user gives a voice input. In this example, the voice input is "Turn on the radio."

[1897] Step 2: The emotion engine recognizes the emotion from the user's voice. In this example, it recognizes "sadness."

[1898] Step 3: The generative AI generates an operation based on the recognized emotion, and provides a result that adapts to the user's emotion. In this example, it generates an operation to turn on the radio, and provides a result that adapts to the user's emotion. Specifically, it generates an operation such as setting the radio channel to a channel that plays sad songs that the user likes.

[1899] "Example 3"

[1900] Step 1: The user gives a voice input. In this example, the user gives the voice input "Turn on the air conditioner."

[1901] Step 2: The emotion engine recognizes the emotion from the user's voice. In this example, it recognizes "anger."

[1902] Step 3: The generative AI generates an operation based on the recognized emotion, and provides a result that adapts the operation to the user's emotion. In this example, it generates an operation to turn on the air conditioner, and provides a result that adapts the operation to the user's emotion. Specifically, it generates an operation such as setting the air conditioner temperature to a low temperature that will make the user feel calm.

[1903] Example 1

[1904] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1905] When elderly people operate IT devices, it is difficult to accurately recognize voice input and generate appropriate operations. In particular, there is a need to support all Japanese dialects, but the diversity and difficulty of dialects reduces the accuracy of voice recognition. In addition, there is also the problem that it is not possible to generate operations that take the user's emotions into account, which results in a poor user experience.

[1906] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1907] In this invention, the server includes a speech recognition unit, a generative artificial intelligence unit, and a learning unit for supporting all Japanese dialects. This allows for accurate recognition of speech input when elderly people operate information technology devices, and enables dialect-compatible speech recognition. Furthermore, by including an emotion engine that recognizes the user's emotions, it becomes possible to generate operations according to the user's emotions, improving the user experience.

[1908] "Speech recognition means" is a technology for converting voice input into text data.

[1909] "Generative artificial intelligence means" refers to artificial intelligence technology for generating appropriate operations based on the results of speech recognition.

[1910] "Learning method to support all Japanese dialects" is a technology that trains a speech recognition model to recognize the various dialects used in Japan.

[1911] "Dialect training" refers to the technique of specifically training a speech recognition model to recognize a particular dialect, especially a difficult dialect.

[1912] "Means for practical application of speech recognition" refers to the technology for applying speech recognition technology to actual applications and systems.

[1913] An "emotion engine" is a technology that analyzes emotions from a user's voice and generates appropriate operations based on those emotions.

[1914] MODE FOR CARRYING OUT THE INVENTION

[1915] This invention is a system that includes a speech recognition means, a generative artificial intelligence means, a learning means for dealing with all dialects of Japan, a means for learning particularly difficult dialects, a means for putting speech recognition into practical use, and an emotion engine. A specific embodiment of this system will be described below.

[1916] Acquiring voice input

[1917] Subject: User

[1918] The user speaks into the device, for example, to give instructions such as "Turn on the TV." The device is equipped with a high-performance microphone that picks up the user's voice with high accuracy.

[1919] Sending audio data

[1920] Subject: Terminal

[1921] The device transmits the acquired voice data to the server, where it is converted into an appropriate format and sent to the server via the network.

[1922] Voice Recognition

[1923] Subject: Server

[1924] The server converts the received voice data into text data using a voice recognition means. Specifically, it converts the voice into text using voice recognition software (for example, Google Cloud Speech-to-Text API).

[1925] Dialect-aware learning

[1926] Subject: Server

[1927] The server has training tools to handle all dialects in Japan, and in particular trains speech recognition models to recognize difficult dialects (e.g., Tsugaru dialect).

[1928] emotion recognition

[1929] Subject: Terminal

[1930] The device has an emotion engine that recognizes emotions from the user's voice. Specific...

Claims

1. A voice recognition means for recognizing voice input when an elderly person operates a smartphone by himself / herself; means for identifying a dialect spoken by the senior citizen based on the recognized speech input; A means for generating an operation instruction for the smartphone using a prompt sentence and a generative AI, the prompt sentence generating an operation instruction for the smartphone based on the recognized voice input and the identified dialect; A means for operating the smartphone according to the generated operation instruction for the smartphone; A system including:

2. The voice recognition means further recognizes a voice input when the elderly person asks a question about a product in a physical store, 2. The system of claim 1, further comprising: means for generating an answer to the question using a prompt and the generative AI to generate an answer to the question based on the recognized speech input and the identified dialect.

3. means for recognizing an emotional state of the senior using an emotion engine; The system of claim 1, wherein the generating means generates operation instructions for the smartphone using a prompt sentence that generates operation instructions for the smartphone based on the recognized voice input, the identified dialect, and the recognized emotional state of the elderly person, and the generation system AI.

Images