System

Abstract

A system is provided.SOLUTION: The system includes a voice recognition means, a means for storing personal data in a database, a multilingual translation means, an image recognition means, an environment detection means using an olfactory sensor, and a means for performing an operation in cooperation with another system.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, the number of elderly people and people living alone is increasing, resulting in a more serious problem of social isolation. Furthermore, existing robots have difficulty identifying individuals, engaging in natural conversations, and supporting multiple languages. Furthermore, few robots are equipped with olfactory sensors that can sensitively respond to environmental changes, and safety measures are insufficient. This has resulted in a lack of adequate daily support and interactive services, leading to a decline in quality of life. [Means for solving the problem]

[0005] The present invention uses a voice recognition means to recognize a user's voice and converts the voice data into text data. It also includes a means for storing personal data in a database and manages personal information with the user's consent. A multilingual translation means supports more than 15 languages, enabling smooth communication with the user. Furthermore, an image recognition means can identify the user's face and provide personalized support. Additionally, an environmental detection means using an olfactory sensor ensures safety and issues appropriate warnings in response to environmental changes such as unusual odors. The system also includes a means for linking with other systems, such as smartphone apps and autonomous driving systems. This provides a multifunctional next-generation robot and improves the quality of life for users.

[0006] "Speech recognition means" refers to a device or software that has the function of capturing a user's voice with a microphone and converting it into text data.

[0007] "Means for storing personal data in a database" refers to a device or software that has the function of storing personal information obtained from users in a database for secure management.

[0008] "Multilingual translation means" refers to a device or software that has the function of translating input text data into multiple languages.

[0009] "Image recognition means" refers to a device or software that has the function of identifying a user or object from image data captured by a camera.

[0010] An "environmental detection means using an olfactory sensor" is a device or software that uses a sensor to detect specific odors present in the surrounding environment and has the function of acquiring and analyzing corresponding data.

[0011] "Means for operating in cooperation with other systems" refers to devices or software that have the function of communicating with other systems, such as smartphone apps or autonomous driving systems, and performing operations in cooperation with them. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0013] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0016] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0017] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0018] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0020] [First embodiment]

[0021] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0022] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0025] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0028] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0032] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0033] An embodiment of the next-generation robot "maid-type" system of the present invention is described in detail below. This system includes a voice recognition unit, a unit for storing personal data in a database, a multilingual translation unit, an image recognition unit, an environment detection unit using an olfactory sensor, and a unit for operating in cooperation with other systems.

[0034] 1. Voice Recognition Method

[0035] User: The user speaks to the robot, for example, "Hello, my name is Taro Tanaka."

[0036] Terminal: The robot's microphone captures the voice. The captured voice is converted into text data by a voice recognition engine, and the text data "Taro Tanaka" is generated.

[0037] 2. How personal data is stored in the database

[0038] Terminal: The converted text data is sent to the server via an HTTP request, which includes metadata such as the user ID.

[0039] Server: The server analyzes the received text data and stores it in a database. For example, the user name "Taro Tanaka" and the corresponding ID are stored.

[0040] 3. Multilingual Translation Tools

[0041] Terminal: Obtains user input as text data, and if multilingual support is required, sends the text data to the server. For example, it asks, "Do you speak English?"

[0042] Server: The server receives the text data and calls the translation API to translate it. The Japanese text "Do you speak English?" is translated into English "Do you speak English?" and the translated data is sent to the device.

[0043] 4. Image Recognition Methods

[0044] User: The user faces the robot's camera.

[0045] Device: The camera captures the user's face and sends the image data to a server. The image recognition system analyzes the face and matches it with a database to identify the individual.

[0046] Server: The image recognition algorithm recognizes the user's face as "Taro Tanaka" and sends the user ID to the device.

[0047] 5. Environmental detection using olfactory sensors

[0048] Terminal: When the olfactory sensor detects an unusual odor, it sends the data to the server in real time. For example, if it detects a specific odor like a gas leak.

[0049] Server: The server analyzes the odor data it receives and identifies the cause by comparing it with known odor patterns. For example, it may identify a "gas leak" and send that data to the device.

[0050] Terminal: Based on the warning information from the server, a speech synthesis engine is used to generate a warning voice saying, "There is a possibility of a gas leak. Please ventilate immediately," and this is played from the speaker.

[0051] 6. Means of cooperation with other systems

[0052] User: Sends the command "Start cleaning" to the robot via the smartphone app.

[0053] Terminal: Sends instructions received from the app to the server.

[0054] Server: Analyzes the received instruction and sends an instruction to start cleaning mode to the terminal.

[0055] Device: The cleaning function starts, and when cleaning is completed, the information is sent to the server. The server notifies the smartphone app of the completion information, and the user receives a notification that "cleaning is completed."

[0056] As described above, the next-generation robot "maid-type" system of the present invention integrates multiple functions to support users' daily lives in many ways and improve their quality of life.

[0057] The processing flow will be explained below.

[0058] 1. Speech recognition processing

[0059] Step 1:

[0060] User: Talks to the robot and says, "Hello, my name is Taro Tanaka."

[0061] Step 2:

[0062] Device: Uses a microphone to capture the user's voice and sends the voice data to a speech recognition engine.

[0063] Step 3:

[0064] Terminal: The voice recognition engine converts the voice data into text data, generating the text data "Taro Tanaka."

[0065] 2. Processing of the means by which personal data is stored in a database

[0066] Step 1:

[0067] Terminal: Sends an HTTP request to the server containing the text data "Taro Tanaka." The request also contains metadata such as the user ID.

[0068] Step 2:

[0069] Server: Analyzes the received text data and stores it in a database along with the user ID.

[0070] Step 3:

[0071] Server: After saving is complete, it sends a response to the terminal indicating that the process was successful.

[0072] 3. Processing of multilingual translation means

[0073] Step 1:

[0074] User: Talks to the robot and asks, "Can you speak English?"

[0075] Step 2:

[0076] Terminal: Captures voice with a microphone and converts it into text data using a speech recognition engine. The converted text data is sent to the server.

[0077] Step 3:

[0078] Server: Sends the received text data to the translation API and performs the translation.

[0079] Step 4:

[0080] Server: Receives the translated text data "Do you speak English?" and sends it to the terminal.

[0081] Step 5:

[0082] Terminal: The translated text data is passed to a speech synthesis engine to generate speech data.

[0083] Step 6:

[0084] Device: Play the generated voice "Yes, I can speak English." from the speaker.

[0085] 4. Image Recognition Method Processing

[0086] Step 1:

[0087] User: Face the robot's camera.

[0088] Step 2:

[0089] Device: The camera captures the user's facial image and sends the image data to the server.

[0090] Step 3:

[0091] Server: Analyzes the received image data using an image recognition engine and identifies the user ID.

[0092] Step 4:

[0093] Server: Sends the identified user ID to the terminal.

[0094] Step 5:

[0095] Terminal: Based on the user ID received from the server, it matches it with a registered user name, generates a personalized response such as "You're Tanaka Taro, right?" and creates speech using a speech synthesis engine.

[0096] Step 6:

[0097] Device: Play the created audio "Hello, Taro Tanaka" from the speaker.

[0098] 5. Processing of environmental detection means using olfactory sensors

[0099] Step 1:

[0100] Terminal: The olfactory sensor detects strange odors in the environment.

[0101] Step 2:

[0102] Terminal: Transmits detected odor data to the server in real time.

[0103] Step 3:

[0104] Server: Analyzes the received odor data and compares it with known odor patterns.

[0105] Step 4:

[0106] Server: Identifies the cause of the strange odor and sends data such as "gas leak" to the terminal.

[0107] Step 5:

[0108] Device: Based on the received data, the speech synthesis engine generates a warning voice saying, "There is a possibility of a gas leak. Please ventilate immediately."

[0109] Step 6:

[0110] Device: Play the generated warning sound through the speaker.

[0111] 6. Processing of means for interoperating with other systems

[0112] Step 1:

[0113] User: Uses the smartphone app to send the command to the robot to "start cleaning."

[0114] Step 2:

[0115] Terminal: Sends instructions received from the app to the server.

[0116] Step 3:

[0117] Server: Analyzes the received instructions and instructs the cleaning robot to start cleaning mode.

[0118] Step 4:

[0119] Device: Follow the instructions to start cleaning mode.

[0120] Step 5:

[0121] Terminal: After cleaning is completed, the information is sent to the server.

[0122] Step 6:

[0123] Server: Receives the completion information and sends a notification to the smartphone app.

[0124] Step 7:

[0125] User: Receives a notification on the smartphone app that "cleaning is complete."

[0126] Example 1

[0127] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0128] Current robot systems use separate technologies such as voice recognition, image recognition, and environmental detection, making it difficult to integrate these pieces of information, resulting in an inconsistent user experience. Furthermore, they lack the ability to respond to odor detection in real time and provide easy user control, making them unable to effectively support users in their daily lives. Furthermore, their ability to communicate in multiple languages ​​to accommodate internationalization is insufficient, meaning more advanced support is required.

[0129] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0130] In this invention, the server includes a voice recognition unit, a unit for storing personal data in a database, a multilingual translation unit, an image recognition unit, an environment detection unit using an olfactory sensor, a unit for operating in cooperation with other systems, a unit for allowing a user to send instructions to the robot using a smartphone app and for the robot to operate based on those instructions, a unit for analyzing data in real time when an abnormal odor is detected and generating an alarm sound to notify the user, a unit for storing text data acquired by voice recognition in a database in association with a user ID, and a unit for displaying the translated text data to the user. This makes it possible to integrate various recognition technologies, environment detection, and user interfaces to respond to various user requests in real time and improve quality of life.

[0131] "Speech recognition means" refers to technology that captures the user's voice and converts it into text data.

[0132] "Means for storing personal data in a database" refers to the technology for storing text data obtained by a voice recognition means in a database in association with a user ID.

[0133] "Multilingual translation means" refers to a technology that translates text in one language into another language.

[0134] "Image recognition means" refers to technology that analyzes image data captured by a camera and recognizes specific objects (e.g., faces).

[0135] "Environmental detection means using olfactory sensors" refers to technology that uses odor-detecting sensors to detect specific odors and analyze their causes.

[0136] "Means of operating in cooperation with other systems" refers to technology that exchanges data with different systems and performs multiple operations in cooperation with them.

[0137] "A means by which a user sends instructions to a robot using a smartphone app, and the robot then performs an action based on those instructions" refers to a technology in which a user sends operating instructions to a robot via a smartphone app, and the robot performs an action in accordance with those instructions.

[0138] "Means for analyzing data in real time when an abnormal odor is detected and generating an audio warning to notify the user" refers to a technology that, when an olfactory sensor detects an abnormal odor, analyzes the data to identify the cause, generates an audio warning to notify the user in real time.

[0139] "Means of storing text data obtained by voice recognition in a database in association with a user ID" refers to a technology that stores text data obtained by voice recognition technology in a database together with the corresponding user ID.

[0140] The term "means for displaying translated text data to the user" refers to a technology for displaying text data translated by the multilingual translation means in a form that can be confirmed by the user.

[0141] The present invention relates to a next-generation robot system that supports users' daily lives. The system includes a voice recognition unit, a unit for storing personal data in a database, a multilingual translation unit, an image recognition unit, an environment detection unit using an olfactory sensor, a unit for operating in conjunction with other systems, a unit for sending instructions to the robot via a smartphone app and having the robot perform actions based on those instructions, a unit for analyzing data in real time when an abnormal odor is detected and generating an alarm sound to notify the user, a unit for storing text data acquired by voice recognition in a database in association with a user ID, and a unit for displaying the translated text data to the user.

[0142] 1. Voice Recognition Method

[0143] When a user speaks to the robot, the robot's microphone captures the voice. The voice data is converted into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text API). For example, if a user says, "Hello, my name is Tanaka Taro," the text data "Hello, my name is Tanaka Taro" is generated.

[0144] 2. How personal data is stored in the database

[0145] The converted text data is sent to the server via an HTTP request. The server analyzes the received data and stores the user name and ID in a database (e.g., MySQL). This allows the information obtained through voice recognition to be managed as personal data.

[0146] 3. Multilingual Translation Tools

[0147] When translating user input text into multiple languages, the text data is sent to the server. The server calls a translation API (e.g., Google Cloud Translation API) to obtain the translation result. For example, the text "Can you speak English?" is translated to "Do you speak English?" The translation result is displayed to the user.

[0148] 4. Image Recognition Methods

[0149] When a user points their face at the robot's camera, the camera captures their facial image. The image data is sent to a server and analyzed using an image recognition algorithm (e.g., Azure Face API). The server then matches the individual against a database and sends the user ID to the device.

[0150] 5. Environmental detection using olfactory sensors

[0151] When the robot's olfactory sensors detect an unusual odor, the data is sent to a server in real time. The server analyzes the data and identifies the source of the odor. For example, if a gas leak is detected, the server sends that information to the device. The device uses a speech synthesis engine (e.g., Amazon Polly) to generate a warning voice saying, "There is a possibility of a gas leak. Please ventilate immediately," and plays it over the speaker.

[0152] 6. Means of cooperation with other systems

[0153] The user sends a command to the robot via a smartphone app to "start cleaning." The command from the app is sent to a server, which analyzes it and then issues a command to the robot to start cleaning. Once cleaning is complete, the device sends the information back to the server, and the server sends a completion notification to the smartphone app. The user then receives a notification that "cleaning is complete."

[0154] Examples and prompts

[0155] Example 1: When a user says, "Where am I now?", the speech recognition engine generates text data of "Where am I now?", sends related information to the server, and returns the answer.

[0156] Example 2: When a user says, "What do you call this in English?", the translation result is displayed to the user as "What do you call this in English?" after going through speech recognition, translation, and display.

[0157] Example prompt for a generative AI model:

[0158] "Please translate the text into English. The original text is 'How do you say this in English?'"

[0159] "Please tell me the procedure for analyzing the speech recognition results and saving them to the database."

[0160] The system of the present invention is an integrated multi-functional robot system that responds to various user requests in real time and improves the quality of life.

[0161] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0162] 1. Voice Recognition Method

[0163] Step 1:

[0164] The user speaks to the robot.

[0165] Input: Voice data (e.g., "Hello, my name is Taro Tanaka")

[0166] Output: None

[0167] Specific action: The user speaks aloud to the robot.

[0168] Step 2:

[0169] The device captures audio using the robot's microphone.

[0170] Input: Audio data

[0171] Output: Raw audio file

[0172] What it does: The microphone collects audio data and creates an audio file in the internal memory.

[0173] Step 3:

[0174] The device's voice recognition engine converts the captured voice data into text data.

[0175] Input: raw audio file

[0176] Output: Text data (e.g. "Hello, my name is Taro Tanaka")

[0177] What happens: A speech recognition engine (e.g., Google Cloud Speech-to-Text API) analyzes the audio file and generates corresponding text data.

[0178] 2. How personal data is stored in the database

[0179] Step 1:

[0180] The terminal sends the converted text data and metadata such as the user ID to the server in the form of an HTTP request.

[0181] Input: Text data, user ID

[0182] Output: HTTP request

[0183] Specific behavior: The text data and user ID are included in the body of the HTTP request and sent to the server.

[0184] Step 2:

[0185] The server analyzes the received HTTP request and extracts the content.

[0186] Input: HTTP request

[0187] Output: User name, User ID

[0188] What happens: The server parses the body of the HTTP request and extracts the username and user ID.

[0189] Step 3:

[0190] The server stores the extracted data in a database.

[0191] Input: Username, User ID

[0192] Output: New record to the database

[0193] Specific behavior: The username and user ID are added as a new record to the database (e.g. MySQL).

[0194] 3. Multilingual Translation Tools

[0195] Step 1:

[0196] The terminal obtains the user's input text and, if multilingual support is required, sends it to the server.

[0197] Input: Text data (e.g., "Do you speak English?")

[0198] Output: HTTP request

[0199] Specific operation: The device creates and sends an HTTP request to send text data to the server.

[0200] Step 2:

[0201] The server receives the text data and calls the translation API to translate it into the specified target language.

[0202] Input: HTTP request, text data

[0203] Output: Translated text data (e.g., "Do you speak English?")

[0204] Specific operation: Send a request to a translation API (e.g., Google Cloud Translation API) and obtain the translation result.

[0205] Step 3:

[0206] The server returns the translation results to the terminal.

[0207] Input: Translated text data

[0208] Output: HTTP response

[0209] Specific operation: Generates an HTTP response containing the translation result and sends it to the device.

[0210] Step 4:

[0211] The terminal displays the translation results to the user.

[0212] Input: Translated text data

[0213] Output: The text that is displayed in the user interface

[0214] Specific operation: The translation results are displayed on the user interface (display or voice reading device).

[0215] 4. Image Recognition Methods

[0216] Step 1:

[0217] The user faces the robot's camera.

[0218] Input: Face image

[0219] Output: None

[0220] Specific action: The user faces the camera directly.

[0221] Step 2:

[0222] The device captures a facial image using the robot's camera.

[0223] Input: Face image

[0224] Output: Image file

[0225] What happens: The camera captures a facial image and generates an image file.

[0226] Step 3:

[0227] The terminal transmits the image data to the server.

[0228] Input: Image file

[0229] Output: HTTP request

[0230] Specific operation: Sends an image file to the server in the form of an HTTP request.

[0231] Step 4:

[0232] The server analyzes the data using image recognition algorithms.

[0233] Input: Image file

[0234] Output: User ID

[0235] Specific operation: Analyzes the image using an image recognition algorithm (e.g. Azure Face API) and identifies the user.

[0236] Step 5:

[0237] The server sends the user ID to the terminal.

[0238] Input: User ID

[0239] Output: HTTP response

[0240] Specific operation: Generate an HTTP response including the user ID and send it to the terminal.

[0241] 5. Environmental detection using olfactory sensors

[0242] Step 1:

[0243] The device detects strange odors using an olfactory sensor.

[0244] Input: Environmental data (odor data)

[0245] Output: Odor detection event

[0246] How it works: The olfactory sensor detects specific odors in the environment.

[0247] Step 2:

[0248] The device transmits the detection data to the server in real time.

[0249] Input: Off-odor data

[0250] Output: HTTP request

[0251] Specific operation: Odor data is sent to the server in real time in the form of an HTTP request.

[0252] Step 3:

[0253] The server analyzes the odor data and compares it with known odor patterns to identify the cause.

[0254] Input: Off-odor data

[0255] Output: Data on the cause of the odor (e.g., gas leak)

[0256] Specific operation: Odor data is compared with a database of known odor patterns to identify the cause.

[0257] Step 4:

[0258] The server sends data on the cause of the odor to the terminal.

[0259] Input: Odor source data

[0260] Output: HTTP response

[0261] Specific operation: Generate an HTTP response including data on the cause of the odor and send it to the terminal.

[0262] Step 5:

[0263] The device generates and plays an audio warning.

[0264] Input: HTTP response (odor cause data)

[0265] Output: Audio warning

[0266] Specific behavior: Generates a warning message using a speech synthesis engine (e.g., Amazon Polly) and plays it over the speaker.

[0267] 6. Means of cooperation with other systems

[0268] Step 1:

[0269] The user sends instructions to the robot via a smartphone app.

[0270] Input: Instruction data (e.g. "Start cleaning")

[0271] Output: HTTP request

[0272] Specific operation: The user operates the app and sends commands to the robot.

[0273] Step 2:

[0274] The device sends the instructions received from the app to the server.

[0275] Input: HTTP request

[0276] Output: HTTP request

[0277] Specific operation: The device receives instructions from the app and forwards them to the server.

[0278] Step 3:

[0279] The server analyzes the received instructions and sends appropriate commands to the terminal.

[0280] Input: HTTP request (instruction data)

[0281] Output: Command data

[0282] Specific operation: The server analyzes the instruction data, generates the necessary commands, and sends them to the terminal.

[0283] Step 4:

[0284] The terminal will trigger the robot to start the cleaning function.

[0285] Input: Command data (e.g. "Start cleaning")

[0286] Output: Robot movement

[0287] Specific operation: The terminal transmits commands to the robot to start the cleaning function.

[0288] Step 5:

[0289] The terminal transmits cleaning completion data to the server.

[0290] Input: Cleaning completion data

[0291] Output: HTTP request

[0292] Specific operation: When cleaning is completed, the device sends the information to the server.

[0293] Step 6:

[0294] The server notifies the app of the cleaning completion data.

[0295] Input: HTTP request (cleaning completion data)

[0296] Output: Smartphone app notification

[0297] Specific operation: The server sends cleaning completion data to the app and notifies the user.

[0298] (Application example 1)

[0299] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0300] Conventional customer service systems in brick-and-mortar stores lack multilingual support for customer service and make it difficult to provide personalized service to repeat customers. Furthermore, emergency response capabilities, such as environmental and odor detection, are inadequate. For these reasons, there is a need to build a next-generation customer service system that can support multiple languages ​​in brick-and-mortar stores, provide personalized service for each individual customer, and ensure environmental safety.

[0301] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0302] In this invention, the server includes a voice recognition means, a means for storing personal data in a database, a multilingual translation means, an image recognition means, an environment detection means using an olfactory sensor, a means for operating in cooperation with other systems, and a means for functioning as a customer service assistant in a physical store. This makes it possible to respond to customers in multiple languages ​​in a physical store, provide personalized services according to each customer, and ensure a safe environment.

[0303] "Speech recognition means" refers to technology that takes voice input and converts that voice into text data.

[0304] "Means for storing personal data in a database" refers to the technology that collects information about users and stores it in an organized form in a database.

[0305] "Multilingual translation means" refers to a technology that translates input text data into multiple different languages.

[0306] "Image recognition means" refers to technology that analyzes image data acquired through devices such as cameras and identifies specific people or objects.

[0307] "Environmental detection means using olfactory sensors" refers to technology that uses sensors that detect various odors to detect specific environmental changes or unusual odors.

[0308] "Means for operating in cooperation with other systems" refers to technology for communicating with external systems or devices and executing specific operations.

[0309] "Means to function as a customer service assistant in a physical store" refers to technology that handles customer service within a physical store and provides multilingual support and personalized service.

[0310] This invention is a system that realizes efficient customer service and multilingual support by using a next-generation robot "maid-type" system as a customer service assistant in a brick-and-mortar store. This system includes a voice recognition means, a means for storing personal data in a database, a multilingual translation means, an image recognition means, an environmental detection means using an olfactory sensor, and a means for operating in cooperation with other systems.

[0311] Voice recognition means

[0312] When a user speaks to the robot, the voice is captured by the device's microphone, and a speech recognition engine (such as Google Web Speech API) converts the voice into text data and sends it to the server.

[0313] How personal data is stored in a database

[0314] The device sends the converted text data to the server via an HTTP request, where it is analyzed along with metadata such as the user ID and stored in a database. This storage process uses a relational or NoSQL database.

[0315] Multilingual translation tools

[0316] When a user makes a multilingual inquiry to the robot, the device sends the text data to the server. The server then calls a translation API to translate the input text data into the required language. The translated text data is then returned to the device and provided to the customer.

[0317] Image Recognition Method

[0318] When a user points their face at the robot's camera, the device's camera captures their face image, which is then sent to a server where image recognition algorithms (such as OpenCV or TensorFlow) are used to analyze the face and match it with a database to identify the user.

[0319] Environmental detection using olfactory sensors

[0320] When the device's built-in olfactory sensor detects an unusual odor, it sends the data to the server in real time. The server analyzes the data and generates a warning if it matches the odor pattern. This warning is output as voice by a speech synthesis engine and notified to the user.

[0321] A means of interacting with other systems

[0322] When a user sends a command to the robot via a smartphone app, the command is sent to the device. The server analyzes the received command and instructs the device to perform the appropriate operation. For example, if a command to start cleaning is sent, the robot will enter cleaning mode, and after cleaning is completed, the information will be sent to the server and the user will be notified.

[0323] Specific examples

[0324] When a customer asks the robot, "Where is this product in stock?", the voice is converted into text by a speech recognition device. The text data is then sent to the server, and related information is retrieved from the database and provided to the customer. In addition, if multilingual support is required, the text is translated into the appropriate language by a translation API.

[0325] Example prompts

[0326] "Write a Python program that uses speech recognition to convert what a customer says to a robot into text data, and then send that data along with the customer ID to a server for storage. The program will use the Google Web Speech API for speech recognition and the requests library to send the data to the server."

[0327] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0328] Step 1:

[0329] The user speaks to the robot. The input is the user's voice information, and the device's microphone captures this voice. Next, the voice data is converted into text data using a speech recognition engine (Google Web Speech API). The output is text data.

[0330] Step 2:

[0331] The terminal sends the converted text data and metadata (e.g., user ID) to the server as an HTTP request. The input is the text data and metadata, and the sent data is saved and analyzed on the server. The output is the saved data.

[0332] Step 3:

[0333] If a user makes a query that requires multilingual support, the device sends the text data back to the server. The input is the text data of the user's query, and the server calls the translation API to translate it into the specified language. The output is the text data in the translated language.

[0334] Step 4:

[0335] The server sends the translated data to the terminal, which then converts it into speech using a speech synthesis engine and transmits it to the user through a speaker. The input is the translated text data, and the output is the audio information provided to the user.

[0336] Step 5:

[0337] When a user points their face at the robot's camera, the device's camera captures a facial image and sends the image data to the server. The input is the captured facial image, and the server identifies the user using an image recognition algorithm (such as OpenCV or TensorFlow). The output is the identified user ID.

[0338] Step 6:

[0339] When the olfactory sensor installed in the device detects an abnormal odor, it sends the data to the server in real time. The input is the abnormal odor data detected by the olfactory sensor, and the server analyzes the data by comparing it with known abnormal odor patterns. The output is the identification of the abnormal odor and a warning message.

[0340] Step 7:

[0341] When the server identifies an abnormal odor, it sends a warning message to the device, which then uses a speech synthesis engine to generate a warning voice and notify the user through the speaker.The input is a warning message about the abnormal odor, and the output is the warning voice provided to the user.

[0342] Step 8:

[0343] When a user sends a command to a robot via a smartphone app, the command is sent to the server via the terminal. The input is the command data from the smartphone app, the server analyzes the received command, and sends appropriate operation instructions to the terminal. The output is the operation result executed by the terminal.

[0344] Step 9:

[0345] For example, when receiving the instruction to "start cleaning," the device starts cleaning mode, and when cleaning is complete, it sends that information to the server. The server notifies the smartphone app of the completion information, and the user receives a notification that "cleaning is complete." The input is the completion report data, and the output is the completion notification provided to the user.

[0346] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0347] A detailed description will be given of an embodiment of a next-generation robot "maid-type" system according to the present invention. This system includes a voice recognition unit, a unit for storing personal data in a database, a multilingual translation unit, an image recognition unit, an environment detection unit using an olfactory sensor, a unit for operating in cooperation with other systems, and an emotion engine for recognizing the user's emotions.

[0348] 1. Voice Recognition Method

[0349] User: The user speaks to the robot, for example, "Hello, my name is Taro Tanaka."

[0350] Terminal: The robot's microphone captures the voice. The captured voice is converted into text data by a voice recognition engine, and the text data "Taro Tanaka" is generated.

[0351] 2. How personal data is stored in the database

[0352] Terminal: The converted text data is sent to the server via an HTTP request, which includes metadata such as the user ID.

[0353] Server: The server analyzes the received text data and stores it in a database along with the user ID. For example, the user name "Taro Tanaka" and the corresponding ID are stored.

[0354] 3. Multilingual Translation Tools

[0355] Terminal: Obtains user input as text data, and if multilingual support is required, sends the text data to the server. For example, it asks, "Do you speak English?"

[0356] Server: The server receives the text data and calls the translation API to translate it. The Japanese text "Do you speak English?" is translated into English "Do you speak English?" and the translated data is sent to the device.

[0357] 4. Image Recognition Methods

[0358] User: The user faces the robot's camera.

[0359] Device: The camera captures the user's face and sends the image data to a server. The image recognition system analyzes the face and matches it with a database to identify the individual.

[0360] Server: The image recognition algorithm recognizes the user's face as "Taro Tanaka" and sends the user ID to the device.

[0361] 5. Environmental detection using olfactory sensors

[0362] Terminal: When the olfactory sensor detects an unusual odor, it sends the data to the server in real time. For example, if it detects a specific odor like a gas leak.

[0363] Server: The server analyzes the odor data it receives and identifies the cause by comparing it with known odor patterns. For example, it may identify a "gas leak" and send that data to the device.

[0364] Terminal: Based on the warning information from the server, a speech synthesis engine is used to generate a warning voice saying, "There is a possibility of a gas leak. Please ventilate immediately," and this is played from the speaker.

[0365] 6. Means of cooperation with other systems

[0366] User: Sends the command "Start cleaning" to the robot via the smartphone app.

[0367] Terminal: Sends instructions received from the app to the server.

[0368] Server: Analyzes the received instruction and sends an instruction to start cleaning mode to the terminal.

[0369] Device: The cleaning function starts, and when cleaning is completed, the information is sent to the server. The server notifies the smartphone app of the completion information, and the user receives a notification that "cleaning is completed."

[0370] 7. Emotional Engine Means

[0371] User: Emotions are expressed through the user's tone of voice and facial expressions when speaking to the robot.

[0372] On the device: The emotion recognition engine analyzes the user's tone of voice and facial expressions to detect their emotional state. For example, it determines that the user is "angry" based on their tone of voice and facial expressions.

[0373] Device: Sends an HTTP request containing the detected emotion data to the server.

[0374] Server: The server analyzes the received emotion data, stores it in a database, and generates appropriate response data based on the emotion data.

[0375] Terminal: Based on the response data received from the server, the terminal uses a speech synthesis engine to generate an appropriate voice response, such as "Taro Tanaka, what's the matter?"

[0376] Device: The generated voice is played back through the speaker, and the dialogue is tailored to the user's emotions.

[0377] As explained above in this embodiment, the next-generation robot "maid-type" system integrates multiple functions to support users' daily lives in many ways and improve their quality of life. In particular, the implementation of an emotion engine enables flexible responses according to emotions, resulting in a more natural and friendly user experience.

[0378] The processing flow will be explained below.

[0379] 1. Speech recognition processing

[0380] Step 1:

[0381] User: Talks to the robot and says, "Hello, my name is Taro Tanaka."

[0382] Step 2:

[0383] Device: Uses a microphone to capture the user's voice and sends the voice data to a speech recognition engine.

[0384] Step 3:

[0385] Terminal: The voice recognition engine converts the voice data into text data, generating the text data "Taro Tanaka."

[0386] 2. Processing of the means by which personal data is stored in a database

[0387] Step 1:

[0388] Terminal: Sends an HTTP request to the server containing the text data "Taro Tanaka." The request also contains metadata such as the user ID.

[0389] Step 2:

[0390] Server: Analyzes the received text data and stores it in a database along with the user ID.

[0391] Step 3:

[0392] Server: After saving is complete, it sends a response to the terminal indicating that the process was successful.

[0393] 3. Processing of multilingual translation means

[0394] Step 1:

[0395] User: Talks to the robot and asks, "Can you speak English?"

[0396] Step 2:

[0397] Terminal: Captures voice with a microphone and converts it into text data using a speech recognition engine. The converted text data is sent to the server.

[0398] Step 3:

[0399] Server: Sends the received text data to the translation API and performs the translation.

[0400] Step 4:

[0401] Server: Receives the translated text data "Do you speak English?" and sends it to the terminal.

[0402] Step 5:

[0403] Terminal: The translated text data is passed to a speech synthesis engine to generate speech data.

[0404] Step 6:

[0405] Device: Play the generated voice "Yes, I can speak English." from the speaker.

[0406] 4. Image Recognition Method Processing

[0407] Step 1:

[0408] User: Face the robot's camera.

[0409] Step 2:

[0410] Device: The camera captures the user's facial image and sends the image data to the server.

[0411] Step 3:

[0412] Server: Analyzes the received image data using an image recognition engine and identifies the user ID.

[0413] Step 4:

[0414] Server: Sends the identified user ID to the terminal.

[0415] Step 5:

[0416] Terminal: Based on the user ID received from the server, it matches it with a registered user name, generates a personalized response such as "You're Tanaka Taro, right?" and creates speech using a speech synthesis engine.

[0417] Step 6:

[0418] Device: Play the created audio "Hello, Taro Tanaka" from the speaker.

[0419] 5. Processing of environmental detection means using olfactory sensors

[0420] Step 1:

[0421] Terminal: The olfactory sensor detects strange odors in the environment.

[0422] Step 2:

[0423] Terminal: Transmits detected odor data to the server in real time.

[0424] Step 3:

[0425] Server: Analyzes the received odor data and compares it with known odor patterns.

[0426] Step 4:

[0427] Server: Identifies the cause of the strange odor and sends data such as "gas leak" to the terminal.

[0428] Step 5:

[0429] Device: Based on the received data, the speech synthesis engine generates a warning voice saying, "There is a possibility of a gas leak. Please ventilate immediately."

[0430] Step 6:

[0431] Device: Play the generated warning sound through the speaker.

[0432] 6. Processing of means for interoperating with other systems

[0433] Step 1:

[0434] User: Uses the smartphone app to send the command to the robot to "start cleaning."

[0435] Step 2:

[0436] Terminal: Sends instructions received from the app to the server.

[0437] Step 3:

[0438] Server: Analyzes the received instructions and instructs the cleaning robot to start cleaning mode.

[0439] Step 4:

[0440] Device: Follow the instructions to start cleaning mode.

[0441] Step 5:

[0442] Terminal: After cleaning is completed, the information is sent to the server.

[0443] Step 6:

[0444] Server: Receives the completion information and sends a notification to the smartphone app.

[0445] Step 7:

[0446] User: Receives a notification on the smartphone app that "cleaning is complete."

[0447] 7. Processing of Emotion Engine Means

[0448] Step 1:

[0449] User: The user expresses emotions through tone of voice and facial expressions when speaking to the robot.

[0450] Step 2:

[0451] On the device: The emotion engine captures the user's tone of voice and facial expressions and analyzes their emotional state. For example, it determines that the user is "angry" based on their tone of voice and facial expressions.

[0452] Step 3:

[0453] Terminal: Transmits the analyzed emotion data to the server.

[0454] Step 4:

[0455] Server: Receives emotion data, analyzes it, stores it in a database, and generates appropriate response data based on the emotion data.

[0456] Step 5:

[0457] Server: Sends the generated response data to the terminal.

[0458] Step 6:

[0459] Terminal: Based on the response data received from the server, the terminal uses a speech synthesis engine to generate an appropriate voice response, such as "Taro Tanaka, what's the matter?"

[0460] Step 7:

[0461] Device: The generated voice is played back through the speaker, and the dialogue is tailored to the user's emotions.

[0462] Example 2

[0463] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0464] Current home robots are unable to provide integrated functions such as natural dialogue with users, multilingual support, facial recognition, emotion recognition, and real-time environmental detection using environmental sensors. Therefore, in order to improve users' quality of life, it is necessary to integrate these diverse functions into a single system and ensure their smooth collaboration.

[0465] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a voice recognition means, a means for storing text data in a database, a multilingual translation means, an image recognition means for recognizing faces, an environment detection means used to detect abnormal odors, a means for operating in cooperation with other systems, and an emotion recognition means for analyzing emotions and responding appropriately. This makes it possible to perform natural dialogue with the user, detect anomalies, and provide multifunctional support in an integrated manner.

[0466] "Speech recognition means" is a technology that captures the voice uttered by the user and converts the voice data into text data.

[0467] "Means for saving text data in a database" refers to a technique for sending the converted text data to a server via a request and saving it in a database.

[0468] "Multilingual translation means" refers to a technology for translating text data into other languages.

[0469] "Image recognition means for recognizing faces" is a technology that analyzes facial images captured by a camera and identifies individuals.

[0470] "Environmental detection means used to detect anomalous odors" refers to technology that uses an olfactory sensor to detect anomalous odors in the environment and analyzes the data.

[0471] "Means of operating in cooperation with other systems" refers to technology that communicates with external systems and applications, exchanging instructions and information to perform operations.

[0472] "Emotion recognition means for analyzing emotions and responding appropriately" is a technology that analyzes emotions from the user's tone of voice and facial expressions and generates an appropriate response according to those emotions.

[0473] MODE FOR CARRYING OUT THE INVENTION

[0474] An embodiment of the next-generation robot "maid-type" system of the present invention will now be described in detail. This system includes a voice recognition unit, a unit for storing text data in a database, a multilingual translation unit, an image recognition unit for recognizing faces, an environment detection unit used to detect odors, a unit for operating in cooperation with other systems, and an emotion recognition unit for analyzing emotions and responding appropriately.

[0475] Voice recognition means

[0476] The user speaks to the robot. For example, "Hello, my name is Tanaka Taro." The device uses the robot's built-in microphone to capture the voice, and sends the voice data to the Google Cloud Speech-to-Text API, where it is converted into text data in real time. In this way, the text data "Tanaka Taro" is generated.

[0477] A means of storing text data in a database

[0478] The device sends the generated text data to the server as an HTTP request. This request also includes metadata such as the user ID. The server analyzes the received text data and stores it in a MySQL database along with the user ID. For example, "User ID: 1234" and "Name: Taro Tanaka" are saved.

[0479] Multilingual translation tools

[0480] The user speaks to the robot, "Can you speak English?" The device acquires the text data, and if multilingual support is required, sends the data to the server via an HTTP request. The server then sends the received data to the Google Cloud Translation API, which translates the Japanese "Can you speak English?" into English "Do you speak English?" The translation result is then sent to the device, and the robot responds to the user with "Do you speak English?"

[0481] Image recognition method for recognizing faces

[0482] The user turns their face towards the robot's camera. The device's camera captures the facial image and sends the image data to the server via an HTTP request. The server uses an image recognition system to analyze the facial image and compare it with a database to identify the individual "Taro Tanaka." The identified user ID is then sent to the device.

[0483] Environmental sensing means used to detect off-odors

[0484] When the device's olfactory sensor detects a gas leak, it sends the data to the server in real time. The server analyzes the received odor data, and if it identifies a gas leak, it sends a warning to the device. Based on the warning information from the server, the device generates an audio warning saying, "There is a possibility of a gas leak. Please ventilate immediately," and plays it through the speaker.

[0485] A means of interacting with other systems

[0486] The user issues a command to "start cleaning" on the smartphone app. The device receives the command from the app and sends it to the server via an HTTP request. The server analyzes the command and instructs the device to start cleaning mode. The device starts the cleaning function and sends that information to the server when cleaning is complete. The server then sends a completion notification to the smartphone app, and the user receives a notification that "cleaning is complete."

[0487] Emotion recognition means for analyzing emotions and responding appropriately

[0488] When the user speaks to the robot, they express their emotions through their voice tone and facial expressions. The device uses an emotion recognition engine to analyze the voice tone and facial expressions to generate emotional data about the user. This data is then sent to the server via an HTTP request. The server analyzes the received emotional data and generates an appropriate response, such as "Taro Tanaka, what's the matter?" The device uses a speech synthesis engine to generate a response based on the response data received from the server, and plays it back from the speaker.

[0489] Prompt Sentence Examples

[0490] "Hello, my name is Tanaka Taro. Do you speak English?"

[0491] This system integrates natural dialogue with the user, anomaly detection, and multifunctional support, improving the quality of everyday life.

[0492] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0493] Program processing flow and specific explanation

[0494] Voice recognition means

[0495] Step 1:

[0496] The user speaks to the robot, saying something like, "Hello, my name is Taro Tanaka."

[0497] Step 2:

[0498] The device captures voice using a built-in microphone, and the input voice data is sent to a voice recognition engine.

[0499] Step 3:

[0500] The device converts the captured audio into text data using the Google Cloud Speech-to-Text API. The input is audio data, and the output is the text data "Taro Tanaka."

[0501] A means of storing text data in a database

[0502] Step 4:

[0503] The terminal creates an HTTP request containing the converted text data "Taro Tanaka." The input is the text data and metadata such as the user ID.

[0504] Step 5:

[0505] The terminal sends an HTTP request to the "Apache Tomcat" server. The input is the HTTP request, and the output is the result of the request sent to the server.

[0506] Step 6:

[0507] The server analyzes the received request and saves the text data and user ID in a MySQL database. The input is the HTTP request, and the output is the result saved in the database.

[0508] Multilingual translation tools

[0509] Step 7:

[0510] The user asks the robot, "Can you speak English?"

[0511] Step 8:

[0512] The terminal creates an HTTP request including the text data and sends it to the server. The input is the text data, and the output is the result of the request sent to the server.

[0513] Step 9:

[0514] The server calls the Google Cloud Translation API to translate the Japanese text "Do you speak English?" into English "Do you speak English?" The input is Japanese text and the output is English text.

[0515] Step 10:

[0516] The server sends the translated text data to the terminal. The input is the translation result, and the output is the data sent to the terminal.

[0517] Step 11:

[0518] The terminal responds to the user by voice, asking, "Do you speak English?" The input is English text, and the output is a voice response.

[0519] Image recognition method for recognizing faces

[0520] Step 12:

[0521] The user faces the robot's camera.

[0522] Step 13:

[0523] The device captures a facial image with a camera and sends the image data to the server via an HTTP request. The input is the facial image data, and the output is the result of the request sent to the server.

[0524] Step 14:

[0525] The server analyzes the facial image using an image recognition system and compares it with a database to identify the individual named "Taro Tanaka." The input is facial image data, and the output is the identification result.

[0526] Step 15:

[0527] The server sends the identified user ID to the terminal. The input is the identification result, and the output is the transmission to the terminal.

[0528] Environmental sensing means used to detect off-odors

[0529] Step 16:

[0530] When the olfactory sensor detects a gas leak, the device sends the data to the server in real time. The input is the odor sensor data, and the output is the request sent to the server.

[0531] Step 17:

[0532] The server analyzes the received odor data and compares it with known odor patterns to identify a "gas leak." The input is the odor data, and the output is the analysis results.

[0533] Step 18:

[0534] When the server identifies a gas leak, it sends the warning information to the terminal. The input is the analysis result, and the output is the message sent to the terminal.

[0535] Step 19:

[0536] The terminal generates a voice warning based on the warning information and plays a voice message saying, "There is a possibility of a gas leak. Please ventilate immediately." The input is the warning information and the output is the voice warning.

[0537] A means of interacting with other systems

[0538] Step 20:

[0539] The user issues a command to "start cleaning" via the smartphone app.

[0540] Step 21:

[0541] The device receives instructions from the smartphone app and sends them to the server via an HTTP request. The input is the instruction data, and the output is the result of the request sent to the server.

[0542] Step 22:

[0543] The server analyzes the received command and instructs the terminal to start cleaning mode. The input is the command data, and the output is sent to the terminal.

[0544] Step 23:

[0545] The terminal starts the cleaning function and sends the cleaning completion information to the server when the cleaning is completed. The input is the cleaning completion data, and the output is the request sent to the server.

[0546] Step 24:

[0547] The server sends a completion notification to the smartphone app, and the user receives a notification that "cleaning is complete." The input is the completion data, and the output is the notification.

[0548] Emotion recognition means for analyzing emotions and responding appropriately

[0549] Step 25:

[0550] When a user speaks to a robot, they express their emotions through their tone of voice and facial expressions.

[0551] Step 26:

[0552] The device analyzes voice tones and facial expressions using an emotion recognition engine, generates emotional data, and sends it to the server via an HTTP request. The input is voice tones and facial expression data, and the output is emotional data.

[0553] Step 27:

[0554] The server analyzes the received emotion data and generates an appropriate response, such as "Taro Tanaka, what's wrong?" The input is emotion data, and the output is response data.

[0555] Step 28:

[0556] The server sends the response data to the terminal. The input is the response data, and the output is the data sent to the terminal.

[0557] Step 29:

[0558] The device generates a response using a speech synthesis engine and plays it back through the speaker. The input is the response data, and the output is the voice response.

[0559] (Application example 2)

[0560] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0561] Modern factories require efficient systems to improve productivity and ensure worker safety. However, current robot systems and automation solutions are often single-function and unable to handle multiple tasks simultaneously and flexibly. Furthermore, there is a lack of systems that can handle environments with a mix of multinational workers and smoothly detect and respond to abnormal situations. This leads to a decline in the operational efficiency of the entire factory.

[0562] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0563] In this invention, the server includes a voice recognition means, a means for storing personal data in a database, a multilingual translation means, an image recognition means, an environment detection means using an olfactory sensor, a means for operating in cooperation with other systems, an emotion engine means for recognizing the user's emotions, and an automatic operation means for supporting factory work. This makes it possible to perform multiple tasks in an integrated and efficient manner, while supporting a variety of languages, improving the safety of the work environment, and enabling flexible responses that take into account the emotional state of workers.

[0564] - "Speech recognition means" refers to technology that captures a user's voice and converts it into text data.

[0565] "Means for storing personal data in a database" refers to technology that stores information related to individuals, such as a user's name or ID, in a database.

[0566] "Multilingual translation means" is a technology that translates text entered in one language into another language.

[0567] "Image recognition means" refers to technology that identifies specific objects or people from image data captured by a camera or other sensor.

[0568] "Environmental detection means using olfactory sensors" is a technology that detects specific odors and gases and analyzes the data.

[0569] "Means for operating in cooperation with other systems" refers to technology for exchanging instructions and data in cooperation with other information systems and devices.

[0570] The "emotion engine means for recognizing user emotions" is a technology that analyzes the user's tone of voice and facial expressions to identify their emotional state.

[0571] "Automatic operation means for supporting factory work" refers to technology that automatically executes specific work within a factory and provides efficient support.

[0572] In this invention, a next-generation robot system is used to efficiently support factory work. First, an overview of the entire system will be described below.

[0573] The server includes a voice recognition means, a means for storing personal data in a database, a multilingual translation means, an image recognition means, an environment detection means using an olfactory sensor, a means for operating in cooperation with other systems, an emotion engine means for recognizing the user's emotions, and an automatic operation means for supporting factory work.

[0574] Hardware

[0575] The following hardware is used to implement this system:

[0576] Smartphone: User interface and audio capture.

[0577] Robot: Equipped with various sensors (audio, image, olfactory) and actuators.

[0578] software

[0579] The system software includes:

[0580] Flask: A web application framework that acts as an API server.

[0581] SpeechRecognition: A speech recognition library that uses the Google Web Speech API.

[0582] GoogleTrans: A multilingual translation library that uses the Google Translate API.

[0583] ImageRecognizer: A custom library for recognition, specifically OpenCV.

[0584] EmotionRecognizer: A library for emotion recognition, specifically using TensorFlow and Keras.

[0585] Data Flow and Processing

[0586] 1. Voice data: When a user speaks into the smartphone, the voice recognition library converts the speech into text data, which is sent to the server and the corresponding instructions are sent to the robot.

[0587] 2. Personal data: Personal information such as the user's name and ID is stored on the server, enabling customized responses to specific users.

[0588] 3. Multilingual Translation: When users give instructions in different languages, the text data is translated by the multilingual translation library and converted into the appropriate language.

[0589] 4. Image Recognition: When a user points the robot's camera at a specific area or object, the image recognition library analyzes the data and performs the corresponding action.

[0590] 5. Environmental detection: When the robot's olfactory sensors detect an unusual odor, the data is sent to a server, and an alert is issued based on the analysis results.

[0591] 6. Emotion Recognition: Recognize the user's emotions from their tone of voice and facial expressions and respond appropriately.

[0592] 7. Automated operation: Specific tasks within the factory are carried out automatically to provide efficient support.

[0593] Specific examples

[0594] For example, a factory worker might use a smartphone app to give a robot a voice command such as "Clean the warehouse." This voice is analyzed by a speech recognition library and converted into text data. This data is then sent to a server, and the robot begins cleaning the warehouse in accordance with the command.

[0595] Prompt Sentence Examples

[0596] "Imagine a smartphone app for voice control of next-generation factory robots. Include features like voice recognition, image recognition, emotion recognition, multilingual translation, environmental monitoring, and integration with other systems."

[0597] The above is a specific embodiment for applying a next-generation robot system to factory work. This system enables multiple tasks to be performed in an integrated and efficient manner, supports a variety of languages, enhances the safety of the work environment, and enables flexible responses that take into consideration the feelings of workers.

[0598] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0599] Step 1:

[0600] Input: The user gives voice instructions to the smartphone.

[0601] Processing: The device captures the audio and converts it into text using the SpeechRecognition library. Specifically, the smartphone's microphone captures the audio data and passes it to the SpeechRecognition library.

[0602] Output: The converted text data is generated.

[0603] Step 2:

[0604] Input: Generated text data.

[0605] Processing: The device sends text data to the server via an HTTP request. The HTTP request also includes metadata such as the user ID. Specifically, the device app calls the data sending function to the server.

[0606] Output: Text data and metadata are sent to the server.

[0607] Step 3:

[0608] Input: The text data sent to the server.

[0609] Processing: The server analyzes the text data, and if it determines that multilingual translation is necessary, it calls the multilingual translation library (GoogleTrans) to perform the translation. Specifically, it calls the translation API on the server and converts the text into the specified language.

[0610] Output: The translated text data.

[0611] Step 4:

[0612] Input: Translated text data.

[0613] Processing: The server sends the translated data to the terminal again using an HTTP request. Specifically, the server returns the data to the terminal using the HTTP sending function on the server.

[0614] Output: The translated text data is received on the terminal.

[0615] Step 5:

[0616] Input: Translation text and images received on the device.

[0617] Processing: The user points the robot at a designated area and captures the image data, which is then sent to the device's image recognition library (ImageRecognizer) for analysis. Specifically, the device's camera function is used to capture the image, and the image recognition library identifies specific objects and areas.

[0618] Output: The analyzed image data is generated and sent to the server.

[0619] Step 6:

[0620] Input: Parsed image data.

[0621] Processing: The data is processed on the server and stored in a database together with environmental information. Specifically, the analyzed image data and environmental information are recorded in the database using the server's data storage function.

[0622] Output: Environmental data stored in a database.

[0623] Step 7:

[0624] Input: Odor data sent from the robot's olfactory sensor.

[0625] Processing: The odor data is analyzed on the server and matched with known odor patterns to identify the cause. Specifically, the odor analysis algorithm on the server compares the data with odor patterns in the database to identify the cause.

[0626] Output: Identified odor causes and associated warning data.

[0627] Step 8:

[0628] Input: Data on identified off-odor causes.

[0629] Processing: The server generates warning data and sends it to the terminal via an HTTP request. Specifically, the warning generation algorithm on the server creates a warning message based on the cause data and sends it to the terminal.

[0630] Output: The warning message sent to the terminal.

[0631] Step 9:

[0632] Input: Emotion data from the user (voice and facial expressions).

[0633] Processing: The device's emotion recognition engine (EmotionRecognizer) analyzes the emotion data to determine the user's emotional state. Specifically, it analyzes voice tone and facial expression data to identify the emotional state.

[0634] Output: The determined emotional state data is generated and sent to the server.

[0635] Step 10:

[0636] Input: Determined emotional state data.

[0637] Processing: The server analyzes the emotion data and generates an appropriate response message. Specifically, a response generation algorithm on the server creates an appropriate response message based on the emotion.

[0638] Output: The voice message generated using the speech synthesis engine is sent to the terminal and played back to the user.

[0639] Step 11:

[0640] Input: Automatic operation instructions for factory work.

[0641] Processing: The server analyzes the automatic operation instructions and sends the instructions to the corresponding robot. Specifically, the server uses a control algorithm to send the instructions to the robot's control system.

[0642] Output: The robot performs factory work according to instructions and returns the resulting data to the server.

[0643] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0644] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0645] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0646] [Second embodiment]

[0647] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0648] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0649] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0650] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0651] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0652] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0653] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0654] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0655] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0656] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0657] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0658] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0659] An embodiment of the next-generation robot "maid-type" system of the present invention is described in detail below. This system includes a voice recognition unit, a unit for storing personal data in a database, a multilingual translation unit, an image recognition unit, an environment detection unit using an olfactory sensor, and a unit for operating in cooperation with other systems.

[0660] 1. Voice Recognition Method

[0661] User: The user speaks to the robot, for example, "Hello, my name is Taro Tanaka."

[0662] Terminal: The robot's microphone captures the voice. The captured voice is converted into text data by a voice recognition engine, and the text data "Taro Tanaka" is generated.

[0663] 2. How personal data is stored in the database

[0664] Terminal: The converted text data is sent to the server via an HTTP request, which includes metadata such as the user ID.

[0665] Server: The server analyzes the received text data and stores it in a database. For example, the user name "Taro Tanaka" and the corresponding ID are stored.

[0666] 3. Multilingual Translation Tools

[0667] Terminal: Obtains user input as text data, and if multilingual support is required, sends the text data to the server. For example, it asks, "Do you speak English?"

[0668] Server: The server receives the text data and calls the translation API to translate it. The Japanese text "Do you speak English?" is translated into English "Do you speak English?" and the translated data is sent to the device.

[0669] 4. Image Recognition Methods

[0670] User: The user faces the robot's camera.

[0671] Device: The camera captures the user's face and sends the image data to a server. The image recognition system analyzes the face and matches it with a database to identify the individual.

[0672] Server: The image recognition algorithm recognizes the user's face as "Taro Tanaka" and sends the user ID to the device.

[0673] 5. Environmental detection using olfactory sensors

[0674] Terminal: When the olfactory sensor detects an unusual odor, it sends the data to the server in real time. For example, if it detects a specific odor like a gas leak.

[0675] Server: The server analyzes the odor data it receives and identifies the cause by comparing it with known odor patterns. For example, it may identify a "gas leak" and send that data to the device.

[0676] Terminal: Based on the warning information from the server, a speech synthesis engine is used to generate a warning voice saying, "There is a possibility of a gas leak. Please ventilate immediately," and this is played from the speaker.

[0677] 6. Means of cooperation with other systems

[0678] User: Sends the command "Start cleaning" to the robot via the smartphone app.

[0679] Terminal: Sends instructions received from the app to the server.

[0680] Server: Analyzes the received instruction and sends an instruction to start cleaning mode to the terminal.

[0681] Device: The cleaning function starts, and when cleaning is completed, the information is sent to the server. The server notifies the smartphone app of the completion information, and the user receives a notification that "cleaning is completed."

[0682] As described above, the next-generation robot "maid-type" system of the present invention integrates multiple functions to support users' daily lives in many ways and improve their quality of life.

[0683] The processing flow will be explained below.

[0684] 1. Speech recognition processing

[0685] Step 1:

[0686] User: Talks to the robot and says, "Hello, my name is Taro Tanaka."

[0687] Step 2:

[0688] Device: Uses a microphone to capture the user's voice and sends the voice data to a speech recognition engine.

[0689] Step 3:

[0690] Terminal: The voice recognition engine converts the voice data into text data, generating the text data "Taro Tanaka."

[0691] 2. Processing of the means by which personal data is stored in a database

[0692] Step 1:

[0693] Terminal: Sends an HTTP request to the server containing the text data "Taro Tanaka." The request also contains metadata such as the user ID.

[0694] Step 2:

[0695] Server: Analyzes the received text data and stores it in a database along with the user ID.

[0696] Step 3:

[0697] Server: After saving is complete, it sends a response to the terminal indicating that the process was successful.

[0698] 3. Processing of multilingual translation means

[0699] Step 1:

[0700] User: Talks to the robot and asks, "Can you speak English?"

[0701] Step 2:

[0702] Terminal: Captures voice with a microphone and converts it into text data using a speech recognition engine. The converted text data is sent to the server.

[0703] Step 3:

[0704] Server: Sends the received text data to the translation API and performs the translation.

[0705] Step 4:

[0706] Server: Receives the translated text data "Do you speak English?" and sends it to the terminal.

[0707] Step 5:

[0708] Terminal: The translated text data is passed to a speech synthesis engine to generate speech data.

[0709] Step 6:

[0710] Device: Play the generated voice "Yes, I can speak English." from the speaker.

[0711] 4. Image Recognition Method Processing

[0712] Step 1:

[0713] User: Face the robot's camera.

[0714] Step 2:

[0715] Device: The camera captures the user's facial image and sends the image data to the server.

[0716] Step 3:

[0717] Server: Analyzes the received image data using an image recognition engine and identifies the user ID.

[0718] Step 4:

[0719] Server: Sends the identified user ID to the terminal.

[0720] Step 5:

[0721] Terminal: Based on the user ID received from the server, it matches it with a registered user name, generates a personalized response such as "You're Tanaka Taro, right?" and creates speech using a speech synthesis engine.

[0722] Step 6:

[0723] Device: Play the created audio "Hello, Taro Tanaka" from the speaker.

[0724] 5. Processing of environmental detection means using olfactory sensors

[0725] Step 1:

[0726] Terminal: The olfactory sensor detects strange odors in the environment.

[0727] Step 2:

[0728] Terminal: Transmits detected odor data to the server in real time.

[0729] Step 3:

[0730] Server: Analyzes the received odor data and compares it with known odor patterns.

[0731] Step 4:

[0732] Server: Identifies the cause of the strange odor and sends data such as "gas leak" to the terminal.

[0733] Step 5:

[0734] Device: Based on the received data, the speech synthesis engine generates a warning voice saying, "There is a possibility of a gas leak. Please ventilate immediately."

[0735] Step 6:

[0736] Device: Play the generated warning sound through the speaker.

[0737] 6. Processing of means for interoperating with other systems

[0738] Step 1:

[0739] User: Uses the smartphone app to send the command to the robot to "start cleaning."

[0740] Step 2:

[0741] Terminal: Sends instructions received from the app to the server.

[0742] Step 3:

[0743] Server: Analyzes the received instructions and instructs the cleaning robot to start cleaning mode.

[0744] Step 4:

[0745] Device: Follow the instructions to start cleaning mode.

[0746] Step 5:

[0747] Terminal: After cleaning is completed, the information is sent to the server.

[0748] Step 6:

[0749] Server: Receives the completion information and sends a notification to the smartphone app.

[0750] Step 7:

[0751] User: Receives a notification on the smartphone app that "cleaning is complete."

[0752] Example 1

[0753] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0754] Current robot systems use separate technologies such as voice recognition, image recognition, and environmental detection, making it difficult to integrate these pieces of information, resulting in an inconsistent user experience. Furthermore, they lack the ability to respond to odor detection in real time and provide easy user control, making them unable to effectively support users in their daily lives. Furthermore, their ability to communicate in multiple languages ​​to accommodate internationalization is insufficient, meaning more advanced support is required.

[0755] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0756] In this invention, the server includes a voice recognition unit, a unit for storing personal data in a database, a multilingual translation unit, an image recognition unit, an environment detection unit using an olfactory sensor, a unit for operating in cooperation with other systems, a unit for allowing a user to send instructions to the robot using a smartphone app and for the robot to operate based on those instructions, a unit for analyzing data in real time when an abnormal odor is detected and generating an alarm sound to notify the user, a unit for storing text data acquired by voice recognition in a database in association with a user ID, and a unit for displaying the translated text data to the user. This makes it possible to integrate various recognition technologies, environment detection, and user interfaces to respond to various user requests in real time and improve quality of life.

[0757] "Speech recognition means" refers to technology that captures the user's voice and converts it into text data.

[0758] "Means for storing personal data in a database" refers to the technology for storing text data obtained by a voice recognition means in a database in association with a user ID.

[0759] "Multilingual translation means" refers to a technology that translates text in one language into another language.

[0760] "Image recognition means" refers to technology that analyzes image data captured by a camera and recognizes specific objects (e.g., faces).

[0761] "Environmental detection means using olfactory sensors" refers to technology that uses odor-detecting sensors to detect specific odors and analyze their causes.

[0762] "Means of operating in cooperation with other systems" refers to technology that exchanges data with different systems and performs multiple operations in cooperation with them.

[0763] "A means by which a user sends instructions to a robot using a smartphone app, and the robot then performs an action based on those instructions" refers to a technology in which a user sends operating instructions to a robot via a smartphone app, and the robot performs an action in accordance with those instructions.

[0764] "Means for analyzing data in real time when an abnormal odor is detected and generating an audio warning to notify the user" refers to a technology that, when an olfactory sensor detects an abnormal odor, analyzes the data to identify the cause, generates an audio warning to notify the user in real time.

[0765] "Means of storing text data obtained by voice recognition in a database in association with a user ID" refers to a technology that stores text data obtained by voice recognition technology in a database together with the corresponding user ID.

[0766] The term "means for displaying translated text data to the user" refers to a technology for displaying text data translated by the multilingual translation means in a form that can be confirmed by the user.

[0767] The present invention relates to a next-generation robot system that supports users' daily lives. The system includes a voice recognition unit, a unit for storing personal data in a database, a multilingual translation unit, an image recognition unit, an environment detection unit using an olfactory sensor, a unit for operating in conjunction with other systems, a unit for sending instructions to the robot via a smartphone app and having the robot perform actions based on those instructions, a unit for analyzing data in real time when an abnormal odor is detected and generating an alarm sound to notify the user, a unit for storing text data acquired by voice recognition in a database in association with a user ID, and a unit for displaying the translated text data to the user.

[0768] 1. Voice Recognition Method

[0769] When a user speaks to the robot, the robot's microphone captures the voice. The voice data is converted into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text API). For example, if a user says, "Hello, my name is Tanaka Taro," the text data "Hello, my name is Tanaka Taro" is generated.

[0770] 2. How personal data is stored in the database

[0771] The converted text data is sent to the server via an HTTP request. The server analyzes the received data and stores the user name and ID in a database (e.g., MySQL). This allows the information obtained through voice recognition to be managed as personal data.

[0772] 3. Multilingual Translation Tools

[0773] When translating user input text into multiple languages, the text data is sent to the server. The server calls a translation API (e.g., Google Cloud Translation API) to obtain the translation result. For example, the text "Can you speak English?" is translated to "Do you speak English?" The translation result is displayed to the user.

[0774] 4. Image Recognition Methods

[0775] When a user points their face at the robot's camera, the camera captures their facial image. The image data is sent to a server and analyzed using an image recognition algorithm (e.g., Azure Face API). The server then matches the individual against a database and sends the user ID to the device.

[0776] 5. Environmental detection using olfactory sensors

[0777] When the robot's olfactory sensors detect an unusual odor, the data is sent to a server in real time. The server analyzes the data and identifies the source of the odor. For example, if a gas leak is detected, the server sends that information to the device. The device uses a speech synthesis engine (e.g., Amazon Polly) to generate a warning voice saying, "There is a possibility of a gas leak. Please ventilate immediately," and plays it over the speaker.

[0778] 6. Means of cooperation with other systems

[0779] The user sends a command to the robot via a smartphone app to "start cleaning." The command from the app is sent to a server, which analyzes it and then issues a command to the robot to start cleaning. Once cleaning is complete, the device sends the information back to the server, and the server sends a completion notification to the smartphone app. The user then receives a notification that "cleaning is complete."

[0780] Examples and prompts

[0781] Example 1: When a user says, "Where am I now?", the speech recognition engine generates text data of "Where am I now?", sends related information to the server, and returns the answer.

[0782] Example 2: When a user says, "What do you call this in English?", the translation result is displayed to the user as "What do you call this in English?" after going through speech recognition, translation, and display.

[0783] Example prompt for a generative AI model:

[0784] "Please translate the text into English. The original text is 'How do you say this in English?'"

[0785] "Please tell me the procedure for analyzing the speech recognition results and saving them to the database."

[0786] The system of the present invention is an integrated multi-functional robot system that responds to various user requests in real time and improves the quality of life.

[0787] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0788] 1. Voice Recognition Method

[0789] Step 1:

[0790] The user speaks to the robot.

[0791] Input: Voice data (e.g., "Hello, my name is Taro Tanaka")

[0792] Output: None

[0793] Specific action: The user speaks aloud to the robot.

[0794] Step 2:

[0795] The device captures audio using the robot's microphone.

[0796] Input: Audio data

[0797] Output: Raw audio file

[0798] What it does: The microphone collects audio data and creates an audio file in the internal memory.

[0799] Step 3:

[0800] The device's voice recognition engine converts the captured voice data into text data.

[0801] Input: raw audio file

[0802] Output: Text data (e.g. "Hello, my name is Taro Tanaka")

[0803] What happens: A speech recognition engine (e.g., Google Cloud Speech-to-Text API) analyzes the audio file and generates corresponding text data.

[0804] 2. How personal data is stored in the database

[0805] Step 1:

[0806] The terminal sends the converted text data and metadata such as the user ID to the server in the form of an HTTP request.

[0807] Input: Text data, user ID

[0808] Output: HTTP request

[0809] Specific behavior: The text data and user ID are included in the body of the HTTP request and sent to the server.

[0810] Step 2:

[0811] The server analyzes the received HTTP request and extracts the content.

[0812] Input: HTTP request

[0813] Output: User name, User ID

[0814] What happens: The server parses the body of the HTTP request and extracts the username and user ID.

[0815] Step 3:

[0816] The server stores the extracted data in a database.

[0817] Input: Username, User ID

[0818] Output: New record to the database

[0819] Specific behavior: The username and user ID are added as a new record to the database (e.g. MySQL).

[0820] 3. Multilingual Translation Tools

[0821] Step 1:

[0822] The terminal obtains the user's input text and, if multilingual support is required, sends it to the server.

[0823] Input: Text data (e.g., "Do you speak English?")

[0824] Output: HTTP request

[0825] Specific operation: The device creates and sends an HTTP request to send text data to the server.

[0826] Step 2:

[0827] The server receives the text data and calls the translation API to translate it into the specified target language.

[0828] Input: HTTP request, text data

[0829] Output: Translated text data (e.g., "Do you speak English?")

[0830] Specific operation: Send a request to a translation API (e.g., Google Cloud Translation API) and obtain the translation result.

[0831] Step 3:

[0832] The server returns the translation results to the terminal.

[0833] Input: Translated text data

[0834] Output: HTTP response

[0835] Specific operation: Generates an HTTP response containing the translation result and sends it to the device.

[0836] Step 4:

[0837] The terminal displays the translation results to the user.

[0838] Input: Translated text data

[0839] Output: The text that is displayed in the user interface

[0840] Specific operation: The translation results are displayed on the user interface (display or voice reading device).

[0841] 4. Image Recognition Methods

[0842] Step 1:

[0843] The user faces the robot's camera.

[0844] Input: Face image

[0845] Output: None

[0846] Specific action: The user faces the camera directly.

[0847] Step 2:

[0848] The device captures a facial image using the robot's camera.

[0849] Input: Face image

[0850] Output: Image file

[0851] What happens: The camera captures a facial image and generates an image file.

[0852] Step 3:

[0853] The terminal transmits the image data to the server.

[0854] Input: Image file

[0855] Output: HTTP request

[0856] Specific operation: Sends an image file to the server in the form of an HTTP request.

[0857] Step 4:

[0858] The server analyzes the data using image recognition algorithms.

[0859] Input: Image file

[0860] Output: User ID

[0861] Specific operation: Analyzes the image using an image recognition algorithm (e.g. Azure Face API) and identifies the user.

[0862] Step 5:

[0863] The server sends the user ID to the terminal.

[0864] Input: User ID

[0865] Output: HTTP response

[0866] Specific operation: Generate an HTTP response including the user ID and send it to the terminal.

[0867] 5. Environmental detection using olfactory sensors

[0868] Step 1:

[0869] The device detects strange odors using an olfactory sensor.

[0870] Input: Environmental data (odor data)

[0871] Output: Odor detection event

[0872] How it works: The olfactory sensor detects specific odors in the environment.

[0873] Step 2:

[0874] The device transmits the detection data to the server in real time.

[0875] Input: Off-odor data

[0876] Output: HTTP request

[0877] Specific operation: Odor data is sent to the server in real time in the form of an HTTP request.

[0878] Step 3:

[0879] The server analyzes the odor data and compares it with known odor patterns to identify the cause.

[0880] Input: Off-odor data

[0881] Output: Data on the cause of the odor (e.g., gas leak)

[0882] Specific operation: Odor data is compared with a database of known odor patterns to identify the cause.

[0883] Step 4:

[0884] The server sends data on the cause of the odor to the terminal.

[0885] Input: Odor source data

[0886] Output: HTTP response

[0887] Specific operation: Generate an HTTP response including data on the cause of the odor and send it to the terminal.

[0888] Step 5:

[0889] The device generates and plays an audio warning.

[0890] Input: HTTP response (odor cause data)

[0891] Output: Audio warning

[0892] Specific behavior: Generates a warning message using a speech synthesis engine (e.g., Amazon Polly) and plays it over the speaker.

[0893] 6. Means of cooperation with other systems

[0894] Step 1:

[0895] The user sends instructions to the robot via a smartphone app.

[0896] Input: Instruction data (e.g. "Start cleaning")

[0897] Output: HTTP request

[0898] Specific operation: The user operates the app and sends commands to the robot.

[0899] Step 2:

[0900] The device sends the instructions received from the app to the server.

[0901] Input: HTTP request

[0902] Output: HTTP request

[0903] Specific operation: The device receives instructions from the app and forwards them to the server.

[0904] Step 3:

[0905] The server analyzes the received instructions and sends appropriate commands to the terminal.

[0906] Input: HTTP request (instruction data)

[0907] Output: Command data

[0908] Specific operation: The server analyzes the instruction data, generates the necessary commands, and sends them to the terminal.

[0909] Step 4:

[0910] The terminal will trigger the robot to start the cleaning function.

[0911] Input: Command data (e.g. "Start cleaning")

[0912] Output: Robot movement

[0913] Specific operation: The terminal transmits commands to the robot to start the cleaning function.

[0914] Step 5:

[0915] The terminal transmits cleaning completion data to the server.

[0916] Input: Cleaning completion data

[0917] Output: HTTP request

[0918] Specific operation: When cleaning is completed, the device sends the information to the server.

[0919] Step 6:

[0920] The server notifies the app of the cleaning completion data.

[0921] Input: HTTP request (cleaning completion data)

[0922] Output: Smartphone app notification

[0923] Specific operation: The server sends cleaning completion data to the app and notifies the user.

[0924] (Application example 1)

[0925] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0926] Conventional customer service systems in brick-and-mortar stores lack multilingual support for customer service and make it difficult to provide personalized service to repeat customers. Furthermore, emergency response capabilities, such as environmental and odor detection, are inadequate. For these reasons, there is a need to build a next-generation customer service system that can support multiple languages ​​in brick-and-mortar stores, provide personalized service for each individual customer, and ensure environmental safety.

[0927] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0928] In this invention, the server includes a voice recognition means, a means for storing personal data in a database, a multilingual translation means, an image recognition means, an environment detection means using an olfactory sensor, a means for operating in cooperation with other systems, and a means for functioning as a customer service assistant in a physical store. This makes it possible to respond to customers in multiple languages ​​in a physical store, provide personalized services according to each customer, and ensure a safe environment.

[0929] "Speech recognition means" refers to technology that takes voice input and converts that voice into text data.

[0930] "Means for storing personal data in a database" refers to the technology that collects information about users and stores it in an organized form in a database.

[0931] "Multilingual translation means" refers to a technology that translates input text data into multiple different languages.

[0932] "Image recognition means" refers to technology that analyzes image data acquired through devices such as cameras and identifies specific people or objects.

[0933] "Environmental detection means using olfactory sensors" refers to technology that uses sensors that detect various odors to detect specific environmental changes or unusual odors.

[0934] "Means for operating in cooperation with other systems" refers to technology for communicating with external systems or devices and executing specific operations.

[0935] "Means to function as a customer service assistant in a physical store" refers to technology that handles customer service within a physical store and provides multilingual support and personalized service.

[0936] This invention is a system that realizes efficient customer service and multilingual support by using a next-generation robot "maid-type" system as a customer service assistant in a brick-and-mortar store. This system includes a voice recognition means, a means for storing personal data in a database, a multilingual translation means, an image recognition means, an environmental detection means using an olfactory sensor, and a means for operating in cooperation with other systems.

[0937] Voice recognition means

[0938] When a user speaks to the robot, the voice is captured by the device's microphone, and a speech recognition engine (such as Google Web Speech API) converts the voice into text data and sends it to the server.

[0939] How personal data is stored in a database

[0940] The device sends the converted text data to the server via an HTTP request, where it is analyzed along with metadata such as the user ID and stored in a database. This storage process uses a relational or NoSQL database.

[0941] Multilingual translation tools

[0942] When a user makes a multilingual inquiry to the robot, the device sends the text data to the server. The server then calls a translation API to translate the input text data into the required language. The translated text data is then returned to the device and provided to the customer.

[0943] Image Recognition Method

[0944] When a user points their face at the robot's camera, the device's camera captures their face image, which is then sent to a server where image recognition algorithms (such as OpenCV or TensorFlow) are used to analyze the face and match it with a database to identify the user.

[0945] Environmental detection using olfactory sensors

[0946] When the device's built-in olfactory sensor detects an unusual odor, it sends the data to the server in real time. The server analyzes the data and generates a warning if it matches the odor pattern. This warning is output as voice by a speech synthesis engine and notified to the user.

[0947] A means of interacting with other systems

[0948] When a user sends a command to the robot via a smartphone app, the command is sent to the device. The server analyzes the received command and instructs the device to perform the appropriate operation. For example, if a command to start cleaning is sent, the robot will enter cleaning mode, and after cleaning is completed, the information will be sent to the server and the user will be notified.

[0949] Specific examples

[0950] When a customer asks the robot, "Where is this product in stock?", the voice is converted into text by a speech recognition device. The text data is then sent to the server, and related information is retrieved from the database and provided to the customer. In addition, if multilingual support is required, the text is translated into the appropriate language by a translation API.

[0951] Example prompts

[0952] "Write a Python program that uses speech recognition to convert what a customer says to a robot into text data, and then send that data along with the customer ID to a server for storage. The program will use the Google Web Speech API for speech recognition and the requests library to send the data to the server."

[0953] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0954] Step 1:

[0955] The user speaks to the robot. The input is the user's voice information, and the device's microphone captures this voice. Next, the voice data is converted into text data using a speech recognition engine (Google Web Speech API). The output is text data.

[0956] Step 2:

[0957] The terminal sends the converted text data and metadata (e.g., user ID) to the server as an HTTP request. The input is the text data and metadata, and the sent data is saved and analyzed on the server. The output is the saved data.

[0958] Step 3:

[0959] If a user makes a query that requires multilingual support, the device sends the text data back to the server. The input is the text data of the user's query, and the server calls the translation API to translate it into the specified language. The output is the text data in the translated language.

[0960] Step 4:

[0961] The server sends the translated data to the terminal, which then converts it into speech using a speech synthesis engine and transmits it to the user through a speaker. The input is the translated text data, and the output is the audio information provided to the user.

[0962] Step 5:

[0963] When a user points their face at the robot's camera, the device's camera captures a facial image and sends the image data to the server. The input is the captured facial image, and the server identifies the user using an image recognition algorithm (such as OpenCV or TensorFlow). The output is the identified user ID.

[0964] Step 6:

[0965] When the olfactory sensor installed in the device detects an abnormal odor, it sends the data to the server in real time. The input is the abnormal odor data detected by the olfactory sensor, and the server analyzes the data by comparing it with known abnormal odor patterns. The output is the identification of the abnormal odor and a warning message.

[0966] Step 7:

[0967] When the server identifies an abnormal odor, it sends a warning message to the device, which then uses a speech synthesis engine to generate a warning voice and notify the user through the speaker.The input is a warning message about the abnormal odor, and the output is the warning voice provided to the user.

[0968] Step 8:

[0969] When a user sends a command to a robot via a smartphone app, the command is sent to the server via the terminal. The input is the command data from the smartphone app, the server analyzes the received command, and sends appropriate operation instructions to the terminal. The output is the operation result executed by the terminal.

[0970] Step 9:

[0971] For example, when receiving the instruction to "start cleaning," the device starts cleaning mode, and when cleaning is complete, it sends that information to the server. The server notifies the smartphone app of the completion information, and the user receives a notification that "cleaning is complete." The input is the completion report data, and the output is the completion notification provided to the user.

[0972] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0973] A detailed description will be given of an embodiment of a next-generation robot "maid-type" system according to the present invention. This system includes a voice recognition unit, a unit for storing personal data in a database, a multilingual translation unit, an image recognition unit, an environment detection unit using an olfactory sensor, a unit for operating in cooperation with other systems, and an emotion engine for recognizing the user's emotions.

[0974] 1. Voice Recognition Method

[0975] User: The user speaks to the robot, for example, "Hello, my name is Taro Tanaka."

[0976] Terminal: The robot's microphone captures the voice. The captured voice is converted into text data by a voice recognition engine, and the text data "Taro Tanaka" is generated.

[0977] 2. How personal data is stored in the database

[0978] Terminal: The converted text data is sent to the server via an HTTP request, which includes metadata such as the user ID.

[0979] Server: The server analyzes the received text data and stores it in a database along with the user ID. For example, the user name "Taro Tanaka" and the corresponding ID are stored.

[0980] 3. Multilingual Translation Tools

[0981] Terminal: Obtains user input as text data, and if multilingual support is required, sends the text data to the server. For example, it asks, "Do you speak English?"

[0982] Server: The server receives the text data and calls the translation API to translate it. The Japanese text "Do you speak English?" is translated into English "Do you speak English?" and the translated data is sent to the device.

[0983] 4. Image Recognition Methods

[0984] User: The user faces the robot's camera.

[0985] Device: The camera captures the user's face and sends the image data to a server. The image recognition system analyzes the face and matches it with a database to identify the individual.

[0986] Server: The image recognition algorithm recognizes the user's face as "Taro Tanaka" and sends the user ID to the device.

[0987] 5. Environmental detection using olfactory sensors

[0988] Terminal: When the olfactory sensor detects an unusual odor, it sends the data to the server in real time. For example, if it detects a specific odor like a gas leak.

[0989] Server: The server analyzes the odor data it receives and identifies the cause by comparing it with known odor patterns. For example, it may identify a "gas leak" and send that data to the device.

[0990] Terminal: Based on the warning information from the server, a speech synthesis engine is used to generate a warning voice saying, "There is a possibility of a gas leak. Please ventilate immediately," and this is played from the speaker.

[0991] 6. Means of cooperation with other systems

[0992] User: Sends the command "Start cleaning" to the robot via the smartphone app.

[0993] Terminal: Sends instructions received from the app to the server.

[0994] Server: Analyzes the received instruction and sends an instruction to start cleaning mode to the terminal.

[0995] Device: The cleaning function starts, and when cleaning is completed, the information is sent to the server. The server notifies the smartphone app of the completion information, and the user receives a notification that "cleaning is completed."

[0996] 7. Emotional Engine Means

[0997] User: Emotions are expressed through the user's tone of voice and facial expressions when speaking to the robot.

[0998] On the device: The emotion recognition engine analyzes the user's tone of voice and facial expressions to detect their emotional state. For example, it determines that the user is "angry" based on their tone of voice and facial expressions.

[0999] Device: Sends an HTTP request containing the detected emotion data to the server.

[1000] Server: The server analyzes the received emotion data, stores it in a database, and generates appropriate response data based on the emotion data.

[1001] Terminal: Based on the response data received from the server, the terminal uses a speech synthesis engine to generate an appropriate voice response, such as "Taro Tanaka, what's the matter?"

[1002] Device: The generated voice is played back through the speaker, and the dialogue is tailored to the user's emotions.

[1003] As explained above in this embodiment, the next-generation robot "maid-type" system integrates multiple functions to support users' daily lives in many ways and improve their quality of life. In particular, the implementation of an emotion engine enables flexible responses according to emotions, resulting in a more natural and friendly user experience.

[1004] The processing flow will be explained below.

[1005] 1. Speech recognition processing

[1006] Step 1:

[1007] User: Talks to the robot and says, "Hello, my name is Taro Tanaka."

[1008] Step 2:

[1009] Device: Uses a microphone to capture the user's voice and sends the voice data to a speech recognition engine.

[1010] Step 3:

[1011] Terminal: The voice recognition engine converts the voice data into text data, generating the text data "Taro Tanaka."

[1012] 2. Processing of the means by which personal data is stored in a database

[1013] Step 1:

[1014] Terminal: Sends an HTTP request to the server containing the text data "Taro Tanaka." The request also contains metadata such as the user ID.

[1015] Step 2:

[1016] Server: Analyzes the received text data and stores it in a database along with the user ID.

[1017] Step 3:

[1018] Server: After saving is complete, it sends a response to the terminal indicating that the process was successful.

[1019] 3. Processing of multilingual translation means

[1020] Step 1:

[1021] User: Talks to the robot and asks, "Can you speak English?"

[1022] Step 2:

[1023] Terminal: Captures voice with a microphone and converts it into text data using a speech recognition engine. The converted text data is sent to the server.

[1024] Step 3:

[1025] Server: Sends the received text data to the translation API and performs the translation.

[1026] Step 4:

[1027] Server: Receives the translated text data "Do you speak English?" and sends it to the terminal.

[1028] Step 5:

[1029] Terminal: The translated text data is passed to a speech synthesis engine to generate speech data.

[1030] Step 6:

[1031] Device: Play the generated voice "Yes, I can speak English." from the speaker.

[1032] 4. Image Recognition Method Processing

[1033] Step 1:

[1034] User: Face the robot's camera.

[1035] Step 2:

[1036] Device: The camera captures the user's facial image and sends the image data to the server.

[1037] Step 3:

[1038] Server: Analyzes the received image data using an image recognition engine and identifies the user ID.

[1039] Step 4:

[1040] Server: Sends the identified user ID to the terminal.

[1041] Step 5:

[1042] Terminal: Based on the user ID received from the server, it matches it with a registered user name, generates a personalized response such as "You're Tanaka Taro, right?" and creates speech using a speech synthesis engine.

[1043] Step 6:

[1044] Device: Play the created audio "Hello, Taro Tanaka" from the speaker.

[1045] 5. Processing of environmental detection means using olfactory sensors

[1046] Step 1:

[1047] Terminal: The olfactory sensor detects strange odors in the environment.

[1048] Step 2:

[1049] Terminal: Transmits detected odor data to the server in real time.

[1050] Step 3:

[1051] Server: Analyzes the received odor data and compares it with known odor patterns.

[1052] Step 4:

[1053] Server: Identifies the cause of the strange odor and sends data such as "gas leak" to the terminal.

[1054] Step 5:

[1055] Device: Based on the received data, the speech synthesis engine generates a warning voice saying, "There is a possibility of a gas leak. Please ventilate immediately."

[1056] Step 6:

[1057] Device: Play the generated warning sound through the speaker.

[1058] 6. Processing of means for interoperating with other systems

[1059] Step 1:

[1060] User: Uses the smartphone app to send the command to the robot to "start cleaning."

[1061] Step 2:

[1062] Terminal: Sends instructions received from the app to the server.

[1063] Step 3:

[1064] Server: Analyzes the received instructions and instructs the cleaning robot to start cleaning mode.

[1065] Step 4:

[1066] Device: Follow the instructions to start cleaning mode.

[1067] Step 5:

[1068] Terminal: After cleaning is completed, the information is sent to the server.

[1069] Step 6:

[1070] Server: Receives the completion information and sends a notification to the smartphone app.

[1071] Step 7:

[1072] User: Receives a notification on the smartphone app that "cleaning is complete."

[1073] 7. Processing of Emotion Engine Means

[1074] Step 1:

[1075] User: The user expresses emotions through tone of voice and facial expressions when speaking to the robot.

[1076] Step 2:

[1077] On the device: The emotion engine captures the user's tone of voice and facial expressions and analyzes their emotional state. For example, it determines that the user is "angry" based on their tone of voice and facial expressions.

[1078] Step 3:

[1079] Terminal: Transmits the analyzed emotion data to the server.

[1080] Step 4:

[1081] Server: Receives emotion data, analyzes it, stores it in a database, and generates appropriate response data based on the emotion data.

[1082] Step 5:

[1083] Server: Sends the generated response data to the terminal.

[1084] Step 6:

[1085] Terminal: Based on the response data received from the server, the terminal uses a speech synthesis engine to generate an appropriate voice response, such as "Taro Tanaka, what's the matter?"

[1086] Step 7:

[1087] Device: The generated voice is played back through the speaker, and the dialogue is tailored to the user's emotions.

[1088] Example 2

[1089] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1090] Current home robots are unable to provide integrated functions such as natural dialogue with users, multilingual support, facial recognition, emotion recognition, and real-time environmental detection using environmental sensors. Therefore, in order to improve users' quality of life, it is necessary to integrate these diverse functions into a single system and ensure their smooth collaboration.

[1091] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a voice recognition means, a means for storing text data in a database, a multilingual translation means, an image recognition means for recognizing faces, an environment detection means used to detect abnormal odors, a means for operating in cooperation with other systems, and an emotion recognition means for analyzing emotions and responding appropriately. This makes it possible to perform natural dialogue with the user, detect anomalies, and provide multifunctional support in an integrated manner.

[1092] "Speech recognition means" is a technology that captures the voice uttered by the user and converts the voice data into text data.

[1093] "Means for saving text data in a database" refers to a technique for sending the converted text data to a server via a request and saving it in a database.

[1094] "Multilingual translation means" refers to a technology for translating text data into other languages.

[1095] "Image recognition means for recognizing faces" is a technology that analyzes facial images captured by a camera and identifies individuals.

[1096] "Environmental detection means used to detect anomalous odors" refers to technology that uses an olfactory sensor to detect anomalous odors in the environment and analyzes the data.

[1097] "Means of operating in cooperation with other systems" refers to technology that communicates with external systems and applications, exchanging instructions and information to perform operations.

[1098] "Emotion recognition means for analyzing emotions and responding appropriately" is a technology that analyzes emotions from the user's tone of voice and facial expressions and generates an appropriate response according to those emotions.

[1099] MODE FOR CARRYING OUT THE INVENTION

[1100] An embodiment of the next-generation robot "maid-type" system of the present invention will now be described in detail. This system includes a voice recognition unit, a unit for storing text data in a database, a multilingual translation unit, an image recognition unit for recognizing faces, an environment detection unit used to detect odors, a unit for operating in cooperation with other systems, and an emotion recognition unit for analyzing emotions and responding appropriately.

[1101] Voice recognition means

[1102] The user speaks to the robot. For example, "Hello, my name is Tanaka Taro." The device uses the robot's built-in microphone to capture the voice, and sends the voice data to the Google Cloud Speech-to-Text API, where it is converted into text data in real time. In this way, the text data "Tanaka Taro" is generated.

[1103] A means of storing text data in a database

[1104] The device sends the generated text data to the server as an HTTP request. This request also includes metadata such as the user ID. The server analyzes the received text data and stores it in a MySQL database along with the user ID. For example, "User ID: 1234" and "Name: Taro Tanaka" are saved.

[1105] Multilingual translation tools

[1106] The user speaks to the robot, "Can you speak English?" The device acquires the text data, and if multilingual support is required, sends the data to the server via an HTTP request. The server then sends the received data to the Google Cloud Translation API, which translates the Japanese "Can you speak English?" into English "Do you speak English?" The translation result is then sent to the device, and the robot responds to the user with "Do you speak English?"

[1107] Image recognition method for recognizing faces

[1108] The user turns their face towards the robot's camera. The device's camera captures the facial image and sends the image data to the server via an HTTP request. The server uses an image recognition system to analyze the facial image and compare it with a database to identify the individual "Taro Tanaka." The identified user ID is then sent to the device.

[1109] Environmental sensing means used to detect off-odors

[1110] When the device's olfactory sensor detects a gas leak, it sends the data to the server in real time. The server analyzes the received odor data, and if it identifies a gas leak, it sends a warning to the device. Based on the warning information from the server, the device generates an audio warning saying, "There is a possibility of a gas leak. Please ventilate immediately," and plays it through the speaker.

[1111] A means of interacting with other systems

[1112] The user issues a command to "start cleaning" on the smartphone app. The device receives the command from the app and sends it to the server via an HTTP request. The server analyzes the command and instructs the device to start cleaning mode. The device starts the cleaning function and sends that information to the server when cleaning is complete. The server then sends a completion notification to the smartphone app, and the user receives a notification that "cleaning is complete."

[1113] Emotion recognition means for analyzing emotions and responding appropriately

[1114] When the user speaks to the robot, they express their emotions through their voice tone and facial expressions. The device uses an emotion recognition engine to analyze the voice tone and facial expressions to generate emotional data about the user. This data is then sent to the server via an HTTP request. The server analyzes the received emotional data and generates an appropriate response, such as "Taro Tanaka, what's the matter?" The device uses a speech synthesis engine to generate a response based on the response data received from the server, and plays it back from the speaker.

[1115] Prompt Sentence Examples

[1116] "Hello, my name is Tanaka Taro. Do you speak English?"

[1117] This system integrates natural dialogue with the user, anomaly detection, and multifunctional support, improving the quality of everyday life.

[1118] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1119] Program processing flow and specific explanation

[1120] Voice recognition means

[1121] Step 1:

[1122] The user speaks to the robot, saying something like, "Hello, my name is Taro Tanaka."

[1123] Step 2:

[1124] The device captures voice using a built-in microphone, and the input voice data is sent to a voice recognition engine.

[1125] Step 3:

[1126] The device converts the captured audio into text data using the Google Cloud Speech-to-Text API. The input is audio data, and the output is the text data "Taro Tanaka."

[1127] A means of storing text data in a database

[1128] Step 4:

[1129] The terminal creates an HTTP request containing the converted text data "Taro Tanaka." The input is the text data and metadata such as the user ID.

[1130] Step 5:

[1131] The terminal sends an HTTP request to the "Apache Tomcat" server. The input is the HTTP request, and the output is the result of the request sent to the server.

[1132] Step 6:

[1133] The server analyzes the received request and saves the text data and user ID in a MySQL database. The input is the HTTP request, and the output is the result saved in the database.

[1134] Multilingual translation tools

[1135] Step 7:

[1136] The user asks the robot, "Can you speak English?"

[1137] Step 8:

[1138] The terminal creates an HTTP request including the text data and sends it to the server. The input is the text data, and the output is the result of the request sent to the server.

[1139] Step 9:

[1140] The server calls the Google Cloud Translation API to translate the Japanese text "Do you speak English?" into English "Do you speak English?" The input is Japanese text and the output is English text.

[1141] Step 10:

[1142] The server sends the translated text data to the terminal. The input is the translation result, and the output is the data sent to the terminal.

[1143] Step 11:

[1144] The terminal responds to the user by voice, asking, "Do you speak English?" The input is English text, and the output is a voice response.

[1145] Image recognition method for recognizing faces

[1146] Step 12:

[1147] The user faces the robot's camera.

[1148] Step 13:

[1149] The device captures a facial image with a camera and sends the image data to the server via an HTTP request. The input is the facial image data, and the output is the result of the request sent to the server.

[1150] Step 14:

[1151] The server analyzes the facial image using an image recognition system and compares it with a database to identify the individual named "Taro Tanaka." The input is facial image data, and the output is the identification result.

[1152] Step 15:

[1153] The server sends the identified user ID to the terminal. The input is the identification result, and the output is the transmission to the terminal.

[1154] Environmental sensing means used to detect off-odors

[1155] Step 16:

[1156] When the olfactory sensor detects a gas leak, the device sends the data to the server in real time. The input is the odor sensor data, and the output is the request sent to the server.

[1157] Step 17:

[1158] The server analyzes the received odor data and compares it with known odor patterns to identify a "gas leak." The input is the odor data, and the output is the analysis results.

[1159] Step 18:

[1160] When the server identifies a gas leak, it sends the warning information to the terminal. The input is the analysis result, and the output is the message sent to the terminal.

[1161] Step 19:

[1162] The terminal generates a voice warning based on the warning information and plays a voice message saying, "There is a possibility of a gas leak. Please ventilate immediately." The input is the warning information and the output is the voice warning.

[1163] A means of interacting with other systems

[1164] Step 20:

[1165] The user issues a command to "start cleaning" via the smartphone app.

[1166] Step 21:

[1167] The device receives instructions from the smartphone app and sends them to the server via an HTTP request. The input is the instruction data, and the output is the result of the request sent to the server.

[1168] Step 22:

[1169] The server analyzes the received command and instructs the terminal to start cleaning mode. The input is the command data, and the output is sent to the terminal.

[1170] Step 23:

[1171] The terminal starts the cleaning function and sends the cleaning completion information to the server when the cleaning is completed. The input is the cleaning completion data, and the output is the request sent to the server.

[1172] Step 24:

[1173] The server sends a completion notification to the smartphone app, and the user receives a notification that "cleaning is complete." The input is the completion data, and the output is the notification.

[1174] Emotion recognition means for analyzing emotions and responding appropriately

[1175] Step 25:

[1176] When a user speaks to a robot, they express their emotions through their tone of voice and facial expressions.

[1177] Step 26:

[1178] The device analyzes voice tones and facial expressions using an emotion recognition engine, generates emotional data, and sends it to the server via an HTTP request. The input is voice tones and facial expression data, and the output is emotional data.

[1179] Step 27:

[1180] The server analyzes the received emotion data and generates an appropriate response, such as "Taro Tanaka, what's wrong?" The input is emotion data, and the output is response data.

[1181] Step 28:

[1182] The server sends the response data to the terminal. The input is the response data, and the output is the data sent to the terminal.

[1183] Step 29:

[1184] The device generates a response using a speech synthesis engine and plays it back through the speaker. The input is the response data, and the output is the voice response.

[1185] (Application example 2)

[1186] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1187] Modern factories require efficient systems to improve productivity and ensure worker safety. However, current robot systems and automation solutions are often single-function and unable to handle multiple tasks simultaneously and flexibly. Furthermore, there is a lack of systems that can handle environments with a mix of multinational workers and smoothly detect and respond to abnormal situations. This leads to a decline in the operational efficiency of the entire factory.

[1188] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1189] In this invention, the server includes a voice recognition means, a means for storing personal data in a database, a multilingual translation means, an image recognition means, an environment detection means using an olfactory sensor, a means for operating in cooperation with other systems, an emotion engine means for recognizing the user's emotions, and an automatic operation means for supporting factory work. This makes it possible to perform multiple tasks in an integrated and efficient manner, while supporting a variety of languages, improving the safety of the work environment, and enabling flexible responses that take into account the emotional state of workers.

[1190] - "Speech recognition means" refers to technology that captures a user's voice and converts it into text data.

[1191] "Means for storing personal data in a database" refers to technology that stores information related to individuals, such as a user's name or ID, in a database.

[1192] "Multilingual translation means" is a technology that translates text entered in one language into another language.

[1193] "Image recognition means" refers to technology that identifies specific objects or people from image data captured by a camera or other sensor.

[1194] "Environmental detection means using olfactory sensors" is a technology that detects specific odors and gases and analyzes the data.

[1195] "Means for operating in cooperation with other systems" refers to technology for exchanging instructions and data in cooperation with other information systems and devices.

[1196] The "emotion engine means for recognizing user emotions" is a technology that analyzes the user's tone of voice and facial expressions to identify their emotional state.

[1197] "Automatic operation means for supporting factory work" refers to technology that automatically executes specific work within a factory and provides efficient support.

[1198] In this invention, a next-generation robot system is used to efficiently support factory work. First, an overview of the entire system will be described below.

[1199] The server includes a voice recognition means, a means for storing personal data in a database, a multilingual translation means, an image recognition means, an environment detection means using an olfactory sensor, a means for operating in cooperation with other systems, an emotion engine means for recognizing the user's emotions, and an automatic operation means for supporting factory work.

[1200] Hardware

[1201] The following hardware is used to implement this system:

[1202] Smartphone: User interface and audio capture.

[1203] Robot: Equipped with various sensors (audio, image, olfactory) and actuators.

[1204] software

[1205] The system software includes:

[1206] Flask: A web application framework that acts as an API server.

[1207] SpeechRecognition: A speech recognition library that uses the Google Web Speech API.

[1208] GoogleTrans: A multilingual translation library that uses the Google Translate API.

[1209] ImageRecognizer: A custom library for recognition, specifically OpenCV.

[1210] EmotionRecognizer: A library for emotion recognition, specifically using TensorFlow and Keras.

[1211] Data Flow and Processing

[1212] 1. Voice data: When a user speaks into the smartphone, the voice recognition library converts the speech into text data, which is sent to the server and the corresponding instructions are sent to the robot.

[1213] 2. Personal data: Personal information such as the user's name and ID is stored on the server, enabling customized responses to specific users.

[1214] 3. Multilingual Translation: When users give instructions in different languages, the text data is translated by the multilingual translation library and converted into the appropriate language.

[1215] 4. Image Recognition: When a user points the robot's camera at a specific area or object, the image recognition library analyzes the data and performs the corresponding action.

[1216] 5. Environmental detection: When the robot's olfactory sensors detect an unusual odor, the data is sent to a server, and an alert is issued based on the analysis results.

[1217] 6. Emotion Recognition: Recognize the user's emotions from their tone of voice and facial expressions and respond appropriately.

[1218] 7. Automated operation: Specific tasks within the factory are carried out automatically to provide efficient support.

[1219] Specific examples

[1220] For example, a factory worker might use a smartphone app to give a robot a voice command such as "Clean the warehouse." This voice is analyzed by a speech recognition library and converted into text data. This data is then sent to a server, and the robot begins cleaning the warehouse in accordance with the command.

[1221] Prompt Sentence Examples

[1222] "Imagine a smartphone app for voice control of next-generation factory robots. Include features like voice recognition, image recognition, emotion recognition, multilingual translation, environmental monitoring, and integration with other systems."

[1223] The above is a specific embodiment for applying a next-generation robot system to factory work. This system enables multiple tasks to be performed in an integrated and efficient manner, supports a variety of languages, enhances the safety of the work environment, and enables flexible responses that take into consideration the feelings of workers.

[1224] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1225] Step 1:

[1226] Input: The user gives voice instructions to the smartphone.

[1227] Processing: The device captures the audio and converts it into text using the SpeechRecognition library. Specifically, the smartphone's microphone captures the audio data and passes it to the SpeechRecognition library.

[1228] Output: The converted text data is generated.

[1229] Step 2:

[1230] Input: Generated text data.

[1231] Processing: The device sends text data to the server via an HTTP request. The HTTP request also includes metadata such as the user ID. Specifically, the device app calls the data sending function to the server.

[1232] Output: Text data and metadata are sent to the server.

[1233] Step 3:

[1234] Input: The text data sent to the server.

[1235] Processing: The server analyzes the text data, and if it determines that multilingual translation is necessary, it calls the multilingual translation library (GoogleTrans) to perform the translation. Specifically, it calls the translation API on the server and converts the text into the specified language.

[1236] Output: The translated text data.

[1237] Step 4:

[1238] Input: Translated text data.

[1239] Processing: The server sends the translated data to the terminal again using an HTTP request. Specifically, the server returns the data to the terminal using the HTTP sending function on the server.

[1240] Output: The translated text data is received on the terminal.

[1241] Step 5:

[1242] Input: Translation text and images received on the device.

[1243] Processing: The user points the robot at a designated area and captures the image data, which is then sent to the device's image recognition library (ImageRecognizer) for analysis. Specifically, the device's camera function is used to capture the image, and the image recognition library identifies specific objects and areas.

[1244] Output: The analyzed image data is generated and sent to the server.

[1245] Step 6:

[1246] Input: Parsed image data.

[1247] Processing: The data is processed on the server and stored in a database together with environmental information. Specifically, the analyzed image data and environmental information are recorded in the database using the server's data storage function.

[1248] Output: Environmental data stored in a database.

[1249] Step 7:

[1250] Input: Odor data sent from the robot's olfactory sensor.

[1251] Processing: The odor data is analyzed on the server and matched with known odor patterns to identify the cause. Specifically, the odor analysis algorithm on the server compares the data with odor patterns in the database to identify the cause.

[1252] Output: Identified odor causes and associated warning data.

[1253] Step 8:

[1254] Input: Data on identified off-odor causes.

[1255] Processing: The server generates warning data and sends it to the terminal via an HTTP request. Specifically, the warning generation algorithm on the server creates a warning message based on the cause data and sends it to the terminal.

[1256] Output: The warning message sent to the terminal.

[1257] Step 9:

[1258] Input: Emotion data from the user (voice and facial expressions).

[1259] Processing: The device's emotion recognition engine (EmotionRecognizer) analyzes the emotion data to determine the user's emotional state. Specifically, it analyzes voice tone and facial expression data to identify the emotional state.

[1260] Output: The determined emotional state data is generated and sent to the server.

[1261] Step 10:

[1262] Input: Determined emotional state data.

[1263] Processing: The server analyzes the emotion data and generates an appropriate response message. Specifically, a response generation algorithm on the server creates an appropriate response message based on the emotion.

[1264] Output: The voice message generated using the speech synthesis engine is sent to the terminal and played back to the user.

[1265] Step 11:

[1266] Input: Automatic operation instructions for factory work.

[1267] Processing: The server analyzes the automatic operation instructions and sends the instructions to the corresponding robot. Specifically, the server uses a control algorithm to send the instructions to the robot's control system.

[1268] Output: The robot performs factory work according to instructions and returns the resulting data to the server.

[1269] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1270] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1271] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1272] [Third embodiment]

[1273] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1274] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1275] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1276] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1277] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1278] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1279] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1280] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1281] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1282] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1283] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1284] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1285] An embodiment of the next-generation robot "maid-type" system of the present invention is described in detail below. This system includes a voice recognition unit, a unit for storing personal data in a database, a multilingual translation unit, an image recognition unit, an environment detection unit using an olfactory sensor, and a unit for operating in cooperation with other systems.

[1286] 1. Voice Recognition Method

[1287] User: The user speaks to the robot, for example, "Hello, my name is Taro Tanaka."

[1288] Terminal: The robot's microphone captures the voice. The captured voice is converted into text data by a voice recognition engine, and the text data "Taro Tanaka" is generated.

[1289] 2. How personal data is stored in the database

[1290] Terminal: The converted text data is sent to the server via an HTTP request, which includes metadata such as the user ID.

[1291] Server: The server analyzes the received text data and stores it in a database. For example, the user name "Taro Tanaka" and the corresponding ID are stored.

[1292] 3. Multilingual Translation Tools

[1293] Terminal: Obtains user input as text data, and if multilingual support is required, sends the text data to the server. For example, it asks, "Do you speak English?"

[1294] Server: The server receives the text data and calls the translation API to translate it. The Japanese text "Do you speak English?" is translated into English "Do you speak English?" and the translated data is sent to the device.

[1295] 4. Image Recognition Methods

[1296] User: The user faces the robot's camera.

[1297] Device: The camera captures the user's face and sends the image data to a server. The image recognition system analyzes the face and matches it with a database to identify the individual.

[1298] Server: The image recognition algorithm recognizes the user's face as "Taro Tanaka" and sends the user ID to the device.

[1299] 5. Environmental detection using olfactory sensors

[1300] Terminal: When the olfactory sensor detects an unusual odor, it sends the data to the server in real time. For example, if it detects a specific odor like a gas leak.

[1301] Server: The server analyzes the odor data it receives and identifies the cause by comparing it with known odor patterns. For example, it may identify a "gas leak" and send that data to the device.

[1302] Terminal: Based on the warning information from the server, a speech synthesis engine is used to generate a warning voice saying, "There is a possibility of a gas leak. Please ventilate immediately," and this is played from the speaker.

[1303] 6. Means of cooperation with other systems

[1304] User: Sends the command "Start cleaning" to the robot via the smartphone app.

[1305] Terminal: Sends instructions received from the app to the server.

[1306] Server: Analyzes the received instruction and sends an instruction to start cleaning mode to the terminal.

[1307] Device: The cleaning function starts, and when cleaning is completed, the information is sent to the server. The server notifies the smartphone app of the completion information, and the user receives a notification that "cleaning is completed."

[1308] As described above, the next-generation robot "maid-type" system of the present invention integrates multiple functions to support users' daily lives in many ways and improve their quality of life.

[1309] The processing flow will be explained below.

[1310] 1. Speech recognition processing

[1311] Step 1:

[1312] User: Talks to the robot and says, "Hello, my name is Taro Tanaka."

[1313] Step 2:

[1314] Device: Uses a microphone to capture the user's voice and sends the voice data to a speech recognition engine.

[1315] Step 3:

[1316] Terminal: The voice recognition engine converts the voice data into text data, generating the text data "Taro Tanaka."

[1317] 2. Processing of the means by which personal data is stored in a database

[1318] Step 1:

[1319] Terminal: Sends an HTTP request to the server containing the text data "Taro Tanaka." The request also contains metadata such as the user ID.

[1320] Step 2:

[1321] Server: Analyzes the received text data and stores it in a database along with the user ID.

[1322] Step 3:

[1323] Server: After saving is complete, it sends a response to the terminal indicating that the process was successful.

[1324] 3. Processing of multilingual translation means

[1325] Step 1:

[1326] User: Talks to the robot and asks, "Can you speak English?"

[1327] Step 2:

[1328] Terminal: Captures voice with a microphone and converts it into text data using a speech recognition engine. The converted text data is sent to the server.

[1329] Step 3:

[1330] Server: Sends the received text data to the translation API and performs the translation.

[1331] Step 4:

[1332] Server: Receives the translated text data "Do you speak English?" and sends it to the terminal.

[1333] Step 5:

[1334] Terminal: The translated text data is passed to a speech synthesis engine to generate speech data.

[1335] Step 6:

[1336] Device: Play the generated voice "Yes, I can speak English." from the speaker.

[1337] 4. Image Recognition Method Processing

[1338] Step 1:

[1339] User: Face the robot's camera.

[1340] Step 2:

[1341] Device: The camera captures the user's facial image and sends the image data to the server.

[1342] Step 3:

[1343] Server: Analyzes the received image data using an image recognition engine and identifies the user ID.

[1344] Step 4:

[1345] Server: Sends the identified user ID to the terminal.

[1346] Step 5:

[1347] Terminal: Based on the user ID received from the server, it matches it with a registered user name, generates a personalized response such as "You're Tanaka Taro, right?" and creates speech using a speech synthesis engine.

[1348] Step 6:

[1349] Device: Play the created audio "Hello, Taro Tanaka" from the speaker.

[1350] 5. Processing of environmental detection means using olfactory sensors

[1351] Step 1:

[1352] Terminal: The olfactory sensor detects strange odors in the environment.

[1353] Step 2:

[1354] Terminal: Transmits detected odor data to the server in real time.

[1355] Step 3:

[1356] Server: Analyzes the received odor data and compares it with known odor patterns.

[1357] Step 4:

[1358] Server: Identifies the cause of the strange odor and sends data such as "gas leak" to the terminal.

[1359] Step 5:

[1360] Device: Based on the received data, the speech synthesis engine generates a warning voice saying, "There is a possibility of a gas leak. Please ventilate immediately."

[1361] Step 6:

[1362] Device: Play the generated warning sound through the speaker.

[1363] 6. Processing of means for interoperating with other systems

[1364] Step 1:

[1365] User: Uses the smartphone app to send the command to the robot to "start cleaning."

[1366] Step 2:

[1367] Terminal: Sends instructions received from the app to the server.

[1368] Step 3:

[1369] Server: Analyzes the received instructions and instructs the cleaning robot to start cleaning mode.

[1370] Step 4:

[1371] Device: Follow the instructions to start cleaning mode.

[1372] Step 5:

[1373] Terminal: After cleaning is completed, the information is sent to the server.

[1374] Step 6:

[1375] Server: Receives the completion information and sends a notification to the smartphone app.

[1376] Step 7:

[1377] User: Receives a notification on the smartphone app that "cleaning is complete."

[1378] Example 1

[1379] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1380] Current robot systems use separate technologies such as voice recognition, image recognition, and environmental detection, making it difficult to integrate these pieces of information, resulting in an inconsistent user experience. Furthermore, they lack the ability to respond to odor detection in real time and provide easy user control, making them unable to effectively support users in their daily lives. Furthermore, their ability to communicate in multiple languages ​​to accommodate internationalization is insufficient, meaning more advanced support is required.

[1381] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1382] In this invention, the server includes a voice recognition unit, a unit for storing personal data in a database, a multilingual translation unit, an image recognition unit, an environment detection unit using an olfactory sensor, a unit for operating in cooperation with other systems, a unit for allowing a user to send instructions to the robot using a smartphone app and for the robot to operate based on those instructions, a unit for analyzing data in real time when an abnormal odor is detected and generating an alarm sound to notify the user, a unit for storing text data acquired by voice recognition in a database in association with a user ID, and a unit for displaying the translated text data to the user. This makes it possible to integrate various recognition technologies, environment detection, and user interfaces to respond to various user requests in real time and improve quality of life.

[1383] "Speech recognition means" refers to technology that captures the user's voice and converts it into text data.

[1384] "Means for storing personal data in a database" refers to the technology for storing text data obtained by a voice recognition means in a database in association with a user ID.

[1385] "Multilingual translation means" refers to a technology that translates text in one language into another language.

[1386] "Image recognition means" refers to technology that analyzes image data captured by a camera and recognizes specific objects (e.g., faces).

[1387] "Environmental detection means using olfactory sensors" refers to technology that uses odor-detecting sensors to detect specific odors and analyze their causes.

[1388] "Means of operating in cooperation with other systems" refers to technology that exchanges data with different systems and performs multiple operations in cooperation with them.

[1389] "A means by which a user sends instructions to a robot using a smartphone app, and the robot then performs an action based on those instructions" refers to a technology in which a user sends operating instructions to a robot via a smartphone app, and the robot performs an action in accordance with those instructions.

[1390] "Means for analyzing data in real time when an abnormal odor is detected and generating an audio warning to notify the user" refers to a technology that, when an olfactory sensor detects an abnormal odor, analyzes the data to identify the cause, generates an audio warning to notify the user in real time.

[1391] "Means of storing text data obtained by voice recognition in a database in association with a user ID" refers to a technology that stores text data obtained by voice recognition technology in a database together with the corresponding user ID.

[1392] The term "means for displaying translated text data to the user" refers to a technology for displaying text data translated by the multilingual translation means in a form that can be confirmed by the user.

[1393] The present invention relates to a next-generation robot system that supports users' daily lives. The system includes a voice recognition unit, a unit for storing personal data in a database, a multilingual translation unit, an image recognition unit, an environment detection unit using an olfactory sensor, a unit for operating in conjunction with other systems, a unit for sending instructions to the robot via a smartphone app and having the robot perform actions based on those instructions, a unit for analyzing data in real time when an abnormal odor is detected and generating an alarm sound to notify the user, a unit for storing text data acquired by voice recognition in a database in association with a user ID, and a unit for displaying the translated text data to the user.

[1394] 1. Voice Recognition Method

[1395] When a user speaks to the robot, the robot's microphone captures the voice. The voice data is converted into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text API). For example, if a user says, "Hello, my name is Tanaka Taro," the text data "Hello, my name is Tanaka Taro" is generated.

[1396] 2. How personal data is stored in the database

[1397] The converted text data is sent to the server via an HTTP request. The server analyzes the received data and stores the user name and ID in a database (e.g., MySQL). This allows the information obtained through voice recognition to be managed as personal data.

[1398] 3. Multilingual Translation Tools

[1399] When translating user input text into multiple languages, the text data is sent to the server. The server calls a translation API (e.g., Google Cloud Translation API) to obtain the translation result. For example, the text "Can you speak English?" is translated to "Do you speak English?" The translation result is displayed to the user.

[1400] 4. Image Recognition Methods

[1401] When a user points their face at the robot's camera, the camera captures their facial image. The image data is sent to a server and analyzed using an image recognition algorithm (e.g., Azure Face API). The server then matches the individual against a database and sends the user ID to the device.

[1402] 5. Environmental detection using olfactory sensors

[1403] When the robot's olfactory sensors detect an unusual odor, the data is sent to a server in real time. The server analyzes the data and identifies the source of the odor. For example, if a gas leak is detected, the server sends that information to the device. The device uses a speech synthesis engine (e.g., Amazon Polly) to generate a warning voice saying, "There is a possibility of a gas leak. Please ventilate immediately," and plays it over the speaker.

[1404] 6. Means of cooperation with other systems

[1405] The user sends a command to the robot via a smartphone app to "start cleaning." The command from the app is sent to a server, which analyzes it and then issues a command to the robot to start cleaning. Once cleaning is complete, the device sends the information back to the server, and the server sends a completion notification to the smartphone app. The user then receives a notification that "cleaning is complete."

[1406] Examples and prompts

[1407] Example 1: When a user says, "Where am I now?", the speech recognition engine generates text data of "Where am I now?", sends related information to the server, and returns the answer.

[1408] Example 2: When a user says, "What do you call this in English?", the translation result is displayed to the user as "What do you call this in English?" after going through speech recognition, translation, and display.

[1409] Example prompt for a generative AI model:

[1410] "Please translate the text into English. The original text is 'How do you say this in English?'"

[1411] "Please tell me the procedure for analyzing the speech recognition results and saving them to the database."

[1412] The system of the present invention is an integrated multi-functional robot system that responds to various user requests in real time and improves the quality of life.

[1413] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1414] 1. Voice Recognition Method

[1415] Step 1:

[1416] The user speaks to the robot.

[1417] Input: Voice data (e.g., "Hello, my name is Taro Tanaka")

[1418] Output: None

[1419] Specific action: The user speaks aloud to the robot.

[1420] Step 2:

[1421] The device captures audio using the robot's microphone.

[1422] Input: Audio data

[1423] Output: Raw audio file

[1424] What it does: The microphone collects audio data and creates an audio file in the internal memory.

[1425] Step 3:

[1426] The device's voice recognition engine converts the captured voice data into text data.

[1427] Input: raw audio file

[1428] Output: Text data (e.g. "Hello, my name is Taro Tanaka")

[1429] What happens: A speech recognition engine (e.g., Google Cloud Speech-to-Text API) analyzes the audio file and generates corresponding text data.

[1430] 2. How personal data is stored in the database

[1431] Step 1:

[1432] The terminal sends the converted text data and metadata such as the user ID to the server in the form of an HTTP request.

[1433] Input: Text data, user ID

[1434] Output: HTTP request

[1435] Specific behavior: The text data and user ID are included in the body of the HTTP request and sent to the server.

[1436] Step 2:

[1437] The server analyzes the received HTTP request and extracts the content.

[1438] Input: HTTP request

[1439] Output: User name, User ID

[1440] What happens: The server parses the body of the HTTP request and extracts the username and user ID.

[1441] Step 3:

[1442] The server stores the extracted data in a database.

[1443] Input: Username, User ID

[1444] Output: New record to the database

[1445] Specific behavior: The username and user ID are added as a new record to the database (e.g. MySQL).

[1446] 3. Multilingual Translation Tools

[1447] Step 1:

[1448] The terminal obtains the user's input text and, if multilingual support is required, sends it to the server.

[1449] Input: Text data (e.g., "Do you speak English?")

[1450] Output: HTTP request

[1451] Specific operation: The device creates and sends an HTTP request to send text data to the server.

[1452] Step 2:

[1453] The server receives the text data and calls the translation API to translate it into the specified target language.

[1454] Input: HTTP request, text data

[1455] Output: Translated text data (e.g., "Do you speak English?")

[1456] Specific operation: Send a request to a translation API (e.g., Google Cloud Translation API) and obtain the translation result.

[1457] Step 3:

[1458] The server returns the translation results to the terminal.

[1459] Input: Translated text data

[1460] Output: HTTP response

[1461] Specific operation: Generates an HTTP response containing the translation result and sends it to the device.

[1462] Step 4:

[1463] The terminal displays the translation results to the user.

[1464] Input: Translated text data

[1465] Output: The text that is displayed in the user interface

[1466] Specific operation: The translation results are displayed on the user interface (display or voice reading device).

[1467] 4. Image Recognition Methods

[1468] Step 1:

[1469] The user faces the robot's camera.

[1470] Input: Face image

[1471] Output: None

[1472] Specific action: The user faces the camera directly.

[1473] Step 2:

[1474] The device captures a facial image using the robot's camera.

[1475] Input: Face image

[1476] Output: Image file

[1477] What happens: The camera captures a facial image and generates an image file.

[1478] Step 3:

[1479] The terminal transmits the image data to the server.

[1480] Input: Image file

[1481] Output: HTTP request

[1482] Specific operation: Sends an image file to the server in the form of an HTTP request.

[1483] Step 4:

[1484] The server analyzes the data using image recognition algorithms.

[1485] Input: Image file

[1486] Output: User ID

[1487] Specific operation: Analyzes the image using an image recognition algorithm (e.g. Azure Face API) and identifies the user.

[1488] Step 5:

[1489] The server sends the user ID to the terminal.

[1490] Input: User ID

[1491] Output: HTTP response

[1492] Specific operation: Generate an HTTP response including the user ID and send it to the terminal.

[1493] 5. Environmental detection using olfactory sensors

[1494] Step 1:

[1495] The device detects strange odors using an olfactory sensor.

[1496] Input: Environmental data (odor data)

[1497] Output: Odor detection event

[1498] How it works: The olfactory sensor detects specific odors in the environment.

[1499] Step 2:

[1500] The device transmits the detection data to the server in real time.

[1501] Input: Off-odor data

[1502] Output: HTTP request

[1503] Specific operation: Odor data is sent to the server in real time in the form of an HTTP request.

[1504] Step 3:

[1505] The server analyzes the odor data and compares it with known odor patterns to identify the cause.

[1506] Input: Off-odor data

[1507] Output: Data on the cause of the odor (e.g., gas leak)

[1508] Specific operation: Odor data is compared with a database of known odor patterns to identify the cause.

[1509] Step 4:

[1510] The server sends data on the cause of the odor to the terminal.

[1511] Input: Odor source data

[1512] Output: HTTP response

[1513] Specific operation: Generate an HTTP response including data on the cause of the odor and send it to the terminal.

[1514] Step 5:

[1515] The device generates and plays an audio warning.

[1516] Input: HTTP response (odor cause data)

[1517] Output: Audio warning

[1518] Specific behavior: Generates a warning message using a speech synthesis engine (e.g., Amazon Polly) and plays it over the speaker.

[1519] 6. Means of cooperation with other systems

[1520] Step 1:

[1521] The user sends instructions to the robot via a smartphone app.

[1522] Input: Instruction data (e.g. "Start cleaning")

[1523] Output: HTTP request

[1524] Specific operation: The user operates the app and sends commands to the robot.

[1525] Step 2:

[1526] The device sends the instructions received from the app to the server.

[1527] Input: HTTP request

[1528] Output: HTTP request

[1529] Specific operation: The device receives instructions from the app and forwards them to the server.

[1530] Step 3:

[1531] The server analyzes the received instructions and sends appropriate commands to the terminal.

[1532] Input: HTTP request (instruction data)

[1533] Output: Command data

[1534] Specific operation: The server analyzes the instruction data, generates the necessary commands, and sends them to the terminal.

[1535] Step 4:

[1536] The terminal will trigger the robot to start the cleaning function.

[1537] Input: Command data (e.g. "Start cleaning")

[1538] Output: Robot movement

[1539] Specific operation: The terminal transmits commands to the robot to start the cleaning function.

[1540] Step 5:

[1541] The terminal transmits cleaning completion data to the server.

[1542] Input: Cleaning completion data

[1543] Output: HTTP request

[1544] Specific operation: When cleaning is completed, the device sends the information to the server.

[1545] Step 6:

[1546] The server notifies the app of the cleaning completion data.

[1547] Input: HTTP request (cleaning completion data)

[1548] Output: Smartphone app notification

[1549] Specific operation: The server sends cleaning completion data to the app and notifies the user.

[1550] (Application example 1)

[1551] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1552] Conventional customer service systems in brick-and-mortar stores lack multilingual support for customer service and make it difficult to provide personalized service to repeat customers. Furthermore, emergency response capabilities, such as environmental and odor detection, are inadequate. For these reasons, there is a need to build a next-generation customer service system that can support multiple languages ​​in brick-and-mortar stores, provide personalized service for each individual customer, and ensure environmental safety.

[1553] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1554] In this invention, the server includes a voice recognition means, a means for storing personal data in a database, a multilingual translation means, an image recognition means, an environment detection means using an olfactory sensor, a means for operating in cooperation with other systems, and a means for functioning as a customer service assistant in a physical store. This makes it possible to respond to customers in multiple languages ​​in a physical store, provide personalized services according to each customer, and ensure a safe environment.

[1555] "Speech recognition means" refers to technology that takes voice input and converts that voice into text data.

[1556] "Means for storing personal data in a database" refers to the technology that collects information about users and stores it in an organized form in a database.

[1557] "Multilingual translation means" refers to a technology that translates input text data into multiple different languages.

[1558] "Image recognition means" refers to technology that analyzes image data acquired through devices such as cameras and identifies specific people or objects.

[1559] "Environmental detection means using olfactory sensors" refers to technology that uses sensors that detect various odors to detect specific environmental changes or unusual odors.

[1560] "Means for operating in cooperation with other systems" refers to technology for communicating with external systems or devices and executing specific operations.

[1561] "Means to function as a customer service assistant in a physical store" refers to technology that handles customer service within a physical store and provides multilingual support and personalized service.

[1562] This invention is a system that realizes efficient customer service and multilingual support by using a next-generation robot "maid-type" system as a customer service assistant in a brick-and-mortar store. This system includes a voice recognition means, a means for storing personal data in a database, a multilingual translation means, an image recognition means, an environmental detection means using an olfactory sensor, and a means for operating in cooperation with other systems.

[1563] Voice recognition means

[1564] When a user speaks to the robot, the voice is captured by the device's microphone, and a speech recognition engine (such as Google Web Speech API) converts the voice into text data and sends it to the server.

[1565] How personal data is stored in a database

[1566] The device sends the converted text data to the server via an HTTP request, where it is analyzed along with metadata such as the user ID and stored in a database. This storage process uses a relational or NoSQL database.

[1567] Multilingual translation tools

[1568] When a user makes a multilingual inquiry to the robot, the device sends the text data to the server. The server then calls a translation API to translate the input text data into the required language. The translated text data is then returned to the device and provided to the customer.

[1569] Image Recognition Method

[1570] When a user points their face at the robot's camera, the device's camera captures their face image, which is then sent to a server where image recognition algorithms (such as OpenCV or TensorFlow) are used to analyze the face and match it with a database to identify the user.

[1571] Environmental detection using olfactory sensors

[1572] When the device's built-in olfactory sensor detects an unusual odor, it sends the data to the server in real time. The server analyzes the data and generates a warning if it matches the odor pattern. This warning is output as voice by a speech synthesis engine and notified to the user.

[1573] A means of interacting with other systems

[1574] When a user sends a command to the robot via a smartphone app, the command is sent to the device. The server analyzes the received command and instructs the device to perform the appropriate operation. For example, if a command to start cleaning is sent, the robot will enter cleaning mode, and after cleaning is completed, the information will be sent to the server and the user will be notified.

[1575] Specific examples

[1576] When a customer asks the robot, "Where is this product in stock?", the voice is converted into text by a speech recognition device. The text data is then sent to the server, and related information is retrieved from the database and provided to the customer. In addition, if multilingual support is required, the text is translated into the appropriate language by a translation API.

[1577] Example prompts

[1578] "Write a Python program that uses speech recognition to convert what a customer says to a robot into text data, and then send that data along with the customer ID to a server for storage. The program will use the Google Web Speech API for speech recognition and the requests library to send the data to the server."

[1579] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1580] Step 1:

[1581] The user speaks to the robot. The input is the user's voice information, and the device's microphone captures this voice. Next, the voice data is converted into text data using a speech recognition engine (Google Web Speech API). The output is text data.

[1582] Step 2:

[1583] The terminal sends the converted text data and metadata (e.g., user ID) to the server as an HTTP request. The input is the text data and metadata, and the sent data is saved and analyzed on the server. The output is the saved data.

[1584] Step 3:

[1585] If a user makes a query that requires multilingual support, the device sends the text data back to the server. The input is the text data of the user's query, and the server calls the translation API to translate it into the specified language. The output is the text data in the translated language.

[1586] Step 4:

[1587] The server sends the translated data to the terminal, which then converts it into speech using a speech synthesis engine and transmits it to the user through a speaker. The input is the translated text data, and the output is the audio information provided to the user.

[1588] Step 5:

[1589] When a user points their face at the robot's camera, the device's camera captures a facial image and sends the image data to the server. The input is the captured facial image, and the server identifies the user using an image recognition algorithm (such as OpenCV or TensorFlow). The output is the identified user ID.

[1590] Step 6:

[1591] When the olfactory sensor installed in the device detects an abnormal odor, it sends the data to the server in real time. The input is the abnormal odor data detected by the olfactory sensor, and the server analyzes the data by comparing it with known abnormal odor patterns. The output is the identification of the abnormal odor and a warning message.

[1592] Step 7:

[1593] When the server identifies an abnormal odor, it sends a warning message to the device, which then uses a speech synthesis engine to generate a warning voice and notify the user through the speaker.The input is a warning message about the abnormal odor, and the output is the warning voice provided to the user.

[1594] Step 8:

[1595] When a user sends a command to a robot via a smartphone app, the command is sent to the server via the terminal. The input is the command data from the smartphone app, the server analyzes the received command, and sends appropriate operation instructions to the terminal. The output is the operation result executed by the terminal.

[1596] Step 9:

[1597] For example, when receiving the instruction to "start cleaning," the device starts cleaning mode, and when cleaning is complete, it sends that information to the server. The server notifies the smartphone app of the completion information, and the user receives a notification that "cleaning is complete." The input is the completion report data, and the output is the completion notification provided to the user.

[1598] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1599] A detailed description will be given of an embodiment of a next-generation robot "maid-type" system according to the present invention. This system includes a voice recognition unit, a unit for storing personal data in a database, a multilingual translation unit, an image recognition unit, an environment detection unit using an olfactory sensor, a unit for operating in cooperation with other systems, and an emotion engine for recognizing the user's emotions.

[1600] 1. Voice Recognition Method

[1601] User: The user speaks to the robot, for example, "Hello, my name is Taro Tanaka."

[1602] Terminal: The robot's microphone captures the voice. The captured voice is converted into text data by a voice recognition engine, and the text data "Taro Tanaka" is generated.

[1603] 2. How personal data is stored in the database

[1604] Terminal: The converted text data is sent to the server via an HTTP request, which includes metadata such as the user ID.

[1605] Server: The server analyzes the received text data and stores it in a database along with the user ID. For example, the user name "Taro Tanaka" and the corresponding ID are stored.

[1606] 3. Multilingual Translation Tools

[1607] Terminal: Obtains user input as text data, and if multilingual support is required, sends the text data to the server. For example, it asks, "Do you speak English?"

[1608] Server: The server receives the text data and calls the translation API to translate it. The Japanese text "Do you speak English?" is translated into English "Do you speak English?" and the translated data is sent to the device.

[1609] 4. Image Recognition Methods

[1610] User: The user faces the robot's camera.

[1611] Device: The camera captures the user's face and sends the image data to a server. The image recognition system analyzes the face and matches it with a database to identify the individual.

[1612] Server: The image recognition algorithm recognizes the user's face as "Taro Tanaka" and sends the user ID to the device.

[1613] 5. Environmental detection using olfactory sensors

[1614] Terminal: When the olfactory sensor detects an unusual odor, it sends the data to the server in real time. For example, if it detects a specific odor like a gas leak.

[1615] Server: The server analyzes the odor data it receives and identifies the cause by comparing it with known odor patterns. For example, it may identify a "gas leak" and send that data to the device.

[1616] Terminal: Based on the warning information from the server, a speech synthesis engine is used to generate a warning voice saying, "There is a possibility of a gas leak. Please ventilate immediately," and this is played from the speaker.

[1617] 6. Means of cooperation with other systems

[1618] User: Sends the command "Start cleaning" to the robot via the smartphone app.

[1619] Terminal: Sends instructions received from the app to the server.

[1620] Server: Analyzes the received instruction and sends an instruction to start cleaning mode to the terminal.

[1621] Device: The cleaning function starts, and when cleaning is completed, the information is sent to the server. The server notifies the smartphone app of the completion information, and the user receives a notification that "cleaning is completed."

[1622] 7. Emotional Engine Means

[1623] User: Emotions are expressed through the user's tone of voice and facial expressions when speaking to the robot.

[1624] On the device: The emotion recognition engine analyzes the user's tone of voice and facial expressions to detect their emotional state. For example, it determines that the user is "angry" based on their tone of voice and facial expressions.

[1625] Device: Sends an HTTP request containing the detected emotion data to the server.

[1626] Server: The server analyzes the received emotion data, stores it in a database, and generates appropriate response data based on the emotion data.

[1627] Terminal: Based on the response data received from the server, the terminal uses a speech synthesis engine to generate an appropriate voice response, such as "Taro Tanaka, what's the matter?"

[1628] Device: The generated voice is played back through the speaker, and the dialogue is tailored to the user's emotions.

[1629] As explained above in this embodiment, the next-generation robot "maid-type" system integrates multiple functions to support users' daily lives in many ways and improve their quality of life. In particular, the implementation of an emotion engine enables flexible responses according to emotions, resulting in a more natural and friendly user experience.

[1630] The processing flow will be explained below.

[1631] 1. Speech recognition processing

[1632] Step 1:

[1633] User: Talks to the robot and says, "Hello, my name is Taro Tanaka."

[1634] Step 2:

[1635] Device: Uses a microphone to capture the user's voice and sends the voice data to a speech recognition engine.

[1636] Step 3:

[1637] Terminal: The voice recognition engine converts the voice data into text data, generating the text data "Taro Tanaka."

[1638] 2. Processing of the means by which personal data is stored in a database

[1639] Step 1:

[1640] Terminal: Sends an HTTP request to the server containing the text data "Taro Tanaka." The request also contains metadata such as the user ID.

[1641] Step 2:

[1642] Server: Analyzes the received text data and stores it in a database along with the user ID.

[1643] Step 3:

[1644] Server: After saving is complete, it sends a response to the terminal indicating that the process was successful.

[1645] 3. Processing of multilingual translation means

[1646] Step 1:

[1647] User: Talks to the robot and asks, "Can you speak English?"

[1648] Step 2:

[1649] Terminal: Captures voice with a microphone and converts it into text data using a speech recognition engine. The converted text data is sent to the server.

[1650] Step 3:

[1651] Server: Sends the received text data to the translation API and performs the translation.

[1652] Step 4:

[1653] Server: Receives the translated text data "Do you speak English?" and sends it to the terminal.

[1654] Step 5:

[1655] Terminal: The translated text data is passed to a speech synthesis engine to generate speech data.

[1656] Step 6:

[1657] Device: Play the generated voice "Yes, I can speak English." from the speaker.

[1658] 4. Image Recognition Method Processing

[1659] Step 1:

[1660] User: Face the robot's camera.

[1661] Step 2:

[1662] Device: The camera captures the user's facial image and sends the image data to the server.

[1663] Step 3:

[1664] Server: Analyzes the received image data using an image recognition engine and identifies the user ID.

[1665] Step 4:

[1666] Server: Sends the identified user ID to the terminal.

[1667] Step 5:

[1668] Terminal: Based on the user ID received from the server, it matches it with a registered user name, generates a personalized response such as "You're Tanaka Taro, right?" and creates speech using a speech synthesis engine.

[1669] Step 6:

[1670] Device: Play the created audio "Hello, Taro Tanaka" from the speaker.

[1671] 5. Processing of environmental detection means using olfactory sensors

[1672] Step 1:

[1673] Terminal: The olfactory sensor detects strange odors in the environment.

[1674] Step 2:

[1675] Terminal: Transmits detected odor data to the server in real time.

[1676] Step 3:

[1677] Server: Analyzes the received odor data and compares it with known odor patterns.

[1678] Step 4:

[1679] Server: Identifies the cause of the strange odor and sends data such as "gas leak" to the terminal.

[1680] Step 5:

[1681] Device: Based on the received data, the speech synthesis engine generates a warning voice saying, "There is a possibility of a gas leak. Please ventilate immediately."

[1682] Step 6:

[1683] Device: Play the generated warning sound through the speaker.

[1684] 6. Processing of means for interoperating with other systems

[1685] Step 1:

[1686] User: Uses the smartphone app to send the command to the robot to "start cleaning."

[1687] Step 2:

[1688] Terminal: Sends instructions received from the app to the server.

[1689] Step 3:

[1690] Server: Analyzes the received instructions and instructs the cleaning robot to start cleaning mode.

[1691] Step 4:

[1692] Device: Follow the instructions to start cleaning mode.

[1693] Step 5:

[1694] Terminal: After cleaning is completed, the information is sent to the server.

[1695] Step 6:

[1696] Server: Receives the completion information and sends a notification to the smartphone app.

[1697] Step 7:

[1698] User: Receives a notification on the smartphone app that "cleaning is complete."

[1699] 7. Processing of Emotion Engine Means

[1700] Step 1:

[1701] User: The user expresses emotions through tone of voice and facial expressions when speaking to the robot.

[1702] Step 2:

[1703] On the device: The emotion engine captures the user's tone of voice and facial expressions and analyzes their emotional state. For example, it determines that the user is "angry" based on their tone of voice and facial expressions.

[1704] Step 3:

[1705] Terminal: Transmits the analyzed emotion data to the server.

[1706] Step 4:

[1707] Server: Receives emotion data, analyzes it, stores it in a database, and generates appropriate response data based on the emotion data.

[1708] Step 5:

[1709] Server: Sends the generated response data to the terminal.

[1710] Step 6:

[1711] Terminal: Based on the response data received from the server, the terminal uses a speech synthesis engine to generate an appropriate voice response, such as "Taro Tanaka, what's the matter?"

[1712] Step 7:

[1713] Device: The generated voice is played back through the speaker, and the dialogue is tailored to the user's emotions.

[1714] Example 2

[1715] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1716] Current home robots are unable to provide integrated functions such as natural dialogue with users, multilingual support, facial recognition, emotion recognition, and real-time environmental detection using environmental sensors. Therefore, in order to improve users' quality of life, it is necessary to integrate these diverse functions into a single system and ensure their smooth collaboration.

[1717] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a voice recognition means, a means for storing text data in a database, a multilingual translation means, an image recognition means for recognizing faces, an environment detection means used to detect abnormal odors, a means for operating in cooperation with other systems, and an emotion recognition means for analyzing emotions and responding appropriately. This makes it possible to perform natural dialogue with the user, detect anomalies, and provide multifunctional support in an integrated manner.

[1718] "Speech recognition means" is a technology that captures the voice uttered by the user and converts the voice data into text data.

[1719] "Means for saving text data in a database" refers to a technique for sending the converted text data to a server via a request and saving it in a database.

[1720] "Multilingual translation means" refers to a technology for translating text data into other languages.

[1721] "Image recognition means for recognizing faces" is a technology that analyzes facial images captured by a camera and identifies individuals.

[1722] "Environmental detection means used to detect anomalous odors" refers to technology that uses an olfactory sensor to detect anomalous odors in the environment and analyzes the data.

[1723] "Means of operating in cooperation with other systems" refers to technology that communicates with external systems and applications, exchanging instructions and information to perform operations.

[1724] "Emotion recognition means for analyzing emotions and responding appropriately" is a technology that analyzes emotions from the user's tone of voice and facial expressions and generates an appropriate response according to those emotions.

[1725] MODE FOR CARRYING OUT THE INVENTION

[1726] An embodiment of the next-generation robot "maid-type" system of the present invention will now be described in detail. This system includes a voice recognition unit, a unit for storing text data in a database, a multilingual translation unit, an image recognition unit for recognizing faces, an environment detection unit used to detect odors, a unit for operating in cooperation with other systems, and an emotion recognition unit for analyzing emotions and responding appropriately.

[1727] Voice recognition means

[1728] The user speaks to the robot. For example, "Hello, my name is Tanaka Taro." The device uses the robot's built-in microphone to capture the voice, and sends the voice data to the Google Cloud Speech-to-Text API, where it is converted into text data in real time. In this way, the text data "Tanaka Taro" is generated.

[1729] A means of storing text data in a database

[1730] The device sends the generated text data to the server as an HTTP request. This request also includes metadata such as the user ID. The server analyzes the received text data and stores it in a MySQL database along with the user ID. For example, "User ID: 1234" and "Name: Taro Tanaka" are saved.

[1731] Multilingual translation tools

[1732] The user speaks to the robot, "Can you speak English?" The device acquires the text data, and if multilingual support is required, sends the data to the server via an HTTP request. The server then sends the received data to the Google Cloud Translation API, which translates the Japanese "Can you speak English?" into English "Do you speak English?" The translation result is then sent to the device, and the robot responds to the user with "Do you speak English?"

[1733] Image recognition method for recognizing faces

[1734] The user turns their face towards the robot's camera. The device's camera captures the facial image and sends the image data to the server via an HTTP request. The server uses an image recognition system to analyze the facial image and compare it with a database to identify the individual "Taro Tanaka." The identified user ID is then sent to the device.

[1735] Environmental sensing means used to detect off-odors

[1736] When the device's olfactory sensor detects a gas leak, it sends the data to the server in real time. The server analyzes the received odor data, and if it identifies a gas leak, it sends a warning to the device. Based on the warning information from the server, the device generates an audio warning saying, "There is a possibility of a gas leak. Please ventilate immediately," and plays it through the speaker.

[1737] A means of interacting with other systems

[1738] The user issues a command to "start cleaning" on the smartphone app. The device receives the command from the app and sends it to the server via an HTTP request. The server analyzes the command and instructs the device to start cleaning mode. The device starts the cleaning function and sends that information to the server when cleaning is complete. The server then sends a completion notification to the smartphone app, and the user receives a notification that "cleaning is complete."

[1739] Emotion recognition means for analyzing emotions and responding appropriately

[1740] When the user speaks to the robot, they express their emotions through their voice tone and facial expressions. The device uses an emotion recognition engine to analyze the voice tone and facial expressions to generate emotional data about the user. This data is then sent to the server via an HTTP request. The server analyzes the received emotional data and generates an appropriate response, such as "Taro Tanaka, what's the matter?" The device uses a speech synthesis engine to generate a response based on the response data received from the server, and plays it back from the speaker.

[1741] Prompt Sentence Examples

[1742] "Hello, my name is Tanaka Taro. Do you speak English?"

[1743] This system integrates natural dialogue with the user, anomaly detection, and multifunctional support, improving the quality of everyday life.

[1744] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1745] Program processing flow and specific explanation

[1746] Voice recognition means

[1747] Step 1:

[1748] The user speaks to the robot, saying something like, "Hello, my name is Taro Tanaka."

[1749] Step 2:

[1750] The device captures voice using a built-in microphone, and the input voice data is sent to a voice recognition engine.

[1751] Step 3:

[1752] The device converts the captured audio into text data using the Google Cloud Speech-to-Text API. The input is audio data, and the output is the text data "Taro Tanaka."

[1753] A means of storing text data in a database

[1754] Step 4:

[1755] The terminal creates an HTTP request containing the converted text data "Taro Tanaka." The input is the text data and metadata such as the user ID.

[1756] Step 5:

[1757] The terminal sends an HTTP request to the "Apache Tomcat" server. The input is the HTTP request, and the output is the result of the request sent to the server.

[1758] Step 6:

[1759] The server analyzes the received request and saves the text data and user ID in a MySQL database. The input is the HTTP request, and the output is the result saved in the database.

[1760] Multilingual translation tools

[1761] Step 7:

[1762] The user asks the robot, "Can you speak English?"

[1763] Step 8:

[1764] The terminal creates an HTTP request including the text data and sends it to the server. The input is the text data, and the output is the result of the request sent to the server.

[1765] Step 9:

[1766] The server calls the Google Cloud Translation API to translate the Japanese text "Do you speak English?" into English "Do you speak English?" The input is Japanese text and the output is English text.

[1767] Step 10:

[1768] The server sends the translated text data to the terminal. The input is the translation result, and the output is the data sent to the terminal.

[1769] Step 11:

[1770] The terminal responds to the user by voice, asking, "Do you speak English?" The input is English text, and the output is a voice response.

[1771] Image recognition method for recognizing faces

[1772] Step 12:

[1773] The user faces the robot's camera.

[1774] Step 13:

[1775] The device captures a facial image with a camera and sends the image data to the server via an HTTP request. The input is the facial image data, and the output is the result of the request sent to the server.

[1776] Step 14:

[1777] The server analyzes the facial image using an image recognition system and compares it with a database to identify the individual named "Taro Tanaka." The input is facial image data, and the output is the identification result.

[1778] Step 15:

[1779] The server sends the identified user ID to the terminal. The input is the identification result, and the output is the transmission to the terminal.

[1780] Environmental sensing means used to detect off-odors

[1781] Step 16:

[1782] When the olfactory sensor detects a gas leak, the device sends the data to the server in real time. The input is the odor sensor data, and the output is the request sent to the server.

[1783] Step 17:

[1784] The server analyzes the received odor data and compares it with known odor patterns to identify a "gas leak." The input is the odor data, and the output is the analysis results.

[1785] Step 18:

[1786] When the server identifies a gas leak, it sends the warning information to the terminal. The input is the analysis result, and the output is the message sent to the terminal.

[1787] Step 19:

[1788] The terminal generates a voice warning based on the warning information and plays a voice message saying, "There is a possibility of a gas leak. Please ventilate immediately." The input is the warning information and the output is the voice warning.

[1789] A means of interacting with other systems

[1790] Step 20:

[1791] The user issues a command to "start cleaning" via the smartphone app.

[1792] Step 21:

[1793] The device receives instructions from the smartphone app and sends them to the server via an HTTP request. The input is the instruction data, and the output is the result of the request sent to the server.

[1794] Step 22:

[1795] The server analyzes the received command and instructs the terminal to start cleaning mode. The input is the command data, and the output is sent to the terminal.

[1796] Step 23:

[1797] The terminal starts the cleaning function and sends the cleaning completion information to the server when the cleaning is completed. The input is the cleaning completion data, and the output is the request sent to the server.

[1798] Step 24:

[1799] The server sends a completion notification to the smartphone app, and the user receives a notification that "cleaning is complete." The input is the completion data, and the output is the notification.

[1800] Emotion recognition means for analyzing emotions and responding appropriately

[1801] Step 25:

[1802] When a user speaks to a robot, they express their emotions through their tone of voice and facial expressions.

[1803] Step 26:

[1804] The device analyzes voice tones and facial expressions using an emotion recognition engine, generates emotional data, and sends it to the server via an HTTP request. The input is voice tones and facial expression data, and the output is emotional data.

[1805] Step 27:

[1806] The server analyzes the received emotion data and generates an appropriate response, such as "Taro Tanaka, what's wrong?" The input is emotion data, and the output is response data.

[1807] Step 28:

[1808] The server sends the response data to the terminal. The input is the response data, and the output is the data sent to the terminal.

[1809] Step 29:

[1810] The device generates a response using a speech synthesis engine and plays it back through the speaker. The input is the response data, and the output is the voice response.

[1811] (Application example 2)

[1812] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1813] Modern factories require efficient systems to improve productivity and ensure worker safety. However, current robot systems and automation solutions are often single-function and unable to handle multiple tasks simultaneously and flexibly. Furthermore, there is a lack of systems that can handle environments with a mix of multinational workers and smoothly detect and respond to abnormal situations. This leads to a decline in the operational efficiency of the entire factory.

[1814] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1815] In this invention, the server includes a voice recognition means, a means for storing personal data in a database, a multilingual translation means, an image recognition means, an environment detection means using an olfactory sensor, a means for operating in cooperation with other systems, an emotion engine means for recognizing the user's emotions, and an automatic operation means for supporting factory work. This makes it possible to perform multiple tasks in an integrated and efficient manner, while supporting a variety of languages, improving the safety of the work environment, and enabling flexible responses that take into account the emotional state of workers.

[1816] - "Speech recognition means" refers to technology that captures a user's voice and converts it into text data.

[1817] "Means for storing personal data in a database" refers to technology that stores information related to individuals, such as a user's name or ID, in a database.

[1818] "Multilingual translation means" is a technology that translates text entered in one language into another language.

[1819] "Image recognition means" refers to technology that identifies specific objects or people from image data captured by a camera or other sensor.

[1820] "Environmental detection means using olfactory sensors" is a technology that detects specific odors and gases and analyzes the data.

[1821] "Means for operating in cooperation with other systems" refers to technology for exchanging instructions and data in cooperation with other information systems and devices.

[1822] The "emotion engine means for recognizing user emotions" is a technology that analyzes the user's tone of voice and facial expressions to identify their emotional state.

[1823] "Automatic operation means for supporting factory work" refers to technology that automatically executes specific work within a factory and provides efficient support.

[1824] In this invention, a next-generation robot system is used to efficiently support factory work. First, an overview of the entire system will be described below.

[1825] The server includes a voice recognition means, a means for storing personal data in a database, a multilingual translation means, an image recognition means, an environment detection means using an olfactory sensor, a means for operating in cooperation with other systems, an emotion engine means for recognizing the user's emotions, and an automatic operation means for supporting factory work.

[1826] Hardware

[1827] The following hardware is used to implement this system:

[1828] Smartphone: User interface and audio capture.

[1829] Robot: Equipped with various sensors (audio, image, olfactory) and actuators.

[1830] software

[1831] The system software includes:

[1832] Flask: A web application framework that acts as an API server.

[1833] SpeechRecognition: A speech recognition library that uses the Google Web Speech API.

[1834] GoogleTrans: A multilingual translation library that uses the Google Translate API.

[1835] ImageRecognizer: A custom library for recognition, specifically OpenCV.

[1836] EmotionRecognizer: A library for emotion recognition, specifically using TensorFlow and Keras.

[1837] Data Flow and Processing

[1838] 1. Voice data: When a user speaks into the smartphone, the voice recognition library converts the speech into text data, which is sent to the server and the corresponding instructions are sent to the robot.

[1839] 2. Personal data: Personal information such as the user's name and ID is stored on the server, enabling customized responses to specific users.

[1840] 3. Multilingual Translation: When users give instructions in different languages, the text data is translated by the multilingual translation library and converted into the appropriate language.

[1841] 4. Image Recognition: When a user points the robot's camera at a specific area or object, the image recognition library analyzes the data and performs the corresponding action.

[1842] 5. Environmental detection: When the robot's olfactory sensors detect an unusual odor, the data is sent to a server, and an alert is issued based on the analysis results.

[1843] 6. Emotion Recognition: Recognize the user's emotions from their tone of voice and facial expressions and respond appropriately.

[1844] 7. Automated operation: Specific tasks within the factory are carried out automatically to provide efficient support.

[1845] Specific examples

[1846] For example, a factory worker might use a smartphone app to give a robot a voice command such as "Clean the warehouse." This voice is analyzed by a speech recognition library and converted into text data. This data is then sent to a server, and the robot begins cleaning the warehouse in accordance with the command.

[1847] Prompt Sentence Examples

[1848] "Imagine a smartphone app for voice control of next-generation factory robots. Include features like voice recognition, image recognition, emotion recognition, multilingual translation, environmental monitoring, and integration with other systems."

[1849] The above is a specific embodiment for applying a next-generation robot system to factory work. This system enables multiple tasks to be performed in an integrated and efficient manner, supports a variety of languages, enhances the safety of the work environment, and enables flexible responses that take into consideration the feelings of workers.

[1850] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1851] Step 1:

[1852] Input: The user gives voice instructions to the smartphone.

[1853] Processing: The device captures the audio and converts it into text using the SpeechRecognition library. Specifically, the smartphone's microphone captures the audio data and passes it to the SpeechRecognition library.

[1854] Output: The converted text data is generated.

[1855] Step 2:

[1856] Input: Generated text data.

[1857] Processing: The device sends text data to the server via an HTTP request. The HTTP request also includes metadata such as the user ID. Specifically, the device app calls the data sending function to the server.

[1858] Output: Text data and metadata are sent to the server.

[1859] Step 3:

[1860] Input: The text data sent to the server.

[1861] Processing: The server analyzes the text data, and if it determines that multilingual translation is necessary, it calls the multilingual translation library (GoogleTrans) to perform the translation. Specifically, it calls the translation API on the server and converts the text into the specified language.

[1862] Output: The translated text data.

[1863] Step 4:

[1864] Input: Translated text data.

[1865] Processing: The server sends the translated data to the terminal again using an HTTP request. Specifically, the server returns the data to the terminal using the HTTP sending function on the server.

[1866] Output: The translated text data is received on the terminal.

[1867] Step 5:

[1868] Input: Translation text and images received on the device.

[1869] Processing: The user points the robot at a designated area and captures the image data, which is then sent to the device's image recognition library (ImageRecognizer) for analysis. Specifically, the device's camera function is used to capture the image, and the image recognition library identifies specific objects and areas.

[1870] Output: The analyzed image data is generated and sent to the server.

[1871] Step 6:

[1872] Input: Parsed image data.

[1873] Processing: The data is processed on the server and stored in a database together with environmental information. Specifically, the analyzed image data and environmental information are recorded in the database using the server's data storage function.

[1874] Output: Environmental data stored in a database.

[1875] Step 7:

[1876] Input: Odor data sent from the robot's olfactory sensor.

[1877] Processing: The odor data is analyzed on the server and matched with known odor patterns to identify the cause. Specifically, the odor analysis algorithm on the server compares the data with odor patterns in the database to identify the cause.

[1878] Output: Identified odor causes and associated warning data.

[1879] Step 8:

[1880] Input: Data on identified off-odor causes.

[1881] Processing: The server generates warning data and sends it to the terminal via an HTTP request. Specifically, the warning generation algorithm on the server creates a warning message based on the cause data and sends it to the terminal.

[1882] Output: The warning message sent to the terminal.

[1883] Step 9:

[1884] Input: Emotion data from the user (voice and facial expressions).

[1885] Processing: The device's emotion recognition engine (EmotionRecognizer) analyzes the emotion data to determine the user's emotional state. Specifically, it analyzes voice tone and facial expression data to identify the emotional state.

[1886] Output: The determined emotional state data is generated and sent to the server.

[1887] Step 10:

[1888] Input: Determined emotional state data.

[1889] Processing: The server analyzes the emotion data and generates an appropriate response message. Specifically, a response generation algorithm on the server creates an appropriate response message based on the emotion.

[1890] Output: The voice message generated using the speech synthesis engine is sent to the terminal and played back to the user.

[1891] Step 11:

[1892] Input: Automatic operation instructions for factory work.

[1893] Processing: The server analyzes the automatic operation instructions and sends the instructions to the corresponding robot. Specifically, the server uses a control algorithm to send the instructions to the robot's control system.

[1894] Output: The robot performs factory work according to instructions and returns the resulting data to the server.

[1895] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1896] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1897] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1898] [Fourth embodiment]

[1899] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1900] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1901] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1902] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1903] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1904] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1905] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1906] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1907] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1908] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1909] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1910] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1911] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1912] An embodiment of the next-generation robot "maid-type" system of the present invention is described in detail below. This system includes a voice recognition unit, a unit for storing personal data in a database, a multilingual translation unit, an image recognition unit, an environment detection unit using an olfactory sensor, and a unit for operating in cooperation with other systems.

[1913] 1. Voice Recognition Method

[1914] User: The user speaks to the robot, for example, "Hello, my name is Taro Tanaka."

[1915] Terminal: The robot's microphone captures the voice. The captured voice is converted into text data by a voice recognition engine, and the text data "Taro Tanaka" is generated.

[1916] 2. How personal data is stored in the database

[1917] Terminal: The converted text data is sent to the server via an HTTP request, which includes metadata such as the user ID.

[1918] Server: The server analyzes the received text data and stores it in a database. For example, the user name "Taro Tanaka" and the corresponding ID are stored.

[1919] 3. Multilingual Translation Tools

[1920] Terminal: Obtains user input as text data, and if multilingual support is required, sends the text data to the server. For example, it asks, "Do you speak English?"

[1921] Server: The server receives the text data and calls the translation API to translate it. The Japanese text "Do you speak English?" is translated into English "Do you speak English?" and the translated data is sent to the device.

[1922] 4. Image Recognition Methods

[1923] User: The user faces the robot's camera.

[1924] Device: The camera captures the user's face and sends the image data to a server. The image recognition system analyzes the face and matches it with a database to identify the individual.

[1925] Server: The image recognition algorithm recognizes the user's face as "Taro Tanaka" and sends the user ID to the device.

[1926] 5. Environmental detection using olfactory sensors

[1927] Terminal: When the olfactory sensor detects an unusual odor, it sends the data to the server in real time. For example, if it detects a specific odor like a gas leak.

[1928] Server: The server analyzes the odor data it receives and identifies the cause by comparing it with known odor patterns. For example, it may identify a "gas leak" and send that data to the device.

[1929] Terminal: Based on the warning information from the server, a speech synthesis engine is used to generate a warning voice saying, "There is a possibility of a gas leak. Please ventilate immediately," and this is played from the speaker.

[1930] 6. Means of cooperation with other systems

[1931] User: Sends the command "Start cleaning" to the robot via the smartphone app.

[1932] Terminal: Sends instructions received from the app to the server.

[1933] Server: Analyzes the received instruction and sends an instruction to start cleaning mode to the terminal.

[1934] Device: The cleaning function starts, and when cleaning is completed, the information is sent to the server. The server notifies the smartphone app of the completion information, and the user receives a notification that "cleaning is completed."

[1935] As described above, the next-generation robot "maid-type" system of the present invention integrates multiple functions to support users' daily lives in many ways and improve their quality of life.

[1936] The processing flow will be explained below.

[1937] 1. Speech recognition processing

[1938] Step 1:

[1939] User: Talks to the robot and says, "Hello, my name is Taro Tanaka."

[1940] Step 2:

[1941] Device: Uses a microphone to capture the user's voice and sends the voice data to a speech recognition engine.

[1942] Step 3:

[1943] Terminal: The voice recognition engine converts the voice data into text data, generating the text data "Taro Tanaka."

[1944] 2. Processing of the means by which personal data is stored in a database

[1945] Step 1:

[1946] Terminal: Sends an HTTP request to the server containing the text data "Taro Tanaka." The request also contains metadata such as the user ID.

[1947] Step 2:

[1948] Server: Analyzes the received text data and stores it in a database along with the user ID.

[1949] Step 3:

[1950] Server: After saving is complete, it sends a response to the terminal indicating that the process was successful.

[1951] 3. Processing of multilingual translation means

[1952] Step 1:

[1953] User: Talks to the robot and asks, "Can you speak English?"

[1954] Step 2:

[1955] Terminal: Captures voice with a microphone and converts it into text data using a speech recognition engine. The converted text data is sent to the server.

[1956] Step 3:

[1957] Server: Sends the received text data to the translation API and performs the translation.

[1958] Step 4:

[1959] Server: Receives the translated text data "Do you speak English?" and sends it to the terminal.

[1960] Step 5:

[1961] Terminal: The translated text data is passed to a speech synthesis engine to generate speech data.

[1962] Step 6:

[1963] Device: Play the generated voice "Yes, I can speak English." from the speaker.

[1964] 4. Image Recognition Method Processing

[1965] Step 1:

[1966] User: Face the robot's camera.

[1967] Step 2:

[1968] Device: The camera captures the user's facial image and sends the image data to the server.

[1969] Step 3:

[1970] Server: Analyzes the received image data using an image recognition engine and identifies the user ID.

[1971] Step 4:

[1972] Server: Sends the identified user ID to the terminal.

[1973] Step 5:

[1974] Terminal: Based on the user ID received from the server, it matches it with a registered user name, generates a personalized response such as "You're Tanaka Taro, right?" and creates speech using a speech synthesis engine.

[1975] Step 6:

[1976] Device: Play the created audio "Hello, Taro Tanaka" from the speaker.

[1977] 5. Processing of environmental detection means using olfactory sensors

[1978] Step 1:

[1979] Terminal: The olfactory sensor detects strange odors in the environment.

[1980] Step 2:

[1981] Terminal: Transmits detected odor data to the server in real time.

[1982] Step 3:

[1983] Server: Analyzes the received odor data and compares it with known odor patterns.

[1984] Step 4:

[1985] Server: Identifies the cause of the strange odor and sends data such as "gas leak" to the terminal.

[1986] Step 5:

[1987] Device: Based on the received data, the speech synthesis engine generates a warning voice saying, "There is a possibility of a gas leak. Please ventilate immediately."

[1988] Step 6:

[1989] Device: Play the generated warning sound through the speaker.

[1990] 6. Processing of means for interoperating with other systems

[1991] Step 1:

[1992] User: Uses the smartphone app to send the command to the robot to "start cleaning."

[1993] Step 2:

[1994] Terminal: Sends instructions received from the app to the server.

[1995] Step 3:

[1996] Server: Analyzes the received instructions and instructs the cleaning robot to start cleaning mode.

[1997] Step 4:

[1998] Device: Follow the instructions to start cleaning mode.

[1999] Step 5:

[2000] Terminal: After cleaning is completed, the information is sent to the server.

[2001] Step 6:

[2002] Server: Receives the completion information and sends a notification to the smartphone app.

[2003] Step 7:

[2004] User: Receives a notification on the smartphone app that "cleaning is complete."

[2005] Example 1

[2006] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2007] Current robot systems use separate technologies such as voice recognition, image recognition, and environmental detection, making it difficult to integrate these pieces of information, resulting in an inconsistent user experience. Furthermore, they lack the ability to respond to odor detection in real time and provide easy user control, making them unable to effectively support users in their daily lives. Furthermore, their ability to communicate in multiple languages ​​to accommodate internationalization is insufficient, meaning more advanced support is required.

[2008] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2009] In this invention, the server includes a voice recognition unit, a unit for storing personal data in a database, a multilingual translation unit, an image recognition unit, an environment detection unit using an olfactory sensor, a unit for operating in cooperation with other systems, a unit for allowing a user to send instructions to the robot using a smartphone app and for the robot to operate based on those instructions, a unit for analyzing data in real time when an abnormal odor is detected and generating an alarm sound to notify the user, a unit for storing text data acquired by voice recognition in a database in association with a user ID, and a unit for displaying the translated text data to the user. This makes it possible to integrate various recognition technologies, environment detection, and user interfaces to respond to various user requests in real time and improve quality of life.

[2010] "Speech recognition means" refers to technology that captures the user's voice and converts it into text data.

[2011] "Means for storing personal data in a database" refers to the technology for storing text data obtained by a voice recognition means in a database in association with a user ID.

[2012] "Multilingual translation means" refers to a technology that translates text in one language into another language.

[2013] "Image recognition means" refers to technology that analyzes image data captured by a camera and recognizes specific objects (e.g., faces).

[2014] "Environmental detection means using olfactory sensors" refers to technology that uses odor-detecting sensors to detect specific odors and analyze their causes.

[2015] "Means of operating in cooperation with other systems" refers to technology that exchanges data with different systems and performs multiple operations in cooperation with them.

[2016] "A means by which a user sends instructions to a robot using a smartphone app, and the robot then performs an action based on those instructions" refers to a technology in which a user sends operating instructions to a robot via a smartphone app, and the robot performs an action in accordance with those instructions.

[2017] "Means for analyzing data in real time when an abnormal odor is detected and generating an audio warning to notify the user" refers to a technology that, when an olfactory sensor detects an abnormal odor, analyzes the data to identify the cause, generates an audio warning to notify the user in real time.

[2018] "Means of storing text data obtained by voice recognition in a database in association with a user ID" refers to a technology that stores text data obtained by voice recognition technology in a database together with the corresponding user ID.

[2019] The term "means for displaying translated text data to the user" refers to a technology for displaying text data translated by the multilingual translation means in a form that can be confirmed by the user.

[2020] The present invention relates to a next-generation robot system that supports users' daily lives. The system includes a voice recognition unit, a unit for storing personal data in a database, a multilingual translation unit, an image recognition unit, an environment detection unit using an olfactory sensor, a unit for operating in conjunction with other systems, a unit for sending instructions to the robot via a smartphone app and having the robot perform actions based on those instructions, a unit for analyzing data in real time when an abnormal odor is detected and generating an alarm sound to notify the user, a unit for storing text data acquired by voice recognition in a database in association with a user ID, and a unit for displaying the translated text data to the user.

[2021] 1. Voice Recognition Method

[2022] When a user speaks to the robot, the robot's microphone captures the voice. The voice data is converted into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text API). For example, if a user says, "Hello, my name is Tanaka Taro," the text data "Hello, my name is Tanaka Taro" is generated.

[2023] 2. How personal data is stored in the database

[2024] The converted text data is sent to the server via an HTTP request. The server analyzes the received data and stores the user name and ID in a database (e.g., MySQL). This allows the information obtained through voice recognition to be managed as personal data.

[2025] 3. Multilingual Translation Tools

[2026] When translating user input text into multiple languages, the text data is sent to the server. The server calls a translation API (e.g., Google Cloud Translation API) to obtain the translation result. For example, the text "Can you speak English?" is translated to "Do you speak English?" The translation result is displayed to the user.

[2027] 4. Image Recognition Methods

[2028] When a user points their face at the robot's camera, the camera captures their facial image. The image data is sent to a server and analyzed using an image recognition algorithm (e.g., Azure Face API). The server then matches the individual against a database and sends the user ID to the device.

[2029] 5. Environmental detection using olfactory sensors

[2030] When the robot's olfactory sensors detect an unusual odor, the data is sent to a server in real time. The server analyzes the data and identifies the source of the odor. For example, if a gas leak is detected, the server sends that information to the device. The device uses a speech synthesis engine (e.g., Amazon Polly) to generate a warning voice saying, "There is a possibility of a gas leak. Please ventilate immediately," and plays it over the speaker.

[2031] 6. Means of cooperation with other systems

[2032] The user sends a command to the robot via a smartphone app to "start cleaning." The command from the app is sent to a server, which analyzes it and then issues a command to the robot to start cleaning. Once cleaning is complete, the device sends the information back to the server, and the server sends a completion notification to the smartphone app. The user then receives a notification that "cleaning is complete."

[2033] Examples and prompts

[2034] Example 1: When a user says, "Where am I now?", the speech recognition engine generates text data of "Where am I now?", sends related information to the server, and returns the answer.

[2035] Example 2: When a user says, "What do you call this in English?", the translation result is displayed to the user as "What do you call this in English?" after going through speech recognition, translation, and display.

[2036] Example prompt for a generative AI model:

[2037] "Please translate the text into English. The original text is 'How do you say this in English?'"

[2038] "Please tell me the procedure for analyzing the speech recognition results and saving them to the database."

[2039] The system of the present invention is an integrated multi-functional robot system that responds to various user requests in real time and improves the quality of life.

[2040] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2041] 1. Voice Recognition Method

[2042] Step 1:

[2043] The user speaks to the robot.

[2044] Input: Voice data (e.g., "Hello, my name is Taro Tanaka")

[2045] Output: None

[2046] Specific action: The user speaks aloud to the robot.

[2047] Step 2:

[2048] The device captures audio using the robot's microphone.

[2049] Input: Audio data

[2050] Output: Raw audio file

[2051] What it does: The microphone collects audio data and creates an audio file in the internal memory.

[2052] Step 3:

[2053] The device's voice recognition engine converts the captured voice data into text data.

[2054] Input: raw audio file

[2055] Output: Text data (e.g. "Hello, my name is Taro Tanaka")

[2056] What happens: A speech recognition engine (e.g., Google Cloud Speech-to-Text API) analyzes the audio file and generates corresponding text data.

[2057] 2. How personal data is stored in the database

[2058] Step 1:

[2059] The terminal sends the converted text data and metadata such as the user ID to the server in the form of an HTTP request.

[2060] Input: Text data, user ID

[2061] Output: HTTP request

[2062] Specific behavior: The text data and user ID are included in the body of the HTTP request and sent to the server.

[2063] Step 2:

[2064] The server analyzes the received HTTP request and extracts the content.

[2065] Input: HTTP request

[2066] Output: User name, User ID

[2067] What happens: The server parses the body of the HTTP request and extracts the username and user ID.

[2068] Step 3:

[2069] The server stores the extracted data in a database.

[2070] Input: Username, User ID

[2071] Output: New record to the database

[2072] Specific behavior: The username and user ID are added as a new record to the database (e.g. MySQL).

[2073] 3. Multilingual Translation Tools

[2074] Step 1:

[2075] The terminal obtains the user's input text and, if multilingual support is required, sends it to the server.

[2076] Input: Text data (e.g., "Do you speak English?")

[2077] Output: HTTP request

[2078] Specific operation: The device creates and sends an HTTP request to send text data to the server.

[2079] Step 2:

[2080] The server receives the text data and calls the translation API to translate it into the specified target language.

[2081] Input: HTTP request, text data

[2082] Output: Translated text data (e.g., "Do you speak English?")

[2083] Specific operation: Send a request to a translation API (e.g., Google Cloud Translation API) and obtain the translation result.

[2084] Step 3:

[2085] The server returns the translation results to the terminal.

[2086] Input: Translated text data

[2087] Output: HTTP response

[2088] Specific operation: Generates an HTTP response containing the translation result and sends it to the device.

[2089] Step 4:

[2090] The terminal displays the translation results to the user.

[2091] Input: Translated text data

[2092] Output: The text that is displayed in the user interface

[2093] Specific operation: The translation results are displayed on the user interface (display or voice reading device).

[2094] 4. Image Recognition Methods

[2095] Step 1:

[2096] The user faces the robot's camera.

[2097] Input: Face image

[2098] Output: None

[2099] Specific action: The user faces the camera directly.

[2100] Step 2:

[2101] The device captures a facial image using the robot's camera.

[2102] Input: Face image

[2103] Output: Image file

[2104] What happens: The camera captures a facial image and generates an image file.

[2105] Step 3:

[2106] The terminal transmits the image data to the server.

[2107] Input: Image file

[2108] Output: HTTP request

[2109] Specific operation: Sends an image file to the server in the form of an HTTP request.

[2110] Step 4:

[2111] The server analyzes the data using image recognition algorithms.

[2112] Input: Image file

[2113] Output: User ID

[2114] Specific operation: Analyzes the image using an image recognition algorithm (e.g. Azure Face API) and identifies the user.

[2115] Step 5:

[2116] The server sends the user ID to the terminal.

[2117] Input: User ID

[2118] Output: HTTP response

[2119] Specific operation: Generate an HTTP response including the user ID and send it to the terminal.

[2120] 5. Environmental detection using olfactory sensors

[2121] Step 1:

[2122] The device detects strange odors using an olfactory sensor.

[2123] Input: Environmental data (odor data)

[2124] Output: Odor detection event

[2125] How it works: The olfactory sensor detects specific odors in the environment.

[2126] Step 2:

[2127] The device transmits the detection data to the server in real time.

[2128] Input: Off-odor data

[2129] Output: HTTP request

[2130] Specific operation: Odor data is sent to the server in real time in the form of an HTTP request.

[2131] Step 3:

[2132] The server analyzes the odor data and compares it with known odor patterns to identify the cause.

[2133] Input: Off-odor data

[2134] Output: Data on the cause of the odor (e.g., gas leak)

[2135] Specific operation: Odor data is compared with a database of known odor patterns to identify the cause.

[2136] Step 4:

[2137] The server sends data on the cause of the odor to the terminal.

[2138] Input: Odor source data

[2139] Output: HTTP response

[2140] Specific operation: Generate an HTTP response including data on the cause of the odor and send it to the terminal.

[2141] Step 5:

[2142] The device generates and plays an audio warning.

[2143] Input: HTTP response (odor cause data)

[2144] Output: Audio warning

[2145] Specific behavior: Generates a warning message using a speech synthesis engine (e.g., Amazon Polly) and plays it over the speaker.

[2146] 6. Means of cooperation with other systems

[2147] Step 1:

[2148] The user sends instructions to the robot via a smartphone app.

[2149] Input: Instruction data (e.g. "Start cleaning")

[2150] Output: HTTP request

[2151] Specific operation: The user operates the app and sends commands to the robot.

[2152] Step 2:

[2153] The device sends the instructions received from the app to the server.

[2154] Input: HTTP request

[2155] Output: HTTP request

[2156] Specific operation: The device receives instructions from the app and forwards them to the server.

[2157] Step 3:

[2158] The server analyzes the received instructions and sends appropriate commands to the terminal.

[2159] Input: HTTP request (instruction data)

[2160] Output: Command data

[2161] Specific operation: The server analyzes the instruction data, generates the necessary commands, and sends them to the terminal.

[2162] Step 4:

[2163] The terminal will trigger the robot to start the cleaning function.

[2164] Input: Command data (e.g. "Start cleaning")

[2165] Output: Robot movement

[2166] Specific operation: The terminal transmits commands to the robot to start the cleaning function.

[2167] Step 5:

[2168] The terminal transmits cleaning completion data to the server.

[2169] Input: Cleaning completion data

[2170] Output: HTTP request

[2171] Specific operation: When cleaning is completed, the device sends the information to the server.

[2172] Step 6:

[2173] The server notifies the app of the cleaning completion data.

[2174] Input: HTTP request (cleaning completion data)

[2175] Output: Smartphone app notification

[2176] Specific operation: The server sends cleaning completion data to the app and notifies the user.

[2177] (Application example 1)

[2178] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2179] Conventional customer service systems in brick-and-mortar stores lack multilingual support for customer service and make it difficult to provide personalized service to repeat customers. Furthermore, emergency response capabilities, such as environmental and odor detection, are inadequate. For these reasons, there is a need to build a next-generation customer service system that can support multiple languages ​​in brick-and-mortar stores, provide personalized service for each individual customer, and ensure environmental safety.

[2180] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2181] In this invention, the server includes a voice recognition means, a means for storing personal data in a database, a multilingual translation means, an image recognition means, an environment detection means using an olfactory sensor, a means for operating in cooperation with other systems, and a means for functioning as a customer service assistant in a physical store. This makes it possible to respond to customers in multiple languages ​​in a physical store, provide personalized services according to each customer, and ensure a safe environment.

[2182] "Speech recognition means" refers to technology that takes voice input and converts that voice into text data.

[2183] "Means for storing personal data in a database" refers to the technology that collects information about users and stores it in an organized form in a database.

[2184] "Multilingual translation means" refers to a technology that translates input text data into multiple different languages.

[2185] "Image recognition means" refers to technology that analyzes image data acquired through devices such as cameras and identifies specific people or objects.

[2186] "Environmental detection means using olfactory sensors" refers to technology that uses sensors that detect various odors to detect specific environmental changes or unusual odors.

[2187] "Means for operating in cooperation with other systems" refers to technology for communicating with external systems or devices and executing specific operations.

[2188] "Means to function as a customer service assistant in a physical store" refers to technology that handles customer service within a physical store and provides multilingual support and personalized service.

[2189] This invention is a system that realizes efficient customer service and multilingual support by using a next-generation robot "maid-type" system as a customer service assistant in a brick-and-mortar store. This system includes a voice recognition means, a means for storing personal data in a database, a multilingual translation means, an image recognition means, an environmental detection means using an olfactory sensor, and a means for operating in cooperation with other systems.

[2190] Voice recognition means

[2191] When a user speaks to the robot, the voice is captured by the device's microphone, and a speech recognition engine (such as Google Web Speech API) converts the voice into text data and sends it to the server.

[2192] How personal data is stored in a database

[2193] The device sends the converted text data to the server via an HTTP request, where it is analyzed along with metadata such as the user ID and stored in a database. This storage process uses a relational or NoSQL database.

[2194] Multilingual translation tools

[2195] When a user makes a multilingual inquiry to the robot, the device sends the text data to the server. The server then calls a translation API to translate the input text data into the required language. The translated text data is then returned to the device and provided to the customer.

[2196] Image Recognition Method

[2197] When a user points their face at the robot's camera, the device's camera captures their face image, which is then sent to a server where image recognition algorithms (such as OpenCV or TensorFlow) are used to analyze the face and match it with a database to identify the user.

[2198] Environmental detection using olfactory sensors

[2199] When the device's built-in olfactory sensor detects an unusual odor, it sends the data to the server in real time. The server analyzes the data and generates a warning if it matches the odor pattern. This warning is output as voice by a speech synthesis engine and notified to the user.

[2200] A means of interacting with other systems

[2201] When a user sends a command to the robot via a smartphone app, the command is sent to the device. The server analyzes the received command and instructs the device to perform the appropriate operation. For example, if a command to start cleaning is sent, the robot will enter cleaning mode, and after cleaning is completed, the information will be sent to the server and the user will be notified.

[2202] Specific examples

[2203] When a customer asks the robot, "Where is this product in stock?", the voice is converted into text by a speech recognition device. The text data is then sent to the server, and related information is retrieved from the database and provided to the customer. In addition, if multilingual support is required, the text is translated into the appropriate language by a translation API.

[2204] Example prompts

[2205] "Write a Python program that uses speech recognition to convert what a customer says to a robot into text data, and then send that data along with the customer ID to a server for storage. The program will use the Google Web Speech API for speech recognition and the requests library to send the data to the server."

[2206] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2207] Step 1:

[2208] The user speaks to the robot. The input is the user's voice information, and the device's microphone captures this voice. Next, the voice data is converted into text data using a speech recognition engine (Google Web Speech API). The output is text data.

[2209] Step 2:

[2210] The terminal sends the converted text data and metadata (e.g., user ID) to the server as an HTTP request. The input is the text data and metadata, and the sent data is saved and analyzed on the server. The output is the saved data.

[2211] Step 3:

[2212] If a user makes a query that requires multilingual support, the device sends the text data back to the server. The input is the text data of the user's query, and the server calls the translation API to translate it into the specified language. The output is the text data in the translated language.

[2213] Step 4:

[2214] The server sends the translated data to the terminal, which then converts it into speech using a speech synthesis engine and transmits it to the user through a speaker. The input is the translated text data, and the output is the audio information provided to the user.

[2215] Step 5:

[2216] When a user points their face at the robot's camera, the device's camera captures a facial image and sends the image data to the server. The input is the captured facial image, and the server identifies the user using an image recognition algorithm (such as OpenCV or TensorFlow). The output is the identified user ID.

[2217] Step 6:

[2218] When the olfactory sensor installed in the device detects an abnormal odor, it sends the data to the server in real time. The input is the abnormal odor data detected by the olfactory sensor, and the server analyzes the data by comparing it with known abnormal odor patterns. The output is the identification of the abnormal odor and a warning message.

[2219] Step 7:

[2220] When the server identifies an abnormal odor, it sends a warning message to the device, which then uses a speech synthesis engine to generate a warning voice and notify the user through the speaker.The input is a warning message about the abnormal odor, and the output is the warning voice provided to the user.

[2221] Step 8:

[2222] When a user sends a command to a robot via a smartphone app, the command is sent to the server via the terminal. The input is the command data from the smartphone app, the server analyzes the received command, and sends appropriate operation instructions to the terminal. The output is the operation result executed by the terminal.

[2223] Step 9:

[2224] For example, when receiving the instruction to "start cleaning," the device starts cleaning mode, and when cleaning is complete, it sends that information to the server. The server notifies the smartphone app of the completion information, and the user receives a notification that "cleaning is complete." The input is the completion report data, and the output is the completion notification provided to the user.

[2225] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2226] A detailed description will be given of an embodiment of a next-generation robot "maid-type" system according to the present invention. This system includes a voice recognition unit, a unit for storing personal data in a database, a multilingual translation unit, an image recognition unit, an environment detection unit using an olfactory sensor, a unit for operating in cooperation with other systems, and an emotion engine for recognizing the user's emotions.

[2227] 1. Voice Recognition Method

[2228] User: The user speaks to the robot, for example, "Hello, my name is Taro Tanaka."

[2229] Terminal: The robot's microphone captures the voice. The captured voice is converted into text data by a voice recognition engine, and the text data "Taro Tanaka" is generated.

[2230] 2. How personal data is stored in the database

[2231] Terminal: The converted text data is sent to the server via an HTTP request, which includes metadata such as the user ID.

[2232] Server: The server analyzes the received text data and stores it in a database along with the user ID. For example, the user name "Taro Tanaka" and the corresponding ID are stored.

[2233] 3. Multilingual Translation Tools

[2234] Terminal: Obtains user input as text data, and if multilingual support is required, sends the text data to the server. For example, it asks, "Do you speak English?"

[2235] Server: The server receives the text data and calls the translation API to translate it. The Japanese text "Do you speak English?" is translated into English "Do you speak English?" and the translated data is sent to the device.

[2236] 4. Image Recognition Methods

[2237] User: The user faces the robot's camera.

[2238] Device: The camera captures the user's face and sends the image data to a server. The image recognition system analyzes the face and matches it with a database to identify the individual.

[2239] Server: The image recognition algorithm recognizes the user's face as "Taro Tanaka" and sends the user ID to the device.

[2240] 5. Environmental detection using olfactory sensors

[2241] Terminal: When the olfactory sensor detects an unusual odor, it sends the data to the server in real time. For example, if it detects a specific odor like a gas leak.

[2242] Server: The server analyzes the odor data it receives and identifies the cause by comparing it with known odor patterns. For example, it may identify a "gas leak" and send that data to the device.

[2243] Terminal: Based on the warning information from the server, a speech synthesis engine is used to generate a warning voice saying, "There is a possibility of a gas leak. Please ventilate immediately," and this is played from the speaker.

[2244] 6. Means of cooperation with other systems

[2245] User: Sends the command "Start cleaning" to the robot via the smartphone app.

[2246] Terminal: Sends instructions received from the app to the server.

[2247] Server: Analyzes the received instruction and sends an instruction to start cleaning mode to the terminal.

[2248] Device: The cleaning function starts, and when cleaning is completed, the information is sent to the server. The server notifies the smartphone app of the completion information, and the user receives a notification that "cleaning is completed."

[2249] 7. Emotional Engine Means

[2250] User: Emotions are expressed through the user's tone of voice and facial expressions when speaking to the robot.

[2251] On the device: The emotion recognition engine analyzes the user's tone of voice and facial expressions to detect their emotional state. For example, it determines that the user is "angry" based on their tone of voice and facial expressions.

[2252] Device: Sends an HTTP request containing the detected emotion data to the server.

[2253] Server: The server analyzes the received emotion data, stores it in a database, and generates appropriate response data based on the emotion data.

[2254] Terminal: Based on the response data received from the server, the terminal uses a speech synthesis engine to generate an appropriate voice response, such as "Taro Tanaka, what's the matter?"

[2255] Device: The generated voice is played back through the speaker, and the dialogue is tailored to the user's emotions.

[2256] As explained above in this embodiment, the next-generation robot "maid-type" system integrates multiple functions to support users' daily lives in many ways and improve their quality of life. In particular, the implementation of an emotion engine enables flexible responses according to emotions, resulting in a more natural and friendly user experience.

[2257] The processing flow will be explained below.

[2258] 1. Speech recognition processing

[2259] Step 1:

[2260] User: Talks to the robot and says, "Hello, my name is Taro Tanaka."

[2261] Step 2:

[2262] Device: Uses a microphone to capture the user's voice and sends the voice data to a speech recognition engine.

[2263] Step 3:

[2264] Terminal: The voice recognition engine converts the voice data into text data, generating the text data "Taro Tanaka."

[2265] 2. Processing of the means by which personal data is stored in a database

[2266] Step 1:

[2267] Terminal: Sends an HTTP request to the server containing the text data "Taro Tanaka." The request also contains metadata such as the user ID.

[2268] Step 2:

[2269] Server: Analyzes the received text data and stores it in a database along with the user ID.

[2270] Step 3:

[2271] Server: After saving is complete, it sends a response to the terminal indicating that the process was successful.

[2272] 3. Processing of multilingual translation means

[2273] Step 1:

[2274] User: Talks to the robot and asks, "Can you speak English?"

[2275] Step 2:

[2276] Terminal: Captures voice with a microphone and converts it into text data using a speech recognition engine. The converted text data is sent to the server.

[2277] Step 3:

[2278] Server: Sends the received text data to the translation API and performs the translation.

[2279] Step 4:

[2280] Server: Receives the translated text data "Do you speak English?" and sends it to the terminal.

[2281] Step 5:

[2282] Terminal: The translated text data is passed to a speech synthesis engine to generate speech data.

[2283] Step 6:

[2284] Device: Play the generated voice "Yes, I can speak English." from the speaker.

[2285] 4. Image Recognition Method Processing

[2286] Step 1:

[2287] User: Face the robot's camera.

[2288] Step 2:

[2289] Device: The camera captures the user's facial image and sends the image data to the server.

[2290] Step 3:

[2291] Server: Analyzes the received image data using an image recognition engine and identifies the user ID.

[2292] Step 4:

[2293] Server: Sends the identified user ID to the terminal.

[2294] Step 5:

[2295] Terminal: Based on the user ID received from the server, it matches it with a registered user name, generates a personalized response such as "You're Tanaka Taro, right?" and creates speech using a speech synthesis engine.

[2296] Step 6:

[2297] Device: Play the created audio "Hello, Taro Tanaka" from the speaker.

[2298] 5. Processing of environmental detection means using olfactory sensors

[2299] Step 1:

[2300] Terminal: The olfactory sensor detects strange odors in the environment.

[2301] Step 2:

[2302] Terminal: Transmits detected odor data to the server in real time.

[2303] Step 3:

[2304] Server: Analyzes the received odor data and compares it with known odor patterns.

[2305] Step 4:

[2306] Server: Identifies the cause of the strange odor and sends data such as "gas leak" to the terminal.

[2307] Step 5:

[2308] Device: Based on the received data, the speech synthesis engine generates a warning voice saying, "There is a possibility of a gas leak. Please ventilate immediately."

[2309] Step 6:

[2310] Device: Play the generated warning sound through the speaker.

[2311] 6. Processing of means for interoperating with other systems

[2312] Step 1:

[2313] User: Uses the smartphone app to send the command to the robot to "start cleaning."

[2314] Step 2:

[2315] Terminal: Sends instructions received from the app to the server.

[2316] Step 3:

[2317] Server: Analyzes the received instructions and instructs the cleaning robot to start cleaning mode.

[2318] Step 4:

[2319] Device: Follow the instructions to start cleaning mode.

[2320] Step 5:

[2321] Terminal: After cleaning is completed, the information is sent to the server.

[2322] Step 6:

[2323] Server: Receives the completion information and sends a notification to the smartphone app.

[2324] Step 7:

[2325] User: Receives a notification on the smartphone app that "cleaning is complete."

[2326] 7. Processing of Emotion Engine Means

[2327] Step 1:

[2328] User: The user expresses emotions through tone of voice and facial expressions when speaking to the robot.

[2329] Step 2:

[2330] On the device: The emotion engine captures the user's tone of voice and facial expressions and analyzes their emotional state. For example, it determines that the user is "angry" based on their tone of voice and facial expressions.

[2331] Step 3:

[2332] Terminal: Transmits the analyzed emotion data to the server.

[2333] Step 4:

[2334] Server: Receives emotion data, analyzes it, stores it in a database, and generates appropriate response data based on the emotion data.

[2335] Step 5:

[2336] Server: Sends the generated response data to the terminal.

[2337] Step 6:

[2338] Terminal: Based on the response data received from the server, the terminal uses a speech synthesis engine to generate an appropriate voice response, such as "Taro Tanaka, what's the matter?"

[2339] Step 7:

[2340] Device: The generated voice is played back through the speaker, and the dialogue is tailored to the user's emotions.

[2341] Example 2

[2342] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2343] Current home robots are unable to provide integrated functions such as natural dialogue with users, multilingual support, facial recognition, emotion recognition, and real-time environmental detection using environmental sensors. Therefore, in order to improve users' quality of life, it is necessary to integrate these diverse functions into a single system and ensure their smooth collaboration.

[2344] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a voice recognition means, a means for storing text data in a database, a multilingual translation means, an image recognition means for recognizing faces, an environment detection means used to detect abnormal odors, a means for operating in cooperation with other systems, and an emotion recognition means for analyzing emotions and responding appropriately. This makes it possible to perform natural dialogue with the user, detect anomalies, and provide multifunctional support in an integrated manner.

[2345] "Speech recognition means" is a technology that captures the voice uttered by the user and converts the voice data into text data.

[2346] "Means for saving text data in a database" refers to a technique for sending the converted text data to a server via a request and saving it in a database.

[2347] "Multilingual translation means" refers to a technology for translating text data into other languages.

[2348] "Image recognition means for recognizing faces" is a technology that analyzes facial images captured by a camera and identifies individuals.

[2349] "Environmental detection means used to detect anomalous odors" refers to technology that uses an olfactory sensor to detect anomalous odors in the environment and analyzes the data.

[2350] "Means of operating in cooperation with other systems" refers to technology that communicates with external systems and applications, exchanging instructions and information to perform operations.

[2351] "Emotion recognition means for analyzing emotions and responding appropriately" is a technology that analyzes emotions from the user's tone of voice and facial expressions and generates an appropriate response according to those emotions.

[2352] MODE FOR CARRYING OUT THE INVENTION

[2353] An embodiment of the next-generation robot "maid-type" system of the present invention will now be described in detail. This system includes a voice recognition unit, a unit for storing text data in a database, a multilingual translation unit, an image recognition unit for recognizing faces, an environment detection unit used to detect odors, a unit for operating in cooperation with other systems, and an emotion recognition unit for analyzing emotions and responding appropriately.

[2354] Voice recognition means

[2355] The user speaks to the robot. For example, "Hello, my name is Tanaka Taro." The device uses the robot's built-in microphone to capture the voice, and sends the voice data to the Google Cloud Speech-to-Text API, where it is converted into text data in real time. In this way, the text data "Tanaka Taro" is generated.

[2356] A means of storing text data in a database

[2357] The device sends the generated text data to the server as an HTTP request. This request also includes metadata such as the user ID. The server analyzes the received text data and stores it in a MySQL database along with the user ID. For example, "User ID: 1234" and "Name: Taro Tanaka" are saved.

[2358] Multilingual translation tools

[2359] The user speaks to the robot, "Can you speak English?" The device acquires the text data, and if multilingual support is required, sends the data to the server via an HTTP request. The server then sends the received data to the Google Cloud Translation API, which translates the Japanese "Can you speak English?" into English "Do you speak English?" The translation result is then sent to the device, and the robot responds to the user with "Do you speak English?"

[2360] Image recognition method for recognizing faces

[2361] The user turns their face towards the robot's camera. The device's camera captures the facial image and sends the image data to the server via an HTTP request. The server uses an image recognition system to analyze the facial image and compare it with a database to identify the individual "Taro Tanaka." The identified user ID is then sent to the device.

[2362] Environmental sensing means used to detect off-odors

[2363] When the device's olfactory sensor detects a gas leak, it sends the data to the server in real time. The server analyzes the received odor data, and if it identifies a gas leak, it sends a warning to the device. Based on the warning information from the server, the device generates an audio warning saying, "There is a possibility of a gas leak. Please ventilate immediately," and plays it through the speaker.

[2364] A means of interacting with other systems

[2365] The user issues a command to "start cleaning" on the smartphone app. The device receives the command from the app and sends it to the server via an HTTP request. The server analyzes the command and instructs the device to start cleaning mode. The device starts the cleaning function and sends that information to the server when cleaning is complete. The server then sends a completion notification to the smartphone app, and the user receives a notification that "cleaning is complete."

[2366] Emotion recognition means for analyzing emotions and responding appropriately

[2367] When the user speaks to the robot, they express their emotions through their voice tone and facial expressions. The device uses an emotion recognition engine to analyze the voice tone and facial expressions to generate emotional data about the user. This data is then sent to the server via an HTTP request. The server analyzes the received emotional data and generates an appropriate response, such as "Taro Tanaka, what's the matter?" The device uses a speech synthesis engine to generate a response based on the response data received from the server, and plays it back from the speaker.

[2368] Prompt Sentence Examples

[2369] "Hello, my name is Tanaka Taro. Do you speak English?"

[2370] This system integrates natural dialogue with the user, anomaly detection, and multifunctional support, improving the quality of everyday life.

[2371] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2372] Program processing flow and specific explanation

[2373] Voice recognition means

[2374] Step 1:

[2375] The user speaks to the robot, saying something like, "Hello, my name is Taro Tanaka."

[2376] Step 2:

[2377] The device captures voice using a built-in microphone, and the input voice data is sent to a voice recognition engine.

[2378] Step 3:

[2379] The device converts the captured audio into text data using the Google Cloud Speech-to-Text API. The input is audio data, and the output is the text data "Taro Tanaka."

[2380] A means of storing text data in a database

[2381] Step 4:

[2382] The terminal creates an HTTP request containing the converted text data "Taro Tanaka." The input is the text data and metadata such as the user ID.

[2383] Step 5:

[2384] The terminal sends an HTTP request to the "Apache Tomcat" server. The input is the HTTP request, and the output is the result of the request sent to the server.

[2385] Step 6:

[2386] The server analyzes the received request and saves the text data and user ID in a MySQL database. The input is the HTTP request, and the output is the result saved in the database.

[2387] Multilingual translation tools

[2388] Step 7:

[2389] The user asks the robot, "Can you speak English?"

[2390] Step 8:

[2391] The terminal creates an HTTP request including the text data and sends it to the server. The input is the text data, and the output is the result of the request sent to the server.

[2392] Step 9:

[2393] The server calls the Google Cloud Translation API to translate the Japanese text "Do you speak English?" into English "Do you speak English?" The input is Japanese text and the output is English text.

[2394] Step 10:

[2395] The server sends the translated text data to the terminal. The input is the translation result, and the output is the data sent to the terminal.

[2396] Step 11:

[2397] The terminal responds to the user by voice, asking, "Do you speak English?" The input is English text, and the output is a voice response.

[2398] Image recognition method for recognizing faces

[2399] Step 12:

[2400] The user faces the robot's camera.

[2401] Step 13:

[2402] The device captures a facial image with a camera and sends the image data to the server via an HTTP request. The input is the facial image data, and the output is the result of the request sent to the server.

[2403] Step 14:

[2404] The server analyzes the facial image using an image recognition system and compares it with a database to identify the individual named "Taro Tanaka." The input is facial image data, and the output is the identification result.

[2405] Step 15:

[2406] The server sends the identified user ID to the terminal. The input is the identification result, and the output is the transmission to the terminal.

[2407] Environmental sensing means used to detect off-odors

[2408] Step 16:

[2409] When the olfactory sensor detects a gas leak, the device sends the data to the server in real time. The input is the odor sensor data, and the output is the request sent to the server.

[2410] Step 17:

[2411] The server analyzes the received odor data and compares it with known odor patterns to identify a "gas leak." The input is the odor data, and the output is the analysis results.

[2412] Step 18:

[2413] When the server identifies a gas leak, it sends the warning information to the terminal. The input is the analysis result, and the output is the message sent to the terminal.

[2414] Step 19:

[2415] The terminal generates a voice warning based on the warning information and plays a voice message saying, "There is a possibility of a gas leak. Please ventilate immediately." The input is the warning information and the output is the voice warning.

[2416] A means of interacting with other systems

[2417] Step 20:

[2418] The user issues a command to "start cleaning" via the smartphone app.

[2419] Step 21:

[2420] The device receives instructions from the smartphone app and sends them to the server via an HTTP request. The input is the instruction data, and the output is the result of the request sent to the server.

[2421] Step 22:

[2422] The server analyzes the received command and instructs the terminal to start cleaning mode. The input is the command data, and the output is sent to the terminal.

[2423] Step 23:

[2424] The terminal starts the cleaning function and sends the cleaning completion information to the server when the cleaning is completed. The input is the cleaning completion data, and the output is the request sent to the server.

[2425] Step 24:

[2426] The server sends a completion notification to the smartphone app, and the user receives a notification that "cleaning is complete." The input is the completion data, and the output is the notification.

[2427] Emotion recognition means for analyzing emotions and responding appropriately

[2428] Step 25:

[2429] When a user speaks to a robot, they express their emotions through their tone of voice and facial expressions.

[2430] Step 26:

[2431] The device analyzes voice tones and facial expressions using an emotion recognition engine, generates emotional data, and sends it to the server via an HTTP request. The input is voice tones and facial expression data, and the output is emotional data.

[2432] Step 27:

[2433] The server analyzes the received emotion data and generates an appropriate response, such as "Taro Tanaka, what's wrong?" The input is emotion data, and the output is response data.

[2434] Step 28:

[2435] The server sends the response data to the terminal. The input is the response data, and the output is the data sent to the terminal.

[2436] Step 29:

[2437] The device generates a response using a speech synthesis engine and plays it back through the speaker. The input is the response data, and the output is the voice response.

[2438] (Application example 2)

[2439] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2440] Modern factories require efficient systems to improve productivity and ensure worker safety. However, current robot systems and automation solutions are often single-function and unable to handle multiple tasks simultaneously and flexibly. Furthermore, there is a lack of systems that can handle environments with a mix of multinational workers and smoothly detect and respond to abnormal situations. This leads to a decline in the operational efficiency of the entire factory.

[2441] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2442] In this invention, the server includes a voice recognition means, a means for storing personal data in a database, a multilingual translation means, an image recognition means, an environment detection means using an olfactory sensor, a means for operating in cooperation with other systems, an emotion engine means for recognizing the user's emotions, and an automatic operation means for supporting factory work. This makes it possible to perform multiple tasks in an integrated and efficient manner, while supporting a variety of languages, improving the safety of the work environment, and enabling flexible responses that take into account the emotional state of workers.

[2443] - "Speech recognition means" refers to technology that captures a user's voice and converts it into text data.

[2444] "Means for storing personal data in a database" refers to technology that stores information related to individuals, such as a user's name or ID, in a database.

[2445] "Multilingual translation means" is a technology that translates text entered in one language into another language.

[2446] "Image recognition means" refers to technology that identifies specific objects or people from image data captured by a camera or other sensor.

[2447] "Environmental detection means using olfactory sensors" is a technology that detects specific odors and gases and analyzes the data.

[2448] "Means for operating in cooperation with other systems" refers to technology for exchanging instructions and data in cooperation with other information systems and devices.

[2449] The "emotion engine means for recognizing user emotions" is a technology that analyzes the user's tone of voice and facial expressions to identify their emotional state. ...

Claims

1. a voice recognition means; a means for storing personal data in a database; Multilingual translation means; Image recognition means; An environment detection means using an olfactory sensor; A system that includes the means to operate in conjunction with other systems.

2. 2. The system according to claim 1, further comprising a personal data registration processing means.

3. 2. The system according to claim 1, further comprising means for issuing an audio warning to the user when an abnormal odor is detected.

Images