System

Abstract

A system is provided.SOLUTION: A voice recognition unit that recognizes an instruction input by voice by a user, an image acquisition unit that captures an image of a surrounding environment, an image analysis unit that analyzes the acquired image data and identifies an obstacle or a step, a route calculation unit that calculates an optimal route to a destination of the user, a movement control unit that controls movement based on the calculated route; A system comprising: communication means for receiving an instruction from a server; and communication means for notifying a user of the instruction received from the server by voice.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The issue is ensuring the safety and guidance of visually impaired people when they go out. Currently, there are very few visually impaired people who use guide dogs, and one of the reasons for this is the lack of guide dog trainers. Another problem is the time and cost involved in training and caring for the dog's excretion. There is a need to improve this situation and provide an environment where more visually impaired people can go out safely. [Means for solving the problem]

[0005] The present invention provides a system including voice recognition means for recognizing instructions input by voice from a user, image acquisition means for capturing images of the surrounding environment, image analysis means for analyzing the captured image data and identifying obstacles and steps, route calculation means for calculating the optimal route to the user's destination, movement control means for controlling movement based on the calculated route, communication means for transmitting the user's instructions recognized by the voice recognition means and data analyzed by the image analysis means to a server and receiving instructions from the server, and communication means for notifying the user by voice of the instructions received from the server. This will solve the problems that visually impaired people encounter when using guide dogs, and make it possible to provide safe and efficient assistance to more visually impaired people.

[0006] A "voice recognition means" is a device or system that converts the voice uttered by the user into digital data and analyzes the data to understand its meaning.

[0007] An "image capture means" is a device or system that captures images of the surrounding environment and generates image data.

[0008] The "image analysis means" is a device or system that analyzes the acquired image data and identifies specific elements such as obstacles and steps.

[0009] The "route calculation means" is a device or system that calculates the optimal route based on the user's current location and destination.

[0010] "Movement control means" refers to a device or system that controls the movement of the robot based on the calculated route.

[0011] The "communication means" is a device or system that transmits data acquired from the voice recognition means or image analysis means to a server and receives instructions from the server.

[0012] A "communication means" is a device or system that notifies the user of instructions received from the server as voice and engages in voice dialogue with the user.

[0013] A "server" is a computer system that processes data, such as storing, analyzing, and learning, and sends instructions to terminals.

[0014] "Users" are visually impaired people who use this system to receive assistance with going out. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] MODE FOR CARRYING OUT THE INVENTION

[0037] The quadrupedal guide dog robot system of the present invention is designed to assist visually impaired people when they go out. This system includes voice recognition means, image acquisition means, image analysis means, route calculation means, movement control means, communication means, and communication means, and each means works in cooperation to guide the visually impaired people safely to their destination.

[0038] Overall system configuration

[0039] Voice recognition means

[0040] It recognizes instructions that users speak (e.g., "I want to go to the nearest supermarket") and converts the speech into text data that is used to analyze the user's intent.

[0041] Image Acquisition Method

[0042] The robot is equipped with a 360-degree camera that captures images of its surroundings in real time, and this image data is used as the basis for understanding the surrounding situation.

[0043] Image analysis methods

[0044] The acquired image data is sent to a server where it is analyzed. This analysis includes identifying obstacles, steps, traffic lights, etc. This information is necessary to calculate a safe guidance route.

[0045] Route calculation method

[0046] The server calculates the optimal route based on the image analysis results and the user's destination information. The calculation also takes into account information such as obstacles, construction areas, and puddles. The calculated route information is sent to the robot.

[0047] Movement Control Means

[0048] The robot begins moving autonomously based on the route information received from the server. As it walks, it dynamically corrects its route based on image data captured in real time.

[0049] communication means

[0050] The user's instructions recognized by the voice recognition means and the data obtained by the image analysis means are transmitted to the server, and communication is also performed to receive instructions from the server and recalculated route information.

[0051] Communication tools

[0052] The robot notifies the user of the instructions received from the server by voice (e.g., "Go straight," "Turn left," etc.). It also confirms the situation and receives further instructions through dialogue with the user.

[0053] Program processing overview

[0054] Voice input and recognition

[0055] 1. User: Says, "I want to go to the nearby supermarket."

[0056] 2. Terminal: The recognition device converts the voice into text data and recognizes the content "Go to the nearby supermarket."

[0057] Acquisition and analysis of the surrounding environment

[0058] 3. Terminal: Captures image data of the surroundings using a 360-degree camera and sends it to the server.

[0059] 4. Server: Analyzes the received image data and identifies obstacles and steps.

[0060] Route calculation and movement control

[0061] 5. Server: Calculates the optimal route based on the analysis results and the user's destination information, and sends that information to the terminal.

[0062] 6. Terminal: Start moving according to the received route, checking the surrounding environment in real time as you go.

[0063] Voice communication

[0064] 7. Terminal: Notifies the user of the current situation and next actions via voice.

[0065] 8. User: Move according to the robot's instructions.

[0066] Example: When a user goes to the supermarket

[0067] 1. User: Says, "I want to go to the nearest supermarket."

[0068] 2. Device: Recognizes voice and captures the surroundings with a 360-degree camera.

[0069] 3. Server: Analyzes the image data and calculates a safe route.

[0070] 4. Terminal: Starts movement based on the calculation results and notifies the user of the next action by voice.

[0071] 5. User: Follow the robot's instructions to safely reach the supermarket.

[0072] In this way, the entire system works together to safely and efficiently guide users to their destination. This system solves the problems faced by conventional guide dogs and provides an environment where visually impaired people can go out more freely.

[0073] The processing flow will be explained below.

[0074] Step 1:

[0075] User: Uses voice input. Says, "I want to go to the nearby supermarket."

[0076] Step 2:

[0077] Device: Captures the user's voice with a built-in microphone.

[0078] Step 3:

[0079] Terminal: The voice recognition system converts the captured voice into text data and interprets the content as "I want to go to the nearby supermarket."

[0080] Step 4:

[0081] Terminal: Analyzes the converted text data and recognizes the user's request.

[0082] Step 5:

[0083] Terminal: Sends the analysis results to the server.

[0084] Step 6:

[0085] Terminal: Captures the current surrounding environment with a 360-degree camera and sends the image data to the server.

[0086] Step 7:

[0087] Server: Analyzes the received image data and maps the current location and environment.

[0088] Step 8:

[0089] Server: Performs image analysis to identify the location of obstacles, steps, and traffic lights.

[0090] Step 9:

[0091] Server: Calculates multiple routes based on the user's destination "supermarket" and current location.

[0092] Step 10:

[0093] Server: From the calculated routes, select the optimal route that avoids obstacles, construction areas, puddles, etc.

[0094] Step 11:

[0095] Server: Sends the selected route information and walking instructions to the device.

[0096] Step 12:

[0097] Terminal: Based on the received route information, the mobility control system starts moving.

[0098] Step 13:

[0099] Terminal: A 360-degree camera scans the surrounding environment in real time to ensure safety.

[0100] Step 14:

[0101] Terminal: Sends captured image data to the server at any time, sharing the latest situation.

[0102] Step 15:

[0103] Terminal: Recognizes obstacles, steps, and traffic lights in real time and guides the user.

[0104] Step 16:

[0105] Device: Speaks to the user about their current progress and next steps, for example, "Go straight" or "Turn left."

[0106] Step 17:

[0107] User: Follow the robot's voice instructions and move.

[0108] Step 18:

[0109] Terminal: If the device encounters an unexpected obstacle or construction area along the way, it sends new image data to the server.

[0110] Step 19:

[0111] Server: Re-analyzes based on the new data and recalculates a new route.

[0112] Step 20:

[0113] Server: Sends the new optimized route information to the device.

[0114] Step 21:

[0115] Device: Resume travel based on new route information.

[0116] Step 22:

[0117] Terminal: When the user arrives at the destination, the terminal notifies the user that "You have arrived at the supermarket."

[0118] Step 23:

[0119] User: Confirms that they have reached the supermarket.

[0120] Step 24:

[0121] Device: Ask the user for feedback on their travel experience. Example: "Did you have any issues during your trip?"

[0122] Step 25:

[0123] User: Provides feedback verbally. Example: "No issues."

[0124] Step 26:

[0125] Terminal: Sends user feedback to the server.

[0126] Step 27:

[0127] Server: Stores the received feedback in a database.

[0128] Step 28:

[0129] Server: Updates the model parameters based on the feedback data to improve performance.

[0130] The above is a specific processing flow of the four-legged guide dog robot system for assisting visually impaired people when they go out.

[0131] Example 1

[0132] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0133] Conventional guide dogs and assistive devices have struggled to adequately support the safe and efficient movement of visually impaired people. In particular, they lacked the ability to grasp the surrounding environment in real time and dynamically correct routes, limiting the ability of visually impaired people to respond to sudden obstacles, unmanned roads, areas under construction, and other situations. Furthermore, they lacked the ability to utilize user feedback to improve the system's efficiency. This made it difficult to provide an environment in which visually impaired people could go out with peace of mind.

[0134] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0135] In this invention, the server includes voice recognition means for recognizing instructions input by voice from the user, image acquisition means for capturing images of the surrounding environment, image analysis means for analyzing the captured image data and identifying obstacles and steps, route calculation means for calculating the optimal route to the user's destination, movement control means for controlling movement based on the calculated route, communication means for transmitting the user's instructions recognized by the voice recognition means and data analyzed by the image analysis means to an information processing device and receiving instructions from the information processing device, communication means for notifying the user of the instructions received from the information processing device by voice, and means for dynamically correcting the route while checking the environment in real time, thereby providing an environment in which visually impaired people can go out with peace of mind.

[0136] The "voice recognition means" is a device that recognizes instructions input by voice from the user and converts them into text data.

[0137] "Image acquisition means" refers to a device that captures images of the surrounding environment and acquires the image data. This device includes a 360-degree camera.

[0138] The "image analysis means" is a device or program for analyzing the acquired image data and identifying obstacles and steps.

[0139] The "route calculation means" is a device or program that calculates the optimal route to the user's destination.

[0140] The "movement control means" is a device or program that controls the movement of the robot based on the calculated route.

[0141] The "communication means" is a device or program for transmitting user instructions recognized by the voice recognition means and data analyzed by the image analysis means to the information processing device, and for receiving instructions from the information processing device.

[0142] The "communication means" is a device or program that notifies the user of instructions received from the information processing device by voice.

[0143] The "means for dynamically correcting the route while checking the environment in real time" refers to a device or program that allows the robot to continuously check the surrounding environment while moving and correct the route as necessary.

[0144] MODE FOR CARRYING OUT THE INVENTION

[0145] The quadrupedal guide dog robot system of the present invention is designed to assist visually impaired people when they go out. This system includes voice recognition means, image acquisition means, image analysis means, route calculation means, movement control means, communication means, and communication means, and each means works in cooperation to guide the visually impaired people safely to their destination.

[0146] Overall system configuration

[0147] The main components of this system are:

[0148] Voice recognition means

[0149] It recognizes instructions spoken by the user and converts the speech into text data. Specifically, it uses a speech recognition system (e.g., Google® Cloud Speech-to-Text service) to recognize the user's voice instructions. This text data is used to analyze the user's intent.

[0150] Image Acquisition Method

[0151] The robot is equipped with a 360-degree camera that captures images of its surroundings in real time, and this image data is used as the basis for understanding the surrounding situation.

[0152] Image analysis methods

[0153] The acquired image data is sent to a server where it is analyzed. This analysis includes identifying obstacles, steps, traffic lights, etc., using object detection algorithms such as YOLO (You Only Look Once). The analysis also generates information on the location of obstacles and safe route options.

[0154] Route calculation method

[0155] The server calculates the optimal route based on the image analysis results and the user's destination information. For example, it uses a path-finding algorithm such as Dijkstra's algorithm. This calculation also takes into account information such as obstacles, construction areas, and puddles. The calculated route information is sent to the robot.

[0156] Movement Control Means

[0157] The robot begins moving autonomously based on the route information received from the server. While moving, it continuously monitors the surrounding environment with a 360-degree camera and corrects its route in real time. Even if an obstacle suddenly appears, it automatically stops and resumes moving according to the re-planned route.

[0158] communication means

[0159] The user's instructions recognized by the voice recognition means and the data analyzed by the image analysis means are transmitted to the server, and communication is also performed to receive instructions and recalculated route information from the server.

[0160] Communication tools

[0161] The robot will then verbally communicate instructions received from the server to the user, such as "go straight" or "turn left," and will also include a learning mechanism to update the system based on user feedback.

[0162] Example: When a user goes to the supermarket

[0163] 1. User: Says "I want to go to the nearest supermarket." The voice recognition means converts this into text data.

[0164] 2. Device: Recognizes voice and captures the surroundings with a 360-degree camera.

[0165] 3. Server: Analyzes image data to identify obstacles and steps. Based on the analysis results, calculates a safe route.

[0166] 4. Terminal: Receives calculated route information and starts moving, monitoring the environment in real time as it moves forward, and notifies the user of the next action via voice.

[0167] 5. User: Follow the robot's instructions to safely reach the supermarket.

[0168] In this way, the entire system works together to safely and efficiently guide users to their destination. This system solves the problems faced by conventional guide dogs and provides an environment where visually impaired people can go out more freely.

[0169] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0170] Step 1:

[0171] User: Enters a voice command such as "I want to go to the nearest supermarket."

[0172] Input: User's voice commands

[0173] Output: Audio input data

[0174] Step 2:

[0175] On the device: A speech recognition system converts speech into text data, for example, using the Google Cloud Speech-to-Text service.

[0176] Input: Voice input data

[0177] Data processing: Converting voice data into text data

[0178] Output: "I want to go to the nearest supermarket."

[0179] How it works: The voice recognizer activates, records speech, and converts it into text.

[0180] Step 3:

[0181] Terminal: Sends the converted text data to the server.

[0182] Input: Text data "I want to go to the nearest supermarket"

[0183] Output: Data sent to the server

[0184] How it works: The communications module packets the text data and sends it over the Internet to a server.

[0185] Step 4:

[0186] Terminal: Captures image data of the surroundings using a 360-degree camera and sends it to the server.

[0187] Input: Surrounding environment

[0188] Output: Image data

[0189] What it does: The camera starts up, continuously captures 360-degree images, and sends the data to a server.

[0190] Step 5:

[0191] Server: Analyzes the received image data and identifies obstacles and steps. Uses object detection algorithms such as YOLO.

[0192] Input: Image data

[0193] Data calculation: Apply object detection algorithm to identify obstacles and steps

[0194] Output: Analysis results (obstacle location information, etc.)

[0195] Operation: Image analysis software processes image data and generates analytical results.

[0196] Step 6:

[0197] Server: Based on the user's destination information "nearby supermarket" and the image analysis results, the optimal route is calculated using Dijkstra's algorithm.

[0198] Input: Destination information, analysis results

[0199] Data calculation: Apply route calculation algorithm

[0200] Output: Optimal route information

[0201] Operation: The route calculation program on the server runs, calculates the optimal route, and generates data.

[0202] Step 7:

[0203] Server: Sends calculated route information to the device.

[0204] Input: Optimal route information

[0205] Output: Data sent to the terminal

[0206] Operation: The communication module packets the route information and sends it to the terminal via the Internet.

[0207] Step 8:

[0208] Terminal: The terminal begins moving autonomously based on the received route information. If an obstacle is detected along the way, the terminal corrects the route in real time.

[0209] Input: Route information, real-time images from a 360-degree camera

[0210] Data calculation: Route correction based on image data

[0211] Output: Movement control command

[0212] Operation: The movement control system is activated and controls the motors and actuators to move the robot.

[0213] Step 9:

[0214] Terminal: Notifies the user of the current situation and next actions via voice.

[0215] Input: Instructions from the server, route information

[0216] Output: Audio notification

[0217] How it works: The voice synthesis system converts instructions into voice and transmits them to the user through the speaker.

[0218] Step 10:

[0219] User: Move according to the robot's voice instructions.

[0220] Input: Voice commands

[0221] Output: User Movement

[0222] Action: The user moves according to the robot's instructions.

[0223] (Application example 1)

[0224] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0225] When visually impaired people go out, existing assistance systems have difficulty quickly recalculating the optimal route in response to changes in the surrounding environment. Furthermore, real-time instructions and notifications of the situation are insufficient, leaving users feeling uneasy. There is a need to provide a system that can solve these problems and enable visually impaired people to travel more safely and efficiently.

[0226] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0227] In this invention, the server includes voice recognition means for recognizing instructions input by voice from the user, image acquisition means for capturing images of the surrounding environment, image analysis means for analyzing the captured image data and identifying obstacles and steps, route calculation means for calculating an optimal route to the user's destination, movement control means for controlling movement based on the calculated route, communication means for transmitting the user's instructions recognized by the voice recognition means and data analyzed by the image analysis means to the server and receiving instructions from the server, communication means for notifying the user of the instructions received from the server by voice, and dynamic re-route calculation means for analyzing the environment and re-calculating the optimal route in real time based on the captured data, thereby enabling safe and efficient movement according to the environment.

[0228] The "voice recognition means" is a device that recognizes instructions input by voice by the user and converts them into text data.

[0229] The "image acquisition means" is a device that takes pictures of the surrounding environment with a camera and acquires image data.

[0230] The "image analysis means" is a device that analyzes the acquired image data and identifies environmental information such as obstacles and steps.

[0231] The "route calculation means" is a device that calculates the optimal route based on the user's destination information.

[0232] "Movement control means" refers to a device that controls the movement of the system based on the calculated route.

[0233] The "communication means" is a device that transmits user instructions recognized by the voice recognition means and data analyzed by the image analysis means to the server, and receives instructions from the server.

[0234] The "communication means" is a device that notifies the user of instructions received from the server by voice and communicates with the user through dialogue.

[0235] A "dynamic re-route calculation means" is a device that analyzes the environment in real time based on acquired data and recalculates the optimal route if necessary.

[0236] The system that realizes this application example is an automated driving guidance system for visually impaired people, and includes as its main components a voice recognition means, an image acquisition means, an image analysis means, a route calculation means, a movement control means, a communication means, a communication means, and a dynamic re-route calculation means.

[0237] Voice recognition means

[0238] The server uses speech recognition software (e.g., the speech_recognition library) to recognize instructions entered by the user through speech and convert them into text data. For example, if a user says, "I want to go to the nearest supermarket," this speech is converted into text and processed within the system.

[0239] Image Acquisition Method

[0240] The device uses a 360-degree camera (e.g., cv2.VideoCapture) to capture the surrounding environment and acquire image data, which is important for the system to understand the visual environment.

[0241] Image analysis methods

[0242] The server uses image analysis software (e.g., OpenCV library) to analyze the acquired image data and identify obstacles, steps, etc. The analyzed data is input into the route calculation means.

[0243] Route calculation method

[0244] The server uses a map service API (e.g., the Google Maps library) to calculate the optimal route based on the user's destination information. The calculated route information is sent to the device.

[0245] Movement Control Means

[0246] The device begins moving autonomously based on the route received from the server, checking environmental data in real time as it moves forward and dynamically correcting its route as necessary.

[0247] communication means

[0248] The server transmits user instructions recognized by the voice recognition means and data analyzed by the image analysis means to the server, and is equipped with communication functions (e.g., Internet connection, Wi-Fi) to receive instructions from the server at the terminal.

[0249] Communication tools

[0250] The device uses a text-to-speech engine (e.g., a TTS engine) to audibly announce to the user the instructions received from the server, for example, informing the user that "the next turn is to the right" on the way to a destination.

[0251] Dynamic reroute calculation method

[0252] The server analyzes the environment in real time based on the acquired data and recalculates the optimal route as needed, allowing the vehicle to respond immediately to situations such as unexpected obstacles or road construction.

[0253] Specific examples

[0254] For example, if a user says, "I want to go to the nearby supermarket," the voice recognition means converts this instruction into text data and sends it to the server. The server uses a map service API to calculate the optimal route and sends this information to the device. The device then takes pictures of the surrounding environment with a 360-degree camera and begins moving while identifying obstacles using image analysis means. As the environment changes, the route is dynamically recalculated and appropriate instructions are provided to the user using a text-to-speech engine.

[0255] Examples of prompts: "I want to go to the nearest supermarket" "Turn right next"

[0256] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0257] Step 1:

[0258] Voice input and recognition

[0259] Subject: User

[0260] How it works: The user says, "I want to go to the nearest supermarket."

[0261] Input: User's voice command

[0262] Data processing: Use speech recognition software (e.g., speech_recognition library) to convert voice instructions into text data.

[0263] Output: Text data "I want to go to a nearby supermarket"

[0264] Step 2:

[0265] Shooting the surrounding environment

[0266] Subject: Device

[0267] What it does: Captures the surrounding environment with a 360-degree camera.

[0268] Input: Surrounding environment information

[0269] Data processing: Use a 360-degree camera (e.g., cv2.VideoCapture) to acquire image data of the environment.

[0270] Output: Image data of the environment

[0271] Step 3:

[0272] Image data analysis

[0273] Subject: Server

[0274] Operation: Analyzes acquired image data and identifies obstacles and steps.

[0275] Input: Image data of the environment

[0276] Data processing: Use image analysis software (e.g., OpenCV library) to identify obstacles, steps, etc.

[0277] Output: Analysis data on obstacles and steps

[0278] Step 4:

[0279] Calculating the best route

[0280] Subject: Server

[0281] Operation: Calculates the optimal route based on the user's destination information.

[0282] Input: User instruction text data and image analysis data

[0283] Data calculation: Uses map service APIs (e.g., Google Maps library) to calculate the optimal route.

[0284] Output: Optimal route information

[0285] Step 5:

[0286] Movement Control

[0287] Subject: Device

[0288] Operation: The robot begins moving autonomously based on the calculated route information.

[0289] Input: Optimal route information

[0290] Data calculation: Using a movement control algorithm, control the movement according to the route.

[0291] Output: Actual movement

[0292] Step 6:

[0293] Real-time environment reanalysis and route recalculation

[0294] Subject: Server

[0295] How it works: Reanalyzes the environment in real time based on acquired data and recalculates the optimal route if necessary.

[0296] Input: Image data of the environment and current route information

[0297] Data calculation: Use dynamic re-route calculation method to recalculate the optimal route.

[0298] Output: Updated optimal route information

[0299] Step 7:

[0300] Communications and Notice of Instructions

[0301] Subject: Server and Terminal

[0302] Operation: New instructions from the server are sent to the terminal, which then notifies the user by voice.

[0303] Input: Updated route information

[0304] Data processing: Using communication functions (e.g., Internet connection, Wi-Fi), instructions from the server are sent to the device, which then uses a TTS engine to provide voice notifications to the user.

[0305] Output: Spoken instructions to the user (e.g., "Turn right next")

[0306] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0307] MODE FOR CARRYING OUT THE INVENTION

[0308] The quadrupedal guide dog robot system of the present invention is designed to assist visually impaired people when they go out, and is also combined with an emotion engine that recognizes the user's emotional state and responds accordingly. This system includes voice recognition means, image acquisition means, image analysis means, route calculation means, movement control means, communication means, and the emotion engine, and each of these means works in cooperation to guide visually impaired people to their destination safely and with a sense of psychological security.

[0309] Overall system configuration

[0310] Voice recognition means

[0311] It recognizes instructions that users speak (e.g., "I want to go to the nearest supermarket") and converts the speech into text data that is used to analyze the user's intent.

[0312] Image Acquisition Method

[0313] The robot is equipped with a 360-degree camera that captures images of its surroundings in real time, and this image data is used as the basis for understanding the surrounding situation.

[0314] Image analysis methods

[0315] The acquired image data is sent to a server where it is analyzed. This analysis includes identifying obstacles, steps, traffic lights, etc. This information is necessary to calculate a safe guidance route.

[0316] Route calculation method

[0317] The server calculates the optimal route based on the image analysis results and the user's destination information. The calculation also takes into account information such as obstacles, construction areas, and puddles. The calculated route information is sent to the robot.

[0318] Movement Control Means

[0319] The robot begins moving autonomously based on the route information received from the server. As it walks, it dynamically corrects its route based on image data captured in real time.

[0320] communication means

[0321] The user's instructions recognized by the voice recognition means and the data obtained by the image analysis means are transmitted to the server, and communication is also performed to receive instructions from the server and recalculated route information.

[0322] Communication tools

[0323] The robot notifies the user of the instructions received from the server by voice (e.g., "Go straight," "Turn left," etc.). It also confirms the situation and receives further instructions through dialogue with the user.

[0324] Emotion Engine

[0325] The robot is equipped with an emotion engine that analyzes the user's voice and facial expression data to recognize the user's emotional state. For example, if the user is feeling anxious, the system can play relaxing music to help the user relax or provide encouraging words such as "It's okay."

[0326] Program processing overview

[0327] Voice Input and Emotion Recognition

[0328] 1. User: Says, "I want to go to the nearby supermarket."

[0329] 2. Device: The built-in microphone and camera capture the user's voice and facial expressions.

[0330] 3. Terminal: The voice recognition system converts the captured voice into text data, and the emotion engine then analyzes the user's emotional state from their voice and facial expressions.

[0331] 4. Terminal: Analyzes the instructions from the text data and sends the emotional state recognized by the emotion engine to the server.

[0332] Environmental awareness and route planning

[0333] 5. Terminal: Captures the current surrounding environment with a 360-degree camera and sends the image data to the server.

[0334] 6. Server: Analyzes the received image data and maps the current location and environment.

[0335] 7. Server: Calculates the optimal route based on the user's instructions and emotional state. The calculation includes adjustments based on the emotional state (e.g., choosing a safe route to reduce the user's anxiety).

[0336] 8. Server: Sends optimal route information and additional instructions according to the emotional state to the device.

[0337] Movement and real-time response

[0338] 9. Terminal: The mobility control system starts moving based on the received route information.

[0339] 10. Terminal: Scans the surrounding environment in real time with a 360-degree camera to ensure safety.

[0340] 11. Terminal: Sends image data and emotional information to the server at any time.

[0341] 12. Terminal: Notifies the user of the ongoing situation and next actions via voice, providing feedback adapted by the emotion engine (e.g., "Go straight," "It's okay," etc.).

[0342] 13. Device: Monitors the user's emotional state and provides relaxing music or encouraging words as needed.

[0343] Example: When a user goes to the supermarket

[0344] 1. User: Says, "I want to go to the nearest supermarket."

[0345] 2. Device: Recognizes voice and analyzes the user's emotional state with an emotion engine. Captures the surroundings with a 360-degree camera.

[0346] 3. Server: Analyzes the image data, calculates a safe route, and instructs the user to play relaxing music to reduce anxiety.

[0347] 4. Device: The device starts moving based on the calculated route and guides the user by saying things like "Go straight" or "Turn left." If it detects an anxious expression, it notifies the user by saying "It's okay, don't worry."

[0348] 5. User: Follow the voice instructions and reach the supermarket with peace of mind.

[0349] In this way, the system of the present invention provides real-time feedback that takes into account the user's emotional state, thereby providing an environment in which visually impaired people can feel safe when going out.

[0350] The processing flow will be explained below.

[0351] Step 1:

[0352] User: Uses voice input. Says, "I want to go to the nearby supermarket."

[0353] Step 2:

[0354] Device: Captures the user's voice with a built-in microphone.

[0355] Step 3:

[0356] Terminal: The voice recognition system converts the captured voice into text data and analyzes the content, such as "I want to go to the nearby supermarket."

[0357] Step 4:

[0358] Device: A 360-degree camera captures the user's facial expressions, and an emotion engine analyzes the facial data to determine the user's emotional state, such as anxiety, relaxation, or joy.

[0359] Step 5:

[0360] Terminal: Sends the analysis results (voice text data and emotional information) to the server.

[0361] Step 6:

[0362] Terminal: Captures the current surrounding environment with a 360-degree camera and sends the image data to the server.

[0363] Step 7:

[0364] Server: Analyzes the received image data and maps the current location and environment.

[0365] Step 8:

[0366] Server: Identifies obstacles, steps, traffic lights, etc. from the results of image analysis.

[0367] Step 9:

[0368] Server: Calculates multiple optimal routes based on the user's destination information, current location information, and emotional state.

[0369] Step 10:

[0370] Server: Selects a safe route that avoids obstacles, construction areas, and puddles, and fine-tunes the route based on the user's emotional state (e.g., if the user is feeling anxious, select a wider, safer path).

[0371] Step 11:

[0372] Server: Sends the selected route information and travel instructions to the terminal.

[0373] Step 12:

[0374] Terminal: The mobility control system starts moving based on the received route information.

[0375] Step 13:

[0376] Terminal: While moving, a 360-degree camera scans the surrounding environment in real time to ensure safety.

[0377] Step 14:

[0378] Device: Captured image data and emotional information are sent to the server at any time, and the latest situation is shared.

[0379] Step 15:

[0380] Terminal: Recognizes obstacles, steps, and traffic lights in real time, and adjusts course to guide the user.

[0381] Step 16:

[0382] Device: Provides audible notification to the user about their current progress and next steps, e.g., "Go straight" or "Turn left."

[0383] Step 17:

[0384] Device: The emotion engine monitors the user's emotional state, and if it detects anxiety or tension, it provides voice feedback to help them relax, such as "It's okay" or "Don't worry."

[0385] Step 18:

[0386] User: Follow the robot's voice instructions and continue moving, with reassuring words from the robot if necessary.

[0387] Step 19:

[0388] Terminal: If the robot encounters an unexpected obstacle or an area under construction along the way, it sends new image data and emotion information to the server.

[0389] Step 20:

[0390] Server: Re-analyzes the new data and calculates a new route, taking into account the user's emotional state to determine the optimal route.

[0391] Step 21:

[0392] Server: Sends the recalculated new route information to the device.

[0393] Step 22:

[0394] Device: Resume travel based on new route information.

[0395] Step 23:

[0396] Terminal: When the user arrives at the destination, the device notifies the user by voice, saying, "You have arrived at the supermarket."

[0397] Step 24:

[0398] User: Confirms that destination has been reached.

[0399] Step 25:

[0400] Device: Ask the user for feedback on their travel experience. Example: "Did you have any issues during your trip?"

[0401] Step 26:

[0402] User: Provide verbal feedback, for example, "No issues" or "I was a little worried."

[0403] Step 27:

[0404] Terminal: Sends user feedback to the server.

[0405] Step 28:

[0406] Server: Stores the received feedback in a database.

[0407] Step 29:

[0408] Server: Updates the model parameters based on the feedback data and reflects them in the next performance improvement.

[0409] The above is the specific processing flow of the visually impaired support system that combines an emotion engine.

[0410] Example 2

[0411] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0412] To support visually impaired people when they go out, it is important to provide not only simple guidance but also adaptive feedback according to the user's emotional state. However, current systems have difficulty recognizing the user's emotional state in real time and providing appropriate support, which makes it difficult to alleviate anxiety and discomfort.

[0413] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0414] In this invention, the server includes a voice recognition means, an image acquisition means, an image analysis means, a route calculation means, a movement control means, a communication means, a communication means, and an emotion engine, which not only calculates and guides a route based on the voice instructions of a visually impaired person, but also analyzes the user's emotional state and provides adaptive feedback in real time.

[0415] A "voice recognition means" is a mechanism that captures voice input from a user and converts the voice into text data.

[0416] The "image acquisition means" is a mechanism that uses a camera mounted on the robot to capture images of the surrounding environment and collect image data.

[0417] The "image analysis means" is a mechanism that analyzes the acquired image data and identifies obstacles, steps, etc.

[0418] The "route calculation means" is a mechanism that calculates the optimal travel route based on the user's destination information and the image analysis results.

[0419] The "movement control means" is a mechanism that controls the movement of the robot based on the calculated route information.

[0420] The "communication means" is a mechanism for transmitting data obtained by the voice recognition means and image analysis means to the server and receiving instructions from the server.

[0421] The "communication means" is a mechanism that notifies the user of instructions received from the server by voice, and confirms the situation or receives further instructions through dialogue with the user.

[0422] The "emotion engine" is a mechanism that analyzes the user's voice and facial expression data to recognize the user's emotional state and provide feedback accordingly.

[0423] MODE FOR CARRYING OUT THE INVENTION

[0424] The quadrupedal guide dog robot system of the present invention is designed to enable visually impaired people to go out safely and is also capable of providing feedback according to the user's emotional state. This system includes a voice recognition unit, an image acquisition unit, an image analysis unit, a route calculation unit, a movement control unit, a communication unit, and an emotion engine. Specific hardware and software configurations for each unit are described below.

[0425] Hardware and Software Configuration

[0426] Voice recognition means

[0427] The device is equipped with a high-performance microphone and voice recognition software that uses generative AI models to analyze and convert users' voice commands into text data.

[0428] Image Acquisition Method

[0429] The device is equipped with a 360-degree camera that captures images of the surrounding environment in real time. This captured data is used by the image analysis method described below.

[0430] Image analysis methods

[0431] The acquired image data is sent to a server where it is analyzed using a high-performance GPU and specific algorithms to identify obstacles and steps.

[0432] Route calculation method

[0433] The server calculates the optimal route based on the image analysis results and the user's destination information. The route calculation also takes into account weather and construction area information. The route calculation algorithm is built on a generative AI model.

[0434] Movement Control Means

[0435] The calculated route information is sent to the terminal, which then controls the robot's movement based on this information. The movement control system dynamically corrects the route while acquiring environmental information in real time.

[0436] communication means

[0437] The data acquired by the voice recognition and image analysis means is transmitted bidirectionally to a server using a secure and fast protocol, such as Wi-Fi or a 5G network.

[0438] Communication tools

[0439] The instructions received from the server are notified to the user by the device through voice. For example, this includes guidance voice such as "Go straight." In addition, it is possible for the user to give instructions or confirm the situation through dialogue.

[0440] Emotion Engine

[0441] The device is equipped with an emotion engine that analyzes the user's voice and facial expression data to recognize the user's emotional state. This engine performs the analysis using a generative AI model, and if the user is feeling anxious, for example, it will play relaxing music or provide encouraging words such as "It's okay."

[0442] Specific examples

[0443] When a user voice-inputs "I want to go to the nearby supermarket," the following operation occurs.

[0444] 1. User: Instructs the device, "I want to go to a nearby supermarket."

[0445] 2. Device: Recognizes voice and analyzes the user's emotional state with an emotion engine. Captures the surroundings with a 360-degree camera.

[0446] 3. Server: Analyzes image data, calculates a safe route, and also recommends relaxing music to reduce the user's anxiety.

[0447] 4. Device: The device starts moving based on the calculated route and guides the user by saying things like "Go straight" or "Turn left." If it detects an anxious expression, it notifies the user by saying "It's okay, don't worry."

[0448] 5. User: Follow the voice instructions and reach the supermarket with peace of mind.

[0449] As described above, the system of the present invention provides real-time feedback and adaptive support to help visually impaired people reach their destination safely.

[0450] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0451] Program processing flow

[0452] Step 1:

[0453] The user speaks, "I want to go to the nearby supermarket."

[0454] Input: User's voice commands

[0455] Output: Audio data

[0456] Specific operation: The user speaks into the device's microphone.

[0457] Step 2:

[0458] The device captures voice with its built-in microphone, and the voice recognition system converts the voice into text data.

[0459] Input: Audio data

[0460] Output: Text data

[0461] How it works: Using a generative AI model, the speech recognition algorithm analyzes the speech and converts it into text.

[0462] Step 3:

[0463] The device captures the user's facial expressions with its built-in camera, and the emotion engine analyzes their emotional state.

[0464] Input: User's facial expression data

[0465] Output: Emotion analysis results

[0466] How it works: The camera captures the user's face, and the generative AI model analyzes the facial expression data to recognize their emotional state.

[0467] Step 4:

[0468] The device sends the converted text data and the emotion analysis results to the server.

[0469] Input: Text data, sentiment analysis results

[0470] Output: Data transmission

[0471] Specific operation: The communication module packages the data and uploads it to the server.

[0472] Step 5:

[0473] The device uses a 360-degree camera to capture images of the current surrounding environment in real time and transmits the image data to a server.

[0474] Input: Image data of the surrounding environment

[0475] Output: Image data transmission

[0476] Specific operation: The camera continuously captures images and the communication module transmits them.

[0477] Step 6:

[0478] The server analyzes the received image data and maps the environment, identifying obstacles, steps, traffic lights, etc.

[0479] Input: Image data

[0480] Output: Environment map, obstacle information

[0481] How it works: Using a high-performance GPU, image analysis algorithms process video data and identify important elements in the environment.

[0482] Step 7:

[0483] The server calculates the optimal route based on the user's intentions and emotional state, taking into account weather information and information about areas under construction.

[0484] Input: User's destination information, emotional state, environment map

[0485] Output: Optimal travel route

[0486] How it works: A route calculation algorithm integrates all the data to generate a safe and efficient route.

[0487] Step 8:

[0488] The server sends the calculated route information and additional instructions according to the emotional state to the terminal.

[0489] Input: Optimal travel route, additional instructions

[0490] Output: Data transmission

[0491] Specific operation: The server's communication module packages the route information and sends it to the terminal.

[0492] Step 9:

[0493] The terminal activates the mobility control system based on the route information received and begins moving autonomously.

[0494] Input: Travel route information

[0495] Output: Robot movement

[0496] Specific behavior: Motors and sensors operate, and the robot moves along the calculated route.

[0497] Step 10:

[0498] The device scans the surrounding environment in real time using a 360-degree camera to check for safety.

[0499] Input: Real-time image of the surrounding environment

[0500] Output: Safety confirmation information

[0501] How it works: Cameras continuously capture images and safety monitoring systems analyze the situation.

[0502] Step 11:

[0503] The image data and emotional information acquired by the terminal are transmitted to the server as needed.

[0504] Input: Real-time image data, emotional information

[0505] Output: Data transmission

[0506] Specific operation: The communication module periodically uploads and updates data.

[0507] Step 12:

[0508] The device notifies the user of the ongoing situation and next actions via voice, providing feedback based on the results analyzed by the emotion engine.

[0509] Input: Progress, next action instructions, emotion analysis results

[0510] Output: Audio instructions and feedback to the user

[0511] Specific operation: The speech synthesis system generates a notification message and transmits it to the user through the speaker.

[0512] Step 13:

[0513] The device continues to monitor the user's emotional state and provides relaxing music and encouraging words.

[0514] Input: Emotional state monitoring data

[0515] Output: Relaxing music, encouraging words

[0516] Specific operation: The emotion engine analyzes emotional data in real time and automatically selects an appropriate response.

[0517] (Application example 2)

[0518] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0519] There is a lack of technological means to assist visually impaired people in going out and navigating stores safely and securely. In particular, there is a lack of systems that can grasp the emotional state of visually impaired people and provide appropriate feedback accordingly. Therefore, it is necessary to reduce anxiety and stress while traveling and enable them to smoothly search for products in physical stores.

[0520] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice recognition unit that recognizes instructions input by voice from the user, an image acquisition unit that captures images of the surrounding environment, an image analysis unit that analyzes the acquired image data and identifies obstacles and steps, a route calculation unit that calculates the optimal route to the user's destination, a movement control unit that controls movement based on the calculated route, a communication unit that transmits the user's instructions recognized by the voice recognition unit and the data analyzed by the image analysis unit to the server and receives instructions from the server, a communication unit that notifies the user of the instructions received from the server by voice, an emotion recognition unit that recognizes the user's emotional state and adapts accordingly, and a feedback unit that provides real-time feedback according to the user's emotional state. This allows visually impaired people to travel with confidence and reach their destination safely. It also enables them to smoothly search for products in physical stores, reducing anxiety and stress.

[0521] "Speech recognition means" refers to a device or software that has the function of sensing voice input from a user and converting it into text data.

[0522] The "image acquisition means" is a device or means for capturing an image of the surrounding environment using a camera or the like and acquiring that information as digital image data.

[0523] "Image analysis means" refers to a device or algorithm that analyzes the captured image data and identifies obstacles, steps, and other features.

[0524] A "route calculation means" is a device or software that calculates the optimal route to the user's destination and provides that information.

[0525] "Movement control means" refers to a device or software that controls the movement of a machine or robot based on calculated route information.

[0526] "Communication means" refers to the device or protocol used to transmit and receive data and instructions between the server and the terminal within the system.

[0527] "Communication means" refers to a device or software that has the function of conveying instructions received from the server to the user by voice or other means.

[0528] "Emotion recognition means" refers to a device or algorithm that analyzes the user's emotions from their tone of voice, facial expressions, etc., and recognizes their state.

[0529] A "feedback means" is a device or software that has the function of providing a response or advice in real time according to the user's emotional state.

[0530] A "learner" is a device or algorithm that has the ability to receive feedback provided by users and update the system's model based on that information.

[0531] A system for implementing this invention includes a voice recognition means, an image acquisition means, an image analysis means, a route calculation means, a movement control means, a communication means, an emotion recognition means, and a feedback means to assist visually impaired people in navigating and searching for products in a physical store.

[0532] The server communicates with a speech recognition system that converts the user's voice input into text data, allowing it to understand and process the user's instructions. The speech recognition system uses the SpeechRecognition library, and the gTTS library is used to generate speech from the text and provide feedback to the user.

[0533] The device is equipped with a camera that captures images of the surrounding environment in real time. The captured image data is analyzed using OpenCV to detect obstacles and steps. In addition, an emotion recognition algorithm called EmotionRecognizer analyzes the user's emotional state from their facial expressions and tone of voice. This allows the device to determine whether the user is feeling anxious and provide appropriate feedback.

[0534] The server calculates the optimal route based on the image analysis results and the user's instructions using a route calculation method. This uses the NavigationAlgorithm. The calculated route information and additional instructions according to the user's emotional state are sent to the device.

[0535] The device operates a mobility control system based on the calculated route. It scans the environment in real time and corrects the route if necessary. It also notifies the user of the next action via voice. If the emotion recognition means detects the user's anxiety, it plays relaxing music or provides encouraging feedback such as "It's okay."

[0536] For example, if a user speaks in a physical store and asks, "Where are the tomatoes on the shelf?", the system will recognize the command, calculate the optimal route, and tell the user, "Go straight, then turn right." When the user reaches the tomato shelf, the system will say, "This is it. The tomatoes are on the shelf on the right."

[0537] An example of a prompt is as follows:

[0538] User: "Where are the tomatoes on the shelf?"

[0539] Application: "Go straight, then turn right."

[0540] (Arrived in front of the tomato shelves)

[0541] "Here it is. The tomatoes are on the shelf on the right."

[0542] User: "Thanks."

[0543] In this way, this invention will help visually impaired people to confidently navigate brick-and-mortar stores and improve the quality of their daily lives.

[0544] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0545] Step 1:

[0546] Action: The user gives a voice command.

[0547] Input: Audio data.

[0548] Data processing: The device's voice recognition system converts voice into text data.

[0549] Output: User's spoken instructions converted into text data.

[0550] Step 2:

[0551] How it works: The device uses a 360-degree camera to capture images of its surroundings.

[0552] Input: Real-time image data of the surrounding environment.

[0553] Data Processing: Image data is captured using an image acquisition means.

[0554] Output: Real-time image data.

[0555] Step 3:

[0556] Operation: The device sends the acquired image data to the server for analysis.

[0557] Input: The captured image data.

[0558] Data processing: Image data is sent to the server.

[0559] Output: Image data for analysis by the server.

[0560] Step 4:

[0561] How it works: The server analyzes image data and identifies obstacles and steps in the surrounding area.

[0562] Input: Image data received by the server.

[0563] Data processing: Analysis algorithms are used to identify obstacles and steps.

[0564] Output: Information on identified obstacles and steps.

[0565] Step 5:

[0566] Operation: The server calculates the optimal route to the user's destination.

[0567] Input: Text data of user's voice instructions and image analysis results.

[0568] Data processing: Calculate the optimal route using a route calculation algorithm.

[0569] Output: Calculated optimal route information.

[0570] Step 6:

[0571] Operation: The server sends the calculated route information to the terminal.

[0572] Input: The calculated optimal route information.

[0573] Data processing: Send route information to the device.

[0574] Output: Route information transferred to the device.

[0575] Step 7:

[0576] Operation: The device starts moving based on the received route information.

[0577] Input: Route information received from the server.

[0578] Data processing: The movement control system controls movement based on the route.

[0579] Output: Move terminal.

[0580] Step 8:

[0581] How it works: While moving, the device continuously scans its surrounding environment in real time.

[0582] Input: Real-time image data of the surrounding environment.

[0583] Data processing: Analyzes information to continuously check safety and avoid obstacles.

[0584] Output: Maintaining a safe travel path.

[0585] Step 9:

[0586] Action: The device will notify the user of the next action by voice.

[0587] Input: Instructions from the server and information on the current movement route.

[0588] Data processing: Generate voice instructions using communication means.

[0589] Output: Voice movement instructions.

[0590] Step 10:

[0591] How it works: The device monitors the user's emotional state in real time and provides adaptive feedback.

[0592] Input: Real-time data on the user's facial expressions and tone of voice.

[0593] Data processing: Emotion recognition algorithms are used to analyze the emotional state and generate appropriate feedback.

[0594] Output: Audio feedback depending on the user's emotional state.

[0595] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0596] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0597] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0598] [Second embodiment]

[0599] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0600] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0601] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0602] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0603] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0604] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0605] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0606] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0607] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0608] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0609] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0610] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0611] MODE FOR CARRYING OUT THE INVENTION

[0612] The quadrupedal guide dog robot system of the present invention is designed to assist visually impaired people when they go out. This system includes voice recognition means, image acquisition means, image analysis means, route calculation means, movement control means, communication means, and communication means, and each means works in cooperation to guide the visually impaired people safely to their destination.

[0613] Overall system configuration

[0614] Voice recognition means

[0615] It recognizes instructions that users speak (e.g., "I want to go to the nearest supermarket") and converts the speech into text data that is used to analyze the user's intent.

[0616] Image Acquisition Method

[0617] The robot is equipped with a 360-degree camera that captures images of its surroundings in real time, and this image data is used as the basis for understanding the surrounding situation.

[0618] Image analysis methods

[0619] The acquired image data is sent to a server where it is analyzed. This analysis includes identifying obstacles, steps, traffic lights, etc. This information is necessary to calculate a safe guidance route.

[0620] Route calculation method

[0621] The server calculates the optimal route based on the image analysis results and the user's destination information. The calculation also takes into account information such as obstacles, construction areas, and puddles. The calculated route information is sent to the robot.

[0622] Movement Control Means

[0623] The robot begins moving autonomously based on the route information received from the server. As it walks, it dynamically corrects its route based on image data captured in real time.

[0624] communication means

[0625] The user's instructions recognized by the voice recognition means and the data obtained by the image analysis means are transmitted to the server, and communication is also performed to receive instructions from the server and recalculated route information.

[0626] Communication tools

[0627] The robot notifies the user of the instructions received from the server by voice (e.g., "Go straight," "Turn left," etc.). It also confirms the situation and receives further instructions through dialogue with the user.

[0628] Program processing overview

[0629] Voice input and recognition

[0630] 1. User: Says, "I want to go to the nearby supermarket."

[0631] 2. Terminal: The recognition device converts the voice into text data and recognizes the content "Go to the nearby supermarket."

[0632] Acquisition and analysis of the surrounding environment

[0633] 3. Terminal: Captures image data of the surroundings using a 360-degree camera and sends it to the server.

[0634] 4. Server: Analyzes the received image data and identifies obstacles and steps.

[0635] Route calculation and movement control

[0636] 5. Server: Calculates the optimal route based on the analysis results and the user's destination information, and sends that information to the terminal.

[0637] 6. Terminal: Start moving according to the received route, checking the surrounding environment in real time as you go.

[0638] Voice communication

[0639] 7. Terminal: Notifies the user of the current situation and next actions via voice.

[0640] 8. User: Move according to the robot's instructions.

[0641] Example: When a user goes to the supermarket

[0642] 1. User: Says, "I want to go to the nearest supermarket."

[0643] 2. Device: Recognizes voice and captures the surroundings with a 360-degree camera.

[0644] 3. Server: Analyzes the image data and calculates a safe route.

[0645] 4. Terminal: Starts movement based on the calculation results and notifies the user of the next action by voice.

[0646] 5. User: Follow the robot's instructions to safely reach the supermarket.

[0647] In this way, the entire system works together to safely and efficiently guide users to their destination. This system solves the problems faced by conventional guide dogs and provides an environment where visually impaired people can go out more freely.

[0648] The processing flow will be explained below.

[0649] Step 1:

[0650] User: Uses voice input. Says, "I want to go to the nearby supermarket."

[0651] Step 2:

[0652] Device: Captures the user's voice with a built-in microphone.

[0653] Step 3:

[0654] Terminal: The voice recognition system converts the captured voice into text data and interprets the content as "I want to go to the nearby supermarket."

[0655] Step 4:

[0656] Terminal: Analyzes the converted text data and recognizes the user's request.

[0657] Step 5:

[0658] Terminal: Sends the analysis results to the server.

[0659] Step 6:

[0660] Terminal: Captures the current surrounding environment with a 360-degree camera and sends the image data to the server.

[0661] Step 7:

[0662] Server: Analyzes the received image data and maps the current location and environment.

[0663] Step 8:

[0664] Server: Performs image analysis to identify the location of obstacles, steps, and traffic lights.

[0665] Step 9:

[0666] Server: Calculates multiple routes based on the user's destination "supermarket" and current location.

[0667] Step 10:

[0668] Server: From the calculated routes, select the optimal route that avoids obstacles, construction areas, puddles, etc.

[0669] Step 11:

[0670] Server: Sends the selected route information and walking instructions to the device.

[0671] Step 12:

[0672] Terminal: Based on the received route information, the mobility control system starts moving.

[0673] Step 13:

[0674] Terminal: A 360-degree camera scans the surrounding environment in real time to ensure safety.

[0675] Step 14:

[0676] Terminal: Sends captured image data to the server at any time, sharing the latest situation.

[0677] Step 15:

[0678] Terminal: Recognizes obstacles, steps, and traffic lights in real time and guides the user.

[0679] Step 16:

[0680] Device: Speaks to the user about their current progress and next steps, for example, "Go straight" or "Turn left."

[0681] Step 17:

[0682] User: Follow the robot's voice instructions and move.

[0683] Step 18:

[0684] Terminal: If the device encounters an unexpected obstacle or construction area along the way, it sends new image data to the server.

[0685] Step 19:

[0686] Server: Re-analyzes based on the new data and recalculates a new route.

[0687] Step 20:

[0688] Server: Sends the new optimized route information to the device.

[0689] Step 21:

[0690] Device: Resume travel based on new route information.

[0691] Step 22:

[0692] Terminal: When the user arrives at the destination, the terminal notifies the user that "You have arrived at the supermarket."

[0693] Step 23:

[0694] User: Confirms that they have reached the supermarket.

[0695] Step 24:

[0696] Device: Ask the user for feedback on their travel experience. Example: "Did you have any issues during your trip?"

[0697] Step 25:

[0698] User: Provides feedback verbally. Example: "No issues."

[0699] Step 26:

[0700] Terminal: Sends user feedback to the server.

[0701] Step 27:

[0702] Server: Stores the received feedback in a database.

[0703] Step 28:

[0704] Server: Updates the model parameters based on the feedback data to improve performance.

[0705] The above is a specific processing flow of the four-legged guide dog robot system for assisting visually impaired people when they go out.

[0706] Example 1

[0707] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0708] Conventional guide dogs and assistive devices have struggled to adequately support the safe and efficient movement of visually impaired people. In particular, they lacked the ability to grasp the surrounding environment in real time and dynamically correct routes, limiting the ability of visually impaired people to respond to sudden obstacles, unmanned roads, areas under construction, and other situations. Furthermore, they lacked the ability to utilize user feedback to improve the system's efficiency. This made it difficult to provide an environment in which visually impaired people could go out with peace of mind.

[0709] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0710] In this invention, the server includes voice recognition means for recognizing instructions input by voice from the user, image acquisition means for capturing images of the surrounding environment, image analysis means for analyzing the captured image data and identifying obstacles and steps, route calculation means for calculating the optimal route to the user's destination, movement control means for controlling movement based on the calculated route, communication means for transmitting the user's instructions recognized by the voice recognition means and data analyzed by the image analysis means to an information processing device and receiving instructions from the information processing device, communication means for notifying the user of the instructions received from the information processing device by voice, and means for dynamically correcting the route while checking the environment in real time, thereby providing an environment in which visually impaired people can go out with peace of mind.

[0711] The "voice recognition means" is a device that recognizes instructions input by voice from the user and converts them into text data.

[0712] "Image acquisition means" refers to a device that captures images of the surrounding environment and acquires the image data. This device includes a 360-degree camera.

[0713] The "image analysis means" is a device or program for analyzing the acquired image data and identifying obstacles and steps.

[0714] The "route calculation means" is a device or program that calculates the optimal route to the user's destination.

[0715] The "movement control means" is a device or program that controls the movement of the robot based on the calculated route.

[0716] The "communication means" is a device or program for transmitting user instructions recognized by the voice recognition means and data analyzed by the image analysis means to the information processing device, and for receiving instructions from the information processing device.

[0717] The "communication means" is a device or program that notifies the user of instructions received from the information processing device by voice.

[0718] The "means for dynamically correcting the route while checking the environment in real time" refers to a device or program that allows the robot to continuously check the surrounding environment while moving and correct the route as necessary.

[0719] MODE FOR CARRYING OUT THE INVENTION

[0720] The quadrupedal guide dog robot system of the present invention is designed to assist visually impaired people when they go out. This system includes voice recognition means, image acquisition means, image analysis means, route calculation means, movement control means, communication means, and communication means, and each means works in cooperation to guide the visually impaired people safely to their destination.

[0721] Overall system configuration

[0722] The main components of this system are:

[0723] Voice recognition means

[0724] It recognizes instructions that a user speaks and converts the speech into text data. Specifically, it uses a speech recognition system (e.g., Google Cloud Speech-to-Text service) to recognize the user's voice instructions. This text data is used to analyze the user's intent.

[0725] Image Acquisition Method

[0726] The robot is equipped with a 360-degree camera that captures images of its surroundings in real time, and this image data is used as the basis for understanding the surrounding situation.

[0727] Image analysis methods

[0728] The acquired image data is sent to a server where it is analyzed. This analysis includes identifying obstacles, steps, traffic lights, etc., using object detection algorithms such as YOLO (You Only Look Once). The analysis also generates location information for obstacles and safe route options.

[0729] Route calculation method

[0730] The server calculates the optimal route based on the image analysis results and the user's destination information. For example, it uses a path-finding algorithm such as Dijkstra's algorithm. This calculation also takes into account information such as obstacles, construction areas, and puddles. The calculated route information is sent to the robot.

[0731] Movement Control Means

[0732] The robot begins moving autonomously based on the route information received from the server. While moving, it continuously monitors the surrounding environment with a 360-degree camera and corrects its route in real time. Even if an obstacle suddenly appears, it automatically stops and resumes moving according to the re-planned route.

[0733] communication means

[0734] The user's instructions recognized by the voice recognition means and the data analyzed by the image analysis means are transmitted to the server, and communication is also performed to receive instructions and recalculated route information from the server.

[0735] Communication tools

[0736] The robot will then verbally communicate instructions received from the server to the user, such as "go straight" or "turn left," and will also include a learning mechanism to update the system based on user feedback.

[0737] Example: When a user goes to the supermarket

[0738] 1. User: Says "I want to go to the nearest supermarket." The voice recognition means converts this into text data.

[0739] 2. Device: Recognizes voice and captures the surroundings with a 360-degree camera.

[0740] 3. Server: Analyzes image data to identify obstacles and steps. Based on the analysis results, calculates a safe route.

[0741] 4. Terminal: Receives calculated route information and starts moving, monitoring the environment in real time as it moves forward, and notifies the user of the next action via voice.

[0742] 5. User: Follow the robot's instructions to safely reach the supermarket.

[0743] In this way, the entire system works together to safely and efficiently guide users to their destination. This system solves the problems faced by conventional guide dogs and provides an environment where visually impaired people can go out more freely.

[0744] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0745] Step 1:

[0746] User: Enters a voice command such as "I want to go to the nearest supermarket."

[0747] Input: User's voice commands

[0748] Output: Audio input data

[0749] Step 2:

[0750] On the device: A speech recognition system converts speech into text data, for example, using the Google Cloud Speech-to-Text service.

[0751] Input: Voice input data

[0752] Data processing: Converting voice data into text data

[0753] Output: "I want to go to the nearest supermarket."

[0754] How it works: The voice recognizer activates, records speech, and converts it into text.

[0755] Step 3:

[0756] Terminal: Sends the converted text data to the server.

[0757] Input: Text data "I want to go to the nearest supermarket"

[0758] Output: Data sent to the server

[0759] How it works: The communications module packets the text data and sends it over the Internet to a server.

[0760] Step 4:

[0761] Terminal: Captures image data of the surroundings using a 360-degree camera and sends it to the server.

[0762] Input: Surrounding environment

[0763] Output: Image data

[0764] What it does: The camera starts up, continuously captures 360-degree images, and sends the data to a server.

[0765] Step 5:

[0766] Server: Analyzes the received image data and identifies obstacles and steps. Uses object detection algorithms such as YOLO.

[0767] Input: Image data

[0768] Data calculation: Apply object detection algorithm to identify obstacles and steps

[0769] Output: Analysis results (obstacle location information, etc.)

[0770] Operation: Image analysis software processes image data and generates analytical results.

[0771] Step 6:

[0772] Server: Based on the user's destination information "nearby supermarket" and the image analysis results, the optimal route is calculated using Dijkstra's algorithm.

[0773] Input: Destination information, analysis results

[0774] Data calculation: Apply route calculation algorithm

[0775] Output: Optimal route information

[0776] Operation: The route calculation program on the server runs, calculates the optimal route, and generates data.

[0777] Step 7:

[0778] Server: Sends calculated route information to the device.

[0779] Input: Optimal route information

[0780] Output: Data sent to the terminal

[0781] Operation: The communication module packets the route information and sends it to the terminal via the Internet.

[0782] Step 8:

[0783] Terminal: The terminal begins moving autonomously based on the received route information. If an obstacle is detected along the way, the terminal corrects the route in real time.

[0784] Input: Route information, real-time images from a 360-degree camera

[0785] Data calculation: Route correction based on image data

[0786] Output: Movement control command

[0787] Operation: The movement control system is activated and controls the motors and actuators to move the robot.

[0788] Step 9:

[0789] Terminal: Notifies the user of the current situation and next actions via voice.

[0790] Input: Instructions from the server, route information

[0791] Output: Audio notification

[0792] How it works: The voice synthesis system converts instructions into voice and transmits them to the user through the speaker.

[0793] Step 10:

[0794] User: Move according to the robot's voice instructions.

[0795] Input: Voice commands

[0796] Output: User Movement

[0797] Action: The user moves according to the robot's instructions.

[0798] (Application example 1)

[0799] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0800] When visually impaired people go out, existing assistance systems have difficulty quickly recalculating the optimal route in response to changes in the surrounding environment. Furthermore, real-time instructions and notifications of the situation are insufficient, leaving users feeling uneasy. There is a need to provide a system that can solve these problems and enable visually impaired people to travel more safely and efficiently.

[0801] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0802] In this invention, the server includes voice recognition means for recognizing instructions input by voice from the user, image acquisition means for capturing images of the surrounding environment, image analysis means for analyzing the captured image data and identifying obstacles and steps, route calculation means for calculating an optimal route to the user's destination, movement control means for controlling movement based on the calculated route, communication means for transmitting the user's instructions recognized by the voice recognition means and data analyzed by the image analysis means to the server and receiving instructions from the server, communication means for notifying the user of the instructions received from the server by voice, and dynamic re-route calculation means for analyzing the environment and re-calculating the optimal route in real time based on the captured data, thereby enabling safe and efficient movement according to the environment.

[0803] The "voice recognition means" is a device that recognizes instructions input by voice by the user and converts them into text data.

[0804] The "image acquisition means" is a device that takes pictures of the surrounding environment with a camera and acquires image data.

[0805] The "image analysis means" is a device that analyzes the acquired image data and identifies environmental information such as obstacles and steps.

[0806] The "route calculation means" is a device that calculates the optimal route based on the user's destination information.

[0807] "Movement control means" refers to a device that controls the movement of the system based on the calculated route.

[0808] The "communication means" is a device that transmits user instructions recognized by the voice recognition means and data analyzed by the image analysis means to the server, and receives instructions from the server.

[0809] The "communication means" is a device that notifies the user of instructions received from the server by voice and communicates with the user through dialogue.

[0810] A "dynamic re-route calculation means" is a device that analyzes the environment in real time based on acquired data and recalculates the optimal route if necessary.

[0811] The system that realizes this application example is an automated driving guidance system for visually impaired people, and includes as its main components a voice recognition means, an image acquisition means, an image analysis means, a route calculation means, a movement control means, a communication means, a communication means, and a dynamic re-route calculation means.

[0812] Voice recognition means

[0813] The server uses speech recognition software (e.g., the speech_recognition library) to recognize instructions entered by the user through speech and convert them into text data. For example, if a user says, "I want to go to the nearest supermarket," this speech is converted into text and processed within the system.

[0814] Image Acquisition Method

[0815] The device uses a 360-degree camera (e.g., cv2.VideoCapture) to capture the surrounding environment and acquire image data, which is important for the system to understand the visual environment.

[0816] Image analysis methods

[0817] The server uses image analysis software (e.g., OpenCV library) to analyze the acquired image data and identify obstacles, steps, etc. The analyzed data is input into the route calculation means.

[0818] Route calculation method

[0819] The server uses a map service API (e.g., the Google Maps library) to calculate the optimal route based on the user's destination information. The calculated route information is sent to the device.

[0820] Movement Control Means

[0821] The device autonomously begins moving based on the route received from the server, checking environmental data in real time as it moves forward and dynamically correcting its route as necessary.

[0822] communication means

[0823] The server transmits user instructions recognized by the voice recognition means and data analyzed by the image analysis means to the server, and is equipped with communication functions (e.g., Internet connection, Wi-Fi) to receive instructions from the server at the terminal.

[0824] Communication tools

[0825] The device uses a text-to-speech engine (e.g., a TTS engine) to audibly announce to the user the instructions received from the server, for example, informing the user that "the next turn is to the right" on the way to a destination.

[0826] Dynamic reroute calculation method

[0827] The server analyzes the environment in real time based on the acquired data and recalculates the optimal route as needed, allowing the vehicle to respond immediately to situations such as unexpected obstacles or road construction.

[0828] Specific examples

[0829] For example, if a user says, "I want to go to the nearby supermarket," the voice recognition means converts this instruction into text data and sends it to the server. The server uses a map service API to calculate the optimal route and sends this information to the device. The device then takes pictures of the surrounding environment with a 360-degree camera and begins moving while identifying obstacles using image analysis means. As the environment changes, the route is dynamically recalculated and appropriate instructions are provided to the user using a text-to-speech engine.

[0830] Examples of prompts: "I want to go to the nearest supermarket" "Turn right next"

[0831] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0832] Step 1:

[0833] Voice input and recognition

[0834] Subject: User

[0835] How it works: The user says, "I want to go to the nearest supermarket."

[0836] Input: User's voice command

[0837] Data processing: Use speech recognition software (e.g., speech_recognition library) to convert voice instructions into text data.

[0838] Output: Text data "I want to go to a nearby supermarket"

[0839] Step 2:

[0840] Shooting the surrounding environment

[0841] Subject: Device

[0842] What it does: Captures the surrounding environment with a 360-degree camera.

[0843] Input: Surrounding environment information

[0844] Data processing: Use a 360-degree camera (e.g., cv2.VideoCapture) to acquire image data of the environment.

[0845] Output: Image data of the environment

[0846] Step 3:

[0847] Image data analysis

[0848] Subject: Server

[0849] Operation: Analyzes acquired image data and identifies obstacles and steps.

[0850] Input: Image data of the environment

[0851] Data processing: Use image analysis software (e.g., OpenCV library) to identify obstacles, steps, etc.

[0852] Output: Analysis data on obstacles and steps

[0853] Step 4:

[0854] Calculating the best route

[0855] Subject: Server

[0856] Operation: Calculates the optimal route based on the user's destination information.

[0857] Input: User instruction text data and image analysis data

[0858] Data calculation: Uses map service APIs (e.g., Google Maps library) to calculate the optimal route.

[0859] Output: Optimal route information

[0860] Step 5:

[0861] Movement Control

[0862] Subject: Device

[0863] Operation: The robot begins moving autonomously based on the calculated route information.

[0864] Input: Optimal route information

[0865] Data calculation: Using a movement control algorithm, control the movement according to the route.

[0866] Output: Actual movement

[0867] Step 6:

[0868] Real-time environment reanalysis and route recalculation

[0869] Subject: Server

[0870] How it works: Reanalyzes the environment in real time based on acquired data and recalculates the optimal route if necessary.

[0871] Input: Image data of the environment and current route information

[0872] Data calculation: Use dynamic re-route calculation method to recalculate the optimal route.

[0873] Output: Updated optimal route information

[0874] Step 7:

[0875] Communications and Notice of Instructions

[0876] Subject: Server and Terminal

[0877] Operation: New instructions from the server are sent to the terminal, which then notifies the user by voice.

[0878] Input: Updated route information

[0879] Data processing: Using communication functions (e.g., Internet connection, Wi-Fi), instructions from the server are sent to the device, which then uses a TTS engine to provide voice notifications to the user.

[0880] Output: Spoken instructions to the user (e.g., "Turn right next")

[0881] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0882] MODE FOR CARRYING OUT THE INVENTION

[0883] The quadrupedal guide dog robot system of the present invention is designed to assist visually impaired people when they go out, and is also combined with an emotion engine that recognizes the user's emotional state and responds accordingly. This system includes voice recognition means, image acquisition means, image analysis means, route calculation means, movement control means, communication means, and the emotion engine, and each of these means works in cooperation to guide visually impaired people to their destination safely and with a sense of psychological security.

[0884] Overall system configuration

[0885] Voice recognition means

[0886] It recognizes instructions that users speak (e.g., "I want to go to the nearest supermarket") and converts the speech into text data that is used to analyze the user's intent.

[0887] Image Acquisition Method

[0888] The robot is equipped with a 360-degree camera that captures images of its surroundings in real time, and this image data is used as the basis for understanding the surrounding situation.

[0889] Image analysis methods

[0890] The acquired image data is sent to a server where it is analyzed. This analysis includes identifying obstacles, steps, traffic lights, etc. This information is necessary to calculate a safe guidance route.

[0891] Route calculation method

[0892] The server calculates the optimal route based on the image analysis results and the user's destination information. The calculation also takes into account information such as obstacles, construction areas, and puddles. The calculated route information is sent to the robot.

[0893] Movement Control Means

[0894] The robot begins moving autonomously based on the route information received from the server. As it walks, it dynamically corrects its route based on image data captured in real time.

[0895] communication means

[0896] The user's instructions recognized by the voice recognition means and the data obtained by the image analysis means are transmitted to the server, and communication is also performed to receive instructions from the server and recalculated route information.

[0897] Communication tools

[0898] The robot notifies the user of the instructions received from the server by voice (e.g., "Go straight," "Turn left," etc.). It also confirms the situation and receives further instructions through dialogue with the user.

[0899] Emotion Engine

[0900] The robot is equipped with an emotion engine that analyzes the user's voice and facial expression data to recognize the user's emotional state. For example, if the user is feeling anxious, the system can play relaxing music to help the user relax or provide encouraging words such as "It's okay."

[0901] Program processing overview

[0902] Voice Input and Emotion Recognition

[0903] 1. User: Says, "I want to go to the nearby supermarket."

[0904] 2. Device: The built-in microphone and camera capture the user's voice and facial expressions.

[0905] 3. Terminal: The voice recognition system converts the captured voice into text data, and the emotion engine then analyzes the user's emotional state from their voice and facial expressions.

[0906] 4. Terminal: Analyzes the instructions from the text data and sends the emotional state recognized by the emotion engine to the server.

[0907] Environmental awareness and route planning

[0908] 5. Terminal: Captures the current surrounding environment with a 360-degree camera and sends the image data to the server.

[0909] 6. Server: Analyzes the received image data and maps the current location and environment.

[0910] 7. Server: Calculates the optimal route based on the user's instructions and emotional state. The calculation includes adjustments based on the emotional state (e.g., choosing a safe route to reduce the user's anxiety).

[0911] 8. Server: Sends optimal route information and additional instructions according to the emotional state to the device.

[0912] Movement and real-time response

[0913] 9. Terminal: The mobility control system starts moving based on the received route information.

[0914] 10. Terminal: Scans the surrounding environment in real time with a 360-degree camera to ensure safety.

[0915] 11. Terminal: Sends image data and emotional information to the server at any time.

[0916] 12. Terminal: Notifies the user of the ongoing situation and next actions via voice, providing feedback adapted by the emotion engine (e.g., "Go straight," "It's okay," etc.).

[0917] 13. Device: Monitors the user's emotional state and provides relaxing music or encouraging words as needed.

[0918] Example: When a user goes to the supermarket

[0919] 1. User: Says, "I want to go to the nearest supermarket."

[0920] 2. Device: Recognizes voice and analyzes the user's emotional state with an emotion engine. Captures the surroundings with a 360-degree camera.

[0921] 3. Server: Analyzes the image data, calculates a safe route, and instructs the user to play relaxing music to reduce anxiety.

[0922] 4. Device: The device starts moving based on the calculated route and guides the user by saying things like "Go straight" or "Turn left." If it detects an anxious expression, it notifies the user by saying "It's okay, don't worry."

[0923] 5. User: Follow the voice instructions and reach the supermarket with peace of mind.

[0924] In this way, the system of the present invention provides real-time feedback that takes into account the user's emotional state, thereby providing an environment in which visually impaired people can feel safe when going out.

[0925] The processing flow will be explained below.

[0926] Step 1:

[0927] User: Uses voice input. Says, "I want to go to the nearby supermarket."

[0928] Step 2:

[0929] Device: Captures the user's voice with a built-in microphone.

[0930] Step 3:

[0931] Terminal: The voice recognition system converts the captured voice into text data and analyzes the content, such as "I want to go to the nearby supermarket."

[0932] Step 4:

[0933] Device: A 360-degree camera captures the user's facial expressions, and an emotion engine analyzes the facial data to determine the user's emotional state, such as anxiety, relaxation, or joy.

[0934] Step 5:

[0935] Terminal: Sends the analysis results (voice text data and emotional information) to the server.

[0936] Step 6:

[0937] Terminal: Captures the current surrounding environment with a 360-degree camera and sends the image data to the server.

[0938] Step 7:

[0939] Server: Analyzes the received image data and maps the current location and environment.

[0940] Step 8:

[0941] Server: Identifies obstacles, steps, traffic lights, etc. from the results of image analysis.

[0942] Step 9:

[0943] Server: Calculates multiple optimal routes based on the user's destination information, current location information, and emotional state.

[0944] Step 10:

[0945] Server: Selects a safe route that avoids obstacles, construction areas, and puddles, and fine-tunes the route based on the user's emotional state (e.g., if the user is feeling anxious, select a wider, safer path).

[0946] Step 11:

[0947] Server: Sends the selected route information and travel instructions to the terminal.

[0948] Step 12:

[0949] Terminal: The mobility control system starts moving based on the received route information.

[0950] Step 13:

[0951] Terminal: While moving, a 360-degree camera scans the surrounding environment in real time to ensure safety.

[0952] Step 14:

[0953] Device: Captured image data and emotional information are sent to the server at any time, and the latest situation is shared.

[0954] Step 15:

[0955] Terminal: Recognizes obstacles, steps, and traffic lights in real time, and adjusts course to guide the user.

[0956] Step 16:

[0957] Device: Provides audible notification to the user about their current progress and next steps, e.g., "Go straight" or "Turn left."

[0958] Step 17:

[0959] Device: The emotion engine monitors the user's emotional state, and if it detects anxiety or tension, it provides voice feedback to help them relax, such as "It's okay" or "Don't worry."

[0960] Step 18:

[0961] User: Follow the robot's voice instructions and continue moving, with reassuring words from the robot if necessary.

[0962] Step 19:

[0963] Terminal: If the robot encounters an unexpected obstacle or an area under construction along the way, it sends new image data and emotion information to the server.

[0964] Step 20:

[0965] Server: Re-analyzes the new data and calculates a new route, taking into account the user's emotional state to determine the optimal route.

[0966] Step 21:

[0967] Server: Sends the recalculated new route information to the device.

[0968] Step 22:

[0969] Device: Resume travel based on new route information.

[0970] Step 23:

[0971] Terminal: When the user arrives at the destination, the device notifies the user by voice, saying, "You have arrived at the supermarket."

[0972] Step 24:

[0973] User: Confirms that destination has been reached.

[0974] Step 25:

[0975] Device: Ask the user for feedback on their travel experience. Example: "Did you have any issues during your trip?"

[0976] Step 26:

[0977] User: Provide verbal feedback, for example, "No issues" or "I was a little worried."

[0978] Step 27:

[0979] Terminal: Sends user feedback to the server.

[0980] Step 28:

[0981] Server: Stores the received feedback in a database.

[0982] Step 29:

[0983] Server: Updates the model parameters based on the feedback data and reflects them in the next performance improvement.

[0984] The above is the specific processing flow of the visually impaired support system that combines an emotion engine.

[0985] Example 2

[0986] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0987] To support visually impaired people when they go out, it is important to provide not only simple guidance but also adaptive feedback according to the user's emotional state. However, current systems have difficulty recognizing the user's emotional state in real time and providing appropriate support, which makes it difficult to alleviate anxiety and discomfort.

[0988] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0989] In this invention, the server includes a voice recognition means, an image acquisition means, an image analysis means, a route calculation means, a movement control means, a communication means, a communication means, and an emotion engine, which not only calculates and guides a route based on the voice instructions of a visually impaired person, but also analyzes the user's emotional state and provides adaptive feedback in real time.

[0990] A "voice recognition means" is a mechanism that captures voice input from a user and converts the voice into text data.

[0991] The "image acquisition means" is a mechanism that uses a camera mounted on the robot to capture images of the surrounding environment and collect image data.

[0992] The "image analysis means" is a mechanism that analyzes the acquired image data and identifies obstacles, steps, etc.

[0993] The "route calculation means" is a mechanism that calculates the optimal travel route based on the user's destination information and the image analysis results.

[0994] The "movement control means" is a mechanism that controls the movement of the robot based on the calculated route information.

[0995] The "communication means" is a mechanism for transmitting data obtained by the voice recognition means and image analysis means to the server and receiving instructions from the server.

[0996] The "communication means" is a mechanism that notifies the user of instructions received from the server by voice, and confirms the situation or receives further instructions through dialogue with the user.

[0997] The "emotion engine" is a mechanism that analyzes the user's voice and facial expression data to recognize the user's emotional state and provide feedback accordingly.

[0998] MODE FOR CARRYING OUT THE INVENTION

[0999] The quadrupedal guide dog robot system of the present invention is designed to enable visually impaired people to go out safely and is also capable of providing feedback according to the user's emotional state. This system includes a voice recognition unit, an image acquisition unit, an image analysis unit, a route calculation unit, a movement control unit, a communication unit, and an emotion engine. Specific hardware and software configurations for each unit are described below.

[1000] Hardware and Software Configuration

[1001] Voice recognition means

[1002] The device is equipped with a high-performance microphone and voice recognition software that uses generative AI models to analyze and convert users' voice commands into text data.

[1003] Image Acquisition Method

[1004] The device is equipped with a 360-degree camera that captures images of the surrounding environment in real time. This captured data is used by the image analysis method described below.

[1005] Image analysis methods

[1006] The acquired image data is sent to a server where it is analyzed using a high-performance GPU and specific algorithms to identify obstacles and steps.

[1007] Route calculation method

[1008] The server calculates the optimal route based on the image analysis results and the user's destination information. The route calculation also takes into account weather and construction area information. The route calculation algorithm is built on a generative AI model.

[1009] Movement Control Means

[1010] The calculated route information is sent to the terminal, which then controls the robot's movement based on this information. The movement control system dynamically corrects the route while acquiring environmental information in real time.

[1011] communication means

[1012] The data acquired by the voice recognition and image analysis means is transmitted bidirectionally to a server using a secure and fast protocol, such as Wi-Fi or a 5G network.

[1013] Communication tools

[1014] The instructions received from the server are notified to the user by the device through voice. For example, this includes guidance voice such as "Go straight." In addition, it is possible for the user to give instructions or confirm the situation through dialogue.

[1015] Emotion Engine

[1016] The device is equipped with an emotion engine that analyzes the user's voice and facial expression data to recognize the user's emotional state. This engine performs the analysis using a generative AI model, and if the user is feeling anxious, for example, it will play relaxing music or provide encouraging words such as "It's okay."

[1017] Specific examples

[1018] When a user voice-inputs "I want to go to the nearby supermarket," the following operation occurs.

[1019] 1. User: Instructs the device, "I want to go to a nearby supermarket."

[1020] 2. Device: Recognizes voice and analyzes the user's emotional state with an emotion engine. Captures the surroundings with a 360-degree camera.

[1021] 3. Server: Analyzes image data, calculates a safe route, and also recommends relaxing music to reduce the user's anxiety.

[1022] 4. Device: The device starts moving based on the calculated route and guides the user by saying things like "Go straight" or "Turn left." If it detects an anxious expression, it notifies the user by saying "It's okay, don't worry."

[1023] 5. User: Follow the voice instructions and reach the supermarket with peace of mind.

[1024] As described above, the system of the present invention provides real-time feedback and adaptive support to help visually impaired people reach their destination safely.

[1025] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1026] Program processing flow

[1027] Step 1:

[1028] The user speaks, "I want to go to the nearby supermarket."

[1029] Input: User's voice commands

[1030] Output: Audio data

[1031] Specific operation: The user speaks into the device's microphone.

[1032] Step 2:

[1033] The device captures voice with its built-in microphone, and the voice recognition system converts the voice into text data.

[1034] Input: Audio data

[1035] Output: Text data

[1036] How it works: Using a generative AI model, the speech recognition algorithm analyzes the speech and converts it into text.

[1037] Step 3:

[1038] The device captures the user's facial expressions with its built-in camera, and the emotion engine analyzes their emotional state.

[1039] Input: User's facial expression data

[1040] Output: Emotion analysis results

[1041] How it works: The camera captures the user's face, and the generative AI model analyzes the facial expression data to recognize their emotional state.

[1042] Step 4:

[1043] The device sends the converted text data and the emotion analysis results to the server.

[1044] Input: Text data, sentiment analysis results

[1045] Output: Data transmission

[1046] Specific operation: The communication module packages the data and uploads it to the server.

[1047] Step 5:

[1048] The device uses a 360-degree camera to capture images of the current surrounding environment in real time and transmits the image data to a server.

[1049] Input: Image data of the surrounding environment

[1050] Output: Image data transmission

[1051] Specific operation: The camera continuously captures images and the communication module transmits them.

[1052] Step 6:

[1053] The server analyzes the received image data and maps the environment, identifying obstacles, steps, traffic lights, etc.

[1054] Input: Image data

[1055] Output: Environment map, obstacle information

[1056] How it works: Using a high-performance GPU, image analysis algorithms process video data and identify important elements in the environment.

[1057] Step 7:

[1058] The server calculates the optimal route based on the user's intentions and emotional state, taking into account weather information and information about areas under construction.

[1059] Input: User's destination information, emotional state, environment map

[1060] Output: Optimal travel route

[1061] How it works: A route calculation algorithm integrates all the data to generate a safe and efficient route.

[1062] Step 8:

[1063] The server sends the calculated route information and additional instructions according to the emotional state to the terminal.

[1064] Input: Optimal travel route, additional instructions

[1065] Output: Data transmission

[1066] Specific operation: The server's communication module packages the route information and sends it to the terminal.

[1067] Step 9:

[1068] The terminal activates the mobility control system based on the route information received and begins moving autonomously.

[1069] Input: Travel route information

[1070] Output: Robot movement

[1071] Specific behavior: Motors and sensors operate, and the robot moves along the calculated route.

[1072] Step 10:

[1073] The device scans the surrounding environment in real time using a 360-degree camera to check for safety.

[1074] Input: Real-time image of the surrounding environment

[1075] Output: Safety confirmation information

[1076] How it works: Cameras continuously capture images and safety monitoring systems analyze the situation.

[1077] Step 11:

[1078] The image data and emotional information acquired by the terminal are transmitted to the server as needed.

[1079] Input: Real-time image data, emotional information

[1080] Output: Data transmission

[1081] Specific operation: The communication module periodically uploads and updates data.

[1082] Step 12:

[1083] The device notifies the user of the ongoing situation and next actions via voice, providing feedback based on the results analyzed by the emotion engine.

[1084] Input: Progress, next action instructions, emotion analysis results

[1085] Output: Audio instructions and feedback to the user

[1086] Specific operation: The speech synthesis system generates a notification message and transmits it to the user through the speaker.

[1087] Step 13:

[1088] The device continues to monitor the user's emotional state and provides relaxing music and encouraging words.

[1089] Input: Emotional state monitoring data

[1090] Output: Relaxing music, encouraging words

[1091] Specific operation: The emotion engine analyzes emotional data in real time and automatically selects an appropriate response.

[1092] (Application example 2)

[1093] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1094] There is a lack of technological means to assist visually impaired people in going out and navigating stores safely and securely. In particular, there is a lack of systems that can grasp the emotional state of visually impaired people and provide appropriate feedback accordingly. Therefore, it is necessary to reduce anxiety and stress while traveling and enable them to smoothly search for products in physical stores.

[1095] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice recognition unit that recognizes instructions input by voice from the user, an image acquisition unit that captures images of the surrounding environment, an image analysis unit that analyzes the acquired image data and identifies obstacles and steps, a route calculation unit that calculates the optimal route to the user's destination, a movement control unit that controls movement based on the calculated route, a communication unit that transmits the user's instructions recognized by the voice recognition unit and the data analyzed by the image analysis unit to the server and receives instructions from the server, a communication unit that notifies the user of the instructions received from the server by voice, an emotion recognition unit that recognizes the user's emotional state and adapts accordingly, and a feedback unit that provides real-time feedback according to the user's emotional state. This allows visually impaired people to travel with confidence and reach their destination safely. It also enables them to smoothly search for products in physical stores, reducing anxiety and stress.

[1096] "Speech recognition means" refers to a device or software that has the function of sensing voice input from a user and converting it into text data.

[1097] The "image acquisition means" is a device or means for capturing an image of the surrounding environment using a camera or the like and acquiring that information as digital image data.

[1098] "Image analysis means" refers to a device or algorithm that analyzes the captured image data and identifies obstacles, steps, and other features.

[1099] A "route calculation means" is a device or software that calculates the optimal route to the user's destination and provides that information.

[1100] "Movement control means" refers to a device or software that controls the movement of a machine or robot based on calculated route information.

[1101] "Communication means" refers to the device or protocol used to transmit and receive data and instructions between the server and the terminal within the system.

[1102] "Communication means" refers to a device or software that has the function of conveying instructions received from the server to the user by voice or other means.

[1103] "Emotion recognition means" refers to a device or algorithm that analyzes the user's emotions from their tone of voice, facial expressions, etc., and recognizes their state.

[1104] A "feedback means" is a device or software that has the function of providing a response or advice in real time according to the user's emotional state.

[1105] A "learner" is a device or algorithm that has the ability to receive feedback provided by users and update the system's model based on that information.

[1106] A system for implementing this invention includes a voice recognition means, an image acquisition means, an image analysis means, a route calculation means, a movement control means, a communication means, an emotion recognition means, and a feedback means to assist visually impaired people in navigating and searching for products in a physical store.

[1107] The server communicates with a speech recognition system that converts the user's voice input into text data, allowing it to understand and process the user's instructions. The speech recognition system uses the SpeechRecognition library, and the gTTS library is used to generate speech from the text and provide feedback to the user.

[1108] The device is equipped with a camera that captures images of the surrounding environment in real time. The captured image data is analyzed using OpenCV to detect obstacles and steps. In addition, an emotion recognition algorithm called EmotionRecognizer analyzes the user's emotional state from their facial expressions and tone of voice. This allows the device to determine whether the user is feeling anxious and provide appropriate feedback.

[1109] The server calculates the optimal route based on the image analysis results and the user's instructions using a route calculation method. This uses the NavigationAlgorithm. The calculated route information and additional instructions according to the user's emotional state are sent to the device.

[1110] The device operates a mobility control system based on the calculated route. It scans the environment in real time and corrects the route if necessary. It also notifies the user of the next action via voice. If the emotion recognition means detects the user's anxiety, it plays relaxing music or provides encouraging feedback such as "It's okay."

[1111] For example, if a user speaks in a physical store and asks, "Where are the tomatoes on the shelf?", the system will recognize the command, calculate the optimal route, and tell the user, "Go straight, then turn right." When the user reaches the tomato shelf, the system will say, "This is it. The tomatoes are on the shelf on the right."

[1112] An example of a prompt is as follows:

[1113] User: "Where are the tomatoes on the shelf?"

[1114] Application: "Go straight, then turn right."

[1115] (Arrived in front of the tomato shelves)

[1116] "Here it is. The tomatoes are on the shelf on the right."

[1117] User: "Thanks."

[1118] In this way, this invention will help visually impaired people to confidently navigate brick-and-mortar stores and improve the quality of their daily lives.

[1119] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1120] Step 1:

[1121] Action: The user gives a voice command.

[1122] Input: Audio data.

[1123] Data processing: The device's voice recognition system converts voice into text data.

[1124] Output: User's spoken instructions converted into text data.

[1125] Step 2:

[1126] How it works: The device uses a 360-degree camera to capture images of its surroundings.

[1127] Input: Real-time image data of the surrounding environment.

[1128] Data Processing: Image data is captured using an image acquisition means.

[1129] Output: Real-time image data.

[1130] Step 3:

[1131] Operation: The device sends the acquired image data to the server for analysis.

[1132] Input: The captured image data.

[1133] Data processing: Image data is sent to the server.

[1134] Output: Image data for analysis by the server.

[1135] Step 4:

[1136] How it works: The server analyzes image data and identifies obstacles and steps in the surrounding area.

[1137] Input: Image data received by the server.

[1138] Data processing: Analysis algorithms are used to identify obstacles and steps.

[1139] Output: Information on identified obstacles and steps.

[1140] Step 5:

[1141] Operation: The server calculates the optimal route to the user's destination.

[1142] Input: Text data of user's voice instructions and image analysis results.

[1143] Data processing: Calculate the optimal route using a route calculation algorithm.

[1144] Output: Calculated optimal route information.

[1145] Step 6:

[1146] Operation: The server sends the calculated route information to the terminal.

[1147] Input: The calculated optimal route information.

[1148] Data processing: Send route information to the device.

[1149] Output: Route information transferred to the device.

[1150] Step 7:

[1151] Operation: The device starts moving based on the received route information.

[1152] Input: Route information received from the server.

[1153] Data processing: The movement control system controls movement based on the route.

[1154] Output: Move terminal.

[1155] Step 8:

[1156] How it works: While moving, the device continuously scans its surrounding environment in real time.

[1157] Input: Real-time image data of the surrounding environment.

[1158] Data processing: Analyzes information to continuously check safety and avoid obstacles.

[1159] Output: Maintaining a safe travel path.

[1160] Step 9:

[1161] Action: The device will notify the user of the next action by voice.

[1162] Input: Instructions from the server and information on the current movement route.

[1163] Data processing: Generate voice instructions using communication means.

[1164] Output: Voice movement instructions.

[1165] Step 10:

[1166] How it works: The device monitors the user's emotional state in real time and provides adaptive feedback.

[1167] Input: Real-time data on the user's facial expressions and tone of voice.

[1168] Data processing: Emotion recognition algorithms are used to analyze the emotional state and generate appropriate feedback.

[1169] Output: Audio feedback depending on the user's emotional state.

[1170] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1171] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1172] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1173] [Third embodiment]

[1174] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1175] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1176] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1177] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1178] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1179] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1180] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1181] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1182] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1183] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1184] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1185] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1186] MODE FOR CARRYING OUT THE INVENTION

[1187] The quadrupedal guide dog robot system of the present invention is designed to assist visually impaired people when they go out. This system includes voice recognition means, image acquisition means, image analysis means, route calculation means, movement control means, communication means, and communication means, and each means works in cooperation to guide the visually impaired people safely to their destination.

[1188] Overall system configuration

[1189] Voice recognition means

[1190] It recognizes instructions that users speak (e.g., "I want to go to the nearest supermarket") and converts the speech into text data that is used to analyze the user's intent.

[1191] Image Acquisition Method

[1192] The robot is equipped with a 360-degree camera that captures images of its surroundings in real time, and this image data is used as the basis for understanding the surrounding situation.

[1193] Image analysis methods

[1194] The acquired image data is sent to a server where it is analyzed. This analysis includes identifying obstacles, steps, traffic lights, etc. This information is necessary to calculate a safe guidance route.

[1195] Route calculation method

[1196] The server calculates the optimal route based on the image analysis results and the user's destination information. The calculation also takes into account information such as obstacles, construction areas, and puddles. The calculated route information is sent to the robot.

[1197] Movement Control Means

[1198] The robot begins moving autonomously based on the route information received from the server. As it walks, it dynamically corrects its route based on image data captured in real time.

[1199] communication means

[1200] The user's instructions recognized by the voice recognition means and the data obtained by the image analysis means are transmitted to the server, and communication is also performed to receive instructions from the server and recalculated route information.

[1201] Communication tools

[1202] The robot notifies the user of the instructions received from the server by voice (e.g., "Go straight," "Turn left," etc.). It also confirms the situation and receives further instructions through dialogue with the user.

[1203] Program processing overview

[1204] Voice input and recognition

[1205] 1. User: Says, "I want to go to the nearby supermarket."

[1206] 2. Terminal: The recognition device converts the voice into text data and recognizes the content "Go to the nearby supermarket."

[1207] Acquisition and analysis of the surrounding environment

[1208] 3. Terminal: Captures image data of the surroundings using a 360-degree camera and sends it to the server.

[1209] 4. Server: Analyzes the received image data and identifies obstacles and steps.

[1210] Route calculation and movement control

[1211] 5. Server: Calculates the optimal route based on the analysis results and the user's destination information, and sends that information to the terminal.

[1212] 6. Terminal: Start moving according to the received route, checking the surrounding environment in real time as you go.

[1213] Voice communication

[1214] 7. Terminal: Notifies the user of the current situation and next actions via voice.

[1215] 8. User: Move according to the robot's instructions.

[1216] Example: When a user goes to the supermarket

[1217] 1. User: Says, "I want to go to the nearest supermarket."

[1218] 2. Device: Recognizes voice and captures the surroundings with a 360-degree camera.

[1219] 3. Server: Analyzes the image data and calculates a safe route.

[1220] 4. Terminal: Starts movement based on the calculation results and notifies the user of the next action by voice.

[1221] 5. User: Follow the robot's instructions to safely reach the supermarket.

[1222] In this way, the entire system works together to safely and efficiently guide users to their destination. This system solves the problems faced by conventional guide dogs and provides an environment where visually impaired people can go out more freely.

[1223] The processing flow will be explained below.

[1224] Step 1:

[1225] User: Uses voice input. Says, "I want to go to the nearby supermarket."

[1226] Step 2:

[1227] Device: Captures the user's voice with a built-in microphone.

[1228] Step 3:

[1229] Terminal: The voice recognition system converts the captured voice into text data and interprets the content as "I want to go to the nearby supermarket."

[1230] Step 4:

[1231] Terminal: Analyzes the converted text data and recognizes the user's request.

[1232] Step 5:

[1233] Terminal: Sends the analysis results to the server.

[1234] Step 6:

[1235] Terminal: Captures the current surrounding environment with a 360-degree camera and sends the image data to the server.

[1236] Step 7:

[1237] Server: Analyzes the received image data and maps the current location and environment.

[1238] Step 8:

[1239] Server: Performs image analysis to identify the location of obstacles, steps, and traffic lights.

[1240] Step 9:

[1241] Server: Calculates multiple routes based on the user's destination "supermarket" and current location.

[1242] Step 10:

[1243] Server: From the calculated routes, select the optimal route that avoids obstacles, construction areas, puddles, etc.

[1244] Step 11:

[1245] Server: Sends the selected route information and walking instructions to the device.

[1246] Step 12:

[1247] Terminal: Based on the received route information, the mobility control system starts moving.

[1248] Step 13:

[1249] Terminal: A 360-degree camera scans the surrounding environment in real time to ensure safety.

[1250] Step 14:

[1251] Terminal: Sends captured image data to the server at any time, sharing the latest situation.

[1252] Step 15:

[1253] Terminal: Recognizes obstacles, steps, and traffic lights in real time and guides the user.

[1254] Step 16:

[1255] Device: Speaks to the user about their current progress and next steps, for example, "Go straight" or "Turn left."

[1256] Step 17:

[1257] User: Follow the robot's voice instructions and move.

[1258] Step 18:

[1259] Terminal: If the device encounters an unexpected obstacle or construction area along the way, it sends new image data to the server.

[1260] Step 19:

[1261] Server: Re-analyzes based on the new data and recalculates a new route.

[1262] Step 20:

[1263] Server: Sends the new optimized route information to the device.

[1264] Step 21:

[1265] Device: Resume travel based on new route information.

[1266] Step 22:

[1267] Terminal: When the user arrives at the destination, the terminal notifies the user that "You have arrived at the supermarket."

[1268] Step 23:

[1269] User: Confirms that they have reached the supermarket.

[1270] Step 24:

[1271] Device: Ask the user for feedback on their travel experience. Example: "Did you have any issues during your trip?"

[1272] Step 25:

[1273] User: Provides feedback verbally. Example: "No issues."

[1274] Step 26:

[1275] Terminal: Sends user feedback to the server.

[1276] Step 27:

[1277] Server: Stores the received feedback in a database.

[1278] Step 28:

[1279] Server: Updates the model parameters based on the feedback data to improve performance.

[1280] The above is a specific processing flow of the four-legged guide dog robot system for assisting visually impaired people when they go out.

[1281] Example 1

[1282] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1283] Conventional guide dogs and assistive devices have struggled to adequately support the safe and efficient movement of visually impaired people. In particular, they lacked the ability to grasp the surrounding environment in real time and dynamically correct routes, limiting the ability of visually impaired people to respond to sudden obstacles, unmanned roads, areas under construction, and other situations. Furthermore, they lacked the ability to utilize user feedback to improve the system's efficiency. This made it difficult to provide an environment in which visually impaired people could go out with peace of mind.

[1284] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1285] In this invention, the server includes voice recognition means for recognizing instructions input by voice from the user, image acquisition means for capturing images of the surrounding environment, image analysis means for analyzing the captured image data and identifying obstacles and steps, route calculation means for calculating the optimal route to the user's destination, movement control means for controlling movement based on the calculated route, communication means for transmitting the user's instructions recognized by the voice recognition means and data analyzed by the image analysis means to an information processing device and receiving instructions from the information processing device, communication means for notifying the user of the instructions received from the information processing device by voice, and means for dynamically correcting the route while checking the environment in real time, thereby providing an environment in which visually impaired people can go out with peace of mind.

[1286] The "voice recognition means" is a device that recognizes instructions input by voice from the user and converts them into text data.

[1287] "Image acquisition means" refers to a device that captures images of the surrounding environment and acquires the image data. This device includes a 360-degree camera.

[1288] The "image analysis means" is a device or program for analyzing the acquired image data and identifying obstacles and steps.

[1289] The "route calculation means" is a device or program that calculates the optimal route to the user's destination.

[1290] The "movement control means" is a device or program that controls the movement of the robot based on the calculated route.

[1291] The "communication means" is a device or program for transmitting user instructions recognized by the voice recognition means and data analyzed by the image analysis means to the information processing device, and for receiving instructions from the information processing device.

[1292] The "communication means" is a device or program that notifies the user of instructions received from the information processing device by voice.

[1293] The "means for dynamically correcting the route while checking the environment in real time" refers to a device or program that allows the robot to continuously check the surrounding environment while moving and correct the route as necessary.

[1294] MODE FOR CARRYING OUT THE INVENTION

[1295] The quadrupedal guide dog robot system of the present invention is designed to assist visually impaired people when they go out. This system includes voice recognition means, image acquisition means, image analysis means, route calculation means, movement control means, communication means, and communication means, and each means works in cooperation to guide the visually impaired people safely to their destination.

[1296] Overall system configuration

[1297] The main components of this system are:

[1298] Voice recognition means

[1299] It recognizes instructions that a user speaks and converts the speech into text data. Specifically, it uses a speech recognition system (e.g., Google Cloud Speech-to-Text service) to recognize the user's voice instructions. This text data is used to analyze the user's intent.

[1300] Image Acquisition Method

[1301] The robot is equipped with a 360-degree camera that captures images of its surroundings in real time, and this image data is used as the basis for understanding the surrounding situation.

[1302] Image analysis methods

[1303] The acquired image data is sent to a server where it is analyzed. This analysis includes identifying obstacles, steps, traffic lights, etc., using object detection algorithms such as YOLO (You Only Look Once). The analysis also generates information on the location of obstacles and safe route options.

[1304] Route calculation method

[1305] The server calculates the optimal route based on the image analysis results and the user's destination information. For example, it uses a path-finding algorithm such as Dijkstra's algorithm. This calculation also takes into account information such as obstacles, construction areas, and puddles. The calculated route information is sent to the robot.

[1306] Movement Control Means

[1307] The robot begins moving autonomously based on the route information received from the server. While moving, it continuously monitors the surrounding environment with a 360-degree camera and corrects its route in real time. Even if an obstacle suddenly appears, it automatically stops and resumes moving according to the re-planned route.

[1308] communication means

[1309] The user's instructions recognized by the voice recognition means and the data analyzed by the image analysis means are transmitted to the server, and communication is also performed to receive instructions and recalculated route information from the server.

[1310] Communication tools

[1311] The robot will then verbally communicate instructions received from the server to the user, such as "go straight" or "turn left," and will also include a learning mechanism to update the system based on user feedback.

[1312] Example: When a user goes to the supermarket

[1313] 1. User: Says "I want to go to the nearest supermarket." The voice recognition means converts this into text data.

[1314] 2. Device: Recognizes voice and captures the surroundings with a 360-degree camera.

[1315] 3. Server: Analyzes image data to identify obstacles and steps. Based on the analysis results, calculates a safe route.

[1316] 4. Terminal: Receives calculated route information and starts moving, monitoring the environment in real time as it moves forward, and notifies the user of the next action via voice.

[1317] 5. User: Follow the robot's instructions to safely reach the supermarket.

[1318] In this way, the entire system works together to safely and efficiently guide users to their destination. This system solves the problems faced by conventional guide dogs and provides an environment where visually impaired people can go out more freely.

[1319] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1320] Step 1:

[1321] User: Enters a voice command such as "I want to go to the nearest supermarket."

[1322] Input: User's voice commands

[1323] Output: Audio input data

[1324] Step 2:

[1325] On the device: A speech recognition system converts speech into text data, for example, using the Google Cloud Speech-to-Text service.

[1326] Input: Voice input data

[1327] Data processing: Converting voice data into text data

[1328] Output: "I want to go to the nearest supermarket."

[1329] How it works: The voice recognizer activates, records speech, and converts it into text.

[1330] Step 3:

[1331] Terminal: Sends the converted text data to the server.

[1332] Input: Text data "I want to go to the nearest supermarket"

[1333] Output: Data sent to the server

[1334] How it works: The communications module packets the text data and sends it over the Internet to a server.

[1335] Step 4:

[1336] Terminal: Captures image data of the surroundings using a 360-degree camera and sends it to the server.

[1337] Input: Surrounding environment

[1338] Output: Image data

[1339] What it does: The camera starts up, continuously captures 360-degree images, and sends the data to a server.

[1340] Step 5:

[1341] Server: Analyzes the received image data and identifies obstacles and steps. Uses object detection algorithms such as YOLO.

[1342] Input: Image data

[1343] Data calculation: Apply object detection algorithm to identify obstacles and steps

[1344] Output: Analysis results (obstacle location information, etc.)

[1345] Operation: Image analysis software processes image data and generates analytical results.

[1346] Step 6:

[1347] Server: Based on the user's destination information "nearby supermarket" and the image analysis results, the optimal route is calculated using Dijkstra's algorithm.

[1348] Input: Destination information, analysis results

[1349] Data calculation: Apply route calculation algorithm

[1350] Output: Optimal route information

[1351] Operation: The route calculation program on the server runs, calculates the optimal route, and generates data.

[1352] Step 7:

[1353] Server: Sends calculated route information to the device.

[1354] Input: Optimal route information

[1355] Output: Data sent to the terminal

[1356] Operation: The communication module packets the route information and sends it to the terminal via the Internet.

[1357] Step 8:

[1358] Terminal: The terminal begins moving autonomously based on the received route information. If an obstacle is detected along the way, the terminal corrects the route in real time.

[1359] Input: Route information, real-time images from a 360-degree camera

[1360] Data calculation: Route correction based on image data

[1361] Output: Movement control command

[1362] Operation: The movement control system is activated and controls the motors and actuators to move the robot.

[1363] Step 9:

[1364] Terminal: Notifies the user of the current situation and next actions via voice.

[1365] Input: Instructions from the server, route information

[1366] Output: Audio notification

[1367] How it works: The voice synthesis system converts instructions into voice and transmits them to the user through the speaker.

[1368] Step 10:

[1369] User: Move according to the robot's voice instructions.

[1370] Input: Voice commands

[1371] Output: User Movement

[1372] Action: The user moves according to the robot's instructions.

[1373] (Application example 1)

[1374] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1375] When visually impaired people go out, existing assistance systems have difficulty quickly recalculating the optimal route in response to changes in the surrounding environment. Furthermore, real-time instructions and notifications of the situation are insufficient, leaving users feeling uneasy. There is a need to provide a system that can solve these problems and enable visually impaired people to travel more safely and efficiently.

[1376] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1377] In this invention, the server includes voice recognition means for recognizing instructions input by voice from the user, image acquisition means for capturing images of the surrounding environment, image analysis means for analyzing the captured image data and identifying obstacles and steps, route calculation means for calculating an optimal route to the user's destination, movement control means for controlling movement based on the calculated route, communication means for transmitting the user's instructions recognized by the voice recognition means and data analyzed by the image analysis means to the server and receiving instructions from the server, communication means for notifying the user of the instructions received from the server by voice, and dynamic re-route calculation means for analyzing the environment and re-calculating the optimal route in real time based on the captured data, thereby enabling safe and efficient movement according to the environment.

[1378] The "voice recognition means" is a device that recognizes instructions input by voice by the user and converts them into text data.

[1379] The "image acquisition means" is a device that takes pictures of the surrounding environment with a camera and acquires image data.

[1380] The "image analysis means" is a device that analyzes the acquired image data and identifies environmental information such as obstacles and steps.

[1381] The "route calculation means" is a device that calculates the optimal route based on the user's destination information.

[1382] "Movement control means" refers to a device that controls the movement of the system based on the calculated route.

[1383] The "communication means" is a device that transmits user instructions recognized by the voice recognition means and data analyzed by the image analysis means to the server, and receives instructions from the server.

[1384] The "communication means" is a device that notifies the user of instructions received from the server by voice and communicates with the user through dialogue.

[1385] A "dynamic re-route calculation means" is a device that analyzes the environment in real time based on acquired data and recalculates the optimal route if necessary.

[1386] The system that realizes this application example is an automated driving guidance system for visually impaired people, and includes as its main components a voice recognition means, an image acquisition means, an image analysis means, a route calculation means, a movement control means, a communication means, a communication means, and a dynamic re-route calculation means.

[1387] Voice recognition means

[1388] The server uses speech recognition software (e.g., the speech_recognition library) to recognize instructions entered by the user through speech and convert them into text data. For example, if a user says, "I want to go to the nearest supermarket," this speech is converted into text and processed within the system.

[1389] Image Acquisition Method

[1390] The device uses a 360-degree camera (e.g., cv2.VideoCapture) to capture the surrounding environment and acquire image data, which is important for the system to understand the visual environment.

[1391] Image analysis methods

[1392] The server uses image analysis software (e.g., OpenCV library) to analyze the acquired image data and identify obstacles, steps, etc. The analyzed data is input into the route calculation means.

[1393] Route calculation method

[1394] The server uses a map service API (e.g., the Google Maps library) to calculate the optimal route based on the user's destination information. The calculated route information is sent to the device.

[1395] Movement Control Means

[1396] The device begins moving autonomously based on the route received from the server, checking environmental data in real time as it moves forward and dynamically correcting its route as necessary.

[1397] communication means

[1398] The server transmits user instructions recognized by the voice recognition means and data analyzed by the image analysis means to the server, and is equipped with communication functions (e.g., Internet connection, Wi-Fi) to receive instructions from the server at the terminal.

[1399] Communication tools

[1400] The device uses a text-to-speech engine (e.g., a TTS engine) to audibly announce to the user the instructions received from the server, for example, informing the user that "the next turn is to the right" on the way to a destination.

[1401] Dynamic reroute calculation method

[1402] The server analyzes the environment in real time based on the acquired data and recalculates the optimal route as needed, allowing the vehicle to respond immediately to situations such as unexpected obstacles or road construction.

[1403] Specific examples

[1404] For example, if a user says, "I want to go to the nearby supermarket," the voice recognition means converts this instruction into text data and sends it to the server. The server uses a map service API to calculate the optimal route and sends this information to the device. The device then takes pictures of the surrounding environment with a 360-degree camera and begins moving while identifying obstacles using image analysis means. As the environment changes, the route is dynamically recalculated and appropriate instructions are provided to the user using a text-to-speech engine.

[1405] Examples of prompts: "I want to go to the nearest supermarket" "Turn right next"

[1406] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1407] Step 1:

[1408] Voice input and recognition

[1409] Subject: User

[1410] How it works: The user says, "I want to go to the nearest supermarket."

[1411] Input: User's voice command

[1412] Data processing: Use speech recognition software (e.g., speech_recognition library) to convert voice instructions into text data.

[1413] Output: Text data "I want to go to a nearby supermarket"

[1414] Step 2:

[1415] Shooting the surrounding environment

[1416] Subject: Device

[1417] What it does: Captures the surrounding environment with a 360-degree camera.

[1418] Input: Surrounding environment information

[1419] Data processing: Use a 360-degree camera (e.g., cv2.VideoCapture) to acquire image data of the environment.

[1420] Output: Image data of the environment

[1421] Step 3:

[1422] Image data analysis

[1423] Subject: Server

[1424] Operation: Analyzes acquired image data and identifies obstacles and steps.

[1425] Input: Image data of the environment

[1426] Data processing: Use image analysis software (e.g., OpenCV library) to identify obstacles, steps, etc.

[1427] Output: Analysis data on obstacles and steps

[1428] Step 4:

[1429] Calculating the best route

[1430] Subject: Server

[1431] Operation: Calculates the optimal route based on the user's destination information.

[1432] Input: User instruction text data and image analysis data

[1433] Data calculation: Uses map service APIs (e.g., Google Maps library) to calculate the optimal route.

[1434] Output: Optimal route information

[1435] Step 5:

[1436] Movement Control

[1437] Subject: Device

[1438] Operation: The robot begins moving autonomously based on the calculated route information.

[1439] Input: Optimal route information

[1440] Data calculation: Using a movement control algorithm, control the movement according to the route.

[1441] Output: Actual movement

[1442] Step 6:

[1443] Real-time environment reanalysis and route recalculation

[1444] Subject: Server

[1445] How it works: Reanalyzes the environment in real time based on acquired data and recalculates the optimal route if necessary.

[1446] Input: Image data of the environment and current route information

[1447] Data calculation: Use dynamic re-route calculation method to recalculate the optimal route.

[1448] Output: Updated optimal route information

[1449] Step 7:

[1450] Communications and Notice of Instructions

[1451] Subject: Server and Terminal

[1452] Operation: New instructions from the server are sent to the terminal, which then notifies the user by voice.

[1453] Input: Updated route information

[1454] Data processing: Using communication functions (e.g., Internet connection, Wi-Fi), instructions from the server are sent to the device, which then uses a TTS engine to provide voice notifications to the user.

[1455] Output: Spoken instructions to the user (e.g., "Turn right next")

[1456] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1457] MODE FOR CARRYING OUT THE INVENTION

[1458] The quadrupedal guide dog robot system of the present invention is designed to assist visually impaired people when they go out, and is also combined with an emotion engine that recognizes the user's emotional state and responds accordingly. This system includes voice recognition means, image acquisition means, image analysis means, route calculation means, movement control means, communication means, and the emotion engine, and each of these means works in cooperation to guide visually impaired people to their destination safely and with a sense of psychological security.

[1459] Overall system configuration

[1460] Voice recognition means

[1461] It recognizes instructions that users speak (e.g., "I want to go to the nearest supermarket") and converts the speech into text data that is used to analyze the user's intent.

[1462] Image Acquisition Method

[1463] The robot is equipped with a 360-degree camera that captures images of its surroundings in real time, and this image data is used as the basis for understanding the surrounding situation.

[1464] Image analysis methods

[1465] The acquired image data is sent to a server where it is analyzed. This analysis includes identifying obstacles, steps, traffic lights, etc. This information is necessary to calculate a safe guidance route.

[1466] Route calculation method

[1467] The server calculates the optimal route based on the image analysis results and the user's destination information. The calculation also takes into account information such as obstacles, construction areas, and puddles. The calculated route information is sent to the robot.

[1468] Movement Control Means

[1469] The robot begins moving autonomously based on the route information received from the server. As it walks, it dynamically corrects its route based on image data captured in real time.

[1470] communication means

[1471] The user's instructions recognized by the voice recognition means and the data obtained by the image analysis means are transmitted to the server, and communication is also performed to receive instructions from the server and recalculated route information.

[1472] Communication tools

[1473] The robot notifies the user of the instructions received from the server by voice (e.g., "Go straight," "Turn left," etc.). It also confirms the situation and receives further instructions through dialogue with the user.

[1474] Emotion Engine

[1475] The robot is equipped with an emotion engine that analyzes the user's voice and facial expression data to recognize the user's emotional state. For example, if the user is feeling anxious, the system can play relaxing music to help the user relax or provide encouraging words such as "It's okay."

[1476] Program processing overview

[1477] Voice Input and Emotion Recognition

[1478] 1. User: Says, "I want to go to the nearby supermarket."

[1479] 2. Device: The built-in microphone and camera capture the user's voice and facial expressions.

[1480] 3. Terminal: The voice recognition system converts the captured voice into text data, and the emotion engine then analyzes the user's emotional state from their voice and facial expressions.

[1481] 4. Terminal: Analyzes the instructions from the text data and sends the emotional state recognized by the emotion engine to the server.

[1482] Environmental awareness and route planning

[1483] 5. Terminal: Captures the current surrounding environment with a 360-degree camera and sends the image data to the server.

[1484] 6. Server: Analyzes the received image data and maps the current location and environment.

[1485] 7. Server: Calculates the optimal route based on the user's instructions and emotional state. The calculation includes adjustments based on the emotional state (e.g., choosing a safe route to reduce the user's anxiety).

[1486] 8. Server: Sends optimal route information and additional instructions according to the emotional state to the device.

[1487] Movement and real-time response

[1488] 9. Terminal: The mobility control system starts moving based on the received route information.

[1489] 10. Terminal: Scans the surrounding environment in real time with a 360-degree camera to ensure safety.

[1490] 11. Terminal: Sends image data and emotional information to the server at any time.

[1491] 12. Terminal: Notifies the user of the ongoing situation and next actions via voice, providing feedback adapted by the emotion engine (e.g., "Go straight," "It's okay," etc.).

[1492] 13. Device: Monitors the user's emotional state and provides relaxing music or encouraging words as needed.

[1493] Example: When a user goes to the supermarket

[1494] 1. User: Says, "I want to go to the nearest supermarket."

[1495] 2. Device: Recognizes voice and analyzes the user's emotional state with an emotion engine. Captures the surroundings with a 360-degree camera.

[1496] 3. Server: Analyzes the image data, calculates a safe route, and instructs the user to play relaxing music to reduce anxiety.

[1497] 4. Device: The device starts moving based on the calculated route and guides the user by saying things like "Go straight" or "Turn left." If it detects an anxious expression, it notifies the user by saying "It's okay, don't worry."

[1498] 5. User: Follow the voice instructions and reach the supermarket with peace of mind.

[1499] In this way, the system of the present invention provides real-time feedback that takes into account the user's emotional state, thereby providing an environment in which visually impaired people can feel safe when going out.

[1500] The processing flow will be explained below.

[1501] Step 1:

[1502] User: Uses voice input. Says, "I want to go to the nearby supermarket."

[1503] Step 2:

[1504] Device: Captures the user's voice with a built-in microphone.

[1505] Step 3:

[1506] Terminal: The voice recognition system converts the captured voice into text data and analyzes the content, such as "I want to go to the nearby supermarket."

[1507] Step 4:

[1508] Device: A 360-degree camera captures the user's facial expressions, and an emotion engine analyzes the facial data to determine the user's emotional state, such as anxiety, relaxation, or joy.

[1509] Step 5:

[1510] Terminal: Sends the analysis results (voice text data and emotional information) to the server.

[1511] Step 6:

[1512] Terminal: Captures the current surrounding environment with a 360-degree camera and sends the image data to the server.

[1513] Step 7:

[1514] Server: Analyzes the received image data and maps the current location and environment.

[1515] Step 8:

[1516] Server: Identifies obstacles, steps, traffic lights, etc. from the results of image analysis.

[1517] Step 9:

[1518] Server: Calculates multiple optimal routes based on the user's destination information, current location information, and emotional state.

[1519] Step 10:

[1520] Server: Selects a safe route that avoids obstacles, construction areas, and puddles, and fine-tunes the route based on the user's emotional state (e.g., if the user is feeling anxious, select a wider, safer path).

[1521] Step 11:

[1522] Server: Sends the selected route information and travel instructions to the terminal.

[1523] Step 12:

[1524] Terminal: The mobility control system starts moving based on the received route information.

[1525] Step 13:

[1526] Terminal: While moving, a 360-degree camera scans the surrounding environment in real time to ensure safety.

[1527] Step 14:

[1528] Device: Captured image data and emotional information are sent to the server at any time, and the latest situation is shared.

[1529] Step 15:

[1530] Terminal: Recognizes obstacles, steps, and traffic lights in real time, and adjusts course to guide the user.

[1531] Step 16:

[1532] Device: Provides audible notification to the user about their current progress and next steps, e.g., "Go straight" or "Turn left."

[1533] Step 17:

[1534] Device: The emotion engine monitors the user's emotional state, and if it detects anxiety or tension, it provides voice feedback to help them relax, such as "It's okay" or "Don't worry."

[1535] Step 18:

[1536] User: Follow the robot's voice instructions and continue moving, with reassuring words from the robot if necessary.

[1537] Step 19:

[1538] Terminal: If the robot encounters an unexpected obstacle or an area under construction along the way, it sends new image data and emotion information to the server.

[1539] Step 20:

[1540] Server: Re-analyzes the new data and calculates a new route, taking into account the user's emotional state to determine the optimal route.

[1541] Step 21:

[1542] Server: Sends the recalculated new route information to the device.

[1543] Step 22:

[1544] Device: Resume travel based on new route information.

[1545] Step 23:

[1546] Terminal: When the user arrives at the destination, the device notifies the user by voice, saying, "You have arrived at the supermarket."

[1547] Step 24:

[1548] User: Confirms that destination has been reached.

[1549] Step 25:

[1550] Device: Ask the user for feedback on their travel experience. Example: "Did you have any issues during your trip?"

[1551] Step 26:

[1552] User: Provide verbal feedback, for example, "No issues" or "I was a little worried."

[1553] Step 27:

[1554] Terminal: Sends user feedback to the server.

[1555] Step 28:

[1556] Server: Stores the received feedback in a database.

[1557] Step 29:

[1558] Server: Updates the model parameters based on the feedback data and reflects them in the next performance improvement.

[1559] The above is the specific processing flow of the visually impaired support system that combines an emotion engine.

[1560] Example 2

[1561] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1562] To support visually impaired people when they go out, it is important to provide not only simple guidance but also adaptive feedback according to the user's emotional state. However, current systems have difficulty recognizing the user's emotional state in real time and providing appropriate support, which makes it difficult to alleviate anxiety and discomfort.

[1563] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1564] In this invention, the server includes a voice recognition means, an image acquisition means, an image analysis means, a route calculation means, a movement control means, a communication means, a communication means, and an emotion engine, which not only calculates and guides a route based on the voice instructions of a visually impaired person, but also analyzes the user's emotional state and provides adaptive feedback in real time.

[1565] A "voice recognition means" is a mechanism that captures voice input from a user and converts the voice into text data.

[1566] The "image acquisition means" is a mechanism that uses a camera mounted on the robot to capture images of the surrounding environment and collect image data.

[1567] The "image analysis means" is a mechanism that analyzes the acquired image data and identifies obstacles, steps, etc.

[1568] The "route calculation means" is a mechanism that calculates the optimal travel route based on the user's destination information and the image analysis results.

[1569] The "movement control means" is a mechanism that controls the movement of the robot based on the calculated route information.

[1570] The "communication means" is a mechanism for transmitting data obtained by the voice recognition means and image analysis means to the server and receiving instructions from the server.

[1571] The "communication means" is a mechanism that notifies the user of instructions received from the server by voice, and confirms the situation or receives further instructions through dialogue with the user.

[1572] The "emotion engine" is a mechanism that analyzes the user's voice and facial expression data to recognize the user's emotional state and provide feedback accordingly.

[1573] MODE FOR CARRYING OUT THE INVENTION

[1574] The quadrupedal guide dog robot system of the present invention is designed to enable visually impaired people to go out safely and is also capable of providing feedback according to the user's emotional state. This system includes a voice recognition unit, an image acquisition unit, an image analysis unit, a route calculation unit, a movement control unit, a communication unit, and an emotion engine. Specific hardware and software configurations for each unit are described below.

[1575] Hardware and Software Configuration

[1576] Voice recognition means

[1577] The device is equipped with a high-performance microphone and voice recognition software that uses generative AI models to analyze and convert users' voice commands into text data.

[1578] Image Acquisition Method

[1579] The device is equipped with a 360-degree camera that captures images of the surrounding environment in real time. This captured data is used by the image analysis method described below.

[1580] Image analysis methods

[1581] The acquired image data is sent to a server where it is analyzed using a high-performance GPU and specific algorithms to identify obstacles and steps.

[1582] Route calculation method

[1583] The server calculates the optimal route based on the image analysis results and the user's destination information. The route calculation also takes into account weather and construction area information. The route calculation algorithm is built on a generative AI model.

[1584] Movement Control Means

[1585] The calculated route information is sent to the terminal, which then controls the robot's movement based on this information. The movement control system dynamically corrects the route while acquiring environmental information in real time.

[1586] communication means

[1587] The data acquired by the voice recognition and image analysis means is transmitted bidirectionally to a server using a secure and fast protocol, such as Wi-Fi or a 5G network.

[1588] Communication tools

[1589] The instructions received from the server are notified to the user by the device through voice. For example, this includes guidance voice such as "Go straight." In addition, it is possible for the user to give instructions or confirm the situation through dialogue.

[1590] Emotion Engine

[1591] The device is equipped with an emotion engine that analyzes the user's voice and facial expression data to recognize the user's emotional state. This engine performs the analysis using a generative AI model, and if the user is feeling anxious, for example, it will play relaxing music or provide encouraging words such as "It's okay."

[1592] Specific examples

[1593] When a user voice-inputs "I want to go to the nearby supermarket," the following operation occurs.

[1594] 1. User: Instructs the device, "I want to go to a nearby supermarket."

[1595] 2. Device: Recognizes voice and analyzes the user's emotional state with an emotion engine. Captures the surroundings with a 360-degree camera.

[1596] 3. Server: Analyzes image data, calculates a safe route, and also recommends relaxing music to reduce the user's anxiety.

[1597] 4. Device: The device starts moving based on the calculated route and guides the user by saying things like "Go straight" or "Turn left." If it detects an anxious expression, it notifies the user by saying "It's okay, don't worry."

[1598] 5. User: Follow the voice instructions and reach the supermarket with peace of mind.

[1599] As described above, the system of the present invention provides real-time feedback and adaptive support to help visually impaired people reach their destination safely.

[1600] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1601] Program processing flow

[1602] Step 1:

[1603] The user speaks, "I want to go to the nearby supermarket."

[1604] Input: User's voice commands

[1605] Output: Audio data

[1606] Specific operation: The user speaks into the device's microphone.

[1607] Step 2:

[1608] The device captures voice with its built-in microphone, and the voice recognition system converts the voice into text data.

[1609] Input: Audio data

[1610] Output: Text data

[1611] How it works: Using a generative AI model, the speech recognition algorithm analyzes the speech and converts it into text.

[1612] Step 3:

[1613] The device captures the user's facial expressions with its built-in camera, and the emotion engine analyzes their emotional state.

[1614] Input: User's facial expression data

[1615] Output: Emotion analysis results

[1616] How it works: The camera captures the user's face, and the generative AI model analyzes the facial expression data to recognize their emotional state.

[1617] Step 4:

[1618] The device sends the converted text data and the emotion analysis results to the server.

[1619] Input: Text data, sentiment analysis results

[1620] Output: Data transmission

[1621] Specific operation: The communication module packages the data and uploads it to the server.

[1622] Step 5:

[1623] The device uses a 360-degree camera to capture images of the current surrounding environment in real time and transmits the image data to a server.

[1624] Input: Image data of the surrounding environment

[1625] Output: Image data transmission

[1626] Specific operation: The camera continuously captures images and the communication module transmits them.

[1627] Step 6:

[1628] The server analyzes the received image data and maps the environment, identifying obstacles, steps, traffic lights, etc.

[1629] Input: Image data

[1630] Output: Environment map, obstacle information

[1631] How it works: Using a high-performance GPU, image analysis algorithms process video data and identify important elements in the environment.

[1632] Step 7:

[1633] The server calculates the optimal route based on the user's intentions and emotional state, taking into account weather information and information about areas under construction.

[1634] Input: User's destination information, emotional state, environment map

[1635] Output: Optimal travel route

[1636] How it works: A route calculation algorithm integrates all the data to generate a safe and efficient route.

[1637] Step 8:

[1638] The server sends the calculated route information and additional instructions according to the emotional state to the terminal.

[1639] Input: Optimal travel route, additional instructions

[1640] Output: Data transmission

[1641] Specific operation: The server's communication module packages the route information and sends it to the terminal.

[1642] Step 9:

[1643] The terminal activates the mobility control system based on the route information received and begins moving autonomously.

[1644] Input: Travel route information

[1645] Output: Robot movement

[1646] Specific behavior: Motors and sensors operate, and the robot moves along the calculated route.

[1647] Step 10:

[1648] The device scans the surrounding environment in real time using a 360-degree camera to check for safety.

[1649] Input: Real-time image of the surrounding environment

[1650] Output: Safety confirmation information

[1651] How it works: Cameras continuously capture images and safety monitoring systems analyze the situation.

[1652] Step 11:

[1653] The image data and emotional information acquired by the terminal are transmitted to the server as needed.

[1654] Input: Real-time image data, emotional information

[1655] Output: Data transmission

[1656] Specific operation: The communication module periodically uploads and updates data.

[1657] Step 12:

[1658] The device notifies the user of the ongoing situation and next actions via voice, providing feedback based on the results analyzed by the emotion engine.

[1659] Input: Progress, next action instructions, emotion analysis results

[1660] Output: Audio instructions and feedback to the user

[1661] Specific operation: The speech synthesis system generates a notification message and transmits it to the user through the speaker.

[1662] Step 13:

[1663] The device continues to monitor the user's emotional state and provides relaxing music and encouraging words.

[1664] Input: Emotional state monitoring data

[1665] Output: Relaxing music, encouraging words

[1666] Specific operation: The emotion engine analyzes emotional data in real time and automatically selects an appropriate response.

[1667] (Application example 2)

[1668] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1669] There is a lack of technological means to assist visually impaired people in going out and navigating stores safely and securely. In particular, there is a lack of systems that can grasp the emotional state of visually impaired people and provide appropriate feedback accordingly. Therefore, it is necessary to reduce anxiety and stress while traveling and enable them to smoothly search for products in physical stores.

[1670] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice recognition unit that recognizes instructions input by voice from the user, an image acquisition unit that captures images of the surrounding environment, an image analysis unit that analyzes the acquired image data and identifies obstacles and steps, a route calculation unit that calculates the optimal route to the user's destination, a movement control unit that controls movement based on the calculated route, a communication unit that transmits the user's instructions recognized by the voice recognition unit and the data analyzed by the image analysis unit to the server and receives instructions from the server, a communication unit that notifies the user of the instructions received from the server by voice, an emotion recognition unit that recognizes the user's emotional state and adapts accordingly, and a feedback unit that provides real-time feedback according to the user's emotional state. This allows visually impaired people to travel with confidence and reach their destination safely. It also enables them to smoothly search for products in physical stores, reducing anxiety and stress.

[1671] "Speech recognition means" refers to a device or software that has the function of sensing voice input from a user and converting it into text data.

[1672] The "image acquisition means" is a device or means for capturing an image of the surrounding environment using a camera or the like and acquiring that information as digital image data.

[1673] "Image analysis means" refers to a device or algorithm that analyzes the captured image data and identifies obstacles, steps, and other features.

[1674] A "route calculation means" is a device or software that calculates the optimal route to the user's destination and provides that information.

[1675] "Movement control means" refers to a device or software that controls the movement of a machine or robot based on calculated route information.

[1676] "Communication means" refers to the device or protocol used to transmit and receive data and instructions between the server and the terminal within the system.

[1677] "Communication means" refers to a device or software that has the function of conveying instructions received from the server to the user by voice or other means.

[1678] "Emotion recognition means" refers to a device or algorithm that analyzes the user's emotions from their tone of voice, facial expressions, etc., and recognizes their state.

[1679] A "feedback means" is a device or software that has the function of providing a response or advice in real time according to the user's emotional state.

[1680] A "learner" is a device or algorithm that has the ability to receive feedback provided by users and update the system's model based on that information.

[1681] A system for implementing this invention includes a voice recognition means, an image acquisition means, an image analysis means, a route calculation means, a movement control means, a communication means, an emotion recognition means, and a feedback means to assist visually impaired people in navigating and searching for products in a physical store.

[1682] The server communicates with a speech recognition system that converts the user's voice input into text data, allowing it to understand and process the user's instructions. The speech recognition system uses the SpeechRecognition library, and the gTTS library is used to generate speech from the text and provide feedback to the user.

[1683] The device is equipped with a camera that captures images of the surrounding environment in real time. The captured image data is analyzed using OpenCV to detect obstacles and steps. In addition, an emotion recognition algorithm called EmotionRecognizer analyzes the user's emotional state from their facial expressions and tone of voice. This allows the device to determine whether the user is feeling anxious and provide appropriate feedback.

[1684] The server calculates the optimal route based on the image analysis results and the user's instructions using a route calculation method. This uses the NavigationAlgorithm. The calculated route information and additional instructions according to the user's emotional state are sent to the device.

[1685] The device operates a mobility control system based on the calculated route. It scans the environment in real time and corrects the route if necessary. It also notifies the user of the next action via voice. If the emotion recognition means detects the user's anxiety, it plays relaxing music or provides encouraging feedback such as "It's okay."

[1686] For example, if a user speaks in a physical store and asks, "Where are the tomatoes on the shelf?", the system will recognize the command, calculate the optimal route, and tell the user, "Go straight, then turn right." When the user reaches the tomato shelf, the system will say, "This is it. The tomatoes are on the shelf on the right."

[1687] An example of a prompt is as follows:

[1688] User: "Where are the tomatoes on the shelf?"

[1689] Application: "Go straight, then turn right."

[1690] (Arrived in front of the tomato shelves)

[1691] "Here it is. The tomatoes are on the shelf on the right."

[1692] User: "Thanks."

[1693] In this way, this invention will help visually impaired people to confidently navigate brick-and-mortar stores and improve the quality of their daily lives.

[1694] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1695] Step 1:

[1696] Action: The user gives a voice command.

[1697] Input: Audio data.

[1698] Data processing: The device's voice recognition system converts voice into text data.

[1699] Output: User's spoken instructions converted into text data.

[1700] Step 2:

[1701] How it works: The device uses a 360-degree camera to capture images of its surroundings.

[1702] Input: Real-time image data of the surrounding environment.

[1703] Data Processing: Image data is captured using an image acquisition means.

[1704] Output: Real-time image data.

[1705] Step 3:

[1706] Operation: The device sends the acquired image data to the server for analysis.

[1707] Input: The captured image data.

[1708] Data processing: Image data is sent to the server.

[1709] Output: Image data for analysis by the server.

[1710] Step 4:

[1711] How it works: The server analyzes image data and identifies obstacles and steps in the surrounding area.

[1712] Input: Image data received by the server.

[1713] Data processing: Analysis algorithms are used to identify obstacles and steps.

[1714] Output: Information on identified obstacles and steps.

[1715] Step 5:

[1716] Operation: The server calculates the optimal route to the user's destination.

[1717] Input: Text data of user's voice instructions and image analysis results.

[1718] Data processing: Calculate the optimal route using a route calculation algorithm.

[1719] Output: Calculated optimal route information.

[1720] Step 6:

[1721] Operation: The server sends the calculated route information to the terminal.

[1722] Input: The calculated optimal route information.

[1723] Data processing: Send route information to the device.

[1724] Output: Route information transferred to the device.

[1725] Step 7:

[1726] Operation: The device starts moving based on the received route information.

[1727] Input: Route information received from the server.

[1728] Data processing: The movement control system controls movement based on the route.

[1729] Output: Move terminal.

[1730] Step 8:

[1731] How it works: While moving, the device continuously scans its surrounding environment in real time.

[1732] Input: Real-time image data of the surrounding environment.

[1733] Data processing: Analyzes information to continuously check safety and avoid obstacles.

[1734] Output: Maintaining a safe travel path.

[1735] Step 9:

[1736] Action: The device will notify the user of the next action by voice.

[1737] Input: Instructions from the server and information on the current movement route.

[1738] Data processing: Generate voice instructions using communication means.

[1739] Output: Voice movement instructions.

[1740] Step 10:

[1741] How it works: The device monitors the user's emotional state in real time and provides adaptive feedback.

[1742] Input: Real-time data on the user's facial expressions and tone of voice.

[1743] Data processing: Emotion recognition algorithms are used to analyze the emotional state and generate appropriate feedback.

[1744] Output: Audio feedback depending on the user's emotional state.

[1745] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1746] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1747] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1748] [Fourth embodiment]

[1749] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1750] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1751] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1752] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1753] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1754] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1755] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1756] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1757] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1758] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1759] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1760] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1761] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1762] MODE FOR CARRYING OUT THE INVENTION

[1763] The quadrupedal guide dog robot system of the present invention is designed to assist visually impaired people when they go out. This system includes voice recognition means, image acquisition means, image analysis means, route calculation means, movement control means, communication means, and communication means, and each means works in cooperation to guide the visually impaired people safely to their destination.

[1764] Overall system configuration

[1765] Voice recognition means

[1766] It recognizes instructions that users speak (e.g., "I want to go to the nearest supermarket") and converts the speech into text data that is used to analyze the user's intent.

[1767] Image Acquisition Method

[1768] The robot is equipped with a 360-degree camera that captures images of its surroundings in real time, and this image data is used as the basis for understanding the surrounding situation.

[1769] Image analysis methods

[1770] The acquired image data is sent to a server where it is analyzed. This analysis includes identifying obstacles, steps, traffic lights, etc. This information is necessary to calculate a safe guidance route.

[1771] Route calculation method

[1772] The server calculates the optimal route based on the image analysis results and the user's destination information. The calculation also takes into account information such as obstacles, construction areas, and puddles. The calculated route information is sent to the robot.

[1773] Movement Control Means

[1774] The robot begins moving autonomously based on the route information received from the server. As it walks, it dynamically corrects its route based on image data captured in real time.

[1775] communication means

[1776] The user's instructions recognized by the voice recognition means and the data obtained by the image analysis means are transmitted to the server, and communication is also performed to receive instructions from the server and recalculated route information.

[1777] Communication tools

[1778] The robot notifies the user of the instructions received from the server by voice (e.g., "Go straight," "Turn left," etc.). It also confirms the situation and receives further instructions through dialogue with the user.

[1779] Program processing overview

[1780] Voice input and recognition

[1781] 1. User: Says, "I want to go to the nearby supermarket."

[1782] 2. Terminal: The recognition device converts the voice into text data and recognizes the content "Go to the nearby supermarket."

[1783] Acquisition and analysis of the surrounding environment

[1784] 3. Terminal: Captures image data of the surroundings using a 360-degree camera and sends it to the server.

[1785] 4. Server: Analyzes the received image data and identifies obstacles and steps.

[1786] Route calculation and movement control

[1787] 5. Server: Calculates the optimal route based on the analysis results and the user's destination information, and sends that information to the terminal.

[1788] 6. Terminal: Start moving according to the received route, checking the surrounding environment in real time as you go.

[1789] Voice communication

[1790] 7. Terminal: Notifies the user of the current situation and next actions via voice.

[1791] 8. User: Move according to the robot's instructions.

[1792] Example: When a user goes to the supermarket

[1793] 1. User: Says, "I want to go to the nearest supermarket."

[1794] 2. Device: Recognizes voice and captures the surroundings with a 360-degree camera.

[1795] 3. Server: Analyzes the image data and calculates a safe route.

[1796] 4. Terminal: Starts movement based on the calculation results and notifies the user of the next action by voice.

[1797] 5. User: Follow the robot's instructions to safely reach the supermarket.

[1798] In this way, the entire system works together to safely and efficiently guide users to their destination. This system solves the problems faced by conventional guide dogs and provides an environment where visually impaired people can go out more freely.

[1799] The processing flow will be explained below.

[1800] Step 1:

[1801] User: Uses voice input. Says, "I want to go to the nearby supermarket."

[1802] Step 2:

[1803] Device: Captures the user's voice with a built-in microphone.

[1804] Step 3:

[1805] Terminal: The voice recognition system converts the captured voice into text data and interprets the content as "I want to go to the nearby supermarket."

[1806] Step 4:

[1807] Terminal: Analyzes the converted text data and recognizes the user's request.

[1808] Step 5:

[1809] Terminal: Sends the analysis results to the server.

[1810] Step 6:

[1811] Terminal: Captures the current surrounding environment with a 360-degree camera and sends the image data to the server.

[1812] Step 7:

[1813] Server: Analyzes the received image data and maps the current location and environment.

[1814] Step 8:

[1815] Server: Performs image analysis to identify the location of obstacles, steps, and traffic lights.

[1816] Step 9:

[1817] Server: Calculates multiple routes based on the user's destination "supermarket" and current location.

[1818] Step 10:

[1819] Server: From the calculated routes, select the optimal route that avoids obstacles, construction areas, puddles, etc.

[1820] Step 11:

[1821] Server: Sends the selected route information and walking instructions to the device.

[1822] Step 12:

[1823] Terminal: Based on the received route information, the mobility control system starts moving.

[1824] Step 13:

[1825] Terminal: A 360-degree camera scans the surrounding environment in real time to ensure safety.

[1826] Step 14:

[1827] Terminal: Sends captured image data to the server at any time, sharing the latest situation.

[1828] Step 15:

[1829] Terminal: Recognizes obstacles, steps, and traffic lights in real time and guides the user.

[1830] Step 16:

[1831] Device: Speaks to the user about their current progress and next steps, for example, "Go straight" or "Turn left."

[1832] Step 17:

[1833] User: Follow the robot's voice instructions and move.

[1834] Step 18:

[1835] Terminal: If the device encounters an unexpected obstacle or construction area along the way, it sends new image data to the server.

[1836] Step 19:

[1837] Server: Re-analyzes based on the new data and recalculates a new route.

[1838] Step 20:

[1839] Server: Sends the new optimized route information to the device.

[1840] Step 21:

[1841] Device: Resume travel based on new route information.

[1842] Step 22:

[1843] Terminal: When the user arrives at the destination, the terminal notifies the user that "You have arrived at the supermarket."

[1844] Step 23:

[1845] User: Confirms that they have reached the supermarket.

[1846] Step 24:

[1847] Device: Ask the user for feedback on their travel experience. Example: "Did you have any issues during your trip?"

[1848] Step 25:

[1849] User: Provides feedback verbally. Example: "No issues."

[1850] Step 26:

[1851] Terminal: Sends user feedback to the server.

[1852] Step 27:

[1853] Server: Stores the received feedback in a database.

[1854] Step 28:

[1855] Server: Updates the model parameters based on the feedback data to improve performance.

[1856] The above is a specific processing flow of the four-legged guide dog robot system for assisting visually impaired people when they go out.

[1857] Example 1

[1858] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1859] Conventional guide dogs and assistive devices have struggled to adequately support the safe and efficient movement of visually impaired people. In particular, they lacked the ability to grasp the surrounding environment in real time and dynamically correct routes, limiting the ability of visually impaired people to respond to sudden obstacles, unmanned roads, areas under construction, and other situations. Furthermore, they lacked the ability to utilize user feedback to improve the system's efficiency. This made it difficult to provide an environment in which visually impaired people could go out with peace of mind.

[1860] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1861] In this invention, the server includes voice recognition means for recognizing instructions input by voice from the user, image acquisition means for capturing images of the surrounding environment, image analysis means for analyzing the captured image data and identifying obstacles and steps, route calculation means for calculating the optimal route to the user's destination, movement control means for controlling movement based on the calculated route, communication means for transmitting the user's instructions recognized by the voice recognition means and data analyzed by the image analysis means to an information processing device and receiving instructions from the information processing device, communication means for notifying the user of the instructions received from the information processing device by voice, and means for dynamically correcting the route while checking the environment in real time, thereby providing an environment in which visually impaired people can go out with peace of mind.

[1862] The "voice recognition means" is a device that recognizes instructions input by voice from the user and converts them into text data.

[1863] "Image acquisition means" refers to a device that captures images of the surrounding environment and acquires the image data. This device includes a 360-degree camera.

[1864] The "image analysis means" is a device or program for analyzing the acquired image data and identifying obstacles and steps.

[1865] The "route calculation means" is a device or program that calculates the optimal route to the user's destination.

[1866] The "movement control means" is a device or program that controls the movement of the robot based on the calculated route.

[1867] The "communication means" is a device or program for transmitting user instructions recognized by the voice recognition means and data analyzed by the image analysis means to the information processing device, and for receiving instructions from the information processing device.

[1868] The "communication means" is a device or program that notifies the user of instructions received from the information processing device by voice.

[1869] The "means for dynamically correcting the route while checking the environment in real time" refers to a device or program that allows the robot to continuously check the surrounding environment while moving and correct the route as necessary.

[1870] MODE FOR CARRYING OUT THE INVENTION

[1871] The quadrupedal guide dog robot system of the present invention is designed to assist visually impaired people when they go out. This system includes voice recognition means, image acquisition means, image analysis means, route calculation means, movement control means, communication means, and communication means, and each means works in cooperation to guide the visually impaired people safely to their destination.

[1872] Overall system configuration

[1873] The main components of this system are:

[1874] Voice recognition means

[1875] It recognizes instructions that a user speaks and converts the speech into text data. Specifically, it uses a speech recognition system (e.g., Google Cloud Speech-to-Text service) to recognize the user's voice instructions. This text data is used to analyze the user's intent.

[1876] Image Acquisition Method

[1877] The robot is equipped with a 360-degree camera that captures images of its surroundings in real time, and this image data is used as the basis for understanding the surrounding situation.

[1878] Image analysis methods

[1879] The acquired image data is sent to a server where it is analyzed. This analysis includes identifying obstacles, steps, traffic lights, etc., using object detection algorithms such as YOLO (You Only Look Once). The analysis also generates information on the location of obstacles and safe route options.

[1880] Route calculation method

[1881] The server calculates the optimal route based on the image analysis results and the user's destination information. For example, it uses a path-finding algorithm such as Dijkstra's algorithm. This calculation also takes into account information such as obstacles, construction areas, and puddles. The calculated route information is sent to the robot.

[1882] Movement Control Means

[1883] The robot begins moving autonomously based on the route information received from the server. While moving, it continuously monitors the surrounding environment with a 360-degree camera and corrects its route in real time. Even if an obstacle suddenly appears, it automatically stops and resumes moving according to the re-planned route.

[1884] communication means

[1885] The user's instructions recognized by the voice recognition means and the data analyzed by the image analysis means are transmitted to the server, and communication is also performed to receive instructions and recalculated route information from the server.

[1886] Communication tools

[1887] The robot will then verbally communicate instructions received from the server to the user, such as "go straight" or "turn left," and will also include a learning mechanism to update the system based on user feedback.

[1888] Example: When a user goes to the supermarket

[1889] 1. User: Says "I want to go to the nearest supermarket." The voice recognition means converts this into text data.

[1890] 2. Device: Recognizes voice and captures the surroundings with a 360-degree camera.

[1891] 3. Server: Analyzes image data to identify obstacles and steps. Based on the analysis results, calculates a safe route.

[1892] 4. Terminal: Receives calculated route information and starts moving, monitoring the environment in real time as it moves forward, and notifies the user of the next action via voice.

[1893] 5. User: Follow the robot's instructions to safely reach the supermarket.

[1894] In this way, the entire system works together to safely and efficiently guide users to their destination. This system solves the problems faced by conventional guide dogs and provides an environment where visually impaired people can go out more freely.

[1895] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1896] Step 1:

[1897] User: Enters a voice command such as "I want to go to the nearest supermarket."

[1898] Input: User's voice commands

[1899] Output: Audio input data

[1900] Step 2:

[1901] On the device: A speech recognition system converts speech into text data, for example, using the Google Cloud Speech-to-Text service.

[1902] Input: Voice input data

[1903] Data processing: Converting voice data into text data

[1904] Output: "I want to go to the nearest supermarket."

[1905] How it works: The voice recognizer activates, records speech, and converts it into text.

[1906] Step 3:

[1907] Terminal: Sends the converted text data to the server.

[1908] Input: Text data "I want to go to the nearest supermarket"

[1909] Output: Data sent to the server

[1910] How it works: The communications module packets the text data and sends it over the Internet to a server.

[1911] Step 4:

[1912] Terminal: Captures image data of the surroundings using a 360-degree camera and sends it to the server.

[1913] Input: Surrounding environment

[1914] Output: Image data

[1915] What it does: The camera starts up, continuously captures 360-degree images, and sends the data to a server.

[1916] Step 5:

[1917] Server: Analyzes the received image data and identifies obstacles and steps. Uses object detection algorithms such as YOLO.

[1918] Input: Image data

[1919] Data calculation: Apply object detection algorithm to identify obstacles and steps

[1920] Output: Analysis results (obstacle location information, etc.)

[1921] Operation: Image analysis software processes image data and generates analytical results.

[1922] Step 6:

[1923] Server: Based on the user's destination information "nearby supermarket" and the image analysis results, the optimal route is calculated using Dijkstra's algorithm.

[1924] Input: Destination information, analysis results

[1925] Data calculation: Apply route calculation algorithm

[1926] Output: Optimal route information

[1927] Operation: The route calculation program on the server runs, calculates the optimal route, and generates data.

[1928] Step 7:

[1929] Server: Sends calculated route information to the device.

[1930] Input: Optimal route information

[1931] Output: Data sent to the terminal

[1932] Operation: The communication module packets the route information and sends it to the terminal via the Internet.

[1933] Step 8:

[1934] Terminal: The terminal begins moving autonomously based on the received route information. If an obstacle is detected along the way, the terminal corrects the route in real time.

[1935] Input: Route information, real-time images from a 360-degree camera

[1936] Data calculation: Route correction based on image data

[1937] Output: Movement control command

[1938] Operation: The movement control system is activated and controls the motors and actuators to move the robot.

[1939] Step 9:

[1940] Terminal: Notifies the user of the current situation and next actions via voice.

[1941] Input: Instructions from the server, route information

[1942] Output: Audio notification

[1943] How it works: The voice synthesis system converts instructions into voice and transmits them to the user through the speaker.

[1944] Step 10:

[1945] User: Move according to the robot's voice instructions.

[1946] Input: Voice commands

[1947] Output: User Movement

[1948] Action: The user moves according to the robot's instructions.

[1949] (Application example 1)

[1950] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1951] When visually impaired people go out, existing assistance systems have difficulty quickly recalculating the optimal route in response to changes in the surrounding environment. Furthermore, real-time instructions and notifications of the situation are insufficient, leaving users feeling uneasy. There is a need to provide a system that can solve these problems and enable visually impaired people to travel more safely and efficiently.

[1952] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1953] In this invention, the server includes voice recognition means for recognizing instructions input by voice from the user, image acquisition means for capturing images of the surrounding environment, image analysis means for analyzing the captured image data and identifying obstacles and steps, route calculation means for calculating an optimal route to the user's destination, movement control means for controlling movement based on the calculated route, communication means for transmitting the user's instructions recognized by the voice recognition means and data analyzed by the image analysis means to the server and receiving instructions from the server, communication means for notifying the user of the instructions received from the server by voice, and dynamic re-route calculation means for analyzing the environment and re-calculating the optimal route in real time based on the captured data, thereby enabling safe and efficient movement according to the environment.

[1954] The "voice recognition means" is a device that recognizes instructions input by voice by the user and converts them into text data.

[1955] The "image acquisition means" is a device that takes pictures of the surrounding environment with a camera and acquires image data.

[1956] The "image analysis means" is a device that analyzes the acquired image data and identifies environmental information such as obstacles and steps.

[1957] The "route calculation means" is a device that calculates the optimal route based on the user's destination information.

[1958] "Movement control means" refers to a device that controls the movement of the system based on the calculated route.

[1959] The "communication means" is a device that transmits user instructions recognized by the voice recognition means and data analyzed by the image analysis means to the server, and receives instructions from the server.

[1960] The "communication means" is a device that notifies the user of instructions received from the server by voice and communicates with the user through dialogue.

[1961] A "dynamic re-route calculation means" is a device that analyzes the environment in real time based on acquired data and recalculates the optimal route if necessary.

[1962] The system that realizes this application example is an automated driving guidance system for visually impaired people, and includes as its main components a voice recognition means, an image acquisition means, an image analysis means, a route calculation means, a movement control means, a communication means, a communication means, and a dynamic re-route calculation means.

[1963] Voice recognition means

[1964] The server uses speech recognition software (e.g., the speech_recognition library) to recognize instructions entered by the user through speech and convert them into text data. For example, if a user says, "I want to go to the nearest supermarket," this speech is converted into text and processed within the system.

[1965] Image Acquisition Method

[1966] The device uses a 360-degree camera (e.g., cv2.VideoCapture) to capture the surrounding environment and acquire image data, which is important for the system to understand the visual environment.

[1967] Image analysis methods

[1968] The server uses image analysis software (e.g., OpenCV library) to analyze the acquired image data and identify obstacles, steps, etc. The analyzed data is input into the route calculation means.

[1969] Route calculation method

[1970] The server uses a map service API (e.g., the Google Maps library) to calculate the optimal route based on the user's destination information. The calculated route information is sent to the device.

[1971] Movement Control Means

[1972] The device begins moving autonomously based on the route received from the server, checking environmental data in real time as it moves forward and dynamically correcting its route as necessary.

[1973] communication means

[1974] The server transmits user instructions recognized by the voice recognition means and data analyzed by the image analysis means to the server, and is equipped with communication functions (e.g., Internet connection, Wi-Fi) to receive instructions from the server at the terminal.

[1975] Communication tools

[1976] The device uses a text-to-speech engine (e.g., a TTS engine) to audibly announce to the user the instructions received from the server, for example, informing the user that "the next turn is to the right" on the way to a destination.

[1977] Dynamic reroute calculation method

[1978] The server analyzes the environment in real time based on the acquired data and recalculates the optimal route as needed, allowing the vehicle to respond immediately to situations such as unexpected obstacles or road construction.

[1979] Specific examples

[1980] For example, if a user says, "I want to go to the nearby supermarket," the voice recognition means converts this instruction into text data and sends it to the server. The server uses a map service API to calculate the optimal route and sends this information to the device. The device then takes pictures of the surrounding environment with a 360-degree camera and begins moving while identifying obstacles using image analysis means. As the environment changes, the route is dynamically recalculated and appropriate instructions are provided to the user using a text-to-speech engine.

[1981] Examples of prompts: "I want to go to the nearest supermarket" "Turn right next"

[1982] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1983] Step 1:

[1984] Voice input and recognition

[1985] Subject: User

[1986] How it works: The user says, "I want to go to the nearest supermarket."

[1987] Input: User's voice command

[1988] Data processing: Use speech recognition software (e.g., speech_recognition library) to convert voice instructions into text data.

[1989] Output: Text data "I want to go to a nearby supermarket"

[1990] Step 2:

[1991] Shooting the surrounding environment

[1992] Subject: Device

[1993] What it does: Captures the surrounding environment with a 360-degree camera.

[1994] Input: Surrounding environment information

[1995] Data processing: Use a 360-degree camera (e.g., cv2.VideoCapture) to acquire image data of the environment.

[1996] Output: Image data of the environment

[1997] Step 3:

[1998] Image data analysis

[1999] Subject: Server

[2000] Operation: Analyzes acquired image data and identifies obstacles and steps.

[2001] Input: Image data of the environment

[2002] Data processing: Use image analysis software (e.g., OpenCV library) to identify obstacles, steps, etc.

[2003] Output: Analysis data on obstacles and steps

[2004] Step 4:

[2005] Calculating the best route

[2006] Subject: Server

[2007] Operation: Calculates the optimal route based on the user's destination information.

[2008] Input: User instruction text data and image analysis data

[2009] Data calculation: Uses map service APIs (e.g., Google Maps library) to calculate the optimal route.

[2010] Output: Optimal route information

[2011] Step 5:

[2012] Movement Control

[2013] Subject: Device

[2014] Operation: The robot begins moving autonomously based on the calculated route information.

[2015] Input: Optimal route information

[2016] Data calculation: Using a movement control algorithm, control the movement according to the route.

[2017] Output: Actual movement

[2018] Step 6:

[2019] Real-time environment reanalysis and route recalculation

[2020] Subject: Server

[2021] How it works: Reanalyzes the environment in real time based on acquired data and recalculates the optimal route if necessary.

[2022] Input: Image data of the environment and current route information

[2023] Data calculation: Use dynamic re-route calculation method to recalculate the optimal route.

[2024] Output: Updated optimal route information

[2025] Step 7:

[2026] Communications and Notice of Instructions

[2027] Subject: Server and Terminal

[2028] Operation: New instructions from the server are sent to the terminal, which then notifies the user by voice.

[2029] Input: Updated route information

[2030] Data processing: Using communication functions (e.g., Internet connection, Wi-Fi), instructions from the server are sent to the device, which then uses a TTS engine to provide voice notifications to the user.

[2031] Output: Spoken instructions to the user (e.g., "Turn right next")

[2032] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2033] MODE FOR CARRYING OUT THE INVENTION

[2034] The quadrupedal guide dog robot system of the present invention is designed to assist visually impaired people when they go out, and is also combined with an emotion engine that recognizes the user's emotional state and responds accordingly. This system includes voice recognition means, image acquisition means, image analysis means, route calculation means, movement control means, communication means, and the emotion engine, and each of these means works in cooperation to guide visually impaired people to their destination safely and with a sense of psychological security.

[2035] Overall system configuration

[2036] Voice recognition means

[2037] It recognizes instructions that users speak (e.g., "I want to go to the nearest supermarket") and converts the speech into text data that is used to analyze the user's intent.

[2038] Image Acquisition Method

[2039] The robot is equipped with a 360-degree camera that captures images of its surroundings in real time, and this image data is used as the basis for understanding the surrounding situation.

[2040] Image analysis methods

[2041] The acquired image data is sent to a server where it is analyzed. This analysis includes identifying obstacles, steps, traffic lights, etc. This information is necessary to calculate a safe guidance route.

[2042] Route calculation method

[2043] The server calculates the optimal route based on the image analysis results and the user's destination information. The calculation also takes into account information such as obstacles, construction areas, and puddles. The calculated route information is sent to the robot.

[2044] Movement Control Means

[2045] The robot begins moving autonomously based on the route information received from the server. As it walks, it dynamically corrects its route based on image data captured in real time.

[2046] communication means

[2047] The user's instructions recognized by the voice recognition means and the data obtained by the image analysis means are transmitted to the server, and communication is also performed to receive instructions from the server and recalculated route information.

[2048] Communication tools

[2049] The robot notifies the user of the instructions received from the server by voice (e.g., "Go straight," "Turn left," etc.). It also confirms the situation and receives further instructions through dialogue with the user.

[2050] Emotion Engine

[2051] The robot is equipped with an emotion engine that analyzes the user's voice and facial expression data to recognize the user's emotional state. For example, if the user is feeling anxious, the system can play relaxing music to help the user relax or provide encouraging words such as "It's okay."

[2052] Program processing overview

[2053] Voice Input and Emotion Recognition

[2054] 1. User: Says, "I want to go to the nearby supermarket."

[2055] 2. Device: The built-in microphone and camera capture the user's voice and facial expressions.

[2056] 3. Terminal: The voice recognition system converts the captured voice into text data, and the emotion engine then analyzes the user's emotional state from their voice and facial expressions.

[2057] 4. Terminal: Analyzes the instructions from the text data and sends the emotional state recognized by the emotion engine to the server.

[2058] Environmental awareness and route planning

[2059] 5. Terminal: Captures the current surrounding environment with a 360-degree camera and sends the image data to the server.

[2060] 6. Server: Analyzes the received image data and maps the current location and environment.

[2061] 7. Server: Calculates the optimal route based on the user's instructions and emotional state. The calculation includes adjustments based on the emotional state (e.g., choosing a safe route to reduce the user's anxiety).

[2062] 8. Server: Sends optimal route information and additional instructions according to the emotional state to the device.

[2063] Movement and real-time response

[2064] 9. Terminal: The mobility control system starts moving based on the received route information.

[2065] 10. Terminal: Scans the surrounding environment in real time with a 360-degree camera to ensure safety.

[2066] 11. Terminal: Sends image data and emotional information to the server at any time.

[2067] 12. Terminal: Notifies the user of the ongoing situation and next actions via voice, providing feedback adapted by the emotion engine (e.g., "Go straight," "It's okay," etc.).

[2068] 13. Device: Monitors the user's emotional state and provides relaxing music or encouraging words as needed.

[2069] Example: When a user goes to the supermarket

[2070] 1. User: Says, "I want to go to the nearest supermarket."

[2071] 2. Device: Recognizes voice and analyzes the user's emotional state with an emotion engine. Captures the surroundings with a 360-degree camera.

[2072] 3. Server: Analyzes the image data, calculates a safe route, and instructs the user to play relaxing music to reduce anxiety.

[2073] 4. Device: The device starts moving based on the calculated route and guides the user by saying things like "Go straight" or "Turn left." If it detects an anxious expression, it notifies the user by saying "It's okay, don't worry."

[2074] 5. User: Follow the voice instructions and reach the supermarket with peace of mind.

[2075] In this way, the system of the present invention provides real-time feedback that takes into account the user's emotional state, thereby providing an environment in which visually impaired people can feel safe when going out.

[2076] The processing flow will be explained below.

[2077] Step 1:

[2078] User: Uses voice input. Says, "I want to go to the nearby supermarket."

[2079] Step 2:

[2080] Device: Captures the user's voice with a built-in microphone.

[2081] Step 3:

[2082] Terminal: The voice recognition system converts the captured voice into text data and analyzes the content, such as "I want to go to the nearby supermarket."

[2083] Step 4:

[2084] Device: A 360-degree camera captures the user's facial expressions, and an emotion engine analyzes the facial data to determine the user's emotional state, such as anxiety, relaxation, or joy.

[2085] Step 5:

[2086] Terminal: Sends the analysis results (voice text data and emotional information) to the server.

[2087] Step 6:

[2088] Terminal: Captures the current surrounding environment with a 360-degree camera and sends the image data to the server.

[2089] Step 7:

[2090] Server: Analyzes the received image data and maps the current location and environment.

[2091] Step 8:

[2092] Server: Identifies obstacles, steps, traffic lights, etc. from the results of image analysis.

[2093] Step 9:

[2094] Server: Calculates multiple optimal routes based on the user's destination information, current location information, and emotional state.

[2095] Step 10:

[2096] Server: Selects a safe route that avoids obstacles, construction areas, and puddles, and fine-tunes the route based on the user's emotional state (e.g., if the user is feeling anxious, select a wider, safer path).

[2097] Step 11:

[2098] Server: Sends the selected route information and travel instructions to the terminal.

[2099] Step 12:

[2100] Terminal: The mobility control system starts moving based on the received route information.

[2101] Step 13:

[2102] Terminal: While moving, a 360-degree camera scans the surrounding environment in real time to ensure safety.

[2103] Step 14:

[2104] Device: Captured image data and emotional information are sent to the server at any time, and the latest situation is shared.

[2105] Step 15:

[2106] Terminal: Recognizes obstacles, steps, and traffic lights in real time, and adjusts course to guide the user.

[2107] Step 16:

[2108] Device: Provides audible notification to the user about their current progress and next steps, e.g., "Go straight" or "Turn left."

[2109] Step 17:

[2110] Device: The emotion engine monitors the user's emotional state, and if it detects anxiety or tension, it provides voice feedback to help them relax, such as "It's okay" or "Don't worry."

[2111] Step 18:

[2112] User: Follow the robot's voice instructions and continue moving, with reassuring words from the robot if necessary.

[2113] Step 19:

[2114] Terminal: If the robot encounters an unexpected obstacle or an area under construction along the way, it sends new image data and emotion information to the server.

[2115] Step 20:

[2116] Server: Re-analyzes the new data and calculates a new route, taking into account the user's emotional state to determine the optimal route.

[2117] Step 21:

[2118] Server: Sends the recalculated new route information to the device.

[2119] Step 22:

[2120] Device: Resume travel based on new route information.

[2121] Step 23:

[2122] Terminal: When the user arrives at the destination, the device notifies the user by voice, saying, "You have arrived at the supermarket."

[2123] Step 24:

[2124] User: Confirms that destination has been reached.

[2125] Step 25:

[2126] Device: Ask the user for feedback on their travel experience. Example: "Did you have any issues during your trip?"

[2127] Step 26:

[2128] User: Provide verbal feedback, for example, "No issues" or "I was a little worried."

[2129] Step 27:

[2130] Terminal: Sends user feedback to the server.

[2131] Step 28:

[2132] Server: Stores the received feedback in a database.

[2133] Step 29:

[2134] Server: Updates the model parameters based on the feedback data and reflects them in the next performance improvement.

[2135] The above is the specific processing flow of the visually impaired support system that combines an emotion engine.

[2136] Example 2

[2137] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2138] To support visually impaired people when they go out, it is important to provide not only simple guidance but also adaptive feedback according to the user's emotional state. However, current systems have difficulty recognizing the user's emotional state in real time and providing appropriate support, which makes it difficult to alleviate anxiety and discomfort.

[2139] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2140] In this invention, the server includes a voice recognition means, an image acquisition means, an image analysis means, a route calculation means, a movement control means, a communication means, a communication means, and an emotion engine, which not only calculates and guides a route based on the voice instructions of a visually impaired person, but also analyzes the user's emotional state and provides adaptive feedback in real time.

[2141] A "voice recognition means" is a mechanism that captures voice input from a user and converts the voice into text data.

[2142] The "image acquisition means" is a mechanism that uses a camera mounted on the robot to capture images of the surrounding environment and collect image data.

[2143] The "image analysis means" is a mechanism that analyzes the acquired image data and identifies obstacles, steps, etc.

[2144] The "route calculation means" is a mechanism that calculates the optimal travel route based on the user's destination information and the image analysis results.

[2145] The "movement control means" is a mechanism that controls the movement of the robot based on the calculated route information.

[2146] The "communication means" is a mechanism for transmitting data obtained by the voice recognition means and image analysis means to the server and receiving instructions from the server.

[2147] The "communication means" is a mechanism that notifies the user of instructions received from the server by voice, and confirms the situation or receives further instructions through dialogue with the user.

[2148] The "emotion engine" is a mechanism that analyzes the user's voice and facial expression data to recognize the user's emotional state and provide feedback accordingly.

[2149] MODE FOR CARRYING OUT THE INVENTION

[2150] The quadrupedal guide dog robot system of the present invention is designed to enable visually impaired people to go out safely and is also capable of providing feedback according to the user's emotional state. This system includes a voice recognition unit, an image acquisition unit, an image analysis unit, a route calculation unit, a movement control unit, a communication unit, and an emotion engine. Specific hardware and software configurations for each unit are described below.

[2151] Hardware and Software Configuration

[2152] Voice recognition means

[2153] The device is equipped with a high-performance microphone and voice recognition software that uses generative AI models to analyze and convert users' voice commands into text data.

[2154] Image Acquisition Method

[2155] The device is equipped with a 360-degree camera that captures images of the surrounding environment in real time. This captured data is used by the image analysis method described below.

[2156] Image analysis methods

[2157] The acquired image data is sent to a server where it is analyzed using a high-performance GPU and specific algorithms to identify obstacles and steps.

[2158] Route calculation method

[2159] The server calculates the optimal route based on the image analysis results and the user's destination information. The route calculation also takes into account weather and construction area information. The route calculation algorithm is built on a generative AI model.

[2160] Movement Control Means

[2161] The calculated route information is sent to the terminal, which then controls the robot's movement based on this information. The movement control system dynamically corrects the route while acquiring environmental information in real time.

[2162] communication means

[2163] The data acquired by the voice recognition and image analysis means is transmitted bidirectionally to a server using a secure and fast protocol, such as Wi-Fi or a 5G network.

[2164] Communication tools

[2165] The instructions received from the server are notified to the user by the device through voice. For example, this includes guidance voice such as "Go straight." In addition, it is possible for the user to give instructions or confirm the situation through dialogue.

[2166] Emotion Engine

[2167] The device is equipped with an emotion engine that analyzes the user's voice and facial expression data to recognize the user's emotional state. This engine performs the analysis using a generative AI model, and if the user is feeling anxious, for example, it will play relaxing music or provide encouraging words such as "It's okay."

[2168] Specific examples

[2169] When a user voice-inputs "I want to go to the nearby supermarket," the following operation occurs.

[2170] 1. User: Instructs the device, "I want to go to a nearby supermarket."

[2171] 2. Device: Recognizes voice and analyzes the user's emotional state with an emotion engine. Captures the surroundings with a 360-degree camera.

[2172] 3. Server: Analyzes image data, calculates a safe route, and also recommends relaxing music to reduce the user's anxiety.

[2173] 4. Device: The device starts moving based on the calculated route and guides the user by saying things like "Go straight" or "Turn left." If it detects an anxious expression, it notifies the user by saying "It's okay, don't worry."

[2174] 5. User: Follow the voice instructions and reach the supermarket with peace of mind.

[2175] As described above, the system of the present invention provides real-time feedback and adaptive support to help visually impaired people reach their destination safely.

[2176] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2177] Program processing flow

[2178] Step 1:

[2179] The user speaks, "I want to go to the nearby supermarket."

[2180] Input: User's voice commands

[2181] Output: Audio data

[2182] Specific operation: The user speaks into the device's microphone.

[2183] Step 2:

[2184] The device captures voice with its built-in microphone, and the voice recognition system converts the voice into text data.

[2185] Input: Audio data

[2186] Output: Text data

[2187] How it works: Using a generative AI model, the speech recognition algorithm analyzes the speech and converts it into text.

[2188] Step 3:

[2189] The device captures the user's facial expressions with its built-in camera, and the emotion engine analyzes their emotional state.

[2190] Input: User's facial expression data

[2191] Output: Emotion analysis results

[2192] How it works: The camera captures the user's face, and the generative AI model analyzes the facial expression data to recognize their emotional state.

[2193] Step 4:

[2194] The device sends the converted text data and the emotion analysis results to the server.

[2195] Input: Text data, sentiment analysis results

[2196] Output: Data transmission

[2197] Specific operation: The communication module packages the data and uploads it to the server.

[2198] Step 5:

[2199] The device uses a 360-degree camera to capture images of the current surrounding environment in real time and transmits the image data to a server.

[2200] Input: Image data of the surrounding environment

[2201] Output: Image data transmission

[2202] Specific operation: The camera continuously captures images and the communication module transmits them.

[2203] Step 6:

[2204] The server analyzes the received image data and maps the environment, identifying obstacles, steps, traffic lights, etc.

[2205] Input: Image data

[2206] Output: Environment map, obstacle information

[2207] How it works: Using a high-performance GPU, image analysis algorithms process video data and identify important elements in the environment.

[2208] Step 7:

[2209] The server calculates the optimal route based on the user's intentions and emotional state, taking into account weather information and information about areas under construction.

[2210] Input: User's destination information, emotional state, environment map

[2211] Output: Optimal travel route

[2212] How it works: A route calculation algorithm integrates all the data to generate a safe and efficient route.

[2213] Step 8:

[2214] The server sends the calculated route information and additional instructions according to the emotional state to the terminal.

[2215] Input: Optimal travel route, additional instructions

[2216] Output: Data transmission

[2217] Specific operation: The server's communication module packages the route information and sends it to the terminal.

[2218] Step 9:

[2219] The terminal activates the mobility control system based on the route information received and begins moving autonomously.

[2220] Input: Travel route information

[2221] Output: Robot movement

[2222] Specific behavior: Motors and sensors operate, and the robot moves along the calculated route.

[2223] Step 10:

[2224] The device scans the surrounding environment in real time using a 360-degree camera to check for safety.

[2225] Input: Real-time image of the surrounding environment

[2226] Output: Safety confirmation information

[2227] How it works: Cameras continuously capture images and safety monitoring systems analyze the situation.

[2228] Step 11:

[2229] The image data and emotional information acquired by the terminal are transmitted to the server as needed.

[2230] Input: Real-time image data, emotional information

[2231] Output: Data transmission

[2232] Specific operation: The communication module periodically uploads and updates data.

[2233] Step 12:

[2234] The device notifies the user of the ongoing situation and next actions via voice, providing feedback based on the results analyzed by the emotion engine.

[2235] Input: Progress, next action instructions, emotion analysis results

[2236] Output: Audio instructions and feedback to the user

[2237] Specific operation: The speech synthesis system generates a notification message and transmits it to the user through the speaker.

[2238] Step 13:

[2239] The device continues to monitor the user's emotional state and provides relaxing music and encouraging words.

[2240] Input: Emotional state monitoring data

[2241] Output: Relaxing music, encouraging words

[2242] Specific operation: The emotion engine analyzes emotional data in real time and automatically selects an appropriate response.

[2243] (Application example 2)

[2244] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2245] There is a lack of technological means to assist visually impaired people in going out and navigating stores safely and securely. In particular, there is a lack of systems that can grasp the emotional state of visually impaired people and provide appropriate feedback accordingly. Therefore, it is necessary to reduce anxiety and stress while traveling and enable them to smoothly search for products in physical stores.

[2246] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice recognition unit that recognizes instructions input by voice from the user, an image acquisition unit that captures images of the surrounding environment, an image analysis unit that analyzes the acquired image data and identifies obstacles and steps, a route calculation unit that calculates the optimal route to the user's destination, a movement control unit that controls movement based on the calculated route, a communication unit that transmits the user's instructions recognized by the voice recognition unit and the data analyzed by the image analysis unit to the server and receives instructions from the server, a communication unit that notifies the user of the instructions received from the server by voice, an emotion recognition unit that recognizes the user's emotional state and adapts accordingly, and a feedback unit that provides real-time feedback according to the user's emotional state. This allows visually impaired people to travel with confidence and reach their destination safely. It also enables them to smoothly search for products in physical stores, reducing anxiety and stress.

[2247] "Speech recognition means" refers to a device or software that has the function of sensing voice input from a user and converting it into text data.

[2248] The "image acquisition means" is a device or means for capturing an image of the surrounding environment using a camera or the like and acquiring that information as digital image data.

[2249] "Image analysis means" refers to a device or algorithm that analyzes the captured image data and identifies obstacles, steps, and other features.

[2250] A "route calculation means" is a device or software that calculates the optimal route to the user's destination and provides that information.

[2251] "Movement control means" refers to a device or software that controls the movement of a machine or robot based on calculated route information.

[2252] "Communication means" refers to the device or protocol used to transmit and receive data and instructions between the server and the terminal within the system.

[2253] "Communication means" refers to a device or software that has the function of conveying instructions received from the server to the user by voice or other means.

[2254] "Emotion recognition means" refers to a device or algorithm that analyzes the user's emotions from their tone of voice, facial expressions, etc., and recognizes their state.

[2255] A "feedback means" is a device or software that has the function of providing a response or advice in real time according to the user's emotional state.

[2256] A "learner" is a device or algorithm that has the ability to receive feedback provided by users and update the system's model based on that information.

[2257] A system for implementing this invention includes a voice recognition means, an image acquisition means, an image analysis means, a route calculation means, a movement control means, a communication means, an emotion recognition means, and a feedback means to assist visually impaired people in navigating and searching for products in a physical store.

[2258] The server communicates with a speech recognition system that converts the user's voice input into text data, allowing it to understand and process the user's instructions. The speech recognition system uses the SpeechRecognition library, and the gTTS library is used to generate speech from the text and provide feedback to the user.

[2259] The device is equipped with a camera that captures images of the surrounding environment in real time. The captured image data is analyzed using OpenCV to detect obstacles and steps. In addition, an emotion recognition algorithm called EmotionRecognizer analyzes the user's emotional state from their facial expressions and tone of voice. This allows the device to determine whether the user is feeling anxious and provide appropriate feedback.

[2260] The server calculates the optimal route based on the image analysis results and the user's instructions using a route calculation method. This uses the NavigationAlgorithm. The calculated route information and additional instructions according to the user's emotional state are sent to the device.

[2261] The device operates a mobility control system based on the calculated route. It scans the environment in real time and corrects the route if necessary. It also notifies the user of the next action via voice. If the emotion recognition means detects the user's anxiety, it plays relaxing music or provides encouraging feedback such as "It's okay."

[2262] For example, if a user speaks in a physical store and asks, "Where are the tomatoes on the shelf?", the system will recognize the command, calculate the optimal route, and tell the user, "Go straight, then turn right." When the user reaches the tomato shelf, the system will say, "This is it. The tomatoes are on the shelf on the right."

[2263] An example of a prompt is as follows:

[2264] User: "Where are the tomatoes on the shelf?"

[2265] Application: "Go straight, then turn right."

[2266] (Arrived in front of the tomato shelves)

[2267] "Here it is. The tomatoes are on the shelf on the right."

[2268] User: "Thanks."

[2269] In this way, this invention will help visually impaired people to confidently navigate brick-and-mortar stores and improve the quality of their daily lives.

[2270] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2271] Step 1:

[2272] Action: The user gives a voice command.

[2273] Input: Audio data.

[2274] Data processing: The device's voice recognition system converts voice into text data.

[2275] Output: User's spoken instructions converted into text data.

[2276] Step 2:

[2277] How it works: The device uses a 360-degree camera to capture images of its surroundings.

[2278] Input: Real-time image data of the surrounding environment.

[2279] Data Processing: Image data is captured using an image acquisition means.

[2280] Output: Real-time image data.

[2281] Step 3:

[2282] Operation: The device sends the acquired image data to the server for analysis.

[2283] Input: The captured image data.

[2284] Data processing: Image data is sent to the server.

[2285] Output: Image data for analysis by the server.

[2286] Step 4:

[2287] How it works: The server analyzes image data and identifies obstacles and steps in the surrounding area.

[2288] Input: Image data received by the server.

[2289] Data processing: Analysis algorithms are used to identify obstacles and steps.

[2290] Output: Information on identified obstacles and steps.

[2291] Step 5:

[2292] Operation: The server calculates the optimal route to the user's destination.

[2293] Input: Text data of user's voice instructions and image analysis results.

[2294] Data processing: Calculate the optimal route using a route calculation algorithm.

[2295] Output: Calculated optimal route information.

[2296] Step 6:

[2297] Operation: The server sends the calculated route information to the terminal.

[2298] Input: The calculated optimal route information.

[2299] Data processing: Send route information to the device.

[2300] Output: Route information transferred to the device.

[2301] Step 7:

[2302] Operation: The device starts moving based on the received route information.

[2303] Input: Route information received from the server.

[2304] Data processing: The movement control system controls movement based on the route.

[2305] Output: Move terminal.

[2306] Step 8:

[2307] How it works: While moving, the device continuously scans its surrounding environment in real time.

[2308] Input: Real-time image data of the surrounding environment.

[2309] Data processing: Analyzes information to continuously check safety and avoid obstacles.

[2310] Output: Maintaining a safe travel path.

[2311] Step 9:

[2312] Action: The device will notify the user of the next action by voice.

[2313] Input: Instructions from the server and information on the current movement route.

[2314] Data processing: Generate voice instructions using communication means.

[2315] Output: Voice movement instructions.

[2316] Step 10:

[2317] How it works: The device monitors the user's emotional state in real time and provides adaptive feedback.

[2318] Input: Real-time data on the user's facial expressions and tone of voice.

[2319] Data processing: Emotion recognition algorithms are used to analyze the emotional state and generate appropriate feedback.

[2320] Output: Audio feedback depending on the user's emotional state.

[2321] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2322] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2323] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2324] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2325] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2326] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2327] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2328] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2329] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2330] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2331] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2332] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2333] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2334] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2335] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2336] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2337] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2338] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2339] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2340] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2341] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2342] The following is further disclosed regarding the above embodiment.

[2343] (Claim 1)

[2344] a voice recognition means for recognizing instructions input by voice from a user;

[2345] image acquisition means for capturing an image of the surrounding environment;

[2346] image analysis means for analyzing the acquired image data and identifying obstacles and steps;

[2347] a route calculation means for calculating an optimal route to a user's destination;

[2348] a movement control means for controlling movement based on the calculated route;

[2349] a communication means for transmitting the user's instruction recognized by the voice recognition means and the data analyzed by the image analysis means to a server and receiving instructions from the server;

[2350] a communication means for notifying the user by voice of the instruction received from the server;

[2351] A system including:

[2352] (Claim 2)

[2353] 10. The system of claim 1, further comprising: learning means for receiving feedback provided by a user and updating the model based on the feedback.

[2354] (Claim 3)

[2355] The system according to claim 1, wherein the acquired image data and route information are transmitted to a server, and if changes are required along the way, a new route is recalculated in real time and instructions are received.

[2356] "Example 1"

[2357] (Claim 1)

[2358] a voice recognition means for recognizing instructions input by voice from a user;

[2359] image acquisition means for capturing an image of the surrounding environment;

[2360] image analysis means for analyzing the acquired image data and identifying obstacles and steps;

[2361] a route calculation means for calculating an optimal route to a user's destination;

[2362] a movement control means for controlling movement based on the calculated route;

[2363] a communication means for transmitting the user's instruction recognized by the voice recognition means and the data analyzed by the image analysis means to an information processing device, and for receiving an instruction from the information processing device;

[2364] a communication means for notifying a user of an instruction received from the information processing device by voice;

[2365] A means to dynamically correct the route while checking the environment in real time,

[2366] A system including:

[2367] (Claim 2)

[2368] 10. The system of claim 1, further comprising: learning means for receiving feedback provided by a user and updating the model based on the feedback.

[2369] (Claim 3)

[2370] The system according to claim 1, wherein the acquired image data and route information are transmitted to an information processing device, and if a change is required along the way, a new route is recalculated in real time and an instruction is received.

[2371] "Application Example 1"

[2372] (Claim 1)

[2373] a voice recognition means for recognizing instructions input by voice from a user;

[2374] image acquisition means for capturing an image of the surrounding environment;

[2375] image analysis means for analyzing the acquired image data and identifying obstacles and steps;

[2376] a route calculation means for calculating an optimal route to a user's destination;

[2377] a movement control means for controlling movement based on the calculated route;

[2378] a communication means for transmitting the user's instruction recognized by the voice recognition means and the data analyzed by the image analysis means to a server and receiving instructions from the server;

[2379] a communication means for notifying the user by voice of the instruction received from the server;

[2380] a dynamic re-route calculation means for analyzing the environment and re-calculating the optimal route in real time based on the acquired data;

[2381] A system including:

[2382] (Claim 2)

[2383] 10. The system of claim 1, further comprising: learning means for receiving feedback provided by a user and updating the model based on the feedback.

[2384] (Claim 3)

[2385] The system according to claim 1, further comprising a dynamic re-route calculation means for transmitting the acquired image data and route information to a server, and for recalculating a new route in real time if a change is required along the way, and for operating in response to instructions.

[2386] "Example 2: Combining Emotion Engines"

[2387] (Claim 1)

[2388] a voice recognition means for recognizing instructions input by voice from a user;

[2389] image acquisition means for capturing an image of the surrounding environment;

[2390] image analysis means for analyzing the acquired image data and identifying obstacles and steps;

[2391] a route calculation means for calculating an optimal route to a user's destination;

[2392] a movement control means for controlling movement based on the calculated route;

[2393] a communication means for transmitting the user's instruction recognized by the voice recognition means and the data analyzed by the image analysis means to a server and receiving instructions from the server;

[2394] a communication means for notifying the user by voice of the instruction received from the server;

[2395] an emotion engine that analyzes the user's emotional state and provides adapted feedback;

[2396] A system including:

[2397] (Claim 2)

[2398] 10. The system of claim 1, further comprising: learning means for receiving feedback provided by a user and updating the model based on the feedback.

[2399] (Claim 3)

[2400] The system according to claim 1, wherein the acquired image data and route information are transmitted to a server, and if changes are required along the way, a new route is recalculated in real time and instructions are received.

[2401] "Application example 2 when combining emotion engines"

[2402] (Claim 1)

[2403] a voice recognition means for recognizing instructions input by voice from a user;

[2404] image acquisition means for capturing an image of the surrounding environment;

[2405] image analysis means for analyzing the acquired image data and identifying obstacles and steps;

[2406] a route calculation means for calculating an optimal route to a user's destination;

[2407] a movement control means for controlling movement based on the calculated route;

[2408] a communication means for transmitting the user's instruction recognized by the voice recognition means and the data analyzed by the image analysis means to a server and receiving instructions from the server;

[2409] a communication means for notifying the user by voice of the instruction received from the server;

[2410] emotion recognition means for recognizing the emotional state of a user and adapting accordingly;

[2411] a feedback means for providing real-time feedback according to the user's emotional state;

[2412] A system including:

[2413] (Claim 2)

[2414] 10. The system of claim 1, further comprising: learning means for receiving feedback provided by a user and updating the model based on the feedback.

[2415] (Claim 3)

[2416] The system according to claim 1, wherein the acquired image data and route information are transmitted to a server, and if changes are required along the way, a new route is recalculated in real time and instructions are received. [Explanation of symbols]

[2417] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a voice recognition means for recognizing instructions input by voice from a user; image acquisition means for capturing an image of the surrounding environment; image analysis means for analyzing the acquired image data and identifying obstacles and steps; a route calculation means for calculating an optimal route to a user's destination; a movement control means for controlling movement based on the calculated route; a communication means for transmitting the user's instruction recognized by the voice recognition means and the data analyzed by the image analysis means to a server and receiving instructions from the server; a communication means for notifying the user by voice of the instruction received from the server; A system including:

2. 10. The system of claim 1, further comprising: learning means for receiving feedback provided by a user and updating the model based on the feedback.

3. 2. The system according to claim 1, wherein the acquired image data and route information are transmitted to a server, and if a change is required along the way, a new route is recalculated in real time and instructions are received.

Images