Information processing system

Abstract

The invention provides an information processing system. The information processing system comprises a voice recognition device used for recognizing a user voice input instruction; the image acquisition device is used for shooting the surrounding environment; the image analysis device is used for analyzing the acquired image data and identifying obstacles and steps; the route calculation device is used for calculating an optimal route from the user to the destination; the movement control device is used for performing movement control according to the calculated route; the communication device is used for sending the user instruction recognized by the voice recognition device and the data analyzed by the image analysis device to a server and receiving the instruction from the server; and the communication device is used for notifying a user of the received instruction from the server in a voice manner.

Description

Technical Field

[0001] The technology disclosed herein relates to an information processing system. Background Technology

[0002] Japanese Patent Application Publication No. 2022-180282 discloses a method for controlling a role-based chatbot executed by at least one processor. The method includes the following steps: receiving a user's speech; adding the user's speech to a prompt word, the prompt word containing instruction statements associated with an explanation of the chatbot's role; encoding the prompt word; and inputting the encoded prompt word into a language model to generate a chatbot response to the user's speech.

[0003] Existing guide dogs suffer from numerous problems, including high maintenance costs, long training cycles, limited environmental adaptability, and difficulty coping with complex and dynamic environments. These issues cause visually impaired individuals to face many inconveniences and safety risks when going out in daily life. In addition, traditional assistive systems struggle to provide real-time feedback and intelligent path adjustments, making it difficult to respond promptly to sudden environmental changes and limiting the independent movement and freedom of life for visually impaired individuals. Summary of the Invention

[0004] This invention provides a system comprising a voice recognition device, an image acquisition device, an image analysis device, a route calculation device, a motion control device, a communication device, and an interaction device. Through voice recognition, the system can accurately understand user needs; through 360-degree imaging and image analysis, it can identify environmental obstacles, steps, and traffic signals in real time; based on server-side path calculation and dynamic route updates, it achieves intelligent obstacle avoidance and efficient guidance in complex environments. Simultaneously, the system has the ability to receive user feedback and automatically learn and optimize its model, continuously improving guidance accuracy and safety, thereby providing visually impaired individuals with safe, intelligent, and highly adaptable assistance services for outings.

[0005] A “speech recognition device” refers to a device or module that can receive and recognize user voice input and convert voice information into text data.

[0006] "Image acquisition device" refers to a device or system that can capture and collect image data of the surrounding environment.

[0007] An "image analysis device" refers to a hardware or software unit that can analyze acquired image data and identify environmental elements such as obstacles and steps.

[0008] A "route calculation device" refers to a processing module or system that automatically calculates the optimal route based on the user's current location and destination information, combined with environmental factors.

[0009] "Motion control device" refers to a unit used to automatically control the main body of the system to move or adjust its direction of travel based on the calculated route information.

[0010] "Communication device" refers to a device or module that enables data transmission and reception between different parts of a system and between the system and a server.

[0011] "Interactive device" refers to a device or module used to notify users of system status, movement instructions, and other information by means of voice or other means, and to enable two-way communication between users and the system.

[0012] A "server" refers to a central processing unit or computer device used to receive, store, analyze data and compute data, and provide instructions and feedback to the system.

[0013] "User" refers to a visually impaired person or related operator who uses this system, inputs their needs via voice, and moves under guidance.

[0014] "Feedback" refers to the opinions or information provided by users regarding system functions, the guidance process, or the user experience during use.

[0015] "Learning device" refers to a functional module or device that automatically adjusts and optimizes system algorithms, model parameters, etc. based on received user feedback. Attached Figure Description

[0016] Figure 1 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the first embodiment.

[0017] Figure 2 This is a conceptual diagram illustrating an example of the main functions of the data processing apparatus and smart device according to the first embodiment.

[0018] Figure 3 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the second embodiment.

[0019] Figure 4 This is a conceptual diagram illustrating an example of the main functions of the data processing device and smart glasses according to the second embodiment.

[0020] Figure 5 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the third embodiment.

[0021] Figure 6 This is a conceptual diagram illustrating an example of the main functions of the data processing apparatus and head-mounted terminal according to the third embodiment.

[0022] Figure 7 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the fourth embodiment.

[0023] Figure 8 This is a conceptual diagram illustrating an example of the main functions of the data processing device and robot according to the fourth embodiment.

[0024] Figure 9 This represents an emotion map that maps multiple emotions.

[0025] Figure 10 This represents an emotion map that maps multiple emotions.

[0026] Figure 11 This is a sequence diagram illustrating the processing flow of the data processing system of the first embodiment.

[0027] Figure 12 This is a sequence diagram illustrating the processing flow of the data processing system in Application Example 1.

[0028] Figure 13 This is a sequence diagram illustrating the processing flow of the data processing system of the second embodiment.

[0029] Figure 14 This is a sequence diagram illustrating the processing flow of the data processing system in Application Example 2. Detailed Implementation

[0030] Hereinafter, an example of an implementation of the system according to the present disclosure will be described with reference to the accompanying drawings.

[0031] First, let me explain the terminology used in the following instructions.

[0032] In the following embodiments, the processor (hereinafter referred to as "processor") with reference numerals may be a single computing device or a combination of multiple computing devices. Furthermore, the processor may be a single computing device or a combination of multiple computing devices. Examples of computing devices include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), etc.

[0033] In the following embodiments, RAM (Random Access Memory), as indicated in the figures, is a memory that temporarily stores information and is used as working memory by the processor.

[0034] In the following embodiments, the memory, as indicated by the reference numerals, is one or more non-volatile storage devices that store various programs and parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), disks (e.g., hard disks), or magnetic tapes.

[0035] In the following embodiments, the communication I / F (Interface) with reference numerals is an interface that includes a communication processor and an antenna, etc. The communication I / F is responsible for communication between multiple computers. As an example of a communication specification applicable to the communication I / F, wireless communication specifications such as 5G (5th Generation Mobile Communication System), Wi-Fi (wireless fidelity) (registered trademark), or Bluetooth (registered trademark) can be listed.

[0036] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. Furthermore, in this specification, when "and / or" connects to express more than three items, the same interpretation as "A and / or B" applies.

[0037] First Implementation Method

[0038] Figure 1 An example of the configuration of the data processing system 10 according to the first embodiment is shown.

[0039] like Figure 1 As shown, the data processing system 10 includes a data processing device 12 and an intelligent device 14. A server can be cited as an example of the data processing device 12.

[0040] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0041] The smart device 14 includes a computer 36, a receiving device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. In addition, the receiving device 38, output device 40, camera 42, and communication I / F 44 are also connected to the bus 52.

[0042] The receiving device 38 includes a touchscreen 38A and a microphone 38B, and receives user input. The touchscreen 38A receives user input via touch by detecting contact with an indicator (e.g., a pen or finger). The microphone 38B receives user input via sound by detecting the user's voice. The control unit 46A in the processor 46 sends data representing the user input received by the touchscreen 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data representing the user input.

[0043] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting data in a form perceptible to the user 20 (e.g., sound and / or text). The display 40A displays visual information such as text and images according to instructions from the processor 46. The speaker 40B outputs sound according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0044] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for sending and receiving various information between processor 46 and processor 28 via network 54.

[0045] Figure 2 The diagram shows an example of the main functions of the data processing device 12 and the smart device 14.

[0046] like Figure 2 As shown, in the data processing apparatus 12, specific processing is performed by the processor 28. A specific processing program 56 is stored in the memory 32. The specific processing program 56 is an example of a "program" as understood in this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0047] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. In the emotion inference function (emotion-specific function) using the emotion-specific model 59, various inferences and predictions related to the user's emotions are performed, including inferences and predictions of the user's emotions, but this is not limited to this example. Furthermore, emotion inference and prediction may also include, for example, emotion analysis (parsing).

[0048] In the smart device 14, the processor 46 performs the acceptance output processing. The memory 50 stores the acceptance output program 60. The acceptance output program 60 is used in conjunction with the data processing system 10 and the specific processing program 56. The processor 46 reads the acceptance output program 60 from the memory 50 and executes the read acceptance output program 60 on the RAM 48. The acceptance output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance output program 60 executed on the RAM 48. Furthermore, the smart device 14 has the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and these models can also be used to perform the same processing as the specific processing unit 290. The acceptance output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance output program 60 executed on the RAM 48.

[0049] Alternatively, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains the processing results (prediction results, etc.) using the data generation model 58 by communicating with the server device that has the data generation model 58. Furthermore, the data processing device 12 may be a server device or a user-held terminal device (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of the processing of the data processing system 10 of the first embodiment will be described.

[0050] Example 1

[0051] The flow of a specific process in Example 1 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart device 14. Furthermore, the data processing device 12 is referred to as the "server," and the smart device 14 is referred to as the "terminal."

[0052] Existing mobility assistance devices, such as guide dogs or traditional assistive tools, struggle to perceive dynamic changes in the user's environment in real time. They are unable to provide efficient and safe path planning for visually impaired users in complex situations such as encountering sudden obstacles, temporarily closed roads, or construction zones. Furthermore, most existing systems lack the automatic collection and utilization of personalized user feedback, failing to dynamically optimize navigation and instructions based on specific user behaviors and needs. Therefore, developing a system capable of dynamic path correction based on real-time environmental information and continuous model optimization through user feedback to provide more intelligent and reliable mobility assistance for visually impaired users is a pressing technical challenge in this field.

[0053] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Embodiment 1 is achieved by the following means.

[0054] In this invention, the server includes a recognition device for acquiring user voice input commands, an acquisition device for collecting surrounding spatial information via an imaging device, an analysis device for analyzing the spatial information and identifying obstacles, a calculation device for calculating and optimizing the path based on user needs and obstacle information, a control device for driving the moving body according to the path results, a communication device for transmitting data and commands between the terminal and the server, a notification device for converting navigation commands into voice notifications to the user, and a correction processing device for real-time path correction. Furthermore, it includes a learning processing device for self-learning and optimizing the path planning model and interaction methods based on user feedback and behavioral data. This enables high-precision perception of the user's real-time environment and dynamic route planning, enhances the system's responsiveness to sudden environmental changes, and improves the personalization and safety of navigation through continuous learning, thereby providing visually impaired users with more intelligent, safe, and user-friendly travel assistance services.

[0055] "Identification device" refers to a device or module that receives and parses signals from user voice input commands and converts the voice information into a recognizable data format.

[0056] "Acquisition device" refers to a device that uses onboard imaging equipment to obtain information data about the surrounding environment or space, including cameras or sensors.

[0057] "Analysis device" refers to a device or module that analyzes and processes acquired spatial information or image data and identifies the location and type of obstacle elements (such as obstacles, road elevation, etc.).

[0058] "Computing device" refers to a device or functional module that uses algorithms to calculate and optimize a movement path based on the user's destination information and the analyzed obstacle element information.

[0059] "Control device" refers to a device or control system that controls and drives a mobile body (such as a robot or auxiliary equipment) to perform corresponding movement actions based on the calculated path results.

[0060] "Communication device" refers to the device or module that enables data transmission between identification devices, analysis devices and information processing devices, as well as the receiving of server instructions.

[0061] "Notification device" refers to a device or broadcasting unit that notifies users of instructions issued by an information processing device via voice or other means.

[0062] "Correction processing device" refers to a device or functional module that corrects the current path based on real-time acquired spatial information data, thereby realizing dynamic obstacle avoidance and route adjustment functions.

[0063] "Learning processing device" refers to a device or functional module that uses user feedback information and historical behavior data to self-optimize and adjust a path planning model or interactive program.

[0064] This invention relates to an assisted travel system based on the coordinated operation of an identification device, an acquisition device, an analysis device, a calculation device, a control device, a communication device, a notification device, a correction processing device, and a learning processing device, which is particularly suitable for intelligent navigation assistance for special groups such as visually impaired persons.

[0065] The server is equipped with a high-performance processor and storage unit to deploy core programs and various software algorithms. The terminal typically integrates a voice acquisition module, a panoramic camera, motion execution components (such as stepper motors, drive mechanisms, etc.), speakers, and input buttons. The server can combine deep learning models, path planning algorithms (such as Dijkstra's algorithm), object detection algorithms (such as YOLO-based object detection software), speech recognition services (such as Google Cloud Speech-to-Text), and speech synthesis services (such as Text-to-Speech modules).

[0066] Users first input travel instructions, such as "Take me to the nearest supermarket," through the terminal's voice acquisition module. The terminal then converts the voice signal into processable text information using a voice recognition service, and transmits it synchronously to the server via a communication device. The terminal also uses a panoramic camera to collect real-time information about the surrounding environment, uploading the image data along with the spatial information data.

[0067] The server analyzes and processes the spatial information collected by the terminal using an analysis device. Object detection algorithms, such as YOLO, identify environmental features like obstacles, steps, and traffic lights, and their coordinates and categories are extracted in real time. The server then utilizes path planning algorithms, such as Dijkstra's algorithm, to calculate and optimize the best path to take based on user instructions and surrounding obstacle information.

[0068] The server transmits the path results, key navigation points, and voice prompts back to the terminal via a communication device. Upon response, the terminal uses a control device to drive the motion module, enabling autonomous movement. During travel, the terminal continuously collects new environmental data and feeds it back to the server. Based on the latest environmental changes, the server can invoke a correction processing device to dynamically adjust the path, maximizing user safety. Any navigation or status information is communicated to the user in real-time via a notification device, such as text-to-speech synthesis, and broadcast through a speaker.

[0069] The system also includes a learning processing unit that automatically collects user feedback during the process, such as evaluations and requests conveyed through voice or keystrokes, as well as historical user behavior data. It then uses a generative artificial intelligence model for self-learning and model optimization. The server analyzes user feedback and automatically optimizes navigation strategies and announcements, enhancing the personalization and intelligence of the assisted experience.

[0070] In a specific implementation, for example, if a user inputs "I want to go to the nearest supermarket," the terminal converts the text to text using Google Cloud Speech-to-Text. A 360-degree camera, such as the RICOH THETA, captures environmental images and sends them to the server. The server uses the YOLO algorithm to identify obstacles and the Dijkstra algorithm to calculate the optimal path. Upon returning to the terminal, it provides a text-to-speech prompt: "Please walk straight ahead for 20 meters and then turn right." When a new obstacle is encountered, the terminal pauses and uploads new data. The server recalculates the new path and informs the user.

[0071] Generative artificial intelligence models can be used to analyze large amounts of environmental and feedback data, thereby continuously improving the intelligence level of processes such as path planning and voice broadcasting.

[0072] Example of a prompt statement:

[0073] Please analyze the following environmental image, identify the locations of all major obstacles, and output their coordinates in a list format.

[0074] Based on the following user commands for speech-to-text conversion and the environmental detection results, please generate the safest and most convenient walking route for the user and output the detailed route steps.

[0075] If the robot encounters a new obstacle during its movement, how can it dynamically adjust its current route to ensure user safety? Please output the steps of the adjusted new route and the voice prompts for the user.

[0076] Through the combination of the above hardware and software and process control, the present invention can provide users with fully automatic, personalized and intelligent travel guidance services.

[0077] use Figure 11 The processing flow is explained.

[0078] Step 1:

[0079] Users input voice commands through the terminal's microphone, such as "I want to go to the nearest supermarket." The input is the user's voice information, and the output is the audio data collected by the terminal. The user's actual action is to issue a clear travel request to the terminal.

[0080] Step 2:

[0081] The terminal uses speech recognition software (such as cloud-based speech recognition services) to convert received audio data into text commands. The input is audio data, which undergoes speech feature analysis and acoustic model comparison, and the output is text data. The terminal's actual action is to invoke the recognition engine to process the user's speech and generate the corresponding text.

[0082] Step 3:

[0083] The terminal sends the text commands generated by voice recognition, along with currently captured environmental image data (obtained by a 360-degree camera), to the server via its wireless communication module. The input consists of text data and environmental image data; the data is processed into packaged and encrypted data packets; and the output is an uplink data stream to the server. The terminal's actual action is to trigger the communication module and send a message.

[0084] Step 4:

[0085] The server receives data packets uploaded by the terminal and first processes the environmental image data. It uses object detection algorithms (such as YOLO) to analyze the image and identify obstacle elements such as obstacles, steps, and construction areas. The input is environmental image data; the data processing involves image feature extraction and classification; the output is structured data containing the type and location of obstacle elements. The server's actual actions include initiating the image analysis process and storing the recognition results in the database.

[0086] Step 5:

[0087] The server combines user text commands with structured information about environmental obstacles, employing a path planning algorithm (such as Dijkstra's algorithm) to calculate the optimal travel route from the current location to the destination. The input includes the user's destination (text) and obstacle information; the data processing involves map generation and path search; and the output is a set of waypoints and navigation commands. The server's actual actions involve calling up path and map libraries to output recommended route data.

[0088] Step 6:

[0089] The server sends waypoints, navigation actions, and voice prompts to the terminal via the communication module. The input is the path and navigation content; the data is processed into a command list and a TTS (Text-to-Speech) template; the output is a data stream sent to the terminal. The server's actual actions involve packaging and pushing the results to the terminal.

[0090] Step 7:

[0091] The terminal parses the path and navigation information sent by the server and uses the control device to drive the robot (or guide device) to move along the planned route. The input is path data and navigation instructions; the data calculation is to compare the current position and generate motion instructions; the output is control signals to the motors or actuators. The actual action of the terminal is to start the motors or actuators, enabling the robot to move autonomously.

[0092] Step 8:

[0093] The terminal continuously captures real-time environmental images via its camera, monitoring for new obstacles. If a new obstacle is detected, it uploads the new image and current path information to the server to request a replanning. The input is the current environmental image and path information; the data processing involves obstacle change detection; and the output is a data packet re-uploaded to the server. The terminal's actual actions are capturing the new environment, uploading data, and pausing movement.

[0094] Step 9:

[0095] If the server receives new obstacles or environmental changes, it re-runs the image processing and path planning algorithms to generate a new optimal route and sends it to the terminal. The inputs are the latest environmental and obstacle information and the current user location; the data calculation is for dynamic path correction; and the output is the new path data. The server's actual actions are automatic path recalculation and push.

[0096] Step 10:

[0097] The terminal converts received navigation information and prompts into speech via a text-to-speech (TTS) module, and then notifies the user of the next action through a speaker. For example, "Please proceed 10 meters and then turn right." The input is navigation instructions, the data is processed into text-to-speech, and the output is an audio signal. The terminal's actual action is to play the spoken content for the user's reference.

[0098] Step 11:

[0099] Users follow the voice prompts on the terminal to safely reach their destination using the mobile assistance device. The input is the terminal's voice output; the user reacts accordingly, and the output is the user's walking motion. The user's actual action is to move forward along the robot's navigation path.

[0100] Step 12:

[0101] The terminal collects user feedback and behavioral information input through button presses, voice feedback, etc., organizes it, and uploads it to the server. The server uses a generative artificial intelligence model to analyze and optimize the data, further improving navigation accuracy and user-friendliness. The input is user feedback data, the data calculation is behavioral modeling and optimization instruction generation, and the output is the optimized service strategy. The actual actions of the terminal and server are collecting feedback and automatic learning and updating.

[0102] Application Example 1

[0103] The process flow corresponding to the specific processing in Use Case 1 will be described below. The various parts of the system described below are implemented by the data processing device 12 and the intelligent device 14. Furthermore, the data processing device 12 is referred to as the "server" and the intelligent device 14 is referred to as the "terminal".

[0104] Existing assistive systems, when guiding visually impaired individuals safely outdoors, struggle to promptly perceive environmental changes, avoid obstacles in real time, and dynamically adjust routes. Furthermore, they often neglect to perceive and care for the user's psychological and emotional state, leading to insufficient safety and psychological comfort during movement. In addition, the lack of an intelligent learning mechanism capable of continuously self-optimizing based on user feedback and experience limits the system's adaptability and personalized service level. Therefore, there is an urgent need for an intelligent guidance and support system that integrates environmental perception, path optimization, emotion recognition, and dynamic learning to improve the travel safety and psychological comfort of visually impaired individuals.

[0105] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Application Example 1 is achieved by the following means.

[0106] In this invention, the server includes a recognition device for recognizing user input voice and facial expressions and acquiring their command content and emotional state; an imaging device for acquiring surrounding environmental information; an analysis device for analyzing environmental information and detecting spatial obstacles; a path calculation device for calculating the optimal path based on the environmental analysis results, user emotions, and destination information, and adaptively adjusting the path; a motion control device for controlling the movement of the mobile body; a communication device for sending various types of data to and receiving information from the server in real time during movement; an interactive device for providing navigation and psychological care to the user; a re-path calculation device for automatically recalculating the path based on environmental and emotional changes; and a learning device for continuously optimizing the guidance model based on user feedback. This allows the system to quickly perceive the surrounding environment and the user's psychological state, intelligently calculate an appropriate safe path, support user emotional care through real-time voice communication, and automatically optimize guidance strategies based on user experience, comprehensively improving the travel safety and psychological well-being of visually impaired individuals.

[0107] A "recognition device" refers to a device that can collect and analyze voice and facial expression information input by a user, thereby recognizing the user's instructions and emotional state.

[0108] "Imaging device" refers to an image acquisition device used to collect information about the user's surrounding environment, including but not limited to cameras or multi-directional imaging sensors.

[0109] "Analysis device" refers to a device that can process the collected environmental information through image processing, obstacle detection, spatial analysis, etc., in order to identify spatial obstacle elements.

[0110] A "path computing device" refers to a computing device that performs path planning based on environmental analysis results, user destination information, and emotional state, and obtains the optimal path solution through adaptive adjustments.

[0111] "Motion control device" refers to a device that controls the movement of a moving body based on the output of a path calculation device, including motion operations such as starting, stopping, turning, and obstacle avoidance.

[0112] "Communication device" refers to a communication module that summarizes the data obtained by the identification device and the analysis device, and is responsible for sending data and receiving instructions with the information processing equipment.

[0113] "Interactive device" refers to a device that can notify users through voice or other means based on the information received, and provide psychological care information or navigation information based on the user's emotional state.

[0114] A "repath calculation device" refers to a device that can monitor new obstacle elements or changes in the user's emotional state in real time and automatically replan the path based on the latest data.

[0115] A “learning device” refers to a device that can collect user experience information and emotional state information, and automatically update system parameters based on generative artificial intelligence models, thereby continuously optimizing guidance performance and psychological care response.

[0116] "Information processing equipment" refers to a comprehensive computing terminal that centrally processes data sent by terminals, generates and distributes instructions, and performs model updates and learning processes.

[0117] "Mobile body" refers to a hardware entity that can autonomously complete movement tasks according to the motion control instructions of the guidance system, including actuators such as guide robots.

[0118] To achieve the present invention, the system comprises an intelligent guidance support platform jointly constructed by the terminal, the server, and the user. The specific structure and functions are as follows.

[0119] The terminal is equipped with audio acquisition hardware (such as a microphone), facial expression acquisition hardware (such as a camera), and environmental imaging hardware (such as a 360-degree panoramic camera). The program is deployed on the terminal's local computing platform and utilizes dedicated software libraries (such as speech_recognition for speech recognition, OpenCV for image processing, and TTS engines such as pyttsx3 for voice broadcasting).

[0120] When a user needs to move, they can express their destination request (e.g., "I want to go to the nearby supermarket") through the terminal's voice input interface. The terminal captures the spoken audio in real time and automatically transcribes it into text commands using speech recognition. Simultaneously, the terminal uses its camera to capture images of the user's facial expressions and analyzes these expressions and voices using a local or cloud-based emotion recognition model to determine the user's current emotional state (e.g., anxiety, reassurance, confusion). Additionally, the terminal uses a 360-degree camera to obtain multi-angle real-time images of the user's current environment for environmental awareness.

[0121] The terminal transmits encrypted data packets, including user command text, emotion status tags, and on-site environment images, to the server in real time via a communication module (such as a Wi-Fi adapter and a standard communication API).

[0122] The server, acting as an information processing device, is centrally configured with a high-performance computing platform, deploying an environment analysis module (based on an OpenCV-based object detection network), a path planning module (integrating map service APIs, such as general map APIs), and a generative artificial intelligence model for optimizing path and emotional support response strategies. The server first detects spatial elements such as obstacles, steps, and traffic lights in the received environmental images and generates structured parsed data. Combining the user's current GPS location, destination request, environmental analysis results, and the user's emotional state, the server comprehensively calculates multiple path options and adaptively selects the optimal path. If the user's emotion is "anxious," it automatically prioritizes routes with sparse crowds, no complex intersections, and quiet and safe environments. Simultaneously, the server generates emotional support messages (such as "Please rest assured, the environment is safe now") and transmits the path planning and psychological reassurance suggestions back to the terminal via a communication interface.

[0123] After receiving feedback from the server, the terminal autonomously completes the movement task using a motion control unit (such as the chassis of a guide robot) based on path and emotional suggestions. Simultaneously, the terminal utilizes TTS (Text-to-Speech) software to broadcast key prompts such as "Please walk straight for 80 meters" and "Don't worry, I'm always with you," ensuring seamless integration of navigation and psychological support. During movement, the terminal continuously collects surrounding images and monitors changes in the user's facial expressions. If it detects new obstacles or new emotional states such as user anxiety, it immediately reports the new data to the server. The server then calculates a new path in real time and sends back correction instructions, enabling the system to achieve real-time dynamic adaptation.

[0124] The system also includes a learning device. The server collects user feedback (such as voice feedback on "mobile experience issues" after task completion) and actual trip emotional data into the database, and drives the self-learning and parameter fine-tuning of the generative artificial intelligence model periodically or upon startup, automatically improving the personalization and accuracy of the next round of navigation and psychological care.

[0125] The system is compatible with various types of mobile devices, including guide robots or walking aids. Through the integrated hardware and software modules, it effectively enables users with visual impairments or low vision to perceive their environment, plan safe and personalized routes, and receive real-time psychological support, thereby improving their safety and sense of security during travel.

[0126] Specific examples:

[0127] The user says through the terminal, "I want to go to the nearest hospital."

[0128] The terminal collects environmental images and photos of facial expressions, identifying them as "the user is a little anxious."

[0129] After analyzing the surrounding roads, the server generates a wide route that is far from the obstacle and provides the following feedback: "Please go straight for 200 meters before turning right. The environment is safe. I have activated the protection mode. Please rest assured."

[0130] Example of a prompt statement:

[0131] "I want to go to the nearby supermarket. Please help me plan the route and provide real-time voice prompts to indicate any changes along the way. Also, please offer comfort based on my mood."

[0132] "Please recommend a safe route for visually impaired people to go to the supermarket. The user is showing mild anxiety."

[0133] How to reassure users and dynamically adjust the route when encountering obstacles?

[0134] use Figure 12 The processing flow is explained.

[0135] Step 1:

[0136] Users issue commands through the terminal's voice input interface, such as saying "I want to go to the nearby supermarket." The input is the user's voice signal. The output is the raw audio data collected by the terminal. The user presses the voice button and clearly expresses their needs.

[0137] Step 2:

[0138] The terminal uses a microphone to capture the user's voice and a camera to capture images of the user's facial expressions. The input consists of the user's voice and facial expressions. The terminal uses the `speech_recognition` library to preprocess, reduce noise, and segment the audio to obtain recognizable audio segments, and temporarily stores the facial expression images. The output is an audio and facial expression image data file that can be used for recognition.

[0139] Step 3:

[0140] The terminal uses the speech_recognition library to perform speech recognition processing on the collected audio, converting the user's speech into command text. Simultaneously, the terminal calls a local or cloud-based emotion recognition model to perform emotion analysis on facial expressions and speech tone. The input consists of the audio and images obtained in step 2. Through data analysis by the artificial intelligence model, the output is a structured user request text (e.g., "I want to go to the nearby supermarket") and an emotion state label (e.g., "anxious").

[0141] Step 4:

[0142] The terminal captures multi-angle photos of the current environment using a 360-degree camera. The input is the surrounding real-world scene. The images undergo basic image enhancement processing using OpenCV (such as color correction and distortion correction). The output is the processed environmental image file.

[0143] Step 5:

[0144] The terminal organizes user command text, emotional state, and environmental images, encrypts and packages them via a communication module (such as Wi-Fi), and then uploads them to the server. The input consists of the three types of structured and unstructured data mentioned above. The communication API completes data synchronization, and the output indicates that the data packet was successfully uploaded and received by the server.

[0145] Step 6:

[0146] The server uses software such as OpenCV to automatically perform spatial scene analysis on uploaded environmental images, including obstacle detection, road type identification, and traffic light recognition. The input is the environmental image uploaded by the terminal. Data computation includes object detection algorithms and image segmentation, and the output is the environmental structure analysis results (e.g., "There is a step 1 meter ahead, and an obstacle 2 meters to the left").

[0147] Step 7:

[0148] The server uses parsed environmental information, user destination text, current location, and user sentiment tags to call the map service API for route planning. Inputs include environmental analysis data and user needs. The server employs a comprehensive algorithm for decision-making, and data processing includes dynamic filtering and adaptive adjustment of multiple routes. The output is the optimal route plan, including route details, inflection points, suggestions, and precautions. Simultaneously, it generates emotional support messages (such as "Don't worry, I'm here with you").

[0149] Step 8:

[0150] The server feeds back the optimal route plan, action instructions, and emotional support messages to the terminal via a communication interface. The input consists of route data and messages; after data encapsulation and transmission, the output is the new navigation and support information received by the terminal.

[0151] Step 9:

[0152] After receiving the data, the terminal calls the motion control unit to automatically advance according to the path plan (such as driving the guide robot forward and turning), and calls the TTS engine to perform voice synthesis to broadcast navigation and care content to the user in real time. The input is the feedback content in step 8, which is executed by the TTS module's audio synthesis and motion control algorithm, and the output is the robot's actions (such as moving forward and turning) and voice broadcast (such as "Please go straight for 80 meters, I am next to you").

[0153] Step 10:

[0154] During movement, the terminal continuously collects environmental images and user facial expressions and uploads them to the server. Upon encountering a new obstacle or experiencing a change in emotion, it immediately initiates a new round of steps 4-9, achieving dynamic path and emotion adaptation. The input consists of real-time environmental and user status data; after data analysis, the output is a new path or reassuring prompts.

[0155] Step 11:

[0156] After the user arrives at their destination, the terminal displays "Destination reached" and solicits user feedback (e.g., "Was this experience satisfactory?"). The user provides feedback via voice, with the input being the user's speech and the output being audio data collected by the terminal.

[0157] Step 12:

[0158] The terminal uploads user feedback audio data to the server, where it performs speech recognition and sentiment analysis, stores the data in an experience database, and fine-tunes the generative AI model based on this data to improve the accuracy of future path and sentiment decisions. The input is user feedback audio; after analysis and learning, the output is updated model parameters and saved experience records.

[0159] Alternatively, an emotion engine for inferring user emotions can be combined. That is, the specific processing unit 290 can also use the emotion-specific model 59 to infer user emotions and perform specific processing using user emotions.

[0160] Example 2

[0161] The flow of a specific process in Example 2 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart device 14. The data processing device 12 will be referred to as the "server," and the smart device 14 as the "terminal."

[0162] Existing assisted mobility systems for visually impaired individuals are limited to simple route guidance, lacking awareness and adaptive feedback regarding the user's real-time emotional state. This makes them ineffective in addressing unexpected obstacles, environmental changes, and user anxiety. Furthermore, traditional systems have limited intelligence in environmental perception, user interaction, and route replanning, resulting in inadequate guarantees of user safety and comfort. Therefore, constructing an intelligent assistive system that comprehensively analyzes user intent, environmental conditions, and real-time emotional state, and utilizes generative artificial intelligence models to provide personalized, safe, and flexible navigation and emotional support, has become a pressing technical challenge in this field.

[0163] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Embodiment 2 is achieved by the following means.

[0164] In this invention, the server includes: a recognition unit for recognizing user voice commands; an acquisition unit for acquiring environmental images; an analysis unit for analyzing obstacles and environmental changes; a path generation unit and a re-path generation unit for generating the optimal path based on user intent and environmental information and replanning it in real time; an adaptive processing unit for acquiring and analyzing the user's emotional state; an input information management unit for receiving and managing various input information; and modules for receiving user feedback and continuously optimizing the generative artificial intelligence model through a learning unit. This enables comprehensive perception of the user's current commands, environmental state, and emotional state, allowing for autonomous adjustment of the path and interactive content based on real-time changes. This provides visually impaired individuals with safe, personalized, real-time, and emotionally supportive intelligent travel support, effectively improving travel safety and comfort.

[0165] The "recognition unit" refers to a functional module used to receive and process the user's input voice information and parse its content in order to extract the user's travel needs or control commands.

[0166] "Acquisition unit" refers to a device used to acquire image information of the user's surrounding environment, including image sensors and their related acquisition and preprocessing functions.

[0167] The “analysis unit” refers to the module that processes the collected environmental image data to identify obstacles, steps, and other environmental elements that may affect the user’s passage.

[0168] The "path generation unit" refers to a functional module that calculates and generates the optimal path from the user's current location to the target location based on the identified user instructions and the analyzed environmental information.

[0169] "Control unit" refers to a device that performs real-time control and adjustment of the motion of the drive unit and robot based on the path information output by the path generation unit.

[0170] "Communication unit" refers to a communication module used for bidirectional data transmission with external information processing devices, including uploading locally collected and processed data and receiving external commands.

[0171] A "notification unit" refers to a device that receives instructions from an external information processing device and notifies the user in the form of audio or other means.

[0172] The "adaptive processing unit" refers to a module that analyzes the user's emotional state using a generative artificial intelligence model based on the acquired user's facial images or voice data, and automatically generates dialogue or guidance content that adapts to the user's current emotions and state.

[0173] The "repath generation unit" refers to a functional module that uses a generative artificial intelligence model on the server or information processing device to update and regenerate the best navigation path and personalized guidance instructions in real time based on the latest environmental information, user status information and external data sources.

[0174] The "input information management unit" refers to a module that expands and manages the data input structure of the adaptive processing unit or the repath generation unit according to different types of input data and business requirements.

[0175] A “learning unit” refers to a functional module used to receive user feedback or monitor changes in the user’s emotional state during the service process, and to optimize and update the parameters or algorithms of a generative artificial intelligence model.

[0176] This invention relates to an intelligent assisted travel system based on a generative artificial intelligence model. The system includes a terminal and a server. Users interact with the system through the terminal, and the terminal and server communicate via a wireless network to collaboratively perform functions such as user identification, environmental perception, route generation, sentiment analysis, and personalized guidance.

[0177] The terminal consists of a high-performance microphone, a 360-degree camera, a speaker, control circuitry (such as a Raspberry Pi or an embedded single-board computer), a wireless communication module, and drive devices (such as stepper motors and sensors). The server uses a high-performance processor and graphics processing unit (such as a GPU server), integrates generative artificial intelligence models, and supports multiple functions such as speech recognition, image analysis, path generation and replanning, and emotion recognition.

[0178] When the system is running, the user speaks their destination or related needs into the terminal's microphone. The terminal captures the speech and converts it into text using embedded speech recognition software (such as an AI speech recognition system based on an open-source model). Simultaneously, the terminal captures the user's facial expressions and images of the surrounding environment through a camera. An emotion engine (AI model) analyzes the user's emotional state, and an environmental recognition model identifies obstacles, steps, and other information in the surrounding environment, transmitting this data to the server in real time.

[0179] The server integrates and analyzes the data uploaded from the terminal using deep learning frameworks (such as TensorFlow and PyTorch), natural language processing modules, emotion recognition models, and path planning algorithms. Based on the user's current needs, environmental obstacle information, user emotional state, and external weather and road information, the server dynamically calculates the optimal walking path and personalized comfort messages using generative artificial intelligence models, and then sends the path and guidance content to the terminal.

[0180] The terminal receives route instructions and customized feedback from the server, drives the guide robot along the recommended path via a drive motor, and broadcasts the instructions and psychological support content to the user in real time through a voice synthesis module. For example, when user anxiety is detected, the system can automatically play soothing music or output a message such as "Don't worry, I will guide you safely to your destination."

[0181] The system also has self-learning capabilities. Feedback and emotional changes generated by users during their travels are collected and used to optimize the parameters of the generative artificial intelligence model, thereby continuously improving the interactive experience and navigation safety.

[0182] The hardware components include a high-performance microphone, a 360-degree camera, a Raspberry Pi single-board computer, stepper motors, wireless communication modules, and GPU servers. The software components include an AI speech recognition system, a generative AI large language model, an image analysis and emotion recognition model, path planning algorithms, and speech synthesis software.

[0183] Specific examples:

[0184] User command: "I want to go to the nearby supermarket."

[0185] The system process is as follows:

[0186] 1. After the terminal collects the voice, it calls the speech recognition model to convert the audio into text.

[0187] 2. The terminal collects facial expressions and analyzes the user's current emotions (such as anxiety).

[0188] 3. The terminal collects 360-degree environmental images and uploads them to the server, which then identifies obstacles and path conditions.

[0189] 4. Based on all the information, the server uses a generative artificial intelligence model to generate the optimal walking route and personalized comfort prompts, such as "Please walk straight for 50 meters, there is a zebra crossing ahead" and "Anxiety detected, playing soothing music for you".

[0190] 5. The terminal plays the corresponding voice content according to the server's instructions and drives the robot to move along the route.

[0191] 6. After receiving voice guidance and psychological reassurance, users can proceed to their destination with peace of mind.

[0192] Examples of prompts from generative artificial intelligence models are as follows:

[0193] "Please convert the audio of the user saying 'I want to go to the nearby supermarket' into the corresponding Chinese text."

[0194] "Please identify whether there are steps, obstacles, or traffic lights in the following 360-degree environmental images, and mark their locations."

[0195] "The user's current facial expression analysis indicates anxiety. Please generate a comforting and encouraging Chinese voice message to help the user move forward positively."

[0196] "Based on the following information, generate the safest walking route suitable for visually impaired users and able to alleviate their anxiety: current location, destination, current mood, and a map showing the distribution of environmental obstacles. Please output detailed route instructions and emotional care suggestions."

[0197] Based on the above specific description, this invention can provide real-time intelligent assisted travel services for visually impaired individuals and other users based on multimodal perception and personalized emotional support, and has broad practical application value.

[0198] use Figure 13 The processing flow is explained.

[0199] Step 1:

[0200] Users input voice commands through the terminal's microphone, such as saying "I want to go to the nearby supermarket".

[0201] Input: The user's audio / voice signal.

[0202] The terminal collects and temporarily stores the user's voice data in preparation for subsequent processing.

[0203] Step 2:

[0204] The terminal calls the speech recognition module to input the collected speech signal into the generative artificial intelligence model for speech-to-text processing.

[0205] Input: Audio / speech signal.

[0206] The terminal performs feature extraction and sequence analysis on the voice data and outputs the corresponding Chinese text.

[0207] Output: The user's text command content.

[0208] Step 3:

[0209] The terminal uses a camera to capture images of the user's face and calls an expression recognition model to analyze the user's current emotional state.

[0210] Input: User's facial image.

[0211] The terminal extracts key facial features and facial expressions using a neural network, and then uses a generative artificial intelligence model to output specific emotion categories, such as anxiety or calmness.

[0212] Output: The user's current sentiment assessment result.

[0213] Step 4:

[0214] The terminal uses a 360-degree camera to capture images of the surrounding environment in real time and packages the captured multi-frame images.

[0215] Input: 360-degree environmental image.

[0216] The terminal performs format conversion and preprocessing on the acquired raw image sequences to facilitate subsequent transmission and analysis.

[0217] Output: Preprocessed image data packets.

[0218] Step 5:

[0219] The terminal uploads user text commands, emotion judgment results, and environmental image data to the server via a wireless communication module.

[0220] Input: User text commands, emotional state, and environmental image data.

[0221] The terminal encrypts and packages the data using network protocols and sends it to the designated interface on the server.

[0222] Output: The uploaded data packet.

[0223] Step 6:

[0224] After receiving various data from the terminal, the server uses high-performance image recognition and target detection models (such as convolutional neural networks based on generative artificial intelligence models) to analyze environmental images and identify dangerous elements such as obstacles and steps.

[0225] Input: Environmental image data.

[0226] The server performs batch processing, feature extraction, and target annotation on image data, and outputs structured environmental obstacle information and maps.

[0227] Output: Environmental information including obstacles, steps, traffic lights, etc.

[0228] Step 7:

[0229] The server combines user command text, emotional state, obstacle information, real-time weather, and external map data, uses a generative artificial intelligence model to plan the optimal path, and generates personalized prompts and comforting content based on the user's emotions.

[0230] Input: User destination text, emotional state, environmental information, external data (such as weather and construction status).

[0231] The server fuses multimodal information, performs complex path planning and dialogue generation calculations, and outputs the most suitable path and interaction content.

[0232] Output: Optimal navigation route, personalized voice prompts or music, and other feedback content.

[0233] Step 8:

[0234] The server packages the optimal path information, voice prompts, comforting words, or music playback instructions, and sends them back to the terminal via the communication network.

[0235] Input: optimal path, personalized suggestions.

[0236] The server formats, encrypts, and compresses the output content before pushing it to the terminal port.

[0237] Output: The data packets sent.

[0238] Step 9:

[0239] After receiving instructions from the server, the terminal activates the motion control system, allowing the guide robot to move automatically according to the planned path, and combines the speech synthesis module to broadcast instructions and comforting words to the user in real time.

[0240] Input: Optimal path information, voice and music commands.

[0241] The terminal unpacks server data, drives motors and sensors to achieve navigation, and broadcasts voice prompts to users through speakers, such as "Please go straight for 50 meters" and "You are safe now, please rest assured".

[0242] Outputs: Robot motion control, voice broadcasting, and music output.

[0243] Step 10:

[0244] The terminal continuously collects environmental images and user facial expression data while moving, and periodically calls the local generative artificial intelligence model to analyze the current situation. If a significant change in the environment or user emotion is detected, data upload is initiated and the server is requested to update the decision.

[0245] Input: Dynamic environment images, user facial expression data.

[0246] The terminal monitors and analyzes continuous data in real time. If it detects changes in obstacles or extreme user anxiety, it will automatically send a new request to the server.

[0247] Output: New data packets uploaded in real time, and server request information.

[0248] Step 11:

[0249] Based on the new environmental information or user status received, the server generates new paths or updates interactive guidance content in real time to maintain optimal navigation safety and a user-friendly experience.

[0250] Input: Real-time environmental change data, user sentiment change results.

[0251] The server runs the path generation unit and the adaptive interaction generation module, outputting the updated navigation path and interaction content.

[0252] Output: New navigation instructions and personalized interactive content.

[0253] Application Example 2

[0254] The process flow corresponding to the specific processing in Use Case 2 will be described below. The various parts of the system described below are implemented by the data processing device 12 and the intelligent device 14. In addition, the data processing device 12 is referred to as the "server" and the intelligent device 14 is referred to as the "terminal".

[0255] Existing technologies supporting visually impaired individuals' travel and independent, convenient, and safe navigation in real-world scenarios (such as stores) are inadequate, particularly in areas such as real-time identification of users' psychological states and providing corresponding feedback based on emotional changes to alleviate anxiety and unease. Furthermore, the lack of an integrated system that combines environmental perception, path planning, real-time obstacle detection, and emotionally adaptive assistance based on generative artificial intelligence means that users still easily experience confusion, fear, and other negative experiences in complex scenarios, impacting their quality of life.

[0256] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Application Example 2 is achieved by the following means.

[0257] In this invention, the server includes an information processing device for recognizing user voice commands, an environmental information acquisition and analysis device for collecting and parsing environmental information, a path information generation device for generating the optimal path based on destination information, a control device for dynamic motion control based on environmental and path information, a communication device for enabling two-way communication of data and commands, an information transmission device for conveying information to the user in voice form, an emotion state inference device for monitoring and inferring the user's psychological state through audio or image features, and a feedback generation device for calling a generative artificial intelligence model to generate personalized feedback prompts and notify the user in real time. This enables comprehensive assistance to users in complex and ever-changing environments, including real-time environmental perception, obstacle detection, path guidance, emotion recognition, and adaptive feedback, significantly improving the sense of security, independence, and travel experience of visually impaired individuals.

[0258] "Information processing device" refers to a hardware or software module that processes input user voice data and converts audio signals into text data, thereby recognizing the content of user commands.

[0259] "Environmental information acquisition device" refers to a device or module that collects image data of the surrounding environment in a time sequence through imaging elements (such as cameras).

[0260] "Environmental information analysis device" refers to the hardware, software, or algorithm that analyzes and processes acquired environmental data (such as images) and identifies objects, elevation differences, and obstacles.

[0261] "Route information generation device" refers to a hardware, software, or algorithm module that automatically generates the optimal movement route based on user-input destination information and current environment information.

[0262] "Control device" refers to a hardware or software system that controls the actions of a user's mobile or smart terminal based on generated path information and environmental data.

[0263] "Communication device" refers to a component or module used to send voice information, environmental information and other data to a server and receive instructions and data from the server to realize bidirectional transmission of information inside and outside the system.

[0264] "Information transmission device" refers to hardware or software that transmits processed information to the user in a timely and clear manner through voice output or other means, such as a speaker or speech synthesis module.

[0265] "Emotional state inference device" refers to a processing module or algorithm that infers a user's current psychological state (such as anxiety, peace of mind, etc.) by analyzing the user's audio or image features.

[0266] "Feedback generation device" refers to a hardware, software, or algorithm module that uses generative artificial intelligence models to dynamically generate personalized feedback prompts based on the user's psychological state and notify the user in real time.

[0267] "Generative artificial intelligence models" refer to artificial intelligence algorithms or systems that can automatically generate output content such as text and speech based on input data, in order to achieve user intent understanding and personalized feedback information.

[0268] To facilitate understanding of the present invention, the system structure, composition, and usage of the present invention will be described in detail below with reference to specific embodiments.

[0269] The system provided by this invention can be applied to scenarios requiring assisted navigation and intelligent feedback, such as those involving visually impaired individuals. Through environmental perception, path planning, emotion recognition, and generative artificial intelligence feedback, it enables users to move autonomously and safely in real-world locations (such as supermarkets, hospitals, and airports).

[0270] This system mainly consists of three parts: a server, terminals (such as smartphones and wearable devices), and users. The specific structure and operation methods are described below.

[0271] The terminal includes a microphone for capturing user voice, a camera (such as a 360-degree panoramic camera or a mobile phone camera) for capturing environmental images, and a speaker for voice output. Internally, the terminal installs speech recognition software (such as the SpeechRecognition library), a speech synthesis module (such as gTTS), image acquisition and analysis tools (such as OpenCV), and emotion recognition software (such as EmotionRecognizer). The terminal can capture user voice input and environmental image data in real time and transmit the processed data to the server via a wireless communication module (such as Wi-Fi, 4G / 5G module).

[0272] The server acts as an information processing device, receiving and storing information uploaded by terminals, and performing comprehensive analysis on multi-source data. The server can utilize path planning software modules (such as NavigationAlgorithm), machine learning platforms (such as generative artificial intelligence models based on TensorFlow or PyTorch), and databases to understand user intent, identify environmental obstacles, calculate the optimal navigation path, and dynamically generate personalized voice prompts and psychological reassurance feedback based on the user's emotional state.

[0273] In practical use, users simply issue verbal commands through the terminal, such as "Where are the tomatoes on the shelf?" The terminal automatically converts the speech into text and transmits it to the server. The server, combined with real-time environmental images, analyzes and determines the optimal path between the user's current location and the target area. Simultaneously, based on environmental analysis and emotion recognition results, the server uses a generative artificial intelligence model to generate prompts adapted to the user's current psychological state, such as "Please walk straight for five meters, then turn right," "There are stairs ahead, please be careful," or "You seem a little nervous, please rest assured, everything is going smoothly at the moment." After receiving the relevant voice commands and prompts, the terminal plays them back to the user in real time and continuously adjusts dynamically based on environmental and emotional information.

[0274] This system is flexible, convenient, and highly adaptable. It can integrate universal hardware and software such as mobile phones, wearable devices, servers, cameras, microphones, speakers, modern communication networks, artificial intelligence algorithms, and data processing tools to meet customized assistance needs in different usage scenarios.

[0275] For example, a user in a supermarket setting might give the following voice command:

[0276] User: "Where are the tomatoes on the shelf?"

[0277] The system responded: "Please go straight, and then turn right at the next intersection."

[0278] When the user arrives at the target area, the system prompts: "You have arrived. The tomatoes are on the shelf to the right."

[0279] Other examples of prompt statements:

[0280] User: "Where is the supermarket exit?"

[0281] The system message reads: "Please proceed along the current corridor. After exiting, turn right to reach the exit."

[0282] When a user is feeling anxious, the system will respond: "We have detected that you are somewhat anxious. Please rest assured that we have planned the safest route for you in real time."

[0283] In summary, this invention provides a comprehensive assistance system capable of real-time environmental perception, understanding of user needs and psychological states, and dynamically generating diverse prompts through a generative artificial intelligence model. This system significantly enhances users' ability to act autonomously and their sense of security in complex environments, and provides high-level intelligent and personalized life assistance services for visually impaired individuals and other users with special needs.

[0284] use Figure 14 The processing flow is explained.

[0285] Step 1:

[0286] Users input voice commands through the terminal's microphone, such as "Where are the tomatoes on the shelf?".

[0287] The input is the user's raw voice signal.

[0288] The terminal calls a speech recognition module (such as SpeechRecognition) to convert the real-time recorded audio data into text, which serves as the user's command content.

[0289] The output is the text instruction to be processed, and the text is saved to the buffer.

[0290] Step 2:

[0291] The terminal uses a camera to capture real-time images or video streams of the current environment.

[0292] The input is image data of the surrounding environment.

[0293] The terminal uses an image preprocessing module (such as using OpenCV to denoise and correct images) to format and store the acquired image frames, ensuring efficient uploading and analysis later.

[0294] The output is a standardized environmental image data file.

[0295] Step 3:

[0296] The terminal packages the text instructions generated in step 1 and the image data obtained in step 2 and sends them to the server via the wireless communication module.

[0297] The input consists of instruction text and environmental image data.

[0298] The terminal uses HTTP or MQTT protocols to upload data, ensuring the security and integrity of data transmission.

[0299] The output is a confirmation message indicating that the upload is complete.

[0300] Step 4:

[0301] The server receives data packets from the terminal, including text commands and environmental images.

[0302] The input is the data packet uploaded by the terminal.

[0303] The server performs semantic understanding of text commands (such as intent parsing) and uses image parsing algorithms (such as OpenCV algorithms) to identify information such as obstacles and spatial layout.

[0304] The output consists of instruction semantic tags and environmental element recognition results.

[0305] Step 5:

[0306] Based on the user's instructions and the analysis results of the current environment, the server calls the path planning module (such as NavigationAlgorithm) to calculate the optimal path from the user's current location to the target.

[0307] The input consists of the target point, current location, and obstacle distribution data.

[0308] The server loads the location map database, performs the shortest or most suitable path algorithm based on the overall environment, and generates path coordinates and segmented action instructions.

[0309] The output includes detailed path planning results and a list of step-by-step guided actions.

[0310] Step 6:

[0311] The server determines whether the terminal has uploaded emotion detection data; if so, it analyzes the user's current emotional state.

[0312] The input consists of user emotion-related data (such as voice characteristics, facial expression images, etc.).

[0313] The server calls a sentiment analysis module (such as EmotionRecognizer) to analyze the current tone of voice and facial expressions to derive the user's emotional label (such as "nervous" or "reassuring").

[0314] The output is the user's mental state label.

[0315] Step 7:

[0316] Based on the route planning results and the user's emotional state, the server uses a generative artificial intelligence model to generate personalized and contextualized prompts, such as "Please go straight, then turn right" or "Rest assured, everything is going smoothly at the moment."

[0317] The input consists of path planning output data and sentiment analysis results.

[0318] The server generates content based on preset templates and self-learning models, producing user-oriented feedback and operation prompt text.

[0319] The output is a personalized voice prompt text.

[0320] Step 8:

[0321] The server sends the generated voice prompts and auxiliary data to the terminal via the communication module.

[0322] The input consists of feedback prompt text and route navigation data.

[0323] The server sends response messages based on the terminal ID, ensuring low-latency and high-availability information delivery.

[0324] The output is a receipt of the sent message.

[0325] Step 9:

[0326] After receiving the prompt text from the server, the terminal calls a speech synthesis module (such as gTTS) to convert the text into playable audio.

[0327] The input is a voice prompt text.

[0328] The terminal completes the text-to-audio conversion process and pushes the audio stream to the speaker or headphones to broadcast guidance and reassurance information in real time.

[0329] The output is a voice prompt that the user can hear.

[0330] Step 10:

[0331] During user movement, the terminal periodically collects real-time environmental images and user voice / expression data, and uploads them to the server to complete continuous environmental and emotional state monitoring.

[0332] The input consists of newly acquired environmental images and emotional feature data.

[0333] The terminal automatically calculates image changes and analyzes user interactions, prompting users to recalculate paths or provide feedback on emotions when necessary.

[0334] The output is dynamically adjusted navigation and real-time feedback results.

[0335] The specific processing unit 290 sends the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires sound representing user input regarding the result of the specific processing. The control unit 46A sends the sound data representing user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0336] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0337] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart device 14 or external devices, and the smart device 14 acquires or collects information required for processing from the data processing device 12 or external devices.

[0338] For example, the collection unit is implemented by the control unit 46A of the smart device 14 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the smart device 14 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the output device 40 of the smart device 14 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0339] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the smart device 14.

[0340] Second Implementation Method

[0341] Figure 3 An example of the configuration of the data processing system 210 according to the second embodiment is shown.

[0342] like Figure 3 As shown, the data processing system 210 includes a data processing device 12 and smart glasses 214. A server can be cited as an example of the data processing device 12.

[0343] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0344] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, and communication I / F 44 are also connected to the bus 52.

[0345] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0346] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to capture images of the user 20's surroundings (e.g., the field of view defined by an angle equivalent to the field of vision of an average healthy person).

[0347] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0348] Figure 4 This illustrates an example of the main functions of the data processing device 12 and the smart glasses 214. For example... Figure 4 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0349] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0350] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. In the emotion inference function (emotion-specific function) using the emotion-specific model 59, various inferences and predictions related to the user's emotions are performed, including inferences and predictions of the user's emotions, but this is not limited to this example. Furthermore, emotion inference and prediction may also include, for example, emotion analysis (parsing).

[0351] In the smart glasses 214, the processor 46 performs reception and output processing. The memory 50 stores the reception and output program 60. The processor 46 reads the reception and output program 60 from the memory 50 and executes the read reception and output program 60 on the RAM 48. The reception and output processing is implemented by the processor 46 operating as a control unit 46A according to the reception and output program 60 executed on the RAM 48. Furthermore, the smart glasses 214 has the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and these models can also be used to perform the same processing as the specific processing unit 290.

[0352] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart glasses 214. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0353] Example 1

[0354] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0355] Application Example 1

[0356] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0357] Example 2

[0358] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0359] Application Example 2

[0360] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0361] The specific processing unit 290 sends the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A outputs the result of the specific processing to the speaker 240. The microphone 238 acquires sound input representing the user's input regarding the result of the specific processing. The control unit 46A sends the sound data representing the user's input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0362] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0363] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or external devices, and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or external devices.

[0364] For example, the collection unit is implemented by the control unit 46A of the smart glasses 214 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the smart glasses 214 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 of the smart glasses 214 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0365] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the smart glasses 214.

[0366] Third Implementation Method

[0367] Figure 5 An example of the configuration of the data processing system 310 according to the third embodiment is shown.

[0368] like Figure 5 As shown, the data processing system 310 includes a data processing device 12 and a head-mounted terminal 314. A server can be cited as an example of the data processing device 12.

[0369] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0370] The head-mounted terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, display 343, and communication I / F 44 are also connected to the bus 52.

[0371] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0372] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to capture images of the user 20's surroundings (e.g., the field of view defined by an angle equivalent to the field of vision of an average healthy person).

[0373] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0374] Figure 6 This illustrates an example of the main functions of the data processing device 12 and the head-mounted terminal 314. For example... Figure 6 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0375] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0376] The memory 32 stores the data generation model 58 and the emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290.

[0377] In the head-mounted terminal 314, the processor 46 performs the acceptance / output processing. The memory 50 stores the acceptance / output program 60. The processor 46 reads the acceptance / output program 60 from the memory 50 and executes the read acceptance / output program 60 on the RAM 48. The acceptance / output processing is implemented by the processor 46 operating as a control unit 46A according to the acceptance / output program 60 executed on the RAM 48.

[0378] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the head-mounted terminal 314. In the following description, the data processing device 12 will be referred to as the "server" and the head-mounted terminal 314 will be referred to as the "terminal".

[0379] Example 1

[0380] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0381] Application Example 1

[0382] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0383] Example 2

[0384] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0385] Application Example 2

[0386] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0387] The specific processing unit 290 sends the result of the specific processing to the head-mounted terminal 314. In the head-mounted terminal 314, the control unit 46A outputs the result of the specific processing to the speaker 240 and the display 343. The microphone 238 acquires sound input representing the user's input regarding the result of the specific processing. The control unit 46A sends the sound data representing the user's input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0388] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0389] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the head-mounted terminal 314, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the head-mounted terminal 314. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the head-mounted terminal 314 or external devices, and the head-mounted terminal 314 acquires or collects information required for processing from the data processing device 12 or external devices.

[0390] For example, the collection unit is implemented by the control unit 46A of the head-mounted terminal 314 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the head-mounted terminal 314 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12 to analyze the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12 to generate a menu using a generation AI. For example, the serving unit is implemented by the speaker 240 and display 343 of the head-mounted terminal 314 or the specific processing unit 290 of the data processing device 12 to provide the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0391] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the head-mounted terminal 314.

[0392] Fourth Implementation Method

[0393] Figure 7 An example of the configuration of the data processing system 410 according to the fourth embodiment is shown.

[0394] like Figure 7 As shown, the data processing system 410 includes a data processing device 12 and a robot 414. A server can be cited as an example of the data processing device 12.

[0395] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0396] Robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, controlled object 443, and communication I / F 44 are also connected to the bus 52.

[0397] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0398] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to photograph the area around robot 414 (e.g., the field of view defined by a perspective equivalent to the field of vision of an average healthy person).

[0399] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0400] The controlled object 443 includes a display device, LEDs (light-emitting diodes) for the eyes, and motors for driving the arms, hands, and feet. The posture or movement of the robot 414 is controlled by controlling the motors in the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. In addition, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0401] Figure 8 This illustrates an example of the main functions of the data processing device 12 and the robot 414. For example... Figure 8 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0402] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0403] The memory 32 stores the data generation model 58 and the emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290.

[0404] In robot 414, the processor 46 performs the acceptance and output processing. The memory 50 stores the acceptance and output program 60. The processor 46 reads the acceptance and output program 60 from the memory 50 and executes the read acceptance and output program 60 on RAM 48. The acceptance and output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance and output program 60 executed on RAM 48.

[0405] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the robot 414. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 will be referred to as the "terminal".

[0406] Example 1

[0407] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0408] Application Example 1

[0409] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0410] Example 2

[0411] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0412] Application Example 2

[0413] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0414] The specific processing unit 290 sends the result of the specific processing to the robot 414. In the robot 414, the control unit 46A outputs the result of the specific processing to the speaker 240 and the controlled object 443. The microphone 238 acquires sound input representing the result of the specific processing. The control unit 46A sends the sound data representing the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0415] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0416] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or external devices, and the robot 414 acquires or collects information required for processing from the data processing device 12 or external devices.

[0417] For example, the collection unit is implemented by the control unit 46A of the robot 414 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the robot 414 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 of the robot 414 and the control object 443 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0418] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the robot 414.

[0419] Furthermore, the emotion-specific model 59, acting as an emotion engine, can determine a user's emotion based on a specific mapping. Specifically, the emotion-specific model 59 can determine a user's emotion based on an emotion graph that serves as a specific mapping (see [reference]). Figure 9 The system determines the user's emotions. Furthermore, the emotion-specific model 59 can similarly determine the robot's emotions, and the specific processing unit 290 performs specific processing based on the robot's emotions.

[0420] Figure 9 This is a diagram representing an emotion map 400 that maps multiple emotions. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotion is. On the outer side of the concentric circles, emotions representing states or behaviors arising from mood are arranged. Emotions are concepts that include feelings and mental states. Emotions generated by reactions occurring in the brain are arranged roughly to the left of the concentric circles. Emotions derived from situational judgments are arranged roughly to the right of the concentric circles. Emotions generated by reactions occurring in the brain and derived from situational judgments are arranged roughly above and below the concentric circles. Furthermore, "pleasant" emotions are arranged above the concentric circles, and "unpleasant" emotions are arranged below them. Thus, in the emotion map 400, multiple emotions are mapped based on the structure that generates emotions, and emotions that are likely to occur simultaneously are mapped close to each other.

[0421] These emotions are distributed at the three o'clock position of the emotion map 400, typically fluctuating between peace and anxiety. In the right half of the emotion map 400, situational awareness dominates over internal sensation, thus resulting in an impression of calm.

[0422] The inner side of the emotion map 400 represents the inner state, while the outer side represents behavior. Therefore, the further outward you are from the emotion map 400, the more visible the emotion becomes (manifested in behavior).

[0423] Here, human emotions are based on various balances such as posture and blood sugar levels. When these balances deviate from an ideal state, it indicates an unpleasant state; when they approach the ideal state, it indicates a pleasant state. Emotions in robots, cars, motorcycles, etc., can also be created in the following way: based on various balances such as posture and remaining battery power, when these balances deviate from an ideal state, it indicates an unpleasant state; when they approach the ideal state, it indicates a pleasant state. Emotion maps can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a Brain Physiological Signal Analysis System for Voice Emotion Recognition and Emotion, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). In the left half of the emotion map, emotions belonging to the sensory-dominated region, called "response," are arranged. Furthermore, in the right half of the emotion map, emotions belonging to the situational cognition-dominated region, called "situation," are arranged.

[0424] In the emotion map, two types of emotions that promote learning are defined. One is a negative emotion on the situational side, in the middle or peripheral region of "repentance" or "reflection." This occurs when the robot experiences negative emotions such as "I don't want to experience this feeling again" or "I don't want to be blamed again." The other is a positive emotion on the response side, near the "desire" region. This occurs when there are positive feelings such as "wanting more" or "wanting to know more."

[0425] The emotion-specific model 59 inputs user input into a pre-trained neural network to obtain emotion values ​​representing each emotion shown in the emotion map 400, thereby determining the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network... Figure 10 As shown in the sentiment graph 900, it was trained in a way that sentiments that are configured close to each other have similar values. Figure 10 The text shows examples of emotions such as "peace of mind", "stability", and "reassurance" that have similar emotion values.

[0426] The above description focuses on the functions of the data processing device 12, but the system of this disclosure is not necessarily installed on a server. The system of this disclosure can also be installed as a general information processing system. This disclosure can also be installed, for example, as a software program running on a personal computer, an application running on a smartphone, etc. The method of this disclosure can also be provided to users in the form of SaaS (Software as a Service).

[0427] In the above embodiments, an example of a specific process being performed by a single computer 22 is given. However, the technology disclosed herein is not limited to this, and the specific process can also be distributed among multiple computers, including computer 22. For example, the data generation model 58 can be located on an external device of the data processing apparatus 12, where data is generated based on the input data.

[0428] In the above embodiments, examples of storing a specific processing program 56 in the memory 32 have been described, but the technology disclosed herein is not limited thereto. For example, the specific processing program 56 may also be stored in a portable computer-readable non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed into the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0429] Alternatively, a specific processing program 56 may be pre-stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 according to the requirements of the data processing device 12.

[0430] In addition, it is not necessary to store all the specific processing program 56 in the storage device such as the server connected to the data processing device 12 via the network 54 or in the memory 32; a portion of the specific processing program 56 may be stored in advance.

[0431] As hardware resources for performing specific processes, various processors, as shown below, can be used. For example, a CPU can be listed as a processor, which functions as a general-purpose processor that performs specific processes by executing software, i.e., a program. Furthermore, processors can be listed as special-purpose circuits such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application-Specific Integrated Circuits), which are processors with circuitry specifically designed to perform specific processes. Each processor has built-in or connected memory, and each processor executes specific processes using that memory.

[0432] The hardware resources for performing a specific process can consist of one of these various processors, or a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resources for performing a specific process can be a single processor.

[0433] As an example of a single processor, there are two approaches: First, a processor is composed of a combination of one or more CPUs and software, which functions as a hardware resource to perform a specific process; second, as represented by a SoC (System-on-a-chip), a processor is used to implement the functionality of the entire system, which includes multiple hardware resources for performing a specific process, using a single IC (Integrated Circuit) chip. In this way, the specific process is implemented by using one or more of the aforementioned processors as hardware resources.

[0434] Furthermore, the hardware architecture of these various processors, more specifically, can utilize circuits that combine semiconductor elements and other circuit components. Moreover, the specific process described above is just one example. Therefore, without departing from the main point, unnecessary steps can certainly be deleted, new steps added, or the processing order changed.

[0435] The descriptions and illustrations above are detailed explanations of a portion of the technology disclosed herein, and are merely one example of the technology disclosed herein. For example, the above descriptions of the structure, function, effect, and results are just one example of the structure, function, effect, and results of a portion of the technology disclosed herein. Therefore, without departing from the spirit of the technology disclosed herein, unnecessary parts may be deleted, new elements added, or replacements may be made to the descriptions and illustrations above. Furthermore, to avoid confusion and facilitate understanding of a portion of the technology disclosed herein, explanations of common technical knowledge that do not require special explanation under the premise of being able to implement the technology disclosed herein have been omitted from the descriptions and illustrations above.

[0436] All documents, patent applications and technical specifications set forth in this specification are incorporated herein by reference to the same extent that each document, patent application and technical specification is specifically and individually described therein and referenced by reference.

[0437] In addition, the following notes are provided in response to the above explanation.

[0438] Example 1

[0439] (Note 1)

[0440] An information processing system includes: a recognition device for acquiring and converting user-inputted commands via voice into information data; an acquisition device for acquiring surrounding spatial information data via an onboard imaging device; an analysis device for analyzing and processing the acquired spatial information data and identifying obstacle elements; a calculation device for calculating and optimizing a movement path based on user destination information and obstacle element information; a control device for driving and controlling a moving body according to the calculated movement path information; a communication device for sending information data acquired by the recognition and analysis devices to the information processing device and receiving control commands from the information processing device; a notification device for converting the command information received by the information processing device into voice and notifying the user; and a correction processing device for real-time path correction using the acquired spatial information data.

[0441] (Note 2)

[0442] The information processing system according to Appendix 1 further includes a learning processing device for acquiring evaluation information provided by the user through operation or input, and for reconstructing or optimizing the program model based on the evaluation information and collected historical behavioral data.

[0443] (Note 3)

[0444] The information processing system according to Appendix 1 further includes functions for sending spatial information data and path information collected by the acquisition device to the information processing device, and for recalculating new paths in real time and receiving control commands when factors change during movement.

[0445] Application Example 1

[0446] (Note 1)

[0447] An information processing system includes: a recognition device for recognizing instruction content and emotional state from user-input voice and facial expression information; an imaging device for acquiring information about the surrounding environment; an analysis device for analyzing the acquired environmental information and recognizing spatial obstacle elements; a path calculation device for calculating the optimal path based on the analysis results, user destination information, and emotional state, and automatically adjusting path selection conditions according to the user's psychological state; a motion control device for controlling the movement of a moving body according to information from the path calculation device; a communication device for sending data obtained through the recognition and analysis devices to an information processing device and receiving instructions from the information processing device; an interactive device for notifying the user of the received information content in the form of voice or other means, and providing psychological care information and navigation information according to the emotional state; and a re-path calculation device for detecting new obstacle elements or changes in the user's emotional state in real time and automatically recalculating the optimal path based on the acquired data.

[0448] (Note 2)

[0449] The information processing system according to Appendix 1 further includes: a learning device for receiving experience information and emotional state information collected from users, automatically updating generative artificial intelligence models and other models using the information, and optimizing the guidance performance or psychological care response of the system.

[0450] (Note 3)

[0451] According to the information processing system described in Appendix 1, the system sends the acquired environmental and path information to the information processing device. When obstacles or changes in emotional state are detected during movement, the system recalculates the new path in real time and dynamically receives and executes guidance instructions and psychological care information.

[0452] Example 2

[0453] (Note 1)

[0454] An information processing system includes: a recognition unit for acquiring sound information and recognizing instruction content; a collection unit for acquiring surrounding environment images and generating environmental information; an analysis unit for analyzing the environmental information to detect obstacles or steps; a path generation unit for generating a path to a destination based on the instruction content recognized by the recognition unit and the environmental information detected by the analysis unit; a control unit for controlling a driving device according to the path; a communication unit for sending data obtained by the recognition unit and the analysis unit to an external information processing device and receiving instruction information from the external information processing device; a notification unit for notifying the user of the instruction from the external information processing device via audio; an adaptive processing unit for acquiring user facial images or sound information, recognizing the user's emotional state using a generative artificial intelligence model, and adaptively generating interactive or guiding content based on the emotional state and user state; a re-path generation unit for generating an optimal path and personalized instructions by employing a generative artificial intelligence model on a server or information processing device, combining environmental information, user state information, and external information sources; and an input information management unit for performing data expansion management on the aforementioned adaptive processing unit or re-path generation unit according to the type of input information.

[0455] (Note 2)

[0456] The information processing system according to Appendix 1 further includes: a learning unit that receives user feedback information or changes in user emotional state during the service process, and uses the feedback information to update the parameters of the generative artificial intelligence model.

[0457] (Note 3)

[0458] The information processing system according to Appendix 1 further includes: a processing unit that, while sending environmental information and path information to an external information processing device, generates new paths or interactive guidance in real time using a generative artificial intelligence model and dynamically updates the instruction content when a change in user status or environmental information is detected.

[0459] Application Example 2

[0460] (Note 1)

[0461] An information processing system includes: an information processing device for inputting user voice data and converting audio signals into text data to identify instruction content; an environmental information acquisition device for acquiring surrounding environmental data in a time sequence through an imaging element; an environmental information analysis device for analyzing the acquired environmental data and identifying objects and elevation differences; a path information generation device for deriving an optimal movement path based on the user's destination information; a control device for controlling movement actions based on the calculated path information and environmental condition analysis results; a communication device for sending voice information and environmental information to the information processing device and receiving path instructions from the device; an information transmission device for conveying the received information to the user through a voice output device; an emotion state inference device for identifying the user's psychological state through audio or image features; and a feedback generation device for generating adaptive feedback prompts based on the user's psychological state using a generative artificial intelligence model and notifying the user.

[0462] (Note 2)

[0463] The information processing system according to Appendix 1 further includes a model updating device for accumulating feedback information obtained from users as learning data and using the data to improve the inference accuracy of the generative artificial intelligence model.

[0464] (Note 3)

[0465] The information processing system according to Appendix 1 further includes the function of sending the environmental information and the path information to the information processing device at any time, dynamically regenerating the path information according to environmental changes, and having the terminal receive the generated instructions accordingly.

Claims

1. An information processing system, characterized in that, include: A voice recognition device for recognizing user voice input commands; An image acquisition device for capturing images of the surrounding environment; Image analysis device for analyzing acquired image data and identifying obstacles and steps; A route calculation device for calculating the optimal route from a user to their destination; A motion control device for controlling movement based on a calculated route; A communication device for sending user commands recognized by the speech recognition device and data parsed by the image parsing device to a server, and for receiving commands from the server; and A communication device for notifying a user of received instructions from the server via voice.

2. The information processing system according to claim 1, characterized in that, It also includes a learning device for receiving user feedback and updating the model based on the feedback.

3. The information processing system according to claim 1, characterized in that, The information processing system can send the acquired image data and route information to the server, and can recalculate the new route in real time and receive instructions when changes are needed along the way.

Images