Information processing system

Abstract

The invention provides an information processing system. The information processing system comprises a device used for acquiring image data through a camera worn by a user; means for acquiring location information; means for transmitting the image data and the location information to a server via a communication network; means for parsing the image data and the position information in the server; the device is used for generating feedback information aiming at the user in the server according to an analysis result; and the device is used for sending the feedback information to the user through the communication network and outputting the feedback information in a voice mode.

Description

Technical Field

[0001] The technology disclosed herein relates to an information processing system. Background Technology

[0002] Japanese Patent Application Publication No. 2022-180282 discloses a method for controlling a role-based chatbot executed by at least one processor. The method includes the following steps: receiving a user's speech; adding the user's speech to a prompt word, the prompt word containing instruction statements associated with an explanation of the chatbot's role; encoding the prompt word; and inputting the encoded prompt word into a language model to generate a chatbot response to the user's speech.

[0003] Currently, visually impaired individuals and other users at risk of travel difficulties in accurately perceiving changes in their surroundings, such as traffic signals, road obstacles, and approaching vehicles. Traditional assistive tools cannot provide real-time, accurate, and comprehensive environmental perception and safety alerts, leading to significant safety risks and mobility inconveniences for users. Therefore, there is an urgent need for an intelligent assistive system that can provide users with real-time environmental perception and generate safe action guidance. Summary of the Invention

[0004] This invention proposes an information processing system that acquires real-time image data and simultaneously collects the user's location information via a user-worn camera, uploading this data to a server in real time via a communication network. The server intelligently analyzes the acquired image data and location information, combining AI image recognition and high-precision map services to detect and assess the user's surrounding environment and potential risk factors. Based on the analysis results, the server automatically generates personalized safety feedback information and promptly sends this feedback to the user's terminal via the communication network, broadcasting it to the user via voice. This system can efficiently and accurately provide users with environmental risk warnings and safety guidance during their walks, significantly improving the safety and convenience of independent travel.

[0005] "User" refers to a person who uses this system and wears a camera to receive feedback information.

[0006] A "camera" is an image acquisition device that can capture images of the environment in front of a user's line of sight.

[0007] "Image data" refers to digital image information captured by a camera and used to reflect the user's surrounding environment.

[0008] "Location information" refers to geographic coordinate data that displays the user's current location, obtained through GPS or other positioning methods.

[0009] "Communication network" refers to wireless or wired communication infrastructure used for exchanging data between terminals and servers, including but not limited to fifth-generation mobile communication systems.

[0010] A "server" refers to a remote computing device that receives image data and location information and parses and processes it.

[0011] "Analysis" refers to the process by which the server identifies, analyzes, and determines the environment of image data and location information.

[0012] "Feedback information" refers to the prompts generated by the server based on the parsing results, used to guide users in security actions.

[0013] "Voice output" refers to the process of using a speaker or headphones to broadcast feedback information to the user in voice form.

[0014] "Wearable devices" refer to smart devices that can be worn directly by users and have functions such as installing cameras.

[0015] The "fifth generation mobile communication system" refers to the 5G wireless communication network that provides high-speed, low-latency data transmission services. Attached Figure Description

[0016] Figure 1 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the first embodiment.

[0017] Figure 2 This is a conceptual diagram illustrating an example of the main functions of the data processing apparatus and smart device according to the first embodiment.

[0018] Figure 3 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the second embodiment.

[0019] Figure 4 This is a conceptual diagram illustrating an example of the main functions of the data processing device and smart glasses according to the second embodiment.

[0020] Figure 5 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the third embodiment.

[0021] Figure 6 This is a conceptual diagram illustrating an example of the main functions of the data processing apparatus and head-mounted terminal according to the third embodiment.

[0022] Figure 7 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the fourth embodiment.

[0023] Figure 8This is a conceptual diagram illustrating an example of the main functions of the data processing device and robot according to the fourth embodiment.

[0024] Figure 9 This represents an emotion map that maps multiple emotions.

[0025] Figure 10 This represents an emotion map that maps multiple emotions.

[0026] Figure 11 This is a sequence diagram illustrating the processing flow of the data processing system of the first embodiment.

[0027] Figure 12 This is a sequence diagram illustrating the processing flow of the data processing system in Application Example 1.

[0028] Figure 13 This is a sequence diagram illustrating the processing flow of the data processing system of the second embodiment.

[0029] Figure 14 This is a sequence diagram illustrating the processing flow of the data processing system in Application Example 2. Detailed Implementation

[0030] Hereinafter, an example of an implementation of the system according to the present disclosure will be described with reference to the accompanying drawings.

[0031] First, let me explain the terminology used in the following instructions.

[0032] In the following embodiments, the processor (hereinafter referred to as "processor") with reference numerals may be a single computing device or a combination of multiple computing devices. Furthermore, the processor may be a single computing device or a combination of multiple computing devices. Examples of computing devices include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), etc.

[0033] In the following embodiments, RAM (Random Access Memory), as indicated in the figures, is a memory that temporarily stores information and is used as working memory by the processor.

[0034] In the following embodiments, the memory, as indicated by the reference numerals, is one or more non-volatile storage devices that store various programs and parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), disks (e.g., hard disks), or magnetic tapes.

[0035] In the following embodiments, the communication I / F (Interface) with reference numerals is an interface that includes a communication processor and an antenna, etc. The communication I / F is responsible for communication between multiple computers. As an example of a communication specification applicable to the communication I / F, wireless communication specifications such as 5G (5th Generation Mobile Communication System), Wi-Fi (wireless fidelity) (registered trademark), or Bluetooth (registered trademark) can be listed.

[0036] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. Furthermore, in this specification, when "and / or" connects to express more than three items, the same interpretation as "A and / or B" applies.

[0037] First Implementation Method

[0038] Figure 1 An example of the configuration of the data processing system 10 according to the first embodiment is shown.

[0039] like Figure 1 As shown, the data processing system 10 includes a data processing device 12 and an intelligent device 14. A server can be cited as an example of the data processing device 12.

[0040] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0041] The smart device 14 includes a computer 36, a receiving device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. In addition, the receiving device 38, output device 40, camera 42, and communication I / F 44 are also connected to the bus 52.

[0042] The receiving device 38 includes a touchscreen 38A and a microphone 38B, and receives user input. The touchscreen 38A receives user input via touch by detecting contact with an indicator (e.g., a pen or finger). The microphone 38B receives user input via sound by detecting the user's voice. The control unit 46A in the processor 46 sends data representing the user input received by the touchscreen 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data representing the user input.

[0043] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting data in a form perceptible to the user 20 (e.g., sound and / or text). The display 40A displays visual information such as text and images according to instructions from the processor 46. The speaker 40B outputs sound according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0044] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for sending and receiving various information between processor 46 and processor 28 via network 54.

[0045] Figure 2 The diagram shows an example of the main functions of the data processing device 12 and the smart device 14.

[0046] like Figure 2 As shown, in the data processing apparatus 12, specific processing is performed by the processor 28. A specific processing program 56 is stored in the memory 32. The specific processing program 56 is an example of a "program" as understood in this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0047] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. In the emotion inference function (emotion-specific function) using the emotion-specific model 59, various inferences and predictions related to the user's emotions are performed, including inferences and predictions of the user's emotions, but this is not limited to this example. Furthermore, emotion inference and prediction may also include, for example, emotion analysis (parsing).

[0048] In the smart device 14, the processor 46 performs the acceptance output processing. The memory 50 stores the acceptance output program 60. The acceptance output program 60 is used in conjunction with the data processing system 10 and the specific processing program 56. The processor 46 reads the acceptance output program 60 from the memory 50 and executes the read acceptance output program 60 on the RAM 48. The acceptance output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance output program 60 executed on the RAM 48. Furthermore, the smart device 14 has the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and these models can also be used to perform the same processing as the specific processing unit 290. The acceptance output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance output program 60 executed on the RAM 48.

[0049] Alternatively, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains the processing results (prediction results, etc.) using the data generation model 58 by communicating with the server device that has the data generation model 58. Furthermore, the data processing device 12 may be a server device or a user-held terminal device (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of the processing of the data processing system 10 of the first embodiment will be described.

[0050] Example 1

[0051] The flow of a specific process in Example 1 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart device 14. Furthermore, the data processing device 12 is referred to as the "server," and the smart device 14 is referred to as the "terminal."

[0052] Existing assistive systems suffer from significant latency and insufficient accuracy in acquiring, analyzing, and generating feedback for users based on the analysis results. This is particularly problematic when assisting visually impaired individuals and other users at risk of injury, as they cannot provide rapid, accurate perception of the surrounding environment or real-time, personalized feedback. Consequently, users struggle to take timely and safe actions when faced with sudden environmental changes (such as traffic light changes, approaching obstacles, or road steps), leading to reduced safety and limited independent movement.

[0053] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Embodiment 1 is achieved by the following means.

[0054] In this invention, the server includes: a device for performing multi-category target detection and environmental analysis on image information and spatial location information acquired in real time through a wearable image acquisition device and a positioning module; a device for spatial data correction using a geographic information database; a device for dynamically generating natural language feedback content tailored to the user's current situation using a generative artificial intelligence model based on environmental analysis results and prompts; and a device for sending the feedback content to the user terminal via an information communication network and outputting it through voice and physical stimulation. This allows users to receive clear and relevant safety prompts in real time, enabling rapid environmental perception and automatic decision-making, effectively improving the independence and safety of their actions.

[0055] "Wearable image acquisition device" refers to a device that can be worn by a user to collect image information of the user's environment in real time.

[0056] "Spatial location information" refers to data that can represent the geographical location of a user, including but not limited to geographic coordinates such as longitude, latitude, and altitude.

[0057] "Information processing device" refers to a computing device that can store, parse, and process acquired image information and spatial location information.

[0058] "Multi-class target recognition" refers to the process of automatically detecting and classifying different types of targets (such as traffic signals, pedestrians, vehicles, obstacles, etc.) in image information.

[0059] "Surrounding environment analysis" refers to the technology of comprehensively processing image information and spatial location information to determine and understand the user's surrounding environment.

[0060] A "geographic information database" is a database system used to store and manage geospatial data, providing reference information for spatial analysis and navigation.

[0061] "Generative artificial intelligence models" refer to artificial intelligence algorithm models that can automatically generate natural language content based on input environmental information and prompts.

[0062] "Prompt statements" are textual messages that are input to generative artificial intelligence models and are used to guide them in generating the required feedback content.

[0063] "Natural language feedback content" refers to prompts or instructions generated by the system based on the user's current environment and expressed in a form of natural language that humans can understand.

[0064] "Information and communication network" refers to the communication infrastructure that enables data transmission between terminals and servers, including wireless communication networks, mobile communication systems, etc.

[0065] "Voice output device" refers to a device that can convert text information into speech and play it, so that users can obtain feedback through hearing.

[0066] "Physical stimulation methods" refer to methods of conveying prompts to users through non-verbal means such as vibration and sound.

[0067] "Real-time coordination and management" refers to the dynamic organization and optimization of various data acquisition, processing, and feedback steps in the system to ensure the overall response speed and stability of the system.

[0068] To better understand the present invention, the embodiments of the present invention will be specifically described below in conjunction with the technical solutions of the present invention.

[0069] This invention relates to a user intelligent assistance system based on a generative artificial intelligence model and multi-source sensor data fusion. The system mainly comprises a wearable image acquisition device, a positioning module, a terminal device, an information processing device (server), an information communication network, and an output device.

[0070] Users wear wearable image acquisition devices (such as head-mounted cameras, smart glasses, etc.) and positioning modules (such as GPS modules). When the user is in motion, the camera can continuously acquire real-time image data of the user's environment at a rate of, for example, 30 frames per second, while the positioning module acquires the current spatial location information in real time.

[0071] The terminal device can be a smartphone, a dedicated portable terminal, etc., responsible for integrating and initially compressing the image data captured by the camera and the spatial location information collected by the positioning module. The terminal then transmits the integrated and compressed data to the server in real time and continuously via high-speed wireless communication networks such as 5G.

[0072] As an information processing device, the server possesses high-performance computing resources (such as a cloud server equipped with a GPU), enabling it to perform the following operations and analyses on the received data. Specifically, the server first uses image analysis algorithms (such as a multi-class object detection algorithm based on the YOLO model) to identify multiple categories of objects in the image, including traffic signals, zebra crossings, obstacles, pedestrians, and stairs. The server further utilizes geographic information databases (such as high-precision map data APIs) and combines them with the user's GPS spatial location information to perform spatial positioning and intelligent correction on the detection results, improving the accuracy of environmental understanding.

[0073] Subsequently, the server converts the structured environmental information into prompts and inputs them into a generative AI model. This generative AI model can be a natural language processing model (such as a large-scale language model), dynamically generating natural language feedback content suitable for the current scenario based on the environmental analysis results and the prompts. For example, when a red light is detected ahead, the server might generate the prompt: "The intersection light is red; please generate a stop-and-wait prompt for the user." The generative AI model's output feedback might be: "The light is red; please wait patiently at the intersection."

[0074] The server encodes the natural language feedback in real time and sends it to the terminal device via the information communication network. Upon receiving the feedback, the terminal uses a built-in or external speech synthesis engine (such as a TTS speech synthesis module, like Baidu TTS or iFlytek TTS) to convert the text feedback into natural speech in real time, and then plays it to the user via Bluetooth, wired headphones, speakers, or other output devices. In special scenarios, such as when there are stairs or potential dangers in front of the user, the terminal can simultaneously activate physical stimulation devices such as vibration motors to help the user understand and perceive changes in the environment.

[0075] The system's advantage lies in its ability to provide fast, accurate, and personalized natural language and multimodal feedback based on multi-source environmental perception data, through efficient data processing and generative artificial intelligence models on the server side, significantly improving the independence and safety of users such as the visually impaired.

[0076] Specific application examples include:

[0077] When a user reaches a road intersection, the server uses camera images and GPS information to determine that the light ahead is red, and outputs "Red light ahead, please stop and wait" through a generative artificial intelligence model.

[0078] When a staircase appears in front of the user, the server parses and generates the prompt message: "There is a staircase 10 meters ahead of the user. Please provide a safety reminder." The server then generates the feedback "There is a staircase 10 meters ahead. Please walk carefully and slowly," and reminds the user through voice and vibration on the terminal.

[0079] The implementation of this invention does not rely on hardware and software from specific manufacturers. Image acquisition devices, positioning modules, terminal devices, and servers can all use existing general-purpose electronic components. High-precision map APIs and speech synthesis software can be selected according to requirements. Generative artificial intelligence models can also be deployed based on mainstream deep learning platforms. Prompt statements can be automatically generated by the server according to the environment and input into the model, realizing fully automated and real-time human-computer interaction assistance.

[0080] use Figure 11 The processing procedure is explained.

[0081] Step 1:

[0082] The user wears a wearable camera and positioning module, and activates the assistance system through the terminal. The input consists of real-time images of the user's surroundings and spatial location information. The user gains a realistic understanding of the external environment through actual walking and operation. The output consists of image data captured by the camera and spatial location information obtained by the positioning module. Specific actions in this stage include the user wearing the device, the terminal application starting, and device self-testing.

[0083] Step 2:

[0084] The terminal receives image data and spatial location information in real time from the camera and positioning module. The input consists of image data and spatial location information. The terminal compresses the raw data (e.g., images use JPEG compression format, and location information is formatted as standard coordinate data) and synchronizes the time, then assembles the data into packets for transmission. The output is a compressed and packaged data packet. Specific actions include the terminal automatically calling the corresponding hardware interface to read the data, compressing the data, and adding time stamps.

[0085] Step 3:

[0086] The terminal uploads data packets to the server in real time via a wireless communication network (such as 5G). The input consists of compressed and packaged image and location data. The terminal prioritizes data packets and uses retransmission mechanisms based on network quality control to ensure stable data transmission. The output is the data packet successfully sent to the server. Specific actions include establishing a network connection, sending data in batches, retransmitting lost packets, and monitoring upload progress.

[0087] Step 4:

[0088] The server receives data packets and performs data decoding and environmental analysis. The input is a compressed data packet uploaded by the terminal. The server first decodes the data to obtain the original image and spatial location information. Then, it uses a multi-class object detection algorithm (such as YOLO) to analyze the image, identifying traffic signals, obstacles, stairs, etc., and uses a geographic information database combined with GPS information for spatial correction and environmental verification. The output is structured environmental information, such as "Red light signal detected ahead." Specific actions include data unpacking, image recognition algorithm invocation, and geographic location matching.

[0089] Step 5:

[0090] The server uses a generative artificial intelligence model to generate natural language feedback suitable for the current scenario based on current environmental information and preset prompts. The input consists of structured environmental information and corresponding prompts (e.g., "The intersection light is red; please generate a stop-and-wait prompt for the user."). The server inputs these prompts into the generative AI model, performs natural language generation inference, and outputs structured, easily understandable text feedback. For example, it might generate a prompt like "The traffic light is red; please stop and wait." Specific actions include automatic prompt generation, AI model invocation, and text post-processing.

[0091] Step 6:

[0092] The server sends the generated text feedback content to the terminal device via a communication network. The input is natural language feedback text. The server encodes and packets the feedback content according to network protocols, and uses priority to ensure the real-time performance and integrity of the feedback content. The output is the feedback data packet uploaded to the terminal. Specific actions include data encoding, network priority transmission, and transmission confirmation.

[0093] Step 7:

[0094] The terminal receives feedback from the server and performs speech synthesis and multimodal alerts. The input is natural language feedback text. The terminal converts the text into speech signals using a TTS (Text-to-Speech) engine and plays it to the user through headphones or speakers. Simultaneously, when the feedback is a high-risk warning, the terminal activates a vibration motor for a physical alert. The output consists of the played voice prompts and physical stimulation signals. Specific actions include using the TTS engine to synthesize speech, outputting audio via Bluetooth or wired connection, and activating the vibration device.

[0095] Step 8:

[0096] Users receive feedback from the system through hearing and touch, and adjust their behavior accordingly. Input is either voice or vibration signals. Based on the feedback, users adjust their walking movements, such as stopping at a red light or slowing down before stairs. Output is the user's actual safety action. Specific actions include listening to voice commands, feeling physical cues, and responding behaviorally.

[0097] Application Example 1

[0098] The process flow corresponding to the specific processing in Use Case 1 will be described below. The various parts of the system described below are implemented by the data processing device 12 and the intelligent device 14. Furthermore, the data processing device 12 is referred to as the "server" and the intelligent device 14 is referred to as the "terminal".

[0099] This invention aims to address the difficulties faced by visually impaired or elderly users in autonomously, safely, and efficiently obtaining navigation and environmental information in real-world environments (such as shops or outdoor locations). Existing navigation systems largely rely on visual information, failing to provide real-time, personalized assistance tailored to the emotional state of visually impaired users or those with limited information-gathering abilities. Furthermore, due to frequent environmental changes, traditional static maps cannot meet users' needs for real-time positioning and risk alerts, severely impacting their autonomous actions and safe travel.

[0100] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Application Example 1 is achieved by the following means.

[0101] In this invention, the server includes a device for analyzing image and location information acquired through a wearable imaging device, and combining target recognition algorithms with geographic information for environmental perception; it further includes a device for generating natural language navigation and safety feedback information based on the analysis results and the user's emotional state using a generative artificial intelligence model; and it also includes a device for analyzing the user's emotional and behavioral state in real time, and adaptively adjusting the feedback content and output method based on this state. This provides users with real-time environmental perception, personalized navigation, and multimodal safety feedback support based on multi-source data fusion, enabling visually impaired and elderly users to autonomously, safely, and efficiently complete destination navigation and risk avoidance.

[0102] "Imaging component" refers to an electronic imaging device that can acquire image information of the user's surrounding environment and can be installed on wearable devices.

[0103] "Location information acquisition device" refers to a hardware device that can acquire user location information in real time, including but not limited to a global positioning system module or other data acquisition equipment that can be used for geographic positioning.

[0104] "Information and communication network" refers to the network infrastructure that enables high-speed, low-latency data transmission, including but not limited to fifth-generation mobile communication systems and wireless networks.

[0105] "Information processing device" refers to a computing device capable of receiving, storing, analyzing and processing data from a terminal, including cloud servers or local servers.

[0106] "Target recognition algorithm" refers to a computational method that uses deep learning or other artificial intelligence technologies to automatically detect and recognize objects in the environment based on acquired image data.

[0107] "Geographic information" refers to data that reflects the user's actual physical location and the characteristics of the surrounding environment, including electronic map information, building layout, etc.

[0108] A "scene analysis device" refers to a device that performs structured analysis of the user's environment based on image and geographic information to obtain environmental semantic content.

[0109] "Generative AI model" refers to an AI system or algorithm model that can automatically generate natural language descriptions or suggestions based on input data.

[0110] "Feedback information" refers to navigation, reminders, and other content generated by the system based on environmental analysis and user status, used to guide user actions or indicate risks.

[0111] "Output device" refers to a device that can present feedback information to the user in the form of voice, text, vibration, images, etc., including headphones, speakers, screens, etc.

[0112] "Emotional state detection device" refers to a sensor and processing unit that can collect and analyze users' physiological and behavioral signals to identify their current emotional state.

[0113] "Adaptive adjustment" refers to the process by which a system dynamically adjusts the content and presentation of feedback information based on the user's emotional state and behavioral characteristics to achieve personalized support.

[0114] The embodiments of the present invention are described below.

[0115] This invention relates to a system and method for assisting visually impaired or elderly users in achieving autonomous and safe navigation and real-time environmental awareness in real-world environments (such as shops or roads). The system comprises a terminal device (e.g., a smartphone or wearable device), an imaging component (e.g., a wearable camera), a location information acquisition device (e.g., a GPS module), an output device (e.g., headphones or speakers), an emotion state detection device (e.g., a heart rate sensor, a microphone, etc.), and a server (cloud or local server).

[0116] The terminal includes hardware such as a wearable camera, GPS module, heart rate sensor, and microphone, as well as corresponding control software. Users wear wearable devices (such as smart glasses, chest-mounted cameras, etc.) and move around in the environment. The terminal control software periodically collects environmental image data, location information, and user audio and physiological signals, and transmits them to the server at high speed and low latency via the fifth-generation mobile communication system (5G).

[0117] On the server side, the following software is deployed using a computer system (such as a high-configuration cloud server): target recognition algorithms (such as YOLOv5), emotion state detection algorithms, geographic information parsing modules (such as calling geographic information service APIs), and generative artificial intelligence models (such as an automatic content generation engine based on a general large-scale language model). The server first performs target recognition on the uploaded image information, combines location information and geographic information data to analyze the user's environment, and further analyzes the user's emotional and physiological data uploaded by the emotion state detection device to determine the user's current psychological and behavioral state.

[0118] Based on the above analysis results, the server writes prompts and automatically generates natural language navigation guidance, risk warnings, and other auxiliary feedback suitable for the current scenario and user state using a generative artificial intelligence model. The generated feedback information may include specific content such as "Fruit area is 10 meters ahead to the left, please avoid it" or "We detected that you are nervous, please wait for the signal here." The server can flexibly adjust the tone and content of the feedback to adapt to different emotional and behavioral states of the user, such as anxiety, confusion, or satisfaction.

[0119] The feedback information is returned to the terminal via the 5G network. The terminal then calls a TTS (Text-to-Speech) module (such as Google TTS, iFlytek TTS, etc.) to convert the text into speech, which is then output to the user in real time through headphones or speakers. When necessary, a vibration motor provides auxiliary reminders. The system ensures that visually impaired or elderly users can obtain environmental and navigation information entirely through voice and touch, without the need for visual operation or complicated input.

[0120] This invention can also be configured with diverse hardware and software based on user needs and practical application scenarios. For example, the terminal device can be smart glasses, wearable bracelets, smartphones, etc., and the server can be local hardware or a remote cloud platform. The target recognition algorithm can adopt mainstream deep learning models, and the generative artificial intelligence model can be any qualified natural language generation network. Specific Implementation

[0122] For example, users can wear a head-mounted camera and Bluetooth headset while shopping in a supermarket. The device captures images of shelves and aisles, obtains GPS location and user heart rate, and then transmits them to the server via 5G. The server uses a target recognition algorithm to identify shelves, obstacles, and user location, generating a prompt such as: "There is a vegetable section 10 meters to your left, please go straight." If the system detects an increased heart rate and the emotion analysis module determines it as tension, it will generate appropriate reassuring guidance: "Please don't be nervous, it's safe ahead, please walk slowly." The prompt is then converted into speech by the TTS module and output to the user.

[0123] For example, if a user is crossing the street and the camera detects a traffic light while the user is anxious, the server generates the message: "The current light is red. Please wait where you are. I will notify you when the light turns green."

[0124] Typical examples of prompts for generative artificial intelligence models include:

[0125] "Please design a system that can recognize the environment and generate navigation instructions for the user to reach their destination."

[0126] "Please generate assistive prompts for visually impaired users to safely locate designated products in the store."

[0127] "Based on the user's emotions and surrounding environment, generate voice navigation suggestions that are suitable for the user's current personal state."

[0128] Through the system described in this invention, visually impaired and elderly users can achieve barrier-free, autonomous, and highly safe mobility and activity experiences, greatly improving the convenience and quality of life.

[0129] use Figure 12 The processing procedure is explained.

[0130] Step 1:

[0131] Users wear a wearable camera and positioning module while walking in their environment. The device periodically captures real-time images via the camera and obtains its current location via the positioning module. The input is the physical environment surrounding the user, and the output is image data and positioning data.

[0132] Step 2:

[0133] The terminal performs preliminary processing on the collected image data and location information, while simultaneously using emotion detection devices (such as heart rate sensors and microphones) to collect the user's physiological and audio data. The input consists of raw images, location data, and raw emotion signals; the output is a structured data packet.

[0134] Step 3:

[0135] The terminal sends the structured data packets to the server via the 5G information communication network. The input is the processed data packets, and the output is the user's environment and status information received by the server.

[0136] Step 4:

[0137] After receiving the data packet, the server uses an object recognition algorithm to detect objects in the image data, combines this with location data for geographic information parsing, and simultaneously assesses the user's state using a sentiment analysis algorithm. The input is a structured data packet (containing image, location, and sentiment data), and the output is a structured result containing environmental objects, location information, and the user's emotional state.

[0138] Step 5:

[0139] Based on the parsed environmental objects, location information, and user emotional state, the server sends prompts to the generative AI model. The model then generates natural language navigation guidance or safety tips suitable for the scenario and user's state. The inputs are scenario analysis and user state data, and the output is the generated personalized text feedback.

[0140] Step 6:

[0141] The server sends the generated text feedback to the terminal via an information communication network. The input is the text feedback data, and the output is the prompt text received by the terminal.

[0142] Step 7:

[0143] The terminal uses a speech synthesis module to convert received text feedback into speech, which is then output to the user through headphones or a speaker. In case of emergency or when the user is nervous, vibration alerts can also be activated simultaneously. The input is text feedback, and the output is a voice prompt or a multimodal feedback signal.

[0144] Step 8:

[0145] Users make corresponding movement or behavioral decisions based on received voice or vibration cues. The input is a voice cue or vibration signal, and the output is the user's movement action (such as moving forward, turning, or stopping).

[0146] Alternatively, an emotion engine for inferring user emotions can be combined. That is, the specific processing unit 290 can also use the emotion-specific model 59 to infer user emotions and perform specific processing using user emotions.

[0147] Example 2

[0148] The flow of a specific process in Example 2 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart device 14. The data processing device 12 will be referred to as the "server," and the smart device 14 as the "terminal."

[0149] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Embodiment 2 is achieved by the following means.

[0150] In this invention, the server includes a device for receiving and parsing environmental images, location coordinates, and emotional state data obtained by the user from physiological sensors or related devices; a device for using image analysis algorithms to perform target recognition and environmental structure analysis, combining geographic coordinate information to obtain the geographic environment, and using emotion analysis algorithms to accurately assess the user's emotional state; and a device for automatically generating feedback information based on the collected and analyzed multi-source data using a generative artificial intelligence model, and adaptively adjusting the feedback content and output method according to the emotional state, sending the feedback to the user terminal through a communication network, and outputting it in multimodal ways such as voice and touch. This enables multi-dimensional intelligent analysis of user environmental perception and personal emotional state, and provides high-risk users with personalized, timely, and diversified safety assistance prompts adapted to their actual situation, significantly improving the level of independent and safe travel or movement.

[0151] "Imaging device" refers to an electronic device that can acquire image information of the surrounding environment, including but not limited to cameras and camcorders.

[0152] "Location information acquisition device" refers to an electronic device capable of collecting spatial geographic coordinate data, including but not limited to GPS receiver modules and geographic information sensors.

[0153] "Physiological information sensors" refer to sensing devices used to detect users' physiological state data, including but not limited to heart rate detectors, skin conductance sensors, and voice analysis modules.

[0154] "Emotional state information" refers to a digital description of a user's current emotions, including tension, calmness, and anxiety, generated by analysis algorithms based on the user's physiological data and behavioral signals.

[0155] "Communication network" refers to the network infrastructure used for data transmission between different devices or systems, including but not limited to wired and wireless communication methods.

[0156] "Information processing device" refers to a computing system that has the ability to receive, parse, and analyze data, and is used to process environmental information, location information, and emotional data.

[0157] "Image analysis algorithm" refers to the computational method used to analyze and identify target objects and environmental features from acquired image data, including but not limited to deep learning models.

[0158] "Emotion analysis algorithms" refer to algorithms that use users' physiological or behavioral data to identify and evaluate emotions, including but not limited to machine learning models and signal processing techniques.

[0159] "Generative AI models" refer to AI systems that are trained on large-scale data and can generate structured text or instructions based on input content, including but not limited to language models.

[0160] "Feedback information" refers to personalized information generated based on a comprehensive analysis of environmental, location, and emotional data, used to guide and prompt users to take safe actions.

[0161] "User terminal" refers to an electronic device that is carried or worn by the user and used to receive, process and output feedback information.

[0162] "Wearable carriers" refer to physical structures that can carry imaging devices and related electronic modules and are worn on the body by users, including glasses, helmets, clothing, etc.

[0163] "Mobile communication methods" refer to communication methods that use wireless signals to exchange data between different mobile devices, including various cellular networks, wireless local area networks, etc.

[0164] A “speech synthesis device” refers to a device or system that can automatically convert text information into audio signals for output, in order to provide voice prompts to users.

[0165] "Haptic cues" refer to feedback methods that convey warnings or information to users through physical means such as vibration and pressure.

[0166] To facilitate others' understanding and implementation of this invention, the embodiments of this invention are now described in detail.

[0167] This system provides real-time safety assistance information to users, especially high-risk groups such as the visually impaired, through three main functional modules: environmental perception, emotion recognition, and intelligent feedback. The system includes terminals (user terminals), various sensing devices, information processing devices (servers), and a communication network.

[0168] Users wear wearable devices, such as glasses, helmets, or clothing, which integrate imaging devices (such as cameras), location information acquisition modules (such as GPS chips), and physiological information sensors (such as heart rate detectors and voice collection modules). The terminal is equipped with hardware such as 5G or other mobile communication modules, speech synthesis devices, and vibration motors.

[0169] The terminal automatically collects the user's current environmental images, spatial location information, and physiological signals. Simultaneously, it uses local or embedded emotion analysis algorithms to preprocess the physiological data, generating preliminary emotional state information. All collected data is packaged by the terminal and sent to the server via a 5G communication network.

[0170] Servers, as information processing devices, include high-performance computing platforms. They run software modules such as image parsing algorithms (e.g., the YOLOv5 model based on deep learning), geographic information parsing software (e.g., general map APIs), sentiment analysis algorithms (e.g., physiological and voiceprint recognition models based on machine learning), and generative artificial intelligence models (e.g., large-scale language generation models).

[0171] The server parses the received data, first using image analysis algorithms to identify road conditions, obstacles, traffic lights, vehicles, and other information in the image; then, combining GPS location information and geographic information APIs, it determines the user's geographical environment and traffic elements; further, through emotion analysis algorithms, it accurately portrays the user's subjective emotional state, such as tension and anxiety. After analyzing and integrating the above multi-source data, the server uses a generative artificial intelligence model to automatically generate personalized, context-appropriate feedback information based on the current danger level, environmental characteristics, and user emotions, and flexibly adjusts the content and expression of the prompts.

[0172] Feedback information is returned to the terminal via the communication network. Upon receiving the feedback, the terminal calls a speech synthesis device to convert the text message into speech in real time and plays it to the user through a speaker or headphones. If the user is in a high-risk situation, a vibration motor can also be activated to achieve multimodal linkage prompts, ensuring that the user can be informed and adopt safety advice in a timely manner in different environments.

[0173] This invention can be applied to various scenarios, such as daily travel for visually impaired individuals, independent activities for the elderly, and temporary emotional disturbances. For example, when a user experiences anxiety in complex traffic environments such as intersections, the system can consider both the real-time environment and the user's state: "Don't worry, the light is red now. Please wait until the green light and I will remind you to cross." Or, when an obstacle is detected ahead: "There is an obstacle ahead. Please turn right."

[0174] The hardware used in this system includes: a wearable camera, a GPS module, a heart rate detector, an integrated communication module, a speaker, and a vibration motor. The software used includes: the YOLOv5 image processing model, a general map API, a sentiment analysis algorithm, a generative artificial intelligence model (large language model), and a speech synthesis program (such as a TTS engine).

[0175] Here are some examples of prompts for generative artificial intelligence models:

[0176] Please explain the overall data flow and algorithm processing of the visually impaired user safe walking assistance system.

[0177] Please provide specific application examples of the emotion engine in a real-time assistance system and an explanation of the feedback mechanism.

[0178] Please provide examples to analyze how this system provides multimodal support for user perception and decision-making when a visually impaired person is crossing the road or encountering an obstacle.

[0179] use Figure 13 The processing procedure is explained.

[0180] Step 1:

[0181] Data is collected at the terminal.

[0182] Input: Environmental images (acquired via camera), spatial location information (obtained via GPS module), and user physiological signals (such as heart rate, voice, etc.).

[0183] The terminal calls the camera API to capture static images once per second, while the GPS module locates the user in real time. The physiological information sensor detects the user's relevant physiological data and uses a local sentiment analysis algorithm to generate emotional state information.

[0184] Output: Acquired image data, location information, and emotional state information.

[0185] Step 2:

[0186] Terminal-level data packaging and transmission.

[0187] Input: Image data, location information, and emotional state information output from step 1.

[0188] The terminal uses a data packaging module to format and compress the above three types of data, package them into data packets, and then calls the 5G communication module to send the data to the server via the network.

[0189] Output: Data packets sent to the server via the 5G network.

[0190] Step 3:

[0191] Server-side data parsing and analysis.

[0192] Input: Data packets from the endpoint (containing image, location information, and emotional state information).

[0193] The server first calls the unpacking program to extract the raw data, then uses the YOLOv5 model to perform object recognition (such as vehicles, traffic lights, and obstacles) on the image data, uses the map API to perform geographic environment recognition on the location data, and uses sentiment analysis algorithms to perform more refined analysis of the user's emotional state.

[0194] Output includes information on objects identified in the environment, environmental characteristics of the user's geographical location, and detailed sentiment analysis results.

[0195] Step 4:

[0196] The server generates feedback information.

[0197] Input: Environmental object information, geographical features, and sentiment analysis results obtained in step 3.

[0198] The server integrates data from multiple sources and uses a generative artificial intelligence model to automatically generate context-appropriate feedback information, and automatically adjusts the message content and tone according to the user's emotions.

[0199] Output: Adaptive, personalized feedback information (text format).

[0200] Step 5:

[0201] The server sends feedback information to the terminal.

[0202] Input: Feedback information generated in step 4.

[0203] The server packages the feedback information and pushes it to the user's terminal in real time via the communication network.

[0204] Output: The feedback data packet received by the terminal.

[0205] Step 6:

[0206] Terminal outputs feedback, and the user receives the notification.

[0207] Input: The feedback data packet received in step 5.

[0208] The terminal parses the packet content, calls the local speech synthesis module to convert text feedback into speech, and plays it through a speaker or headphones. Simultaneously, depending on the feedback content, it activates a vibration motor as needed to achieve multimodal output. Users receive timely environmental cues and action suggestions through hearing and touch.

[0209] Output: Voice prompts and / or vibration cues provided to the user to help them act safely and autonomously.

[0210] Application Example 2

[0211] The process flow corresponding to the specific processing in Use Case 2 will be described below. The various parts of the system described below are implemented by the data processing device 12 and the intelligent device 14. In addition, the data processing device 12 is referred to as the "server" and the intelligent device 14 is referred to as the "terminal".

[0212] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Application Example 2 is achieved by the following means.

[0213] In this invention, the server includes a device for receiving image information, spatial location information, and biometric information collected by an information acquisition device; a device for comprehensively analyzing the image information and spatial location information using object recognition algorithms and spatial discrimination algorithms; and a device for generating personalized feedback information by using a generative artificial intelligence model, taking pre-set prompt statements as input, and combining the user's state recognition results. This enables efficient fusion and analysis of various data, automatically generating feedback information that conforms to the working environment and personnel state, and outputting it in a timely manner in the form of voice or vibration. This allows workers to quickly receive safety alerts and emotional support, reducing accidents and improving work efficiency.

[0214] "Information acquisition device" refers to a device that can be worn by a user and used to collect image information, spatial location information and biometric information, including but not limited to wearable devices such as smart glasses and smartwatches.

[0215] "Image information" refers to visual data acquired in real time through the image acquisition module in the information acquisition device, which is used to reflect the visual conditions of the user's surrounding environment.

[0216] "Spatial location information" refers to data acquired through the positioning module (such as the Global Positioning System module) of an information acquisition device, which can represent the specific latitude, longitude, or location information of the user's location in space.

[0217] "Bioinformation" refers to data that characterizes a user's physiological or emotional state, including but not limited to information obtained through sensors or analysis modules such as heart rate, skin conductance, facial expressions, and voice tone.

[0218] "Communication infrastructure" refers to network facilities used to enable data transmission between information acquisition devices and information processing devices, including mobile communication networks, wireless local area networks, etc.

[0219] "Information processing device" refers to a computing device used to receive, store and analyze data from information acquisition devices, and perform processing such as object recognition, spatial discrimination and state recognition.

[0220] "Object recognition processing" refers to the process of using image recognition algorithms to detect and classify objects in image information, in order to determine whether there are dangerous objects or other important targets.

[0221] "Spatial discrimination processing" refers to the process of determining the nature and risk level of a user's environment by combining spatial location information.

[0222] "State recognition processing" refers to the process of analyzing a user's current emotional, health, or stress state based on bioinformatics.

[0223] "Generative AI processing unit" refers to an AI module that can automatically generate personalized feedback information based on input prompts and multimodal analysis results, such as a generative language model.

[0224] "Prompt statements" refer to pre-set text information used to guide generative artificial intelligence processing units in generating feedback content, which includes elements such as the current environment, objects, and emotions.

[0225] "Feedback information" refers to suggestions, prompts, or warnings generated by an information processing device and output to the user in the form of text, voice, or vibration.

[0226] "Voice output" refers to the process of converting feedback information into sound through speech synthesis technology, which is then played back by the information acquisition device for the user to hear.

[0227] "Vibration output" refers to the process by which the vibration module of an information acquisition device transmits feedback information to the user in the form of vibration.

[0228] The embodiments of the present invention are described below.

[0229] This system mainly consists of an information acquisition device (i.e., a terminal), an information processing device (i.e., a server), and communication infrastructure. Users wear wearable information acquisition devices, such as smart glasses or smartwatches, which include a built-in camera, a GPS module, a physiological signal acquisition module (such as a heart rate sensor, microphone, and accelerometer), a speaker, and a vibration motor. The server is a data processing device with high-performance computing capabilities and can be deployed locally or in the cloud. The communication infrastructure can utilize mobile communication networks (such as 4G / 5G) or wireless LANs to achieve real-time data transmission between the terminal and the server.

[0230] The terminal uses a camera to collect real-time image information of the user's surroundings, obtains spatial location information through a GPS module, and collects biological information such as heart rate, voice tone, and facial expressions through physiological signal sensors. Internally, the terminal runs data acquisition and preliminary analysis software, such as calling open-source deep learning models like TensorFlow to make preliminary judgments about emotional state. The terminal packages the acquired image, spatial location, and biological information into data packets and sends them to the server via wireless communication methods such as 5G.

[0231] After receiving the data uploaded by the client, the server first uses object detection algorithms such as YOLOv5 (based on deep learning platforms such as PyTorch) to perform object recognition on the image, detecting the presence of dangerous objects (such as forklifts, chemical drums, or other moving devices or obstacles). Simultaneously, the server compares the location information with a factory geographic information database to determine the security level of the user's location. The server further analyzes the uploaded biometric information, combining it with a pre-trained emotion recognition model (such as the TensorFlow emotion model) to determine whether the user is currently in a state of tension, stress, fatigue, or calm.

[0232] The server integrates generative artificial intelligence models, such as natural language generation models based on prompts (e.g., large pre-trained language models), and takes the above analysis results as input (including results from object detection, spatial discrimination, and emotion recognition), combining them with predefined prompts to generate specific feedback information. This feedback information can be personalized content such as safety tips, operational suggestions, or emotional reassurance.

[0233] The server transmits the feedback information to the terminal in real time via the communication infrastructure. Upon receiving the feedback, the terminal uses a TTS (Text-to-Speech) module to convert the text-based feedback into natural speech, which is then played to the user through a speaker. If necessary, a vibration motor can be used to accompany the vibration, enabling multimodal warnings and notifications. The terminal can automatically adjust the tone, speed, and output method of the prompts based on the content characteristics. This allows users to receive timely important safety information and emotional support in the work environment, effectively improving safety and work efficiency.

[0234] Here's a specific example: A user is approaching a hazardous area in a factory, and the emotion model detects that the user's physiological information indicates a state of tension. At this point, the terminal sends images, location information, and physiological signals to the server. The server uses YOLOv5 to detect a forklift approaching and inputs the information "forklift detected approaching and user is tense" into the generative AI model. It then combines this information with preset prompts to generate a realistic voice prompt, such as: "A forklift is approaching ahead. Please be careful and remain calm."

[0235] Here are some examples of prompt statements:

[0236] YOLOv5: Detects objects in a factory environment.

[0237] TensorFlow Emotion Model: Analyzes current emotions based on physiological data.

[0238] Scene:

[0239] Object detection result: Dangerous equipment has been detected.

[0240] Emotional state: The user is in a state of tension.

[0241] Please generate Chinese voice prompts for the above scenarios, combining object detection and emotional state.

[0242] Output example: Dangerous equipment is approaching ahead. Please be careful, take a deep breath, and stay calm.

[0243] The solution of this invention combines multiple sensory data and multi-level artificial intelligence analysis to provide users with personalized, real-time safety and emotional feedback, which is highly practical and advanced.

[0244] use Figure 14 The processing procedure is explained.

[0245] Step 1:

[0246] When a user wears a terminal (such as smart glasses), the terminal automatically activates its camera, GPS module, and physiological signal sensors to collect real-time image information, spatial location information, and biological information about the current environment. The input consists of the user's surrounding scene, location information, and physiological parameters; the output is an encapsulated raw data packet. The terminal synchronously samples from various sensors, calls sensor interfaces, and synthesizes a unified data structure.

[0247] Step 2:

[0248] The terminal merges and performs preliminary processing on the collected data, packaging image information, spatial location information, and biological information into a single packet. This packet is then connected to a high-speed wireless network module, such as 5G, and compressed and encapsulated using encryption and data protocols. The input is the raw collected data; the output is the packaged data packet. The terminal establishes a data connection and performs encryption, compression, packaging, and uploading actions.

[0249] Step 3:

[0250] The server receives data packets uploaded by the terminal and uses image processing algorithms (such as YOLOv5) to perform target detection on the image information, identifying dangerous objects or equipment in the scene. Simultaneously, the server performs geographic verification of the spatial location information, comparing it with the user's current area attributes in a factory layout database. The server inputs biometric information into an emotion recognition model (such as a TensorFlow-based emotion model) to assess the user's emotional and physiological state. The input is the uploaded packaged data packet; the output is the detection results (such as whether there are dangerous objects, the user's location, and the user's emotional state). The server performs calculations such as target detection, database queries, and emotion analysis.

[0251] Step 4:

[0252] Based on target detection results, location information, and sentiment state, the server automatically generates a multimodal analysis report and constructs the input for the generative AI model according to a set template and prompts. The server calls the generative AI model and outputs personalized feedback based on multiple input information. The input consists of various analysis results and prompts; the output is natural language feedback text. The server automatically concatenates the prompts, calls the model API, and generates the feedback content.

[0253] Step 5:

[0254] The server pushes the generated feedback text to the terminal via the communication network. The input is the generated feedback content; the output is the feedback message sent to the terminal. The server encapsulates, compresses, and uploads the data to ensure the secure transmission of feedback information.

[0255] Step 6:

[0256] The terminal receives feedback text, invokes its local TTS speech synthesis module to convert the text into speech, and plays it to the user through the speaker. Simultaneously, vibration output can be activated depending on the feedback type. The input is the received feedback message; the output is a user-perceptible voice reminder and / or vibration cue. The terminal parses the content, performs speech synthesis and multimodal prompts, and records the reminder log.

[0257] Step 7:

[0258] Users adjust their behavior accordingly based on the terminal's voice and vibration prompts, such as moving away from dangerous areas or accepting suggested rest. Input consists of the terminal's output prompts; output is the user's immediate reaction and safe actions. By listening to instructions and providing action feedback or behavioral changes, users improve safety and work efficiency.

[0259] The specific processing unit 290 sends the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires sound representing user input regarding the result of the specific processing. The control unit 46A sends the sound data representing user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0260] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0261] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart device 14 or external devices, and the smart device 14 acquires or collects information required for processing from the data processing device 12 or external devices.

[0262] For example, the collection unit is implemented by the control unit 46A of the smart device 14 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the smart device 14 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the output device 40 of the smart device 14 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0263] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the smart device 14.

[0264] Second Implementation Method

[0265] Figure 3 An example of the configuration of the data processing system 210 according to the second embodiment is shown.

[0266] like Figure 3 As shown, the data processing system 210 includes a data processing device 12 and smart glasses 214. A server can be cited as an example of the data processing device 12.

[0267] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0268] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, and communication I / F 44 are also connected to the bus 52.

[0269] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0270] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to capture images of the user 20's surroundings (e.g., the field of view defined by an angle equivalent to the field of vision of an average healthy person).

[0271] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0272] Figure 4 This illustrates an example of the main functions of the data processing device 12 and the smart glasses 214. For example... Figure 4 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0273] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0274] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. In the emotion inference function (emotion-specific function) using the emotion-specific model 59, various inferences and predictions related to the user's emotions are performed, including inferences and predictions of the user's emotions, but this is not limited to this example. Furthermore, emotion inference and prediction may also include, for example, emotion analysis (parsing).

[0275] In the smart glasses 214, the processor 46 performs reception and output processing. The memory 50 stores the reception and output program 60. The processor 46 reads the reception and output program 60 from the memory 50 and executes the read reception and output program 60 on the RAM 48. The reception and output processing is implemented by the processor 46 operating as a control unit 46A according to the reception and output program 60 executed on the RAM 48. Furthermore, the smart glasses 214 has the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and these models can also be used to perform the same processing as the specific processing unit 290.

[0276] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart glasses 214. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0277] Example 1

[0278] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0279] Application Example 1

[0280] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0281] Example 2

[0282] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0283] Application Example 2

[0284] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0285] The specific processing unit 290 sends the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A outputs the result of the specific processing to the speaker 240. The microphone 238 acquires sound input representing the user's input regarding the result of the specific processing. The control unit 46A sends the sound data representing the user's input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0286] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0287] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or external devices, and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or external devices.

[0288] For example, the collection unit is implemented by the control unit 46A of the smart glasses 214 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the smart glasses 214 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 of the smart glasses 214 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0289] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the smart glasses 214.

[0290] Third Implementation Method

[0291] Figure 5 An example of the configuration of the data processing system 310 according to the third embodiment is shown.

[0292] like Figure 5 As shown, the data processing system 310 includes a data processing device 12 and a head-mounted terminal 314. A server can be cited as an example of the data processing device 12.

[0293] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0294] The head-mounted terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, display 343, and communication I / F 44 are also connected to the bus 52.

[0295] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0296] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to capture images of the user 20's surroundings (e.g., the field of view defined by an angle equivalent to the field of vision of an average healthy person).

[0297] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0298] Figure 6 This illustrates an example of the main functions of the data processing device 12 and the head-mounted terminal 314. For example... Figure 6 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0299] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0300] The memory 32 stores the data generation model 58 and the emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290.

[0301] In the head-mounted terminal 314, the processor 46 performs the acceptance / output processing. The memory 50 stores the acceptance / output program 60. The processor 46 reads the acceptance / output program 60 from the memory 50 and executes the read acceptance / output program 60 on the RAM 48. The acceptance / output processing is implemented by the processor 46 operating as a control unit 46A according to the acceptance / output program 60 executed on the RAM 48.

[0302] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the head-mounted terminal 314. In the following description, the data processing device 12 will be referred to as the "server" and the head-mounted terminal 314 will be referred to as the "terminal".

[0303] Example 1

[0304] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0305] Application Example 1

[0306] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0307] Example 2

[0308] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0309] Application Example 2

[0310] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0311] The specific processing unit 290 sends the result of the specific processing to the head-mounted terminal 314. In the head-mounted terminal 314, the control unit 46A outputs the result of the specific processing to the speaker 240 and the display 343. The microphone 238 acquires sound input representing the result of the specific processing. The control unit 46A sends the sound data representing the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0312] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 includes prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0313] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the head-mounted terminal 314, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the head-mounted terminal 314. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the head-mounted terminal 314 or external devices, and the head-mounted terminal 314 acquires or collects information required for processing from the data processing device 12 or external devices.

[0314] For example, the collection unit is implemented by the control unit 46A of the head-mounted terminal 314 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the head-mounted terminal 314 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12 to analyze the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12 to generate a menu using a generation AI. For example, the serving unit is implemented by the speaker 240 and display 343 of the head-mounted terminal 314 or the specific processing unit 290 of the data processing device 12 to provide the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0315] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the head-mounted terminal 314.

[0316] Fourth Implementation Method

[0317] Figure 7 An example of the configuration of the data processing system 410 according to the fourth embodiment is shown.

[0318] like Figure 7 As shown, the data processing system 410 includes a data processing device 12 and a robot 414. A server can be cited as an example of the data processing device 12.

[0319] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0320] Robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, controlled object 443, and communication I / F 44 are also connected to the bus 52.

[0321] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0322] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to photograph the area around robot 414 (e.g., the field of view defined by a perspective equivalent to the field of vision of an average healthy person).

[0323] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0324] The controlled object 443 includes a display device, LEDs (light-emitting diodes) for the eyes, and motors for driving the arms, hands, and feet. The posture or movement of the robot 414 is controlled by controlling the motors in the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. In addition, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0325] Figure 8 This illustrates an example of the main functions of the data processing device 12 and the robot 414. For example... Figure 8 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0326] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0327] The memory 32 stores the data generation model 58 and the emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290.

[0328] In robot 414, the processor 46 performs the acceptance and output processing. The memory 50 stores the acceptance and output program 60. The processor 46 reads the acceptance and output program 60 from the memory 50 and executes the read acceptance and output program 60 on RAM 48. The acceptance and output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance and output program 60 executed on RAM 48.

[0329] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the robot 414. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 will be referred to as the "terminal".

[0330] Example 1

[0331] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0332] Application Example 1

[0333] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0334] Example 2

[0335] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0336] Application Example 2

[0337] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0338] The specific processing unit 290 sends the result of the specific processing to the robot 414. In the robot 414, the control unit 46A outputs the result of the specific processing to the speaker 240 and the controlled object 443. The microphone 238 acquires sound input representing the result of the specific processing. The control unit 46A sends the sound data representing the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0339] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0340] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or external devices, and the robot 414 acquires or collects information required for processing from the data processing device 12 or external devices.

[0341] For example, the collection unit is implemented by the control unit 46A of the robot 414 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the robot 414 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 of the robot 414 and the control object 443 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0342] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the robot 414.

[0343] Furthermore, the emotion-specific model 59, acting as an emotion engine, can determine a user's emotion based on a specific mapping. Specifically, the emotion-specific model 59 can determine a user's emotion based on an emotion graph that serves as a specific mapping (see [reference]). Figure 9 The system determines the user's emotions. Furthermore, the emotion-specific model 59 can similarly determine the robot's emotions, and the specific processing unit 290 performs specific processing based on the robot's emotions.

[0344] Figure 9 This is a diagram representing an emotion map 400 that maps multiple emotions. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotion is. On the outer side of the concentric circles, emotions representing states or behaviors arising from mood are arranged. Emotions are concepts that include feelings and mental states. Emotions generated by reactions occurring in the brain are arranged roughly to the left of the concentric circles. Emotions derived from situational judgments are arranged roughly to the right of the concentric circles. Emotions generated by reactions occurring in the brain and derived from situational judgments are arranged roughly above and below the concentric circles. Furthermore, "pleasant" emotions are arranged above the concentric circles, and "unpleasant" emotions are arranged below them. Thus, in the emotion map 400, multiple emotions are mapped based on the structure that generates emotions, and emotions that are likely to occur simultaneously are mapped close to each other.

[0345] These emotions are distributed at the three o'clock position of the emotion map 400, typically fluctuating between peace and anxiety. In the right half of the emotion map 400, situational awareness dominates over internal sensation, thus resulting in an impression of calm.

[0346] The inner side of the emotion map 400 represents the inner state, while the outer side represents behavior. Therefore, the further outward you are from the emotion map 400, the more visible the emotion becomes (manifested in behavior).

[0347] Here, human emotions are based on various balances such as posture and blood sugar levels. When these balances deviate from an ideal state, it indicates an unpleasant state; when they approach the ideal state, it indicates a pleasant state. Emotions in robots, cars, motorcycles, etc., can also be created in the following way: based on various balances such as posture and remaining battery power, when these balances deviate from an ideal state, it indicates an unpleasant state; when they approach the ideal state, it indicates a pleasant state. Emotion maps can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a Brain Physiological Signal Analysis System for Voice Emotion Recognition and Emotion, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). In the left half of the emotion map, emotions belonging to the sensory-dominated region, called "response," are arranged. Furthermore, in the right half of the emotion map, emotions belonging to the situational cognition-dominated region, called "situation," are arranged.

[0348] In the emotion map, two types of emotions that promote learning are defined. One is a negative emotion on the situational side, in the middle or peripheral region of "repentance" or "reflection." This occurs when the robot experiences negative emotions such as "I don't want to experience this feeling again" or "I don't want to be blamed again." The other is a positive emotion on the response side, near the "desire" region. This occurs when there are positive feelings such as "wanting more" or "wanting to know more."

[0349] The emotion-specific model 59 inputs user input into a pre-trained neural network to obtain emotion values ​​representing each emotion shown in the emotion map 400, thereby determining the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network... Figure 10 As shown in the sentiment graph 900, it was trained in a way that sentiments that are configured close to each other have similar values. Figure 10 The text shows examples of emotions such as "peace of mind", "stability", and "reassurance" that have similar emotion values.

[0350] The above description focuses on the functions of the data processing device 12, but the system of this disclosure is not necessarily installed on a server. The system of this disclosure can also be installed as a general information processing system. This disclosure can also be installed, for example, as a software program running on a personal computer, an application running on a smartphone, etc. The method of this disclosure can also be provided to users in the form of SaaS (Software as a Service).

[0351] In the above embodiments, an example of a specific process being performed by a single computer 22 is given. However, the technology disclosed herein is not limited to this, and the specific process can also be distributed among multiple computers, including computer 22. For example, the data generation model 58 can be located on an external device of the data processing apparatus 12, where data is generated based on the input data.

[0352] In the above embodiments, examples of storing a specific processing program 56 in the memory 32 have been described, but the technology disclosed herein is not limited thereto. For example, the specific processing program 56 may also be stored in a portable computer-readable non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed into the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0353] Alternatively, a specific processing program 56 may be pre-stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 according to the requirements of the data processing device 12.

[0354] In addition, it is not necessary to store all the specific processing program 56 in the storage device such as the server connected to the data processing device 12 via the network 54 or in the memory 32; a portion of the specific processing program 56 may be stored in advance.

[0355] As hardware resources for performing specific processes, various processors, as shown below, can be used. For example, a CPU can be listed as a processor, which functions as a general-purpose processor that performs specific processes by executing software, i.e., a program. Furthermore, processors can be listed as special-purpose circuits such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application-Specific Integrated Circuits), which are processors with circuitry specifically designed to perform specific processes. Each processor has built-in or connected memory, and each processor executes specific processes using that memory.

[0356] The hardware resources for performing a specific process can consist of one of these various processors, or a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resources for performing a specific process can be a single processor.

[0357] As an example of a single processor, there are two approaches: First, a processor is composed of a combination of one or more CPUs and software, which functions as a hardware resource to perform a specific process; second, as represented by a SoC (System-on-a-chip), a processor is used to implement the functionality of the entire system, which includes multiple hardware resources for performing a specific process, using a single IC (Integrated Circuit) chip. In this way, the specific process is implemented by using one or more of the aforementioned processors as hardware resources.

[0358] Furthermore, the hardware architecture of these various processors, more specifically, can utilize circuits that combine semiconductor elements and other circuit components. Moreover, the specific process described above is just one example. Therefore, without departing from the main point, unnecessary steps can certainly be deleted, new steps added, or the processing order changed.

[0359] The descriptions and illustrations above are detailed explanations of a portion of the technology disclosed herein, and are merely one example of the technology disclosed herein. For example, the above descriptions of the structure, function, effect, and results are just one example of the structure, function, effect, and results of a portion of the technology disclosed herein. Therefore, without departing from the spirit of the technology disclosed herein, unnecessary parts may be deleted, new elements added, or replacements may be made to the descriptions and illustrations above. Furthermore, to avoid confusion and facilitate understanding of a portion of the technology disclosed herein, explanations of common technical knowledge that do not require special explanation under the premise of being able to implement the technology disclosed herein have been omitted from the descriptions and illustrations above.

[0360] All documents, patent applications and technical specifications set forth in this specification are incorporated herein by reference to the same extent that each document, patent application and technical specification is specifically and individually described therein and referenced by reference.

[0361] In addition, the following notes are provided in response to the above explanation.

[0362] Example 1

[0363] (Note 1)

[0364] An information processing system includes: a device for acquiring image information using a wearable image acquisition device; a device for acquiring user spatial location information; a device for transmitting the image information and spatial location information to the information processing device in real time via an information communication network; a device for performing multi-category target recognition and surrounding environment analysis on the image information and spatial location information in the information processing device; a device for correcting the results using spatial data obtained from a geographic information database; a device for dynamically generating natural language content for user feedback based on the above analysis results using a generative artificial intelligence model according to prompt statements; a device for transmitting the natural language content to the user via an information communication network and providing prompts via an output device using voice and physical stimuli; and a device for coordinating and managing the above processes in real time to assist the user in independent and safe behavior.

[0365] (Note 2)

[0366] According to the information processing system described in Appendix 1, the wearable image acquisition device is a portable information acquisition device.

[0367] (Note 3)

[0368] According to the information processing system described in Appendix 1, the information communication network is a mobile communication system.

[0369] Application Example 1

[0370] (Note 1)

[0371] An information processing system includes: a device for acquiring image information through an imaging component worn by a user; a location information acquisition device for acquiring user location information; a device for transmitting the image information and the location information to the information processing device via an information communication network; an analysis device for identifying environmental objects using a target recognition algorithm based on the image information and the location information, and performing scene analysis in conjunction with geographic information, within the information processing device; a generation device for generating feedback information containing natural language navigation instructions and safety prompts using a generative artificial intelligence model, based on the analysis results and the user's state, within the information processing device; a device for transmitting the feedback information to an output device via an information communication network and providing the feedback information to the user in a voice or multimodal manner; and a device for acquiring and analyzing the user's physiological and behavioral data through an emotion state detection device, and adaptively adjusting the feedback content and presentation based on the analysis results.

[0372] (Note 2)

[0373] According to the information processing system described in Appendix 1, the imaging component is an imaging device attached to a wearable device.

[0374] (Note 3)

[0375] According to the information processing system described in Appendix 1, the information communication network is a fifth-generation mobile communication system.

[0376] Example 2

[0377] (Note 1)

[0378] An information processing system includes: a device for acquiring environmental image information via an imaging device; a device for acquiring spatial coordinate information via a location information acquisition device; a device for acquiring user physiological data and generating emotional state information via devices such as physiological information sensors; a device for transmitting the image information, the location information, and the emotional state information to the information processing device via a communication network; a device for analyzing the image information using an image analysis algorithm to perform target recognition, acquiring the geographical environment based on the spatial coordinate information, and analyzing the emotional state information using an emotion analysis algorithm within the information processing device; a device for automatically generating feedback information using a generative artificial intelligence model based on the above analysis results, and adaptively adjusting the content and prompting method according to the user's emotional state; and a device for sending the feedback information to a user terminal via a communication network and outputting voice or tactile prompts via a speech synthesis device.

[0379] (Note 2)

[0380] According to the information processing system described in Appendix 1, the imaging device is disposed on a wearable carrier.

[0381] (Note 3)

[0382] According to the information processing system described in Appendix 1, the communication network is a mobile communication method.

[0383] Application Example 2

[0384] (Note 1)

[0385] An information processing system includes: a device for acquiring image information via a user-worn information acquisition device; a device for acquiring spatial location information; a device for acquiring biological information; a device for transmitting the image information, the spatial location information, and the biological information to the information processing device via a communication infrastructure; a device for parsing the image information and the spatial location information, performing object recognition processing and spatial discrimination processing in the information processing device, and further performing user state recognition processing by parsing the biological information; a device for generating personalized feedback information based on the object recognition processing result, the spatial discrimination processing result, and the state recognition processing result, using a generative artificial intelligence processing unit and taking a predefined prompt statement as input; a device for transmitting the feedback information to the information acquisition device via the communication infrastructure; and a device for outputting the received feedback information in the information acquisition device in the form of voice or vibration.

[0386] (Note 2)

[0387] According to the information processing system described in Appendix 1, the information acquisition device is a wearable computing device.

[0388] (Note 3)

[0389] According to the information processing system described in Appendix 1, the communication infrastructure is a mobile communication network.

Claims

1. An information processing system, characterized in that, include: Device for acquiring image data through a user-worn camera; A device for acquiring location information; A device for sending the image data and location information to a server via a communication network; A device for parsing the image data and the location information in the server; A device for generating feedback information for the user based on the parsing results in the server; and A device for sending the feedback information to the user via the communication network and outputting it in a voice manner.

2. The information processing system according to claim 1, characterized in that, The camera is mounted on a wearable device.

3. The information processing system according to claim 1, characterized in that, The communication network is a fifth-generation mobile communication system.

Images