Prompt word generation interaction system under multi-mode perception condition

Abstract

The invention discloses a cue word generation interaction system under a multi-mode perception condition, and relates to the technical field of artificial intelligence, the system comprises a main control module, a multi-mode perception module, a voice interaction module and a state display module; the multi-mode sensing module is used for sensing various environment states; the main control module is used for generating target prompt words according to various environment states; a cooperation instruction corresponding to the target prompt word is sent to a voice interaction module and a state display module; the voice interaction module is used for acquiring questions of the user; executing a cooperation instruction corresponding to the target prompt word; and the state display module is used for executing the cooperation instruction corresponding to the target prompt word. By integrating the multi-mode sensing module, the system can automatically sense the physical environment and the user behavior and dynamically generate the most suitable prompt word according to the physical environment and the user behavior, so that the interaction has more context correlation, and the interaction experience is improved.

Description

A prompt word generation and interaction system under multimodal perception conditions Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a prompt word generation and interaction system under multimodal perception conditions. Background Technology

[0002] Currently, AI voice assistants (such as smart speakers and mobile phone voice assistants) have become an important way of human-computer interaction. Their core technology is cue word engineering, which involves guiding large language models to generate the expected response through specific text commands. However, existing technologies almost entirely rely on pure voice channels for interaction.

[0003] The existing AI assistants based on pure voice interaction have the following core drawbacks, which limit their intelligence level and application scope, and have the following problems: (1) Lack of environmental context awareness: Existing devices cannot perceive the physical environment in which the user is located (such as light and dark, ambient noise, and whether there are specific objects), resulting in their prompts being static or triggered only by voice keywords. For example, in a noisy industrial environment, the device cannot automatically adjust the prompts to activate noise reduction and simplified instruction mode; in a dark environment, it will not automatically switch to a low-volume broadcast mode, which may disturb others. (2) Single and inefficient interaction mode: All interaction initializations must be completed through "wake word + voice command". For complex tasks, users need to say lengthy instructions, which is inefficient. For example, users cannot simply pick up a medicine bottle to make the assistant automatically enter the medicine query mode. (3) Opaque system status: Users cannot get intuitive feedback on what working mode the device is currently in and what type of task it is processing through the device hardware. The interaction process is like a "black box", resulting in an unfriendly experience. Summary of the Invention

[0004] The main objective of this application is to propose a prompt word generation and interaction system under multimodal perception conditions, so as to generate more accurate prompt words and thus improve the intelligent interaction experience.

[0005] To achieve the above objectives, this application proposes a prompt word generation and interaction system under multimodal perception conditions. The system includes: a main control module, a multimodal perception module, a voice interaction module, and a status display module. The multimodal perception module is used to perceive multiple environmental states. The main control module is used to convert the various environmental states into a set of feature labels; fuse the labels in the feature label set into a contextual scene snapshot; inject the contextual scene snapshot into a basic prompt word template of a large language model to obtain a fused prompt word template; adjust the interaction instructions according to the contextual scene snapshot; obtain user questions from the voice interaction module; combine the user questions, the fused prompt word template, and the adjusted interaction instructions into a target prompt word; and send the corresponding collaborative instructions to the voice interaction module and the status display module respectively. The voice interaction module is used to obtain the user questions; execute the collaborative instructions corresponding to the target prompt word; and the status display module is used to execute the collaborative instructions corresponding to the target prompt word.

[0006] In some embodiments, the multimodal sensing module includes an ambient light sensor, a proximity sensor, and an inertial measurement unit; wherein, the ambient light sensor is used to sense the brightness of the current environment, calculate the brightness change trend and absolute brightness value of the current environment, and determine whether the system has been carried into a dark environment; the proximity sensor is used to calculate the distance change rate between the user and the system, and distinguish the user's distance intention based on the distance change rate; wherein, the distance intention includes approaching, moving away, or staying; the inertial measurement unit is used to perform attitude recognition using accelerometer and gyroscope data; wherein, the recognized attitude includes whether the system has been picked up, put down, or is being moved.

[0007] In some embodiments, the main control module includes: a weight allocation unit, used to allocate confidence weights to each of the labels in the feature label set; and a context construction unit, used to construct the context scene snapshot according to the confidence weights of each label.

[0008] In some embodiments, the main control module includes: an intent mapping unit, configured to map the context scene snapshot into an interaction intent; and a prompt word generation unit, configured to inject the interaction intent and the context scene snapshot into the basic prompt word template of the large language model to obtain the fused prompt word template.

[0009] In some embodiments, the main control module includes: a resource scheduling decision unit, configured to send the collaborative instruction in text form to the voice interaction module, along with a playback mode and priority; and to send the collaborative instruction, including display color, breathing mode, and screen brightness, to the status display module.

[0010] In some embodiments, the main control module further includes an interactive closed-loop learning unit, used to record the context scene snapshot and the target prompt word for each interaction, and then optimize the data fusion weight and intent mapping rules of the main control module through a machine learning model.

[0011] In some embodiments, the voice interaction module is configured to: receive a text to be played and a set of metadata instructions from the main control module; play the text to be played using a TTS engine according to the playback style and emotion tags in the metadata instructions; send a playback signal to the status display module at the start of playback to trigger the light strip of the status display module to display a flowing effect or the screen to display a sound wave animation; send a playback end signal to the status display module at the end of playback; and continuously monitor whether new voice input from the user is received during the playback of the text to be played, and then respond to the new voice input.

[0012] In some embodiments, the status display module is configured to: receive the collaborative instruction from the main control module; implement emotional and context-adaptive display logic according to the collaborative instruction; wherein the display logic includes LED display logic and screen display logic; and dynamically display the switching and transition animations between listening, thinking and broadcasting.

[0013] In some embodiments, the system further includes a housing; the main control module, the multimodal perception module, the voice interaction module, and the status display module are disposed within the housing.

[0014] In some embodiments, the housing includes any one of desktop, handheld, or embedded types.

[0015] The embodiments of this application include at least the following beneficial effects: This application provides a prompt word generation and interaction system under multimodal perception conditions. The system of this application includes: a main control module, a multimodal perception module, a voice interaction module, and a status display module; wherein, the multimodal perception module is used to perceive multiple environmental states; the main control module is used to convert various environmental states into a feature label set; fuse each label in the feature label set into a contextual scene snapshot; inject the contextual scene snapshot into the basic prompt word template of the large language model to obtain a fused prompt word template; adjust the interaction instructions according to the contextual scene snapshot; obtain user questions from the voice interaction module; combine the user questions, the fused prompt word template, and the adjusted interaction instructions into a target prompt word; send the collaborative instructions corresponding to the target prompt word to the voice interaction module and the status display module respectively; the voice interaction module is used to obtain user questions; execute the collaborative instructions corresponding to the target prompt word; the status display module is used to execute the collaborative instructions corresponding to the target prompt word. By integrating a multimodal perception module, this application enables the system to automatically perceive the physical environment and user behavior, and dynamically generate the most suitable prompt words accordingly, making the interaction more context-sensitive and improving the interactive experience. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 is a structural schematic diagram of a prompt word generation interaction system under multimodal perception conditions provided in an embodiment of this application; Figure 2 is a flowchart of a prompt word generation interaction system under multimodal perception conditions provided in an embodiment of this application; Figure 3 is an application scenario example of a prompt word generation interaction system under multimodal perception conditions provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0020] Before providing a detailed description of the embodiments of this application, some related technologies involved in the embodiments of this application will be described first, as follows: The technical solution of this application has high cross-industry applicability and can be applied to the following fields: Consumer electronics field: used to develop next-generation smart speakers, smart screens, learning machines, wearable devices, etc. for ordinary consumers, to enhance product differentiation competitiveness and user experience.

[0021] In the field of smart education: We provide core technologies to educational technology companies and develop products such as interactive teaching robots, smart reading pens, and smart classroom control systems to achieve personalized and contextualized teaching.

[0022] In the smart home / building sector: As the top entry point for smart homes, it provides a more intelligent interaction hub for home appliance manufacturers and whole-house smart solution providers, realizing a home environment that truly "understands you".

[0023] In the field of Industrial Internet of Things (IIoT) and smart manufacturing: providing interactive terminals that can "listen, speak, see, and think" for industrial sites, applied to equipment inspection, warehouse management, remote assistance, etc., to improve production efficiency and safety.

[0024] In the field of smart elderly care and healthcare: We provide contactless, proactive solutions for elderly care institutions and community medical centers to care for the elderly and manage their medication, thereby reducing the burden of care and ensuring the safety of seniors.

[0025] To address the shortcomings of the existing technologies, the purpose of this application is to provide a prompt word generation and interaction system under multimodal perception conditions. This system aims to achieve context-aware intelligent interaction: by integrating multiple sensors, the system can automatically perceive the physical environment and user behavior, and dynamically generate the most suitable prompt words accordingly, making the interaction more context-sensitive.

[0026] Expand interaction channels and improve efficiency: Utilize physical sensor information (such as object recognition and user proximity) as trigger conditions to simplify the interaction process and achieve more direct and efficient wake word-free interaction.

[0027] Enhance interactive transparency and user experience: The hardware status display module clearly and intuitively shows the current working mode and status of the system, establishing a more trustworthy interactive relationship between users and AI.

[0028] Referring to Figure 1, this application embodiment provides a prompt word generation interaction system under multimodal perception conditions. The system includes: a main control module, a multimodal perception module, a voice interaction module, and a status display module. The multimodal perception module is used to perceive multiple environmental states. The main control module is used to convert the various environmental states into a feature label set; fuse the labels in the feature label set into a contextual scene snapshot; inject the contextual scene snapshot into a basic prompt word template of a large language model to obtain a fused prompt word template; adjust the interaction instructions according to the contextual scene snapshot; obtain user questions from the voice interaction module; combine the user questions, the fused prompt word template, and the adjusted interaction instructions into a target prompt word; and send the corresponding collaborative instructions to the voice interaction module and the status display module respectively. The voice interaction module is used to obtain the user questions; execute the collaborative instructions corresponding to the target prompt word; and the status display module is used to execute the collaborative instructions corresponding to the target prompt word.

[0029] Optionally, the multimodal perception module includes an ambient light sensor, a proximity sensor, and an inertial measurement unit; wherein, the ambient light sensor is used to sense the brightness of the current environment, calculate the brightness change trend and absolute brightness value of the current environment, and determine whether the system has been carried into a low-light environment; the proximity sensor is used to calculate the distance change rate between the user and the system, and distinguish the user's distance intention based on the distance change rate; wherein, the distance intention includes approaching, moving away, or staying; the inertial measurement unit is used to perform attitude recognition through accelerometer and gyroscope data; wherein, the recognized attitude includes whether the system has been picked up, put down, or is being moved.

[0030] Optionally, the main control module includes: a weight allocation unit, used to allocate confidence weights to each of the labels in the feature label set; and a context construction unit, used to construct the context scene snapshot based on the confidence weights of each label.

[0031] Optionally, the main control module includes: an intent mapping unit, used to map the context scene snapshot into an interaction intent; and a prompt word generation unit, used to inject the interaction intent and the context scene snapshot into the basic prompt word template of the large language model to obtain the fused prompt word template.

[0032] Optionally, the main control module includes: a resource scheduling decision unit, used to send the collaborative instruction in text form to the voice interaction module, along with the playback mode and priority; and to send the collaborative instruction including display color, breathing mode and screen brightness to the status display module.

[0033] Optionally, the main control module further includes an interactive closed-loop learning unit, used to record the context scene snapshot and the target prompt word for each interaction, and then optimize the data fusion weight and intent mapping rules of the main control module through a machine learning model.

[0034] Optionally, the voice interaction module is configured to: receive the text to be played and a set of metadata instructions from the main control module; play the text to be played using a TTS engine according to the playback style and emotion tags in the metadata instructions; send a playback signal to the status display module at the start of playback to trigger the light strip of the status display module to display a flowing effect or the screen to display a sound wave animation; send a playback end signal to the status display module at the end of playback; and continuously monitor whether new voice input from the user is received during the playback of the text to be played, and then respond to the new voice input.

[0035] Optionally, the status display module is used to: receive the collaborative instruction from the main control module; implement emotional and context-adaptive display logic according to the collaborative instruction; wherein the display logic includes LED display logic and screen display logic; and dynamically display the switching and transition animation between listening, thinking and broadcasting.

[0036] Optionally, the system further includes a housing; the main control module, the multimodal perception module, the voice interaction module, and the status display module are disposed within the housing.

[0037] Optionally, the housing may be any one of desktop, handheld, or embedded.

[0038] The following sections will provide a detailed description and explanation of some optional embodiments of this application, using specific application examples.

[0039] This embodiment provides a prompt word generation and interaction system under multimodal perception conditions. Its core idea lies in providing rich multimodal inputs to the prompt word generation engine through innovative hardware structure design, thereby achieving contextual intelligence. The system mainly includes the following parts, and its workflow is shown in Figure 2.

[0040] The core of the current solution lies in building a closed-loop interactive system with continuous perception, intelligent decision-making, and multi-channel natural feedback. The following is a detailed description of the workflow of each module: 1. Main control module - the "intelligent brain" of the system.

[0041] The main control module is not merely a data relay station, but also the cognitive and decision-making center of the entire system. Its innovation lies in its context awareness, intent understanding, and resource scheduling capabilities.

[0042] Detailed workflow: Step 1: Data preprocessing and feature extraction.

[0043] Input: Receive raw data stream from the "Multimodal Sensing Module".

[0044] Ambient light sensor: It not only senses brightness, but also calculates the trend of brightness change (such as from bright to dark) and the absolute brightness value to determine whether the user has entered a dark environment.

[0045] Proximity sensor: Accurately calculates the rate of change in distance between the user and the device, distinguishing between intentions such as "approaching", "moving away" or "staying".

[0046] IMU (Inertial Measurement Unit): Uses accelerometer and gyroscope data to identify attitude (such as whether the device is being picked up, put down, or is being moved).

[0047] Output: Generate a set of standardized, semantic feature vectors, such as {Environment: Dark light, User pose: Still, User distance: Near}.

[0048] Step 2: Multimodal data fusion and context construction (creativity core).

[0049] This step is no longer a simple data overlay, but rather an information fusion based on a weighted decision model.

[0050] How the model works: The system assigns a dynamic weight to each type of sensor data. For example, in night mode, the weight of the ambient light sensor is significantly increased; while when the device is picked up, the weight of the IMU increases.

[0051] Context building: The system combines real-time feature vectors with historical interaction data (such as the content of the last interaction and user habits) to generate a rich "context snapshot." For example, the system will know that "the user is at night, indoors, and picked up the device," which contains more interaction intent than isolated sensor data.

[0052] Step 3: Intelligent prompt word engineering and decision-making.

[0053] Intent Mapping: Maps the "context snapshot" generated in the previous step to specific interaction intents. For example, {Environment: Dark light, User posture: Picking up, User distance: Approaching} might be mapped to "The user is preparing to engage in voice interaction and requires the system to automatically brighten the screen or provide gentle voice feedback."

[0054] Dynamic Prompt Generation: Instead of using fixed prompts, the system automatically injects contextual information as variables into the prompts of the Large Language Model (LLM). Example: The basic prompt might be "Please answer the user's question in a short sentence." The dynamically generated prompt would be: "The ambient light is dim, and the user has just picked up the device. Please answer the following question in a gentle, concise voice style: [User Question]."

[0055] Resource scheduling decision: Based on the decision results, the main control module sends precise instructions to other modules. For example, it not only sends the generated text to the voice module, but also includes parameters: {playback mode: gentle, priority: high}; at the same time, it sends instructions to the status display module: {LED color: light blue, breathing mode: slow, screen brightness: 30%}.

[0056] Step 4: Interactive closed-loop learning.

[0057] The module records the context, decisions, and results of each interaction (such as whether the user will ask the question again soon), and optimizes its data fusion weights and intent mapping rules through lightweight machine learning models (such as reinforcement learning), making the system smarter the more it is used.

[0058] 2. Voice Interaction Module - A human-like and emotional communication bridge.

[0059] The creativity of this module lies in going beyond simple "text-to-speech" to achieve an interactive experience that is emotional, rhythmic, and coordinated with visual states.

[0060] Detailed workflow: Step 1: Receive and parse enhanced instructions.

[0061] Input: The main control module receives not only the text to be broadcast, but also a set of metadata instructions, such as {text content: "The weather is sunny today.", broadcast style: cheerful, speech rate: medium, emotion tag: positive, coordination signal: the LED needs to blink during broadcast}.

[0062] Step 2: Emotional speech synthesis and audio post-processing.

[0063] TTS Engine: Employs an advanced TTS engine that supports emotional speech synthesis. The module directly inputs "broadcast style" and "emotional tags" parameters into the engine to generate expressive speech, rather than just mechanical reading.

[0064] Audio post-processing: Based on ambient light sensor data (ambient noise level), automatically adjust the volume and high / low frequencies of the voice output to ensure clarity in noisy environments and smoothness in quiet environments.

[0065] Step 3: Collaborative feedback and real-time interrupt monitoring.

[0066] In coordination with the status module: When the broadcast begins, a "broadcasting in progress" signal is sent to the status display module, triggering the LED light strip to display a flowing effect or the screen to display a sound wave animation. When the broadcast ends, a "broadcast ended" signal is sent.

[0067] Real-time monitoring: During voice output, the module continuously monitors for new voice input from the user (such as "stop" or "speak faster") and has voice activity detection capabilities to support real-time interruption and response, making the interaction more natural and efficient.

[0068] 3. Status Display Module - Environment Integration and Disturbance-Free Status Prompts.

[0069] Creativity lies in contextualizing, abstracting, and emotionalizing state information, reducing the direct visual impact on users, and providing an "environmentally intelligent" experience.

[0070] Detailed workflow: Step 1: Receive multi-dimensional status instructions.

[0071] Input: The instructions received from the main control module contain rich information dimensions, such as {system status: listening, main context: night mode, coordination signal: needs to be synchronized with voice broadcast}.

[0072] Step 2: Emotional and context-adaptive display logic.

[0073] LED display logic: Standby state: an extremely dark single color dot, breathing slowly, almost invisible, blending into the environment.

[0074] Listening status: Depending on the "main context", if it is night mode, it will be displayed as a soft warm yellow light strip that slowly scans from left to right; if it is day mode, it will be displayed as a brighter blue.

[0075] Processing / Thinking State: The LEDs display a soft, multi-color gradient cycle, simulating the dynamics of "thinking".

[0076] Broadcast status: The brightness of the light strip fluctuates synchronously with the waveform of the voice volume, visualizing the auditory information.

[0077] Screen display logic (if equipped): Instead of displaying complex text, use abstract visualizations, such as concentric circles representing listening, particles converging towards the center representing thinking, and smooth waveforms representing broadcasting.

[0078] The screen brightness is automatically and steplessly adjusted based on data from the ambient light sensor, always maintaining a comfortable level.

[0079] Step 3: Seamless state switching and transition animation.

[0080] The module includes elegant state transition animations to ensure a smooth and natural transition from "listening" to "thinking" and then to "broadcasting," avoiding abrupt changes and enhancing the product's quality and user experience.

[0081] In summary, through the detailed workflow design described above, the creativity of this solution has been significantly enhanced: 1. Intelligence: The main control module has the ability to perceive and make adaptive decisions based on context, rather than fixed rules.

[0082] 2. Humanization: Voice interaction is no longer a cold, impersonal machine voice, but rather a communication with emotion and collaborative feedback.

[0083] 3. Environmentalization: The status display is deeply integrated with the physical environment and usage scenarios, providing a more advanced and comfortable "environmental intelligence" experience.

[0084] 4. Organic Collaboration: The three modules collaborate deeply through a wealth of signals and parameters, forming a smooth, efficient, and dynamic interactive loop.

[0085] This detailed workflow design transformed the solution from a conceptual block diagram into a highly feasible and innovative intelligent interactive system implementation plan.

[0086] The following describes each module: Housing: As the physical carrier and supporting structure of the entire system, its shape can be designed as desktop, handheld, or embedded, depending on the application scenario. The material and internal structure of the housing must provide stable installation and necessary protection for all built-in modules.

[0087] Main Control Module: This is the brain of the system. Its core is a main control microcontroller unit, which is electrically connected to all other modules (i.e., connected and controlled via electrical signals) through interfaces such as the integrated circuit bus on the circuit board. The main control microcontroller unit has pre-set prompt word decision and generation logic. This logic is essentially a set of "condition-action" mapping rules, for example: Condition 1: If the ambient light sensor value < threshold L and the proximity sensor detects no one approaching → Action: Call the low-power standby mode prompt word template.

[0088] Condition 2: If the RFID reader reads a specific item number → Action: Generate and send a prompt such as "Item [XX] identified, what do you need to know?"

[0089] The main microcontroller unit is responsible for executing these rules and sending the final generated prompts, which contain rich context, to the backend AI model.

[0090] Multimodal sensing module: This is the key hardware innovation of this embodiment. It consists of multiple sensors, all of which are fixed in specific positions on the housing and connected to the main control module via ribbon cables.

[0091] Ambient light sensor: Usually installed on the front panel of the housing, it is used to detect the intensity of ambient light, and its analog or digital signal is transmitted to the main control microcontroller unit.

[0092] Proximity sensor: Usually installed in parallel with ambient light sensor, it determines whether a user or object is close to the device by emitting infrared light and detecting the reflection.

[0093] Inertial Measurement Unit: Installed on the mainboard inside the device, it contains an accelerometer and a gyroscope to detect the physical motion of the device, such as being picked up, shaken, or moved.

[0094] Non-contact identification modules (such as RFID readers / near-field communication modules): Their reading antennas need to be integrated into a specific area within the housing (such as the bottom or side) to ensure good reading performance. They are used to read the identity information of electronic tags attached to objects.

[0095] Status display module: Designed to enhance interactive transparency. It can be a small LCD screen or a set of multi-color LED indicator lights, embedded in the housing surface. It is directly controlled by the main microcontroller unit and is used to display visual status information such as "Listening," "Thinking," "Drug Recognition Mode," and "Night Mode."

[0096] Voice interaction module: Includes a microphone and speaker, with openings and grilles designed on the housing for basic voice input and output.

[0097] Workflow details (with reference to Figure 2): Perception phase: The multimodal perception module continuously or passively collects data from the physical world (such as ambient light intensity and object identification information).

[0098] Decision-making and generation phase: This data is sent to the main control module in real time. The decision-making logic inside the main control microcontroller unit selects or dynamically assembles the most suitable prompt words according to preset rules.

[0099] Detailed Explanation of the Decision-Making and Generation Phase: This phase is the "brain" and "soul" of the system. Its goal is to transform raw, low-level sensor data into high-level, context-rich, and semantically rich intelligent prompts. This process is not a simple "if-else" judgment, but a multi-layered, dynamically weighted, and predictive intelligent decision-making process.

[0100] Step 1: Data preprocessing and feature extraction (from signal to semantics).

[0101] The main control module first converts the received raw data stream into "feature labels" with clear semantics.

[0102] Ambient light sensor data → Illumination scene label: Brightness value < threshold L_dark → "Dark environment".

[0103] Threshold L_dark < brightness value < threshold L_bright → "Normal lighting".

[0104] Brightness value > threshold L_bright → "Strong light environment".

[0105] Advanced features: Calculate brightness change trends. For example, if the brightness decreases rapidly over a short period of time, a "ambient light is getting darker" label can be generated for predictive interactions.

[0106] Proximity sensor data → User distance and behavior label: Distance value < threshold D_near → "User is approaching".

[0107] Distance value > threshold D_far → "User is far away".

[0108] Distance value decreases rapidly → "User is approaching".

[0109] The distance value stabilized at D_near → "Users continue to pay attention".

[0110] IMU (Inertial Measurement Unit) Data → Device Status Label.

[0111] A specific acceleration and angular velocity pattern was detected → "Device picked up".

[0112] Accelerometer data is stable, gyroscope data is close to zero → "Equipment is stationary".

[0113] Continuous detection of linear acceleration → "Device is moving".

[0114] This step outputs a set of feature labels, such as: {low light environment, user is approaching, device is picked up}.

[0115] Step 2: Multimodal data fusion and context construction (creativity core).

[0116] This is the most crucial step. The system merges these isolated labels into a unified, actionable "contextual snapshot." This is achieved through a weighted decision matrix.

[0117] 1. Conflict Resolution and Confidence Weighting: Data from different sensors may conflict. For example, the IMU might show "device picked up," but the proximity sensor might show "user moved away." The system assigns a confidence weight to each sensor (based on sensor accuracy and historical reliability). In this case, the IMU might have a higher weight, and the system is more inclined to believe the "device picked up" assessment, interpreting the proximity sensor data as a "false detection" or "specific grip posture."

[0118] 2. Scene pattern recognition: The system has a built-in "scene library" that maps tag combinations to specific interaction scenes.

[0119] Pattern matching example: {Low-light environment, user approaches, device is picked up} → Scenario A: Active interaction mode. Interpretation: The user actively picks up the device in a poorly lit environment with the intention of using it. Decision weight: High. The system should prepare to enter a high-responsiveness state.

[0120] {Normal lighting, user away, device stationary} → Scenario B: Environmental monitoring mode. Interpretation: The device is in standby mode, only monitoring environmental changes. Decision weight: Low. The system maintains low power consumption.

[0121] {Bright light environment, device is moving} → Scenario C: Outdoor mobile mode. Interpretation: The user may use the device outdoors. Decision weight: Medium. The system needs to prioritize the clarity of voice prompts and the maximum screen brightness.

[0122] Step 3: Dynamic prompt word engineering and generation (the embodiment of intelligence).

[0123] The system dynamically assembles prompts and sends them to the Large Language Model (LLM) based on the generated "contextual scene snapshot". This is not a simple string concatenation, but a structured prompt engineering process.

[0124] 1. Role setting and context injection: The first part of the prompt is the fixed system role setting and the context of this interaction.

[0125] Basic template: "You are an intelligent assistant. The context of the current interaction is as follows:"

[0126] Context injection: Fill in the key information from the context object in the previous step. For example: "- Environment: A low-light environment at night. Please keep your answer concise and avoid visual descriptions.\n- User status: The user has just picked up the device and may urgently need information.\n- Device status: The device is ready."

[0127] 2. Command and Style Adjustment: Dynamically adjust commands to the AI ​​based on the scenario.

[0128] Regarding the "proactive interaction mode": "Please answer users' questions directly in a gentle, clear, and concise tone."

[0129] Regarding the "Outdoor Mobile Mode": "Please answer with short, powerful, and concise sentences, avoiding complex sentence structures."

[0130] 3. Final assembly: Combine the above parts with real user questions (obtained from the voice interaction module) to form the final prompt words.

[0131] [Example]: A user asks, "How's the weather today?"

[0132] Unprocessed prompt: "How's the weather today?" (Meaning is average; the AI ​​might reply with a long, detailed text including UV index and probability of precipitation).

[0133] The intelligent prompt after processing by this system is: You are an intelligent assistant. The current interaction context is as follows: Environment: Low-light environment at night. Please keep your answers concise and avoid visual descriptions.

[0134] User status: The user has just picked up the device and may urgently need information.

[0135] Please answer users' questions directly in a gentle, clear, and concise tone.

[0136] User question: What's the weather like today?

[0137] Expected AI response: "Sunny tonight, 15 degrees Celsius. Tomorrow, sunny turning cloudy, 18 to 25 degrees Celsius." (The response is concise, gentle, and gets straight to the point, making it perfect for nighttime listening.)

[0138] Step 4: Distribute collaborative instructions.

[0139] While generating prompts, the main control module sends coordination instructions to other modules to ensure consistency in multimodal feedback.

[0140] Send the following command to the status display module: {Scene: Active Interaction Mode, LED Color: Soft Blue, Brightness: 30%, Mode: Slow Breathing}.

[0141] Send the command to the voice interaction module: {Text: [Generated AI response], TTS mode: Gentle, Speech rate: Medium}.

[0142] In summary, the solution of this embodiment can be applied to the scenario shown in Figure 3.

[0143] In summary, the innovation of this solution in the "decision and generation phase" is reflected in: 1. From signal to semantics: transforming low-level sensor data into high-level scene understanding.

[0144] 2. Weighted Fusion Decision Making: By using credibility weights and scene pattern recognition, it intelligently resolves sensor conflicts and accurately judges user intent.

[0145] 3. Dynamic prompt word engineering: By embedding contextual information as a key parameter into the instructions given to the LLM, the AI's response style and content are fundamentally shaped, making it more considerate, intelligent, and more suited to the current usage scenario.

[0146] 4. Closed-loop learning potential: The system can record the context, prompts, and user satisfaction of each interaction (such as whether the user immediately asks follow-up questions). In the future, machine learning models can be introduced to continuously optimize its decision weights and prompt templates, achieving self-evolution.

[0147] During the display and interaction phase: The main control microcontroller unit drives the status display module to update its status (e.g., the LED turns blue), and simultaneously sends the generated prompts to the AI ​​model. The AI ​​model understands the user's speech within the optimized prompt framework and provides an accurate response, which is then played through the speaker.

[0148] This embodiment, through the aforementioned hardware technology solution, brings the following beneficial effects: From "single-modal" to "multi-modal," the interaction is more intelligent: Existing technologies rely solely on a single voice channel. This embodiment, through hardware innovation, adds multiple physical information channels such as light sensing, distance perception, and object recognition, enabling the generation of prompts to be based on an understanding of the real environment. This achieves a qualitative leap in the interactive experience, moving from "mechanical response" to "proactive contextual intelligence."

[0149] From "passive wake-up" to "active triggering," interaction becomes more efficient: breaking the paradigm of "must speak first." A simple user action (such as picking up a device or approaching an object) can trigger complex interaction processes, greatly simplifying operation steps and improving interaction efficiency, especially in scenarios requiring rapid response, such as industry and healthcare.

[0150] From "black box" to "visualization," interaction becomes more trustworthy: By adding a dedicated status display module, the AI's internal prompts and operating status are made visible. Users can intuitively understand "what the device is doing" and "what mode it is in," reducing confusion and enhancing users' trust and sense of control over the AI.

[0151] Robust hardware foundation protection: The advantages of this embodiment are not achieved solely through software algorithms, but rather stem from the innovative integration and layout of the multimodal sensors at the hardware level and their collaborative working structure with the main control system.

[0152] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0153] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer technical solutions than shown, or a combination of certain technical solutions, or different technical solutions.

[0154] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0155] It should be understood that in this application, "at least one (item)" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0156] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A prompt word generation and interaction system under multimodal perception conditions, characterized in that, The system includes: a main control module, a multimodal perception module, a voice interaction module, and a status display module; wherein, the multimodal perception module is used to perceive multiple environmental states; the main control module is used to convert the various environmental states into a set of feature labels; fuse the labels in the feature label set into a contextual scene snapshot; inject the contextual scene snapshot into the basic prompt word template of the large language model to obtain a fused prompt word template; adjust the interaction instructions according to the contextual scene snapshot; obtain user questions from the voice interaction module; combine the user questions, the fused prompt word template, and the adjusted interaction instructions into a target prompt word; send the collaborative instructions corresponding to the target prompt word to the voice interaction module and the status display module respectively; the voice interaction module is used to obtain the user questions; execute the collaborative instructions corresponding to the target prompt word; the status display module is used to execute the collaborative instructions corresponding to the target prompt word.

2. The interactive system for generating prompt words under multimodal perception conditions according to claim 1, characterized in that, The multimodal perception module includes an ambient light sensor, a proximity sensor, and an inertial measurement unit. The ambient light sensor senses the brightness of the current environment, calculates the brightness change trend and absolute brightness value, and determines whether the system has been carried into a low-light environment. The proximity sensor calculates the rate of change of distance between the user and the system, and distinguishes the user's distance intention based on the rate of change of distance; the distance intention includes approaching, moving away, or remaining stationary. The inertial measurement unit performs attitude recognition using accelerometer and gyroscope data; the recognized attitude includes whether the system has been picked up, put down, or is being moved.

3. The interactive system for generating prompt words under multimodal perception conditions according to claim 1, characterized in that, The main control module includes: a weight allocation unit, used to allocate confidence weights to each of the labels in the feature label set; and a context construction unit, used to construct the context scene snapshot based on the confidence weights of each label.

4. The prompt word generation and interaction system under multimodal perception conditions according to claim 1, characterized in that, The main control module includes: an intent mapping unit, used to map the context scene snapshot into an interaction intent; and a prompt word generation unit, used to inject the interaction intent and the context scene snapshot into the basic prompt word template of the large language model to obtain the fused prompt word template.

5. The interactive system for generating prompt words under multimodal perception conditions according to claim 1, characterized in that, The main control module includes: a resource scheduling decision unit, used to send the collaborative instructions in text form to the voice interaction module, along with the playback mode and priority; and to send the collaborative instructions including display color, breathing mode and screen brightness to the status display module.

6. The interactive system for generating prompt words under multimodal perception conditions according to claim 1, characterized in that, The main control module further includes an interactive closed-loop learning unit, which records the context scene snapshot and the target prompt word for each interaction, and then optimizes the data fusion weight and intent mapping rules of the main control module through a machine learning model.

7. The interactive system for generating prompt words under multimodal perception conditions according to claim 1, characterized in that, The voice interaction module is used to: receive the text to be played and a set of metadata instructions from the main control module; play the text to be played using the TTS engine according to the playback style and emotion tags in the metadata instructions; send a playback signal to the status display module at the start of playback to trigger the light strip of the status display module to present a flowing effect or display a sound wave animation on the screen; send a playback end signal to the status display module at the end of playback; and continuously listen for new voice input from the user during the playback of the text to be played, and then respond to the new voice input.

8. The interactive system for generating prompt words under multimodal perception conditions according to claim 1, characterized in that, The status display module is used to: receive the collaborative instructions from the main control module; implement emotional and context-adaptive display logic according to the collaborative instructions; wherein the display logic includes LED display logic and screen display logic; and dynamically display the switching and transition animations between listening, thinking and broadcasting.

9. A prompt word generation and interaction system under multimodal perception conditions according to any one of claims 1 to 8, characterized in that, The system also includes a housing; the main control module, the multimodal perception module, the voice interaction module and the status display module are disposed within the housing.

10. A prompt word generation and interaction system under multimodal perception conditions according to claim 9, characterized in that, The housing may be desktop, handheld, or embedded.

Images