A portable multi-modal intelligent perception assistant and a working method thereof

Abstract

The application discloses a kind of portable multimodal intelligent perception assistants and its working method, belong to intelligent perception and man-machine interaction technical field.The application includes terminal main body, main control module, storage module, communication module, interactive module, power supply module and multimodal perception module;User thinking modeling unit and personalized decision output unit are integrated in main control module;Multimodal perception module includes audio unit, visual unit, environment detection unit and biological detection unit.The working method of the application includes: multidimensional data acquisition, judges and constructs or calls user thinking model, data cleaning and analysis, model matching, generates and outputs personalized suggestion, collects user feedback and iteratively optimizes model, real-time monitoring network state and automatically switches online / offline mode.The application realizes accurate personalized intelligent assistance by closed-loop iterative optimization and local / cloud collaboration, suitable for daily, outdoor, weak network and other various scenes.

Description

Technical Field

[0001] This invention relates to the field of intelligent sensing and human-computer interaction technology, and in particular to a portable multimodal intelligent sensing assistant and its working method. Background Technology

[0002] With the rapid development of mobile smart devices and artificial intelligence technology, intelligent assistive devices catering to personalized user needs have gradually become a research hotspot. Currently, most smart devices on the market rely on single voice or touch interaction methods, which have the following limitations: 1. The perception dimension is singular, making it impossible to simultaneously collect multi-dimensional data such as user physiological state, environmental information, and behavioral habits, thus making it difficult to build accurate user profiles; 2. The model has weak iteration capabilities and lacks a closed-loop optimization mechanism based on user feedback, making it unable to dynamically adjust the output strategy according to user habits; 3. Poor offline adaptability: Most devices rely on cloud computing, and cannot provide normal service when the network is interrupted, making it difficult to meet the usage needs of complex scenarios such as outdoor and weak network conditions.

[0003] Therefore, there is an urgent need for a portable intelligent sensing assistant that can achieve multimodal perception, personalized modeling, and closed-loop iterative optimization to solve the problems of insufficient perception dimensions, poor adaptability, and unsatisfactory interactive experience in existing technologies. Summary of the Invention

[0004] The purpose of this invention is to provide a portable multimodal intelligent sensing assistant and its working method. Through multimodal data collection, user thinking modeling and closed-loop iterative optimization, it realizes personalized intelligent assistance services, and has the ability to switch between online and offline adaptively, thereby improving the device's scene adaptability and user experience.

[0005] To achieve the above objectives, the present invention provides the following technical solution: 1. A portable multimodal intelligent sensing assistant It includes the terminal body, main control module, storage module, communication module, interaction module, power supply module, and multimodal sensing module; The terminal body is a hardware carrier, and the main control module, storage module, communication module, interaction module, power supply module and multimodal perception module are all integrated within the terminal body; The main control module is electrically connected to the storage module, communication module, interaction module, power supply module and multimodal sensing module respectively; The multimodal perception module includes an audio unit, a vision unit, an environmental detection unit, and a biological detection unit, which are used to collect user voice, visual information, environmental parameters, and physiological data, respectively. The power supply module provides stable power support for the entire system; The main control module is configured to: receive multi-dimensional data collected by the multimodal perception module, construct or retrieve user thinking models, generate personalized suggestions and output them through the interaction module, and iteratively optimize the user thinking model based on user feedback.

[0006] Furthermore, the main control module includes a user thinking modeling unit and a personalized decision output unit; the user thinking modeling unit is used to construct and update the user thinking model, and the personalized decision output unit is used to generate and output personalized suggestions based on the user thinking model.

[0007] Furthermore, the storage module is configured to store user thinking models, multimodal acquisition data, and user feedback data, and supports the main control module in retrieving existing models or storing updated model data.

[0008] Furthermore, the communication module is configured to enable data interaction between the device and an external platform / cloud, and the main control module can automatically switch between online and offline working modes according to the network status.

[0009] Furthermore, the interaction module includes at least one of a voice interaction unit, a display unit, or a touch unit, for outputting information to the user and receiving user feedback.

[0010] 2. A working method for a portable multimodal intelligent sensing assistant Includes the following steps: - S1: Device initialization, completing self-tests and parameter configurations for each module; - S2: The multimodal perception module collects multi-dimensional data about the user and the environment through audio units, vision units, environmental detection units, and biological detection units; - S3: Determine if this is the first time using the model or if an update is needed. If so, execute S4 to build / update the user's mental model; otherwise, execute S5 to retrieve data from the storage module. Existing user mind models; - S6: Clean and parse the collected multimodal data, remove invalid data and extract key features; - S7: Integrate and match the parsed data with the user's mental model to identify the user's state and needs; - S8: Based on the matching results, personalized suggestions are generated through the personalized decision output unit of the main control module; - S9: Output the personalized suggestions to the user through the interaction module; - S10: Collect user feedback on the personalized suggestions; - S11: Determine whether it is necessary to iteratively optimize the user's mental model. If so, execute S12 to update the model and store it in the storage module. - S13: Real-time monitoring of network status, automatically switching between online / offline working modes based on network connectivity; - S14: Return to standby mode, forming a closed-loop intelligent assistance process.

[0011] Compared with the prior art, the present invention has the following beneficial effects: 1. Multimodal perception capability: Through collaborative acquisition of audio, visual, environmental and biological multi-unit data, it achieves the acquisition of user and environmental data in all dimensions, providing comprehensive data support for personalized modeling; 2. Closed-loop iterative optimization: Through a closed-loop process of user feedback and model updates, the device can dynamically adapt to changes in user habits and needs, thereby improving the accuracy of services; 3. Online / Offline Adaptive: Through the coordinated control of the communication module and the main control module, network status monitoring and automatic mode switching are achieved to ensure normal use in weak network or no network scenarios; 4. Portability and practicality: With the terminal as the carrier, the integrated design realizes the miniaturization and portability of the device, which can be widely used in various scenarios such as daily learning, work, and health assistance. Attached Figure Description

[0012] Figure 1 This is a system structure block diagram of the present invention; Figure 2 This is a flowchart of the process of the present invention; In the diagram: 1-Terminal main body; 2-Main control module; 3-Storage module; 4-Communication module; 5-Interaction module; 6-Power supply module; 7-Multimodal perception module; 71-Audio unit; 72-Vision unit; 73-Environmental detection unit; 74-Biological detection unit; S1-Device initialization; S2-Multimodal information acquisition; S3-Determine if it is the first time using / updating the model; S4-Build / update the thinking model; S5-Retrieve the existing thinking model; S6-Data cleaning and parsing; S7-Data and model fusion and matching; S8-Generate personalized suggestions; S9-Interactive output suggestions; S10-Collect user feedback; S11-Determine if iterative optimization of the model is needed; S12-Update the model and store it; S13-Network status monitoring and mode switching; S14-Return to standby state. Detailed Implementation

[0013] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0014] Example 1: Portable Multimodal Intelligent Sensing Assistant like Figure 1As shown, the portable multimodal intelligent sensing assistant of this embodiment includes a terminal body 1, a main control module 2, a storage module 3, a communication module 4, an interaction module 5, a power supply module 6, and a multimodal sensing module 7.

[0015] The main terminal 1 is a small portable device. The main control module 2 uses a low-power microprocessor and integrates a user thinking modeling unit and a personalized decision output unit. The storage module 3 uses a Flash memory to store user models and collected data. The communication module 4 supports Bluetooth and 4G / 5G networks to enable interaction between the device and the cloud. The interaction module 5 includes an OLED display and a microphone to enable information output and voice interaction. The multimodal perception module 7 has an audio unit 71 that collects user voice, a vision unit 72 that collects user behavior and facial information, an environmental detection unit 73 that collects environmental parameters such as temperature, humidity, and light, and a biological detection unit 74 that collects physiological data such as user heart rate and blood oxygen. The power supply module 6 uses a lithium battery to provide power support for the entire system.

[0016] Example 2: Working method of a portable multimodal intelligent sensing assistant like Figure 2 As shown, the workflow of this embodiment is as follows: After the device starts up, it executes S1 initialization to complete module self-test; in S2, the multimodal perception module collects user voice, facial image, ambient temperature and humidity, and heart rate data; in S3, it determines that it is the first use, executes S4 to build an initial user mind model and stores it in the storage module (S5); in S6, it cleans the collected data and removes noisy data; in S7, it fuses and matches the parsed data with the model to identify the user's current state; in S8, it generates personalized health suggestions and outputs them through the display screen (S9); in S10, it collects user feedback, in S11 it determines that the model needs to be optimized, executes S12 to update the model and store it; in S13, it detects a network interruption, automatically switches to offline mode, and uses the local model to provide services; in S14, it returns to standby state, completing one closed-loop process.

Claims

1. A portable multimodal intelligent sensing assistant, characterized in that, It includes a terminal body (1), a main control module (2), a storage module (3), a communication module (4), an interaction module (5), a power supply module (6), and a multimodal sensing module (7); The terminal body (1) is a handheld or wearable structure; The main control module (2) integrates a user thinking modeling unit and a personalized decision output unit; The multimodal sensing module (7) is used to collect multi-dimensional information in real time; The power supply module (6) supplies power to each module.

2. The portable multimodal intelligent sensing assistant according to claim 1, characterized in that, The multimodal sensing module (7) includes an audio unit (71), a visual unit (72), an environmental detection unit (73), and a biological detection unit (74).

3. The portable multimodal intelligent sensing assistant according to claim 1, characterized in that, The storage module (3) is a non-volatile memory used to store user thinking feature models and multimodal perception data.

4. The portable multimodal intelligent sensing assistant according to claim 1, characterized in that, The communication module (4) supports multi-standard network communication and is configured to monitor network status in real time and automatically switch between online and offline modes.

5. A method for operating a portable multimodal intelligent sensing assistant based on any one of claims 1 to 4, characterized in that, Includes the following steps: S1: Device initialization, completing self-tests and parameter configurations for each module; S2: Collect multi-dimensional data of users and environment through the multimodal perception module (7); S3: Determine if this is the first time using the model or if the model needs to be updated. If yes, execute S4; otherwise, execute S5. S4: Build or update user mental models and store them; S5: Retrieve the existing user thinking model in the storage module (3); S6: Clean and analyze the collected multi-dimensional data; S7: Integrate and match the parsed data with the user's mental model; S8: Generate personalized suggestions; S9: Output the personalized suggestions through the interaction module (5); S10: Collect user feedback on the personalized suggestions; S11: Determine whether it is necessary to iteratively optimize the user's mental model; if so, execute S12. S12: Update the user's mental model and store it in the storage module (3); S13: Monitor network status in real time and automatically switch between online / offline working modes based on network connectivity; S14: Return to standby mode.

6. The working method according to claim 5, characterized in that, In step S4, when constructing or updating the user's thinking model, user thinking habits and decision preferences are extracted based on a preset machine learning algorithm.

7. The working method according to claim 5, characterized in that, In step S13, when the network is detected to be available, the system automatically switches to online mode and performs cloud model synchronization and data backup; when the network is unavailable, the system switches to offline mode and runs independently based on the locally stored model.

Images