Facial expression recognition and interaction method based on edge vision and multi-modal model

Abstract

The application relates to the field of artificial intelligence technology and discloses an expression recognition and interaction method based on edge vision and a multi-modal model, which comprises the following steps: S1, collecting an image sequence containing a user and pre-processing the image sequence; S2, inputting the standardized face image data into a visual language large model loaded locally by an edge computing device; S3, analyzing the structured multi-modal interaction instruction and driving multi-modal output hardware integrated by the edge computing device to perform an action of interacting with the user; and S4, based on a preset trigger condition, performing an incremental learning algorithm to update model parameters of the visual language large model online. Through the deployment of the visual language large model on the edge device and the execution of localized closed-loop incremental learning, the application realizes personalized interaction with low delay and high data security, and can generate a multi-modal response of voice, action and visual expression coordination.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a facial expression recognition and interaction method based on edge vision and multimodal models. Background Technology

[0002] In daily life, with the widespread adoption of smart home devices, social robots, and personal assistants, human-machine interaction is becoming increasingly frequent. To transform this interaction from simple command execution into more human-like communication, devices need the ability to understand users' nonverbal cues. Facial expressions are the most direct and richest channel for conveying emotions, intentions, and feedback.

[0003] In existing technologies, a mainstream approach to facial expression recognition leverages the powerful computing capabilities of cloud computing. This approach deploys large-scale, highly complex deep neural network models on cloud servers to centrally process and analyze user facial image data uploaded from edge devices. Thanks to the unrestricted computing resources of server clusters, these large models can be trained on massive and diverse datasets, achieving high classification accuracy and generalization ability when recognizing common, standardized basic facial expression categories.

[0004] However, the aforementioned existing technologies still have some shortcomings in practical applications. First, this technical approach relies on network communication between edge devices and cloud servers. The round-trip data transmission inevitably introduces communication latency, resulting in delayed responses from devices in real-time interactive scenarios. Simultaneously, uploading sensitive biometric information such as user facial images to external servers raises concerns about data privacy and security. Second, these general-purpose models, trained once in the cloud, have fixed functions after deployment. They lack mechanisms for adaptive learning for specific users and cannot capture and adapt to subtle differences in individual expression habits, leading to a lack of personalized interaction. Finally, the interactive output of existing technologies is usually based on a preset, single-modal response triggered by the recognition result, making it difficult to dynamically generate complex interactive behaviors that are temporally and semantically coordinated across multiple modalities such as voice, actions, and screen displays according to specific contexts. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an expression recognition and interaction method based on edge vision and multimodal models. This method solves the problems of high latency and privacy and security risks caused by reliance on cloud computing in existing expression recognition and interaction methods, as well as the problems of local model fixation, inability to perform personalized adaptive learning for specific users, single interaction output, and lack of multimodal collaborative capabilities.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] The first aspect of this invention provides a facial expression recognition and interaction method based on edge vision and a multimodal model, the method comprising:

[0008] S1. Acquire an image sequence containing the user and preprocess the image sequence to generate standardized facial image data representing the user's current state;

[0009] S2. Input the standardized facial image data into the visual language big model loaded locally on the edge computing device, and have the visual language big model analyze it to generate structured multimodal interaction instructions;

[0010] S3. Parse the structured multimodal interaction instructions and drive the multimodal output hardware integrated in the edge computing device to perform actions that interact with the user;

[0011] S4. Collect and process the image sequence of user feedback to generate standardized facial image data of feedback. Record the standardized facial image data, the generated structured multimodal interaction instructions, and the standardized facial image data of feedback together as the interaction history. Based on preset trigger conditions, execute an incremental learning algorithm to update the model parameters of the visual language big model online.

[0012] In one specific embodiment, the step of acquiring an image sequence containing the user and preprocessing the image sequence specifically includes: performing face detection on each image frame in the image frame sequence to locate the bounding box of the user's face; performing facial key point localization within the bounding box; and, based on the facial key points, correcting the face pose by calculating an affine transformation matrix, and normalizing the size and illumination of the image.

[0013] In a specific embodiment, the step of analyzing the visual language big model to generate structured multimodal interaction instructions specifically includes: converting the standardized facial image data into visual feature embeddings through a visual encoder; concatenating the visual feature embeddings with the text feature embeddings corresponding to preset text prompts to form a unified multimodal input sequence; and inputting the multimodal input sequence into the visual language big model to generate the structured multimodal interaction instructions.

[0014] Preferably, the structured multimodal interaction instructions are in JSON format, and the instructions include text content for voice interaction, action identifiers for physical body movements, or visual expression identifiers for screen display.

[0015] In one specific embodiment, the step of parsing the structured multimodal interaction instructions specifically includes: receiving an instruction string in a predetermined JSON format output by the visual language big model; and performing syntactic analysis on the instruction string through a parser module to extract the key-value pairs contained therein, wherein the key-value pairs include text content for voice interaction, action identifiers for physical body movements, and visual expression identifiers for screen display.

[0016] In a specific embodiment, the step of driving the multimodal output hardware integrated in the edge computing device to perform actions that interact with the user specifically includes at least one of the following operations: transmitting the voice interaction text content extracted from the instruction to a local text-to-speech engine to generate audio waveform data that drives the speaker to emit sound; querying the physical limb movement identifiers extracted from the instruction in a locally stored action library to obtain a set of specific servo motor control sequences, and generating pulse width modulation signals according to the control sequences to drive the servo motors to complete physical actions; or, querying the screen visual expression identifiers extracted from the instruction in a locally stored resource library to obtain corresponding animation sequence files, and rendering the image frames of the animation sequence files onto the display screen through a graphics processing unit.

[0017] In one specific embodiment, the step of acquiring and processing the image sequence of user feedback to generate standardized facial image data of feedback specifically includes: after the edge computing device performs an action to interact with the user, after a preset time interval, acquiring a sequence of feedback images containing the user; and performing preprocessing operations on the feedback image sequence, the preprocessing operations including performing face detection, facial key point localization, and pose correction and normalization processing based on affine transformation, to generate the standardized facial image data of feedback.

[0018] In a specific embodiment, the step of performing an incremental learning algorithm to update the model parameters of the visual language large model online includes: when a preset trigger condition is met, sampling a small batch of interaction event tuples from the recorded interaction history to form a training dataset for model fine-tuning; the preset trigger condition includes the device accumulating a preset threshold of interactions or the device entering an idle state and being connected to a power source; using an elastic weight consolidation algorithm to update the model parameters of the visual language large model by constructing a total loss function, which includes a loss calculated on the current training dataset and a quadratic penalty term, which measures the importance of each parameter in the model to existing knowledge and penalizes changes in highly important parameters to suppress forgetting of general knowledge while adapting to new user data; and finally, updating the model parameters of the visual language large model using a stochastic gradient descent algorithm based on the gradient of the total loss function.

[0019] A second aspect of the present invention provides an expression recognition and interaction system based on edge vision and a multimodal model, the system being configured to perform the method described in any of the preceding claims, comprising:

[0020] The front-end perception module collects image sequences containing the user and preprocesses the image sequences to generate standardized facial image data representing the user's current state.

[0021] The edge processing module inputs the standardized facial image data into a locally loaded visual language big model for analysis to generate structured multimodal interaction instructions, records the interaction history, and executes an incremental learning algorithm to update the model parameters of the visual language big model online based on preset trigger conditions.

[0022] The multimodal interaction module parses the structured multimodal interaction instructions and drives its integrated multimodal output hardware to perform actions that interact with the user.

[0023] The data storage unit persistently stores the model parameter file of the large visual language model and the interaction history.

[0024] Preferably, the edge processing module includes a system-on-a-chip consisting of a central processing unit, a graphics processing unit, and a dedicated neural network processing unit; the central processing unit, the graphics processing unit, and the dedicated neural network processing unit work together to perform the analysis and reasoning of the large visual language model and the update calculation of the incremental learning algorithm.

[0025] This invention provides a facial expression recognition and interaction method based on edge vision and a multimodal model. It has the following beneficial effects:

[0026] 1. This invention achieves low-latency response and localized processing of user data by executing all perception, reasoning, interaction, and learning steps locally on an edge computing device. Specifically, the large visual language model is deployed on a local processor after lightweight processing. All image data containing the user's face and interaction logs are processed and stored in the local storage unit, without relying on network or cloud server communication, thereby ensuring the real-time nature of the interaction process and guaranteeing the physical isolation and security of the user's personal data.

[0027] 2. This invention constructs a closed-loop adaptive learning mechanism that incorporates user feedback, enabling the interaction model to undergo personalized iterations for specific users. Specifically, the system records the user's initial facial image, the device-generated interaction commands, and the user's subsequent facial image feedback on that interaction as a complete interaction history tuple, and uses the Elastic Weight Consolidation (EWC) incremental learning algorithm to fine-tune the parameters of the local visual language model online. This method allows the model to learn the specific user's expression habits while suppressing the forgetting of existing general knowledge, thereby gradually adjusting its interaction strategy to adapt to individual users.

[0028] 3. This invention achieves collaborative interaction across three channels—voice, body language, and visual expression—by generating structured multimodal instructions. Specifically, the visual language big data model outputs a unified JSON-formatted instruction containing voice text content, action identifiers, and expression identifiers. After parsing this instruction, the multimodal interaction module can synchronously drive the speaker, servo motor, and display screen to execute a time-coordinated composite interactive action guided by a unified intent, thus forming a multi-dimensional, non-single interactive output. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0030] Figure 2 This is a system architecture diagram of the present invention. Detailed Implementation

[0031] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] Please see Figure 1 , Figure 1This is a flowchart illustrating an expression recognition and interaction method based on edge vision and multimodal models according to an embodiment of the present invention. The method may include the following steps:

[0033] S100, Front-end perception and data acquisition stage. The edge computing device uses its integrated visual sensors, such as a camera module, to capture scenes containing the user's face in real time and acquire a continuous sequence of image frames. For each acquired image frame, a series of preprocessing operations are performed, including face detection to locate the bounding box of the user's face in the image, facial key point localization to obtain the precise coordinates of facial features, and data standardization. The data standardization operation corrects the facial pose through affine transformation and normalizes the image size and illumination, ultimately generating standardized facial image data.

[0034] S200, Execute the localized deep analysis and inference phase. All computations in this phase are performed on the local processor of the edge computing device. The processor loads a pre-deployed, lightweight Visual Language Model (VLM). This VLM receives standardized facial image data output from the front-end perception and data acquisition phase as input. The VLM analyzes the input visual information to infer the user's real-time emotional state and interaction intent, and based on the inference results, generates a set of structured multimodal interaction commands.

[0035] S300, Execution of the multimodal interaction stage. The edge computing device parses the multimodal interaction instructions generated in the previous stage. These instructions contain data for driving different hardware units. Based on the instructions, the device drives its integrated multimodal output hardware. This hardware includes at least one of the following: a speaker for generating speech, one or more servo motors for performing physical actions, or a screen for displaying visual information. By driving this hardware, the device completes the interaction with the user.

[0036] S400, Personalized Learning and Model Adaptive Iteration Phase. The edge computing device continuously records the interaction history in its local memory. This recorded interaction history includes the user's facial image data at specific points in time, the interaction commands generated by the device, and the user's subsequent visual feedback on the interaction. Based on preset trigger conditions, the system uses the stored interaction history data to execute an incremental learning algorithm. This incremental learning algorithm fine-tunes and updates the model parameters of the locally deployed visual language model online. Through this iterative process, the model gradually adapts to the specific user's facial expression habits, thereby adjusting the generation strategy of subsequent interaction commands.

[0037] Please see Figure 1The front-end perception and data acquisition stage (S100) aims to transform the raw image information acquired through the visual sensor into standardized, structured data required for subsequent deep analysis models.

[0038] Step S110: High Frame Rate Image Sequence Acquisition. A video stream containing the user is acquired using a high-definition camera module integrated into the edge computing device, with preset parameters. These parameters include a resolution of at least 1920x1080 pixels to ensure clarity of facial details, and a frame rate of at least 30 frames per second (FPS) to capture continuous dynamic changes in facial expressions. The camera module decodes the acquired video stream into a series of digital image frames arranged in chronological order. ,in Represents a point in time The captured image frame.

[0039] Step S120: Face detection and tracking. For each frame image acquired from the video stream... The system first performs face detection. In this embodiment, an algorithm based on a multi-task cascaded convolutional neural network (MTCNN) is used. This algorithm uses a cascaded structure to progressively filter candidate regions from the image and regress accurate face bounding boxes. The bounding box is defined as follows: In the formula, The coordinates of the top left corner of the frame. and These represent the width and height of the frame, respectively. To ensure tracking stability across consecutive video frames, this embodiment introduces a Kalman filter algorithm. This Kalman filter algorithm is based on the previous moment... Given the bounding box state, predict the current time step. The bounding box position is determined, and the predicted value is corrected and updated using the MTCNN detection results of the current frame, thereby outputting a smooth and continuous face tracking trajectory.

[0040] Step S130: Facial key point localization and feature extraction. Based on the facial bounding box determined in step S120... Internally, the system calls a pre-trained convolutional neural network model to locate facial key points. This convolutional neural network model takes an image within a bounding box as input and outputs an image containing... A collection of facial key points .

[0041] ;

[0042] In the formula, This represents a set of facial landmarks. This set contains complete data on all located facial landmarks. This represents the first facial landmark; This represents the second facial key point; Representing the Key facial features.

[0043] Step S140: Visual Data Standardization. To eliminate data interference caused by user head posture, distance from the camera, and changes in ambient lighting, the extracted face image and its key points must be standardized. This standardization process is divided into geometric standardization and appearance standardization.

[0044] The purpose of geometric normalization is to align pose and scale. This embodiment achieves this by calculating an affine transformation matrix. This is achieved by mapping several detected keypoints (e.g., the center points of both eyes) from their coordinates in the current image to a set of target coordinates predefined in a standard reference coordinate system. This transformation process can be described by the following equation:

[0045] ;

[0046] In the formula, This represents the homogeneous coordinate vector of the transformed point. It is a column vector used to conveniently represent translation operations in two-dimensional linear transformations. Represents the origin point The x-coordinate in the target coordinate system after affine transformation. Represents the origin point The ordinate in the target coordinate system after affine transformation; This represents a 3×3 affine transformation matrix that contains information about all geometric transformations, including rotation, scaling, shearing, and translation. Represents the homogeneous coordinate vector of a point in the original image; Represents the x-coordinate of a pixel in the original image; Represents the ordinate of a pixel in the original image; Represents the affine transformation matrix The specific unfolding form; These are elements of the transformation matrix, primarily affecting scaling and rotation in the horizontal direction; The elements of the transformation matrix primarily affect horizontal shearing and rotation. The elements of the transformation matrix represent the amount of translation in the horizontal direction; These are elements of the transformation matrix, primarily affecting vertical shearing and rotation; These are elements of the transformation matrix, primarily affecting scaling and rotation in the vertical direction; The elements of the transformation matrix represent the translation amount in the vertical direction.

[0047] The purpose of appearance normalization is to reduce the impact of uneven illumination. This embodiment applies gamma correction to the geometrically normalized image. The correction formula is:

[0048] ;

[0049] In the formula, For the input image in coordinates Pixel intensity value at that location, It outputs the pixel intensity value at the corresponding location in the image. It is a constant (usually set to 1). This is the gamma value, a hyperparameter used to adjust image contrast. After all the above steps, the system finally outputs a standardized facial image. This image will serve as input for the next stage of the VLM analysis module.

[0050] Please see Figure 1 The localized deep analysis and inference phase (S200) is then executed. This phase runs on the local processor of the edge computing device and is responsible for processing the standardized facial images output by the front-end module. It performs deep understanding and generates structured instructions to guide subsequent interactions.

[0051] The Visual Language Model (VLM) used in this embodiment consists of a visual encoder and a large language model. To enable deployment on edge devices, this VLM is a version processed using model lightweighting techniques.

[0052] Perform visual information encoding. Standardized facial images. The image is first fed into the visual encoder of the VLM, which in this embodiment is a Visual Transformer (ViT) network. The ViT processes the input two-dimensional image. The image is segmented into a series of fixed-size, non-overlapping image patches. Each patch is then mapped into a one-dimensional vector by a linear projection layer. This process transforms image information into a sequence of vectors. To preserve the spatial location information of the image patches, learnable positional embeddings are added to each vector, ultimately generating a set of visual feature embeddings. .

[0053] ;

[0054] In the formula, This represents the complete processing flow of a visual encoder; It is the input standardized image; It is the output visual feature embedding sequence.

[0055] Perform multimodal input construction and prompting engineering. The system embeds the visual features generated in the previous step. With preset text prompts The input sequences are fused to construct the final input sequence fed into a large language model. The text prompts... This is a piece of natural language text used to guide the model in reasoning for a specific task. The text is first converted into a sequence of integers by a tokenizer, and then mapped to text feature embeddings through a word embedding layer. Subsequently, the visual feature embeddings and text feature embeddings are concatenated to form a unified multimodal input sequence. .

[0056] ;

[0057] In the formula, For a unified multimodal input sequence; This represents a sequence concatenation operation; It is the output visual feature embedding sequence; Embedding text features.

[0058] Perform VLM inference and generate structured instructions. Multimodal input sequence. The large language model portion is fed into the VLM. This large language model employs a multi-head self-attention mechanism, enabling each visual and text embedding in the sequence to interact and be weighted and fused with all other embeddings. The large language model generates the output sequence in an autoregressive manner. That is, when generating each new token, all previously generated tokens are used as context. In this embodiment, the model's output is limited to a predefined JSON (JavaScript Object Notation) format to ensure that it is a machine-readable structured instruction.

[0059] Lightweight model implementation and efficient inference. To ensure the VLM described above can run on edge devices with limited computing power and storage resources, this embodiment employs model quantization technology before model deployment. This technology converts the weight parameters in the model network from 32-bit floating-point numbers (FP32) to 8-bit integers (INT8). The linear quantization process is as follows:

[0060] ;

[0061] In the formula, It is the original floating-point value; It is the quantized integer value; It is a scaling factor calculated based on the range of floating-point values; It is the zero-point offset; The function represents the rounding function, which rounds the result of the numerical calculation within the parentheses to the nearest integer.

[0062] Please see Figure 1 The multimodal interaction execution phase (S300) is responsible for transforming the structured multimodal interaction instructions generated in the localized deep analysis and inference phase (S200) into physical interaction behaviors on the edge computing device.

[0063] The processor on the edge computing device performs the interaction command parsing operation. The processor receives a command string conforming to a predetermined JSON format from the output of the visual language big data model. The parser module performs syntactic analysis on the string, extracting multiple key-value pairs contained therein. These key-value pairs include at least the text content for voice interaction, action identifiers for physical body movements, and visual expression identifiers for screen display.

[0064] The process involves generating and executing voice interaction. After the parser extracts the text content of the voice interaction, it sends it to a locally deployed text-to-speech (TTS) engine. This TTS engine converts the input text string into digitized audio waveform data, such as an audio stream in Pulse Code Modulation (PCM) format. This audio stream is then sent to a digital-to-analog converter (DAC), which converts it into an analog electrical signal. This analog electrical signal ultimately drives the speaker on the device to produce sound, completing the playback of the voice content.

[0065] Simultaneously, the system generates and executes limb movements. The parser extracts the motion identifiers of the physical limb movements. The system queries this identifier in a locally stored motion library, which predefines a mapping between each identifier and a specific set of servo motor control sequences. (Control Sequences) It consists of multiple control points, which define the trajectory and time of the action:

[0066] ;

[0067] In the formula, For a complete control sequence; The first control point in the sequence; The second control point in the sequence; The first in the sequence The last control point; Rotate by the target angle.

[0068] The device's microcontroller generates a corresponding pulse width modulation (PWM) control signal for the specified servo motor based on this sequence. The duty cycle of the PWM signal... Rotation angle relative to the target There exists a definite linear relationship:

[0069] ;

[0070] In the formula, A proportionality constant; It is the offset constant; Rotate by the target angle; This refers to the duty cycle of the PWM signal. It is determined by following the control sequence. It outputs a precise PWM signal, and the microcontroller drives the servo motor to complete preset physical actions such as nodding and waving.

[0071] In addition, the system performs screen visual expression generation and execution. The parser extracts the visual expression identifier. The system queries the locally stored resource library for this identifier to obtain the corresponding animation sequence file. This file contains a series of consecutive image frames. The device's graphics processing unit (GPU) or central processing unit (CPU) is responsible for rendering these image frames sequentially at a preset frame rate and writing them to the frame buffer of the device's display screen. By continuously refreshing the content of the display screen, the device presents dynamic visual expressions on the screen that match the current interaction context.

[0072] Please see Figure 1 The process then proceeds to the personalized learning and model adaptive iteration phase (S400). This phase empowers edge computing devices to self-optimize through interaction with specific users, with all computation and data storage performed locally on the device.

[0073] The system performs user interaction data recording and feedback quantification operations. Each complete interaction event is continuously recorded in the form of a structured log in the device's local non-volatile storage unit. Each interaction event is recorded as a data tuple. Its composition is as follows:

[0074] ;

[0075] In the formula, For data tuples; At a certain point in time A standardized user facial image is captured and processed in step S100; It is the device at a certain point in time. Structured interactive instructions generated for this image; After a time interval following interactive execution Captured and normalized user facial images, which are used as indicators of interaction behavior. Visual feedback.

[0076] Incremental learning triggering and data preparation operations are performed. In this embodiment, the model update process is triggered by preset conditions, including: the cumulative number of interactions completed by the device reaches a preset threshold. Alternatively, the device may enter an idle state and be connected to a power source. When the trigger condition is met, the system extracts a mini-batch of interaction event tuples from the locally stored interaction logs according to a preset sampling strategy, forming a training dataset for model fine-tuning. .

[0077] Perform online model fine-tuning and parameter update operations. This online model fine-tuning and parameter update operation aims to utilize the dataset... Parameters of the locally deployed Visual Language Model (VLM) Update the model. To prevent the model from forgetting its existing general knowledge when learning the personalized expressions of new users, this embodiment uses the Elastic Weight Consolidation (EWC) algorithm. This algorithm is implemented by adding a quadratic penalty term to the loss function. The total loss function optimized when updating parameters... Defined as:

[0078] ;

[0079] In the formula, The total loss function optimized when updating parameters; This represents the current set of model parameters for the VLM; It is the first one One parameter; In the current training dataset The loss calculated above could be, for example, the cross-entropy loss between the model-generated instructions and the expected instructions; This is the first time the model has been solidified after the previous learning task has been completed. The optimal values ​​of each parameter; It is a hyperparameter used to adjust the strength of protection for old knowledge; It is the first Parameters The measure of importance is approximated by the diagonal elements of the Fisher Information Matrix, which reflects the importance of the parameters to the old task.

[0080] Parameter updates are performed using stochastic gradient descent (SGD) or its variants (such as Adam), with the following single-step update rule:

[0081] ;

[0082] In the formula, These are the model parameters for the current iteration step; These are the updated parameters; It is the learning rate; This is the gradient of the total loss function with respect to the current parameters. Through this iterative update process, VLM effectively suppresses the destruction of important old knowledge while adapting to new user data, thus achieving personalized adaptive iteration.

[0083] Please see Figure 2 , Figure 2 This is an embodiment of an expression recognition and interaction system architecture based on edge vision and multimodal models according to the present invention. The system is an integrated edge computing device, which internally includes: a front-end perception module 210, an edge processing module 220, a multimodal interaction module 230, and a data storage unit 240.

[0084] The front-end perception module 210 physically comprises a high-definition camera module, which includes an optical lens and a CMOS image sensor. This front-end perception module 210 is configured to execute all steps of the front-end perception and data acquisition stage (S100) in the aforementioned method embodiment of the present invention, specifically including: acquiring an image sequence containing the user at a preset frame rate, and performing face detection, facial key point localization, and visual data standardization processing in each frame of the image sequence, ultimately outputting standardized facial image data.

[0085] The edge processing module 220 is a system-on-a-chip that integrates a central processing unit (CPU), a graphics processing unit (GPU), and a dedicated neural network processing unit (NPU). This edge processing module 220 is configured to execute the localized deep analysis and inference phase (S200) and the personalized learning and model adaptive iteration phase (S400) in the aforementioned method embodiments of the present invention. Specifically, the edge processing module 220 is responsible for loading and running a lightweight visual language model from the data storage unit 240, analyzing and inferring from the standardized facial image data input by the front-end perception module 210 to generate structured interaction instructions, and executing an incremental learning algorithm based on preset conditions to update the model's parameters.

[0086] The multimodal interaction module 230 physically comprises a speaker driven by an audio codec, one or more servo motors driven by a pulse width modulation (PWM) signal controller, and a display screen controlled by a graphics driving circuit. This multimodal interaction module 230 is configured to execute the multimodal interaction execution phase (S300) in the aforementioned method embodiment of the present invention. Specifically, the multimodal interaction module 230 receives and parses the structured interaction instructions generated by the edge processing module 220, and drives the speaker to generate voice, drives the servo motors to perform physical actions, and drives the display screen to present visual animations according to the instruction content.

[0087] The data storage unit 240 physically comprises random access memory (RAM) and non-volatile flash memory. This data storage unit 240 is configured to: persistently store the operating system, model parameter files of the large visual language model, and user interaction history logs in the non-volatile flash memory; and provide temporary data caching for real-time computation by the edge processing module 220 in the random access memory.

[0088] In this embodiment, the front-end perception module 210, edge processing module 220, multimodal interaction module 230, and data storage unit 240 communicate via an internal data bus. During operation, the front-end perception module 210 sends processed, standardized facial image data to the edge processing module 220. After analysis, the edge processing module 220 sends the generated interaction commands to the multimodal interaction module 230 for execution. Simultaneously, the edge processing module 220 reads model parameters and historical data from the data storage unit 240 and writes new interaction logs and updated model parameters back to the data storage unit 240. The collaborative work of these modules constitutes a complete, locally running adaptive interaction system.

Claims

1. An expression recognition and interaction method based on edge vision and multi-modal model, characterized in that, The method comprises the following steps: S1, collecting an image sequence containing a user, and pre-processing the image sequence to generate standardized facial image data representing the current state of the user; S2, inputting the standardized facial image data into a visual language large model loaded locally by an edge computing device, and analyzing the visual language large model to generate structured multi-modal interaction instructions; S3, parsing the structured multi-modal interaction instructions and driving the multi-modal output hardware integrated by the edge computing device to perform actions of interacting with the user; S4, collecting and processing the image sequence of the user feedback to generate standardized facial image data, recording the standardized facial image data, the generated structured multi-modal interaction instructions, and the standardized facial image data of the feedback together as an interaction history, and based on a preset trigger condition, performing an incremental learning algorithm to update the model parameters of the visual language large model online; The structured multi-modal interaction instructions are in JSON format, and the structured multi-modal interaction instructions include text content for voice interaction, action identifiers for physical limb actions, and visual expression identifiers for screen display; In step S2, the analysis by the visual language large model to generate structured multi-modal interaction instructions specifically includes: Converting the standardized facial image data into visual feature embeddings through a visual encoder; Splicing the visual feature embeddings with text feature embeddings corresponding to preset text prompts to form a unified multi-modal input sequence; Inputting the multi-modal input sequence into the visual language large model to generate the structured multi-modal interaction instructions.

2. The expression recognition and interaction method based on edge vision and multi-modal model according to claim 1, characterized in that, In step S1, the collection of the image sequence containing the user and the pre-processing of the image sequence specifically includes: Performing face detection on each image frame in the image frame sequence to locate the bounding box of the user's face; Performing face key point positioning within the bounding box; Based on the face key points, correcting the face pose by calculating an affine transformation matrix, and normalizing the size and illumination of the image.

3. The expression recognition and interaction method based on edge vision and multi-modal model according to claim 1, characterized in that, In step S3, the parsing of the structured multi-modal interaction instructions specifically includes: Receiving an instruction string in a predetermined JSON format output by the visual language large model; Performing syntax analysis on the instruction string through a parser module to extract key-value pairs contained therein, including text content for voice interaction, action identifiers for physical limb actions, and visual expression identifiers for screen display.

4. The expression recognition and interaction method based on edge vision and multi-modal model according to claim 1, characterized in that, In step S3, the driving of the multi-modal output hardware integrated by the edge computing device to perform actions of interacting with the user specifically includes: Transferring the voice interaction text content extracted from the instruction to a local text-to-speech engine to generate audio waveform data for driving the speaker to sound; The physical limb action identifier extracted from the instruction is queried in a locally stored action library to obtain a specific servo motor control sequence, and a pulse width modulation signal is generated according to the control sequence to drive the servo motor to complete the physical action; The screen visual expression identifier extracted from the instruction is queried in a locally stored resource library to obtain a corresponding animation sequence file, and an image frame of the animation sequence file is rendered to a display screen by a graphics processing unit.

5. The expression recognition and interaction method based on edge vision and multi-modal model according to claim 1, characterized in that, In step S4, the image sequence of the user feedback is collected and processed to generate standardized facial image data of the feedback, specifically including: After the edge computing device performs the action of interacting with the user, a feedback image sequence containing the user is collected after a preset time interval; The feedback image sequence is preprocessed, including performing face detection, facial key point positioning, and pose correction and normalization based on affine transformation, to generate the standardized facial image data of the feedback.

6. The expression recognition and interaction method based on edge vision and multi-modal model according to claim 1, characterized in that, In step S4, the incremental learning algorithm is used to update the model parameters of the visual language large model, specifically including: When a preset trigger condition is met, a small batch of interaction event tuples is sampled from the recorded interaction history to form a training data set for model fine-tuning; The preset trigger condition includes that the number of interactions completed by the device reaches a preset threshold or the device enters an idle state and is connected to a power supply; An elastic weight consolidation algorithm is used to update the model parameters of the visual language large model by constructing a total loss function containing a quadratic penalty term; The quadratic penalty term is used to measure the importance of each parameter in the model to the existing knowledge, and a penalty is imposed on the change of the parameter with high importance, so as to adapt to new user data while inhibiting forgetting of general knowledge; The model parameters of the visual language large model are updated according to the gradient of the total loss function by a stochastic gradient descent algorithm.

7. The expression recognition and interaction system based on edge vision and multi-modal model according to any one of claims 1-6, characterized in that, It includes: The front-end perception module collects an image sequence containing the user and pre-processes the image sequence to generate standardized facial image data representing the current state of the user; The edge processing module inputs the standardized facial image data into a locally loaded visual language large model for analysis to generate structured multi-modal interaction instructions, records the interaction history, and based on a preset trigger condition, performs an incremental learning algorithm to update the model parameters of the visual language large model online; The multi-modal interaction module analyzes the structured multi-modal interaction instructions and drives the integrated multi-modal output hardware to perform actions of interacting with the user; The data storage unit persistently stores the model parameter file of the visual language large model and the interaction history.

8. The edge vision and multi-modal model based expression recognition and interaction system of claim 7, wherein, The edge processing module includes a central processing unit, a graphics processing unit, and a system on a chip of a special neural network processing unit; The central processing unit, the graphics processing unit, and the special neural network processing unit work together to perform analysis and inference of the visual language large model and update calculation of the incremental learning algorithm.

Images