Intelligent interaction system and method based on multi-modal large model

Abstract

The invention relates to the technical field of intelligent interaction systems, and discloses an intelligent interaction system and method based on a multi-modal large model. The system comprises a multi-modal input layer, a context sensing module, a cross-modal fusion module, a dynamic response generation module and a system monitoring module. The multi-modal input layer preprocesses multi-modal data; the context sensing module constructs a dynamic context vector; the cross-modal fusion module fuses the data to generate joint features; the dynamic response generation module generates a response strategy through hierarchical decision; and the system monitoring module monitors the key indexes and triggers an optimization mechanism. All the modules work cooperatively, multi-modal data deep fusion, precise context sensing, efficient dynamic response and system self-optimization are achieved, the accuracy, adaptability and stability of intelligent interaction are improved, and the method can be widely applied to multiple fields such as intelligent home and intelligent customer service.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent interaction systems, and in particular to an intelligent interaction system and method based on a multimodal large model. Background Art

[0002] With the rapid development of artificial intelligence (AI), intelligent interactive systems have become widely used in our lives and work. From smart home devices to intelligent customer service, from smart car systems to virtual assistants, intelligent interactive systems are ubiquitous. However, existing intelligent interactive systems still face numerous challenges, which hinder their further development and the improvement of user experience.

[0003] When it comes to multimodal data processing, most current intelligent interactive systems lack a deep understanding of the integration of data from different modalities. Text, voice, and image data each have unique characteristics and expressions. Effectively integrating these multimodal data to complement and synergize them is a key issue. For example, in a smart home scenario, users may control devices simultaneously through voice commands and image manipulation on a mobile app. Existing systems struggle to accurately integrate these two modalities, leading to inaccuracies in understanding user intent and affecting the accuracy of device control.

[0004] Insufficient contextual awareness is also a major challenge. Intelligent interaction systems need to understand contextual information such as the user's current environment, historical interactions, and device operating status to provide more personalized and intelligent services. However, existing systems often only have access to limited contextual information or are unable to adjust interaction strategies based on this information in real time. Take intelligent customer service as an example. When users ask the same question at different times and in different scenarios, the customer service system is unable to combine the user's previous consultation records and the current context to provide targeted responses, making the interaction process seem stilted and unintelligent.

[0005] Cross-modal fusion technology is still immature. Although some systems have attempted cross-modal fusion, there are deficiencies in feature alignment and semantic fusion. The feature spaces of data from different modalities vary greatly, making it difficult to find an effective way to align and fuse them into a unified feature representation. This results in an inability to fully utilize the advantages of multimodal data when generating interactive responses, and the quality and accuracy of the response content needs to be improved. For example, in intelligent in-vehicle navigation systems, when combining voice navigation instructions and map image information, due to poor cross-modal fusion, there may be inconsistencies between voice prompts and map displays, causing trouble for users.

[0006] In addition, the dynamic response generation mechanism of existing intelligent interaction systems is not flexible enough. They usually rely on preset rules or simple models to generate responses and cannot adaptively adjust according to complex and changing scenarios and user needs. When faced with vague and polysemous inputs from users, the system often fails to give accurate and appropriate responses. Moreover, the system lacks effective self-monitoring and optimization capabilities. When performance degradation or abnormal situations occur, it cannot be detected and repaired in a timely manner, affecting the stability and reliability of the system. Summary of the Invention

[0007] The purpose of the present invention is to provide an intelligent interaction system and method based on a multimodal large model to solve the problems raised in the above background technology.

[0008] To achieve the above object, the present invention provides the following technical solution: An intelligent interaction system based on a multimodal large model, the system includes a multimodal input layer, a context awareness module, a cross-modal fusion module, a dynamic response generation module, and a system monitoring module;

[0009] The multimodal input layer is used to receive multimodal data input by text, voice, image, and sensors, and perform standardized preprocessing on the multimodal data;

[0010] The context awareness module constructs a dynamic context representation vector by collecting user historical behavior data, environmental state information, and device operation parameters;

[0011] The cross-modal fusion module uses a spatio-temporal correlation modeling algorithm to perform feature alignment and semantic fusion on the preprocessed multimodal data and the dynamic context representation vector to generate a cross-modal joint feature;

[0012] The dynamic response generation module generates an interaction response strategy adapted to the current scenario based on the cross-modal joint feature through a hierarchical decision-making mechanism, including voice output, image display, or control instructions;

[0013] The system monitoring module real-time monitors the feature alignment error value of the cross-modal fusion module, the decision delay value of the dynamic response generation module, and the data update frequency of the context awareness module. If the feature alignment error value exceeds a preset error threshold, the decision delay value exceeds a preset delay threshold, or the data update frequency is lower than a preset frequency threshold, a system exception signal is generated and an adaptive optimization mechanism is triggered.

[0014] Preferably, the specific processing process of the multimodal input layer is as follows:

[0015] In the text data processing stage, the input text is segmented and semantically vectorized, and the semantic vector is matched with a preset domain knowledge base to generate a text feature vector;

[0016] In the speech data processing stage, the spectral and intonation features of the speech signal are extracted and combined with the speech-to-text model to generate a speech semantic vector;

[0017] In the image data processing stage, the key objects in the image are identified through the target detection algorithm, and the object attribute features and spatial relationship features are extracted to generate the image feature vector;

[0018] In the sensor data processing stage, the ambient light intensity, temperature value and user position coordinates are collected, and the deviation value of the ambient light intensity from the preset light threshold is marked as the light analysis value, the deviation value of the temperature value from the preset temperature threshold is marked as the temperature analysis value, and the deviation value of the user position coordinates from the preset interaction area is marked as the position analysis value;

[0019] If the illumination analysis value, temperature analysis value or position analysis value exceeds the corresponding preset threshold, an environmental abnormality signal is generated and sent to the dynamic response generation module.

[0020] Preferably, the context awareness module further includes an interaction trajectory analysis unit, and the specific operation process is as follows:

[0021] Collecting the user's interaction operation sequence in the current session, and dividing the interaction operation sequence into multiple interaction segments according to timestamps;

[0022] Extract the operation type, operation duration, and operation object association for each interaction segment, mark the matching degree between the operation type and the preset interaction mode as the mode analysis value, mark the deviation between the operation duration and the preset duration as the duration utility value, and mark the ratio of the operation object association to the preset association threshold as the association analysis value;

[0023] The interaction segment utility value is obtained by weighted calculation of the pattern analysis value, duration utility value and association analysis value. If the interaction segment utility value is lower than the preset utility threshold, an interaction inefficiency signal is generated and sent to the dynamic response generation module to adjust the response strategy.

[0024] Preferably, the cross-modal fusion module adopts a feature alignment method that combines contrastive learning and self-supervised learning, specifically including:

[0025] Construct a cross-modal contrastive learning framework to map text feature vectors, speech semantic vectors, and image feature vectors into a unified embedding space;

[0026] By calculating the cosine similarity between the eigenvectors of different modalities, an inter-modal alignment matrix is generated;

[0027] Using a self-supervised learning algorithm to perform sparse processing on the inter-modality alignment matrix, and eliminating eigenvector pairs with alignment below a preset threshold;

[0028] The filtered feature vectors are weighted and fused to generate the cross-modal joint features.

[0029] Preferably, the cross-modal fusion module further includes an abnormal alignment detection unit, and the specific analysis process is as follows:

[0030] During the feature alignment process, the distribution entropy value of each modal feature vector is calculated in real time, and the deviation value between the distribution entropy value and the preset entropy threshold is marked as the entropy segregation value;

[0031] The sparsification ratio value of the inter-modal alignment degree matrix is collected, and the deviation value between the sparsification ratio value and the preset ratio threshold is marked as the sparsity segregation value;

[0032] If the entropy segregation value or the sparsity segregation value exceeds the corresponding preset threshold, an alignment abnormal signal is generated and the feature realignment mechanism is triggered.

[0033] Preferably, the dynamic response generation module adopts a hierarchical decision-making mechanism, which specifically includes:

[0034] In the primary decision-making layer, according to the cross-modal joint features, a preset response template library is matched to generate a candidate response set;

[0035] In the secondary decision-making layer, based on the dynamic context representation vector of the context awareness module, the scene adaptability and user preference matching degree of each candidate response are calculated;

[0036] In the final decision-making layer, by weighted fusion of the scene adaptability and the user preference matching degree, the optimal response strategy is selected and output to the interaction terminal.

[0037] Preferably, the dynamic response generation module further includes a response effect evaluation unit, and the specific operation process is as follows:

[0038] The feedback data of the user on the output response is collected, and the ratio of the number of confirmation operations to the number of correction operations in the feedback data is marked as the feedback utility value;

[0039] The deviation value between the feedback utility value and the preset utility threshold is marked as the utility segregation value;

[0040] If the utility segregation value exceeds the preset deviation threshold, a response optimization signal is generated and the decision model parameter update mechanism is triggered.

[0041] Preferably, the system monitoring module is communicatively connected to the cross-modal knowledge transfer module, and the specific operation process of the cross-modal knowledge transfer module is as follows:

[0042] A knowledge distillation framework is constructed, and the multi-modal large model in the cross-modal fusion module is used as the teacher model, and the lightweight inference model is used as the student model;

[0043] Generate the knowledge transfer loss value by comparing the output feature distribution of the teacher model and the student model;

[0044] If the knowledge transfer loss value exceeds a preset loss threshold, a migration abnormality signal is generated and a distillation parameter optimization mechanism is triggered.

[0045] Preferably, the cross-modal knowledge transfer module further includes a transfer effect verification unit, and the specific analysis process is as follows:

[0046] During the preset verification period, the accuracy improvement and inference speed change of the student model in multimodal tasks are collected;

[0047] The ratio of the accuracy improvement value to the preset improvement threshold is marked as the migration gain value, and the deviation value of the inference speed change value from the preset speed threshold is marked as the speed analysis value;

[0048] If the migration gain value is lower than a preset gain threshold or the speed analysis value exceeds a preset deviation threshold, a migration tuning signal is generated and sent to the system monitoring module.

[0049] Preferably, the system monitoring module is in communication with the interaction quality summary module, and the specific analysis process of the interaction quality summary module is as follows:

[0050] Setting a detection period, and collecting the number of system abnormality signals and response optimization signals generated during the detection period;

[0051] The ratio of the number of system abnormal signals to the detection duration is marked as the system abnormality frequency value, and the ratio of the number of response optimization signals to the number of system abnormal signals is marked as the optimization utility value;

[0052] The interaction quality evaluation value is obtained by weighted calculation of the system abnormality frequency value and the optimization utility value. If the interaction quality evaluation value exceeds the preset quality threshold, a system maintenance warning signal is generated and sent to the management terminal.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] In multi-modal data processing, the multi-modal input layer performs standardized preprocessing on text, speech, images, and sensor data. During text processing, after word segmentation and semantic vectorization, it matches with a preset domain knowledge base to generate text feature vectors, enabling the system to accurately extract key information from text. For example, in an intelligent office scenario, it can accurately understand instructions related to document content. During speech processing, it extracts spectral and intonation features and combines them with a speech-to-text model to generate semantic vectors, capable of capturing emotions and intentions in speech. For instance, in an intelligent customer service scenario, it can better respond to customer emotions. Images are processed through object detection algorithms to identify key objects and extract feature vectors of attributes and spatial relationships, enabling quick identification of abnormal situations in intelligent security monitoring. During sensor data processing, data deviations such as light, temperature, and position are marked, and abnormal signals are generated when exceeding the threshold to ensure the stability of environmental monitoring and interaction. For example, in a smart home, devices are automatically adjusted according to environmental changes.

[0055] The context awareness module constructs a dynamic context representation vector by collecting the user's historical behavior, environmental status, and device operation parameters, and also includes an interaction trajectory analysis unit. It analyzes the user's interaction operation sequence, divides segments to extract features such as operation type, duration, and object association degree, calculates the utility value, and generates an inefficient signal to adjust the response strategy when the utility value is low. In intelligent education software, it optimizes the presentation of teaching content and tutoring methods according to students' learning operations to achieve personalized learning support.

[0056] The cross-modal fusion module uses a spatio-temporal correlation modeling algorithm for feature alignment and semantic fusion to generate cross-modal joint features. A feature alignment method combining contrastive learning and self-supervised learning maps vectors of different modalities to a unified space, calculates the cosine similarity to generate an alignment degree matrix, and performs weighted fusion after self-supervised sparsification processing to improve the quality of feature fusion. The abnormal alignment detection unit calculates the entropy deviation value and the sparsity deviation value, and triggers a realignment mechanism when exceeding the threshold to ensure accurate and stable feature alignment. In intelligent design software, it better fuses the designer's voice description, sketch images, and text requirements to assist in generating a design plan that better meets the requirements.

[0057] The dynamic response generation module generates an interaction response strategy adapted to the scenario based on the cross-modal joint features through a hierarchical decision-making mechanism. The primary decision-making layer matches a preset template library to generate candidate responses. The secondary decision-making layer calculates the scene adaptation degree and the user preference matching degree in combination with the context vector. The final decision-making layer performs weighted fusion to select the optimal strategy for output. The response effect evaluation unit calculates the feedback utility value and the utility deviation value based on the user feedback data, and updates the decision model parameters when exceeding the threshold. In an intelligent marketing system, it accurately pushes product information that meets the user's interests and scenarios to improve the marketing effect.

[0058] The system monitoring module monitors key metrics in real time and triggers the adaptive optimization mechanism when anomalies are detected. It communicates with the cross-modal knowledge transfer module to build a knowledge distillation framework, calculates the knowledge transfer loss value by comparing the outputs of the teacher and student models, and optimizes the distillation parameters when the threshold is exceeded to improve the performance of the lightweight model. The migration effect verification unit collects the accuracy improvement value and inference speed change value of the student model, and sends a tuning signal when the standard is not met. It communicates with the interaction quality summary module, collects the number of anomaly and optimization signals to calculate relevant metrics for evaluating the interaction quality, and sends a maintenance warning signal when the threshold is exceeded to ensure the stable operation of the system and guarantee the fluency and accuracy of massive user interactions in a large-scale intelligent interaction platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 FIG. is a working principle diagram of the intelligent interaction system based on the multi-modal large model of the present invention;

[0060] Figure 2 FIG. is a flowchart of the interaction trajectory analysis of the context awareness module;

[0061] Figure 3 FIG. is a flowchart of feature alignment of the cross-modal fusion module;

[0062] Figure 4 FIG. is a flowchart of the operation of the cross-modal knowledge transfer module. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0063] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0064] Please refer to Figure 1-4 , the present invention provides a technical solution: an intelligent interaction system based on a multi-modal large model, the system includes:

[0065] Multi-modal input layer: responsible for receiving multi-modal data input such as text, voice, image, and sensor. After receiving the data, it will perform standard preprocessing on these data and convert them into a format that the system can further process.

[0066] Context awareness module: constructs a dynamic context representation vector by collecting user historical behavior data, environmental state information, and device operation parameters. This vector will be updated in real time as these data change, so as to accurately reflect the current interaction environment.

[0067] Cross-modal fusion module: Adopting a spatio-temporal correlation modeling algorithm, it performs feature alignment and semantic fusion on the preprocessed multi-modal data and the dynamic context representation vector to generate cross-modal joint features. This process enables the organic combination of data from different modalities, providing more comprehensive information for subsequent decision-making.

[0068] Dynamic response generation module: Based on the cross-modal joint features, it generates an interaction response strategy adapted to the current scenario through a hierarchical decision-making mechanism, including voice output, image display, or control instructions, etc., to meet the interaction needs of users in different scenarios.

[0069] System monitoring module: It monitors in real-time the feature alignment error value of the cross-modal fusion module, the decision-making delay value of the dynamic response generation module, and the data update frequency of the context awareness module. Once the feature alignment error value exceeds the preset error threshold, the decision-making delay value exceeds the preset delay threshold, or the data update frequency is lower than the preset frequency threshold, it will generate a system anomaly signal and trigger an adaptive optimization mechanism to ensure that the system is always in the best operating state.

[0070] The present invention will be further described below in conjunction with Embodiments 1 to 6:

[0071] Embodiment 1:

[0072] In an actual application scenario, the processing process of the multi-modal input layer plays a crucial role. Taking the smart home control scenario as an example, when a user interacts with the intelligent interaction system, the multi-modal input layer processes the received text, voice, image, and sensor data respectively.

[0073] In the text data processing stage, assume that the user inputs a text command like "Turn on the living room light" through a mobile phone APP. First, the system will tokenize the input text, splitting it into words such as "Turn on", "living room", and "light". Then, using semantic vectorization technology, these words will be transformed into corresponding semantic vectors. Next, the system will match the semantic vectors with a pre-set domain knowledge base. In the pre-set domain knowledge base of smart home, semantic information of various device control instructions, room names, and device names is stored. Through matching, the system can accurately identify the room location corresponding to "living room" and the device corresponding to "light", and then generate a text feature vector containing device control intent, location information, etc. In the voice data processing stage, the user speaks the voice command "Turn on the living room light". The system will first extract the spectral features of the voice signal, such as obtaining the energy distribution of the voice signal at different frequencies through technologies like Fourier transform, and at the same time extract intonation features to judge information such as the tone and emotion of the user when speaking. Then, combined with a speech-to-text model, the voice signal will be converted into text form, and based on this, a voice semantic vector will be generated. For example, after the speech-to-text model converts the voice into the text "Turn on the living room light", in a similar way to the text data processing, a voice semantic vector containing control intent and device location information, etc. will be generated.

[0074] In the image data processing stage, assume that the intelligent interaction system is equipped with a camera for identifying user gestures or objects in the surrounding environment. When the camera captures an image, the system will use object detection algorithms, such as object detection models based on deep learning (such as YOLO, Faster R-CNN, etc.), to identify the key objects in the image. If the user makes a gesture pointing to the living room light, the system will identify the gesture as the key object and extract its attribute features (such as the shape and action type of the gesture) and spatial relationship features (such as the relative position relationship between the gesture and the living room light in the image), and then generate an image feature vector. In the sensor data processing stage, the system will collect information such as ambient light intensity, temperature value, and user position coordinates. For example, if the ambient light sensor detects that the current light intensity is 50 lux and the pre-set light threshold is 100 lux, then the light analysis value is 50 - 100 = -50 (here assume that the light analysis value is the difference between the current light intensity and the pre-set light threshold). If the light analysis value exceeds the pre-set threshold (assuming the deviation range of the pre-set light threshold is ±20 lux, and -50 is out of the range at this time), an environmental anomaly signal will be generated and sent to the dynamic response generation module. Similarly, after collecting the temperature value and user position coordinates, the temperature analysis value and position analysis value are calculated respectively. If they exceed the corresponding pre-set thresholds, environmental anomaly signals will also be sent. In this way, the system can timely sense environmental changes and provide a basis for subsequent interaction decisions.

[0075] Example 2:

[0076] When a user has a conversation with an intelligent customer service, the interaction trajectory analysis unit will collect the sequence of interaction operations of the user in the current session. Suppose when the user communicates with the intelligent customer service, first asks "Why hasn't the product I purchased been shipped yet", then asks "Can I apply for a refund", and then requests the intervention of a human customer service. This series of operations will be divided into multiple interaction segments according to timestamps.

[0077] For each interaction segment, the system will extract the operation type, operation duration, and the relevance degree of the operation object. For example, in the first interaction segment, the operation type is asking about the product shipping status, the operation duration is assumed to be 5 seconds (the time interval from when the user sends the inquiry message to when the system receives the message), and the relevance degree of the operation object refers to the closeness of the content the user asks about to the business object of product shipping, which is obtained by matching and calculating with the preset business knowledge and is assumed to be 0.8 (the value range is 0 - 1, and 1 means completely relevant).

[0078] Mark the matching degree between the operation type and the preset interaction mode as the mode analysis value. In the intelligent customer service scenario, the preset interaction modes may include consultation, complaint, after - sales, etc. For the operation type of "asking about the product shipping status", the matching degree with the consultation - type interaction mode is assumed to be 0.9 (also in the value range of 0 - 1), that is, the mode analysis value is 0.9. Mark the deviation value between the operation duration and the preset duration as the time - effect value. Suppose the preset inquiry - type operation duration is 3 - 8 seconds, then the time - effect value of this 5 - second operation duration is 5 - 5 = 0 (here it is assumed that the time - effect value is the difference between the operation duration and the middle value of the preset duration). Mark the ratio of the relevance degree of the operation object to the preset relevance threshold as the relevance analysis value. Suppose the preset relevance threshold is 0.6, then the relevance analysis value is 0.8÷0.6≈1.33.

[0079] Calculate the interaction segment utility value through weighted calculation of the mode analysis value, time - effect value, and relevance analysis value. Suppose the weighted coefficients are 0.5, 0.2, and 0.3 respectively (which can be adjusted according to actual business needs), then the interaction segment utility value = 0.9×0.5 + 0×0.2 + 1.33×0.3 = 0.45 + 0 + 0.399 = 0.849. If the preset utility threshold is 0.8, at this time the interaction segment utility value is higher than the preset utility threshold, and the system operates normally. However, if the utility value of a certain interaction segment is lower than the preset utility threshold, for example, in subsequent interactions, the user repeatedly asks some unimportant questions, resulting in a decrease in the relevance degree of the operation object, and the calculated result of the interaction segment utility value is 0.7, lower than the preset utility threshold, then an interaction inefficiency signal will be generated and sent to the dynamic response generation module. After receiving the signal, the dynamic response generation module will adjust the response strategy, such as actively guiding the user to enter the discussion of key issues, or prompting the user to consult more relevant questions to improve the interaction efficiency.

[0080] Example 3:

[0081] In the intelligent security monitoring scenario, the cross-modal fusion module adopts a feature alignment method that combines contrastive learning and self-supervised learning. Suppose the system obtains a surveillance video (containing image and voice information), and at the same time receives an event description in text form (such as "suspicious person found wandering in the parking lot").

[0082] First, construct a cross-modal contrastive learning framework. Map the text feature vector, speech semantic vector, and image feature vector to a unified embedding space. In terms of text processing, after operations such as word segmentation and semantic vectorization on the text "suspicious person found wandering in the parking lot", a text feature vector is obtained. In terms of speech processing, extract the spectrum and intonation features of the speech from the surveillance video and convert them into a speech semantic vector. In terms of images, identify objects such as people and vehicles in the video through an object detection algorithm, and extract object attributes and spatial relationship features to obtain an image feature vector. Then, map these feature vectors of different modalities to a unified low-dimensional vector space so that they have comparable feature representations in this space.

[0083] By calculating the cosine similarity between feature vectors of different modalities, generate an inter-modal alignment degree matrix. For example, calculate the cosine similarity between the text feature vector and the image feature vector. Suppose the text feature vector is A = [a1, a2, …, a n , and the image feature vector is B = [b1, b2, …, b n , then the cosine similarity And so on, calculate the similarity between all feature vectors of different modalities to form a matrix representing the degree of inter-modal alignment.

[0084] Use a self-supervised learning algorithm to sparsify the inter-modal alignment degree matrix and eliminate feature vector pairs with an alignment degree lower than a preset threshold. Suppose the preset threshold is 0.5. For feature vector pairs with a similarity lower than 0.5, it is considered that the correlation between them is low, and they are excluded from subsequent processing, which can reduce the influence of noise and redundant information.

[0085] Finally, perform weighted fusion on the filtered feature vectors to generate a cross-modal joint feature. Suppose the weight of the text feature vector is 0.3, the weight of the speech semantic vector is 0.2, and the weight of the image feature vector is 0.5 (which can be adjusted according to the actual situation), then the cross-modal joint feature F = 0.3A + 0.2B + 0.5C (C is the image feature vector).

[0086] The cross-modal fusion module also includes an abnormal alignment detection unit. During the feature alignment process, calculate the distribution entropy value of each modal feature vector in real time. The distribution entropy value is used to measure the uncertainty of the feature vector distribution. Suppose the probability distribution of the feature vector X is p(x i ), then its distribution entropy value (where m is the number of elements in the eigenvector). The deviation value between the distribution entropy value and the preset entropy threshold is marked as the entropy segregation value. At the same time, the sparsification ratio value of the inter-modal alignment matrix is collected, and the deviation value between it and the preset ratio threshold is marked as the sparsity segregation value. If the entropy segregation value or the sparsity segregation value exceeds the corresponding preset threshold, for example, the entropy segregation value exceeds ±0.2 of the preset threshold, or the sparsity segregation value exceeds ±0.1 of the preset ratio threshold, an alignment anomaly signal is generated and the feature realignment mechanism is triggered to re-align and fuse the features to ensure the accuracy of cross-modal fusion.

[0087] Example 4:

[0088] In an intelligent vehicle interaction system, the hierarchical decision-making mechanism and the response effect evaluation unit of the dynamic response generation module have specific applications. When the vehicle is in motion, the system receives multi-modal inputs from the user, such as the user saying "I want to listen to songs by a certain singer" (voice input), and at the same time, the vehicle's position sensor provides the current position information (sensor input).

[0089] In the primary decision-making layer, the preset response template library is matched according to the cross-modal joint features. Suppose the cross-modal joint features include information such as the user's music preference (extracted from the voice semantic vector) and the vehicle's current position (extracted from the sensor data eigenvector). The preset response template library stores response templates for various different scenarios, such as music playback, navigation guidance, vehicle control, etc. Based on these features, the system filters out a set of candidate responses related to music playback from the template library, which may include "Playing a list of popular songs by a certain singer for you" and "Searching for and playing the album of a certain singer".

[0090] In the secondary decision-making layer, based on the dynamic context representation vector of the context awareness module, the scene suitability and user preference matching degree of each candidate response are calculated. For example, the dynamic context representation vector includes information such as the vehicle's current driving state (whether it is driving on the highway, whether it is congested, etc.) and the user's historical music playback habits. For the candidate response "Playing a list of popular songs by a certain singer for you", if the vehicle is currently in a high-speed driving state, its scene suitability may be relatively low to avoid distracting the driver; however, if the user often listens to popular songs, its user preference matching degree may be relatively high. Through a series of calculations and evaluations, the scene suitability and user preference matching degree of each candidate response are determined.

[0091] In the ultimate decision-making layer, by weighted-fusing the scene adaptability and the user preference matching degree, the optimal response strategy is selected and output to the interaction terminal. Assume that the weight of the scene adaptability is 0.4 and the weight of the user preference matching degree is 0.6, and calculate the comprehensive score of each candidate response. For example, if the scene adaptability of "Playing the popular song list of a certain singer for you" is 0.3 and the user preference matching degree is 0.8, then its comprehensive score = 0.3×0.4 + 0.8×0.6 = 0.12 + 0.48 = 0.6. Compare the comprehensive scores of other candidate responses, select the response strategy with the highest score. For example, finally determine "Playing the classic song list of a certain singer for you (with a slower rhythm, suitable for listening while driving)", and send this instruction to the in-vehicle audio system for playback.

[0092] The dynamic response generation module also includes a response effect evaluation unit. After the system plays a song, collect the feedback data of the user on the output response. If the user frequently switches songs (the number of correction operations is relatively large), or does not perform confirmation operations (such as liking, collecting, etc.), mark the ratio of the number of confirmation operations to the number of correction operations in the feedback data as the feedback utility value. Assume that the number of confirmation operations is 2 times and the number of correction operations is 8 times, then the feedback utility value is 2÷8 = 0.25. Mark the deviation value between the feedback utility value and the preset utility threshold as the utility analysis value. Assume that the preset utility threshold is 0.5, then the utility analysis value is 0.25 - 0.5 = -0.25. If the utility analysis value exceeds the preset deviation threshold (assume it is ±0.1), then generate a response optimization signal and trigger the decision model parameter update mechanism. The system will re-evaluate the user's music preferences and adjust parameters such as the weights of the scene adaptability and the user preference matching degree in the decision model, so as to provide a response that better meets the user's needs in subsequent interactions.

[0093] Example 5:

[0094] In the intelligent medical image diagnosis assistance system, the system monitoring module and the cross-modal knowledge transfer module work together. In this system, the cross-modal fusion module uses a large multi-modal medical image analysis model (as the teacher model). At the same time, in order to improve the real-time performance of the system and its running efficiency on mobile devices, a lightweight inference model (as the student model) is introduced.

[0095] First, construct a knowledge distillation framework, using the multi-modal large model in the cross-modal fusion module as the teacher model and the lightweight inference model as the student model. During the training process, the teacher model processes medical image data (including multi-modal data such as X-ray, CT, MRI image data, and related medical record text data), and outputs the feature distribution. The student model also processes the same data and outputs the feature distribution.

[0096] Generate the knowledge transfer loss value by comparing the output feature distributions of the teacher model and the student model. Assume that the output feature vector of the teacher model is T = [t1, t2, …, t n , and the output feature vector of the student model is S = [s1, s2, …, s n . Loss functions such as mean squared error (MSE) can be used to calculate the knowledge transfer loss value If the knowledge transfer loss value exceeds the preset loss threshold (assumed to be 0.05), a transfer anomaly signal is generated and the distillation parameter optimization mechanism is triggered. The system will adjust the parameters in the knowledge distillation process, such as the temperature parameter (used to control the smoothness of soft labels in knowledge distillation), learning rate, etc., to improve the learning effect of the student model on the knowledge of the teacher model.

[0097] The cross-modal knowledge transfer module also includes a transfer effect verification unit. During the preset verification period, collect the accuracy improvement value and the inference speed change value of the student model in the multi-modal task. For example, when testing 100 medical image diagnosis tasks, the accuracy of the student model before knowledge transfer is 70%, and after knowledge transfer, the accuracy is improved to 75%. Then the accuracy improvement value is 75% - 70% = 5%. At the same time, record the inference speed of the student model. Assume that it takes 5 seconds to infer one image before transfer and 4 seconds after transfer. The inference speed change value is 4 - 5 = -1 second (here the speed change value is the speed after transfer minus the speed before transfer). Mark the ratio of the accuracy improvement value to the preset improvement threshold as the transfer gain value. Assume that the preset improvement threshold is 3%, then the transfer gain value is 5% ÷ 3% ≈ 1.67. Mark the deviation value of the inference speed change value from the preset speed threshold as the speed analysis value. Assume that the preset speed threshold deviation range is ±1 second. At this time, -1 second does not exceed the range. If the transfer gain value is lower than the preset gain threshold (such as lower than 1.2) or the speed analysis value exceeds the preset deviation threshold, a transfer tuning signal is generated and sent to the system monitoring module. After receiving the signal, the system monitoring module will further adjust the relevant parameters of knowledge transfer or optimize the student model to improve the knowledge transfer effect.

[0098] Example 6:

[0099] In the scenario of an intelligent office interaction system, the system monitoring module and the interaction quality summary module work together to comprehensively evaluate and manage the overall interaction quality of the system.

[0100] Set the detection period as a working day, that is, from 9:00 am to 5:00 pm, a total of 8 hours (T = 8 hours). During this detection period, the system monitoring module monitors the feature alignment error value of the cross-modal fusion module, the decision-making delay value of the dynamic response generation module, and the data update frequency of the context awareness module in real time. Once these indicators are abnormal, a system abnormal signal will be generated. Suppose that within a certain working day, the system generates a total of N = 15 system abnormal signals.

[0101] At the same time, the response effect evaluation unit in the dynamic response generation module analyzes according to the feedback data of the user on the output response. If the feedback data shows that the user is not satisfied with the response, a response optimization signal will be generated. Assume that within this working day, a total of M = 9 response optimization signals are generated.

[0102] Mark the ratio of the number of system abnormal signals to the detection duration as the system abnormal frequency value (F), and its calculation formula is: Substituting the data, we can get This value reflects the frequency of system abnormalities per unit time.

[0103] Mark the ratio of the number of response optimization signals to the number of system abnormal signals as the optimization utility value (U), and the calculation formula is: After substituting the data, This value reflects the effect of adjustment through the optimization mechanism after the system appears abnormal.

[0104] The interaction quality evaluation value (Q) is obtained by weighted calculation of the system abnormal frequency value and the optimization utility value. Let the weighted coefficient of the system abnormal frequency value be w1 = 0.7, and the weighted coefficient of the optimization utility value be w2 = 0.3 (the weighted coefficients can be adjusted according to actual business requirements and importance). The calculation formula is: Q = w1F + w2U. Substituting F = 1.875, U = 0.6, w1 = 0.7, and w2 = 0.3 into the formula, we can get: Q = 0.7×1.875 + 0.3×0.6 = 1.3125 + 0.18 = 1.4925.

[0105] Preset the preset quality threshold of the interaction quality evaluation value as Q th = 1.2 (this threshold can be set according to the expected interaction quality standard of the system). When the calculated interaction quality evaluation value Q exceeds the preset quality threshold Q th That is, Q > Q th , the interaction quality summary module will generate a system maintenance warning signal and send this signal to the management terminal. After receiving the signal, the management terminal can arrange technicians to check, optimize, and maintain the system to ensure that the intelligent office interaction system can continuously and stably provide high-quality interaction services and ensure the smooth progress of the office process.

[0106] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0107] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent interaction system based on a multimodal large model, characterized in that, It includes a multi-modal input layer, a context awareness module, a cross-modal fusion module, a dynamic response generation module, and a system monitoring module; The multi-modal input layer is used to receive multi-modal data input by text, speech, images, and sensors, and perform standardized preprocessing on the multi-modal data; The context awareness module constructs a dynamic context representation vector by collecting the user's historical behavior data, environmental state information, and device operation parameters; The cross-modal fusion module uses a spatio-temporal correlation modeling algorithm to perform feature alignment and semantic fusion on the preprocessed multi-modal data and the dynamic context representation vector to generate cross-modal joint features; The dynamic response generation module generates an interaction response strategy adapted to the current scenario based on the cross-modal joint features through a hierarchical decision-making mechanism, including voice output, image display, or control instructions; The system monitoring module monitors in real time the feature alignment error value of the cross-modal fusion module, the decision delay value of the dynamic response generation module, and the data update frequency of the context awareness module. If the feature alignment error value exceeds the preset error threshold, the decision delay value exceeds the preset delay threshold, or the data update frequency is lower than the preset frequency threshold, a system anomaly signal is generated and an adaptive optimization mechanism is triggered.

2. An intelligent interaction system based on a multimodal large model according to claim 1, characterized in that, The specific processing process of the multi-modal input layer is as follows: In the text data processing stage, the input text is tokenized and semantically vectorized, and the semantic vector is matched with a preset domain knowledge base to generate a text feature vector; In the speech data processing stage, the spectral features and intonation features of the speech signal are extracted, and a speech semantic vector is generated in combination with a speech-to-text model; In the image data processing stage, key objects in the image are identified through an object detection algorithm, and object attribute features and spatial relationship features are extracted to generate an image feature vector; In the sensor data processing stage, the environmental light intensity, temperature value, and user position coordinates are collected. The deviation value between the environmental light intensity and the preset light threshold is marked as the light analysis value, the deviation value between the temperature value and the preset temperature threshold is marked as the temperature analysis value, and the deviation value between the user position coordinates and the preset interaction area is marked as the position analysis value; If the light analysis value, temperature analysis value, or position analysis value exceeds the corresponding preset threshold, an environmental anomaly signal is generated and sent to the dynamic response generation module.

3. An intelligent interaction system based on a multimodal large model according to claim 1, characterized in that, The context awareness module further includes an interaction trajectory analysis unit, and the specific operation process is as follows: Collect the interaction operation sequence of the user in the current session, and divide the interaction operation sequence into multiple interaction segments according to the time stamp; For each interaction segment, the operation type, operation duration, and operation object correlation are extracted. The matching degree between the operation type and the preset interaction mode is marked as the mode analysis value, the deviation value between the operation duration and the preset duration is marked as the time utility value, and the ratio of the operation object correlation to the preset correlation threshold is marked as the correlation analysis value; The interaction segment utility value is obtained by weighted calculation of the mode analysis value, time utility value, and correlation analysis value. If the interaction segment utility value is lower than the preset utility threshold, an interaction inefficiency signal is generated and sent to the dynamic response generation module to adjust the response strategy.

4. An intelligent interaction system based on a multimodal large model according to claim 1, characterized in that, The cross-modal fusion module adopts a feature alignment method that combines contrastive learning and self-supervised learning, specifically including: Construct a cross-modal contrastive learning framework to map text feature vectors, speech semantic vectors, and image feature vectors to a unified embedding space; Generate an inter-modal alignment matrix by calculating the cosine similarity between feature vectors of different modalities; Use a self-supervised learning algorithm to sparsify the inter-modal alignment matrix and eliminate feature vector pairs with alignment below a preset threshold; Perform weighted fusion on the filtered feature vectors to generate the cross-modal joint features.

5. An intelligent interaction system based on a multimodal large model according to claim 4, characterized in that, The cross-modal fusion module also includes an abnormal alignment detection unit, and the specific analysis process is as follows: During the feature alignment process, calculate the distribution entropy value of each modal feature vector in real time, and mark the deviation value between the distribution entropy value and the preset entropy threshold as the entropy analysis value; Collect the sparsity ratio value of the inter-modal alignment matrix, and mark the deviation value between the sparsity ratio value and the preset ratio threshold as the sparsity analysis value; If the entropy analysis value or the sparsity analysis value exceeds the corresponding preset threshold, generate an alignment abnormal signal and trigger the feature re-alignment mechanism.

6. An intelligent interaction system based on a multimodal large model according to claim 1, characterized in that, The dynamic response generation module adopts a hierarchical decision-making mechanism, specifically including: In the primary decision-making layer, match the preset response template library according to the cross-modal joint features to generate a candidate response set; In the secondary decision-making layer, calculate the scene adaptability and user preference matching degree of each candidate response based on the dynamic context representation vector of the context awareness module; In the final decision-making layer, select the optimal response strategy by weighted fusion of the scene adaptability and user preference matching degree and output it to the interaction terminal.

7. An intelligent interaction system based on a multi-modal large model according to claim 6, characterized in that, The dynamic response generation module also includes a response effect evaluation unit, and the specific operation process is as follows: Collect the feedback data of the user on the output response, and mark the ratio of the number of confirmation operations to the number of correction operations in the feedback data as the feedback utility value; Mark the deviation value between the feedback utility value and the preset utility threshold as the utility analysis value; If the utility analysis value exceeds the preset deviation threshold, generate a response optimization signal and trigger the decision model parameter update mechanism.

8. An intelligent interaction system based on a multimodal large model according to claim 1, characterized in that, The system monitoring module is communicatively connected to the cross-modal knowledge transfer module, and the specific operation process of the cross-modal knowledge transfer module is as follows: Construct a knowledge distillation framework, using the multi-modal large model in the cross-modal fusion module as the teacher model and the lightweight inference model as the student model; Generate a knowledge transfer loss value by comparing the output feature distributions of the teacher model and the student model; If the knowledge transfer loss value exceeds the preset loss threshold, generate a transfer abnormal signal and trigger the distillation parameter optimization mechanism.

9. An intelligent interaction system based on a multimodal large model according to claim 8, characterized in that, The cross-modal knowledge transfer module also includes a transfer effect verification unit, and the specific analysis process is as follows: During a preset verification period, collect the accuracy improvement value and inference speed change value of the student model in multi-modal tasks; Mark the ratio of the accuracy improvement value to the preset improvement threshold as the transfer gain value, and mark the deviation value between the inference speed change value and the preset speed threshold as the speed analysis value; If the migration gain value is lower than the preset gain threshold or the speed analysis value exceeds the preset deviation threshold, a migration tuning signal is generated and sent to the system monitoring module.

10. An intelligent interaction system based on a multimodal large model according to claim 1, characterized in that, The system monitoring module is communicatively connected to the interaction quality aggregation module, and the specific analysis process of the interaction quality aggregation module is as follows: Set a detection period, and collect the number of system exception signals and the number of response optimization signals generated during the detection period; Mark the ratio of the number of system exception signals to the detection duration as the system exception frequency value, and mark the ratio of the number of response optimization signals to the number of system exception signals as the optimization utility value; Obtain the interaction quality evaluation value by weighted calculation of the system exception frequency value and the optimization utility value. If the interaction quality evaluation value exceeds the preset quality threshold, a system maintenance warning signal is generated and sent to the management terminal.

Images