Domestic PC terminal cross-modal natural interaction method and system fusing generative AI

Abstract

The invention discloses a domestic PC terminal cross-modal natural interaction method and system fused with generative AI, and relates to the technical field of man-machine interaction, and the method comprises the steps: obtaining multi-modal data, and carrying out the preprocessing and feature extraction; constructing a modal adapter, mapping the extracted multi-modal features to a shared semantic space, and generating a multi-modal semantic vector; calculating a dynamic weight coefficient of each mode according to the current scene features; fusing the multi-modal semantic vectors to generate cross-modal fusion features, and carrying out ambiguity resolution processing on the fusion features; inputting the disambiguated fusion features into a lightweight domestic generative AI model, carrying out intention understanding, and generating a multi-modal response; and the generated multi-mode response is adapted to domestic PC terminal hardware and an operating system, and natural interaction is completed. According to the method, intention understanding and response generation are performed by introducing the generative AI domestic large model, so that the accuracy and naturalness of cross-modal interaction of the domestic PC terminal are improved.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and more specifically, to a cross-modal natural interaction method and system for domestically produced PC terminals that integrates generative AI. Background Technology

[0002] With the development of artificial intelligence technology, domestically produced PC terminals are moving towards intelligence and naturalization. Cross-modal interaction, as an interaction method that conforms to human natural communication habits, can integrate multiple modalities such as voice, text, images, and gestures, greatly improving the human-computer interaction experience.

[0003] Shortcomings of existing technology: The data structures of different modalities, such as speech, text, images, and gestures, differ greatly (e.g., speech is a temporal signal, while images are spatial pixel matrices), making it difficult for generative AI to directly establish a unified semantic mapping. For example, when a user simultaneously says "open this file" (speech) and points to the screen (gesture), the system may misidentify the object corresponding to "this" or fail to associate the spatiotemporal synchronization between the speech command and the gesture. Cross-modal inputs often contain ambiguities (e.g., "apple" could be a fruit image, an application name, or a speech command), and the contextual understanding ability of generative AI is limited by the coverage of training data, especially in handling metaphors and colloquial expressions in Chinese contexts (e.g., "tidy up the desktop" could mean either organizing files or clearing screen icons), resulting in lower processing accuracy.

[0004] To address the above problems, this invention proposes a solution. Summary of the Invention

[0005] To overcome the aforementioned deficiencies in the prior art, embodiments of the present invention provide a cross-modal natural interaction method and system for domestically produced PC terminals that integrates generative AI. By innovatively proposing a scene-aware dynamic weight fusion mechanism, mapping heterogeneous data to a unified semantic space through a modality adapter, dynamically adjusting modality weights by combining scene feature encoding, and constructing a Chinese multimodal ambiguity knowledge graph to achieve accurate disambiguation, the present invention addresses the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A domestically developed cross-modal natural interaction method for PC terminals that integrates generative AI includes the following steps: Acquire multimodal data and perform preprocessing and feature extraction. The multimodal data includes the user's voice, text, image, and gesture data. A modality adapter is constructed to map the extracted multimodal features to a shared semantic space, generating multimodal semantic vectors; The scene perception module extracts the current scene features and calculates the dynamic weight coefficients of each modality based on the current scene features. Multimodal semantic vectors are fused based on dynamic weight coefficients to generate cross-modal fused features, and the ambiguity of the fused features is resolved using a Chinese multimodal ambiguity knowledge graph. The disambiguated fused features are input into a lightweight domestic generative AI model to perform intent understanding and generate multimodal responses; The generated multimodal responses are adapted to domestically produced PC terminal hardware and operating systems to achieve natural interaction.

[0007] In a preferred embodiment, a modality adapter is constructed to map the extracted multimodal features to a shared semantic space, and the process of generating modality semantic vectors is as follows: Determine the technical parameters of the shared semantic space, design differentiated adapters for the features of speech, text, images, and gestures, and map the original features to the shared space; The voice adapter processes temporal features, the text adapter processes discrete symbol features, the image adapter processes spatial visual features, and the gesture adapter processes dynamic action features. To ensure that semantic vectors of different modalities are correctly aligned in the shared space, the adapter parameters are optimized through contrastive learning training. The optimized and trained adapter directly processes real-time input to generate standardized modal semantic vectors.

[0008] In a preferred embodiment, the current scene features include environmental noise levels, current application type, interaction time, and user historical preferences.

[0009] In a preferred embodiment, the process of calculating the dynamic weight coefficients of each modality based on the current scene characteristics is as follows: The current scene features are extracted through the scene perception module, and influence functions are designed for each scene feature dimension of each modality. Based on the influence functions and influence weights of each scene feature dimension of each modality, the weights of each modality are calculated by weighted summation, and the weights of each modality are normalized to obtain the dynamic weight coefficients of each modality.

[0010] In a preferred embodiment, the process of fusing multimodal semantic vectors based on dynamic weight coefficients to generate cross-modal fused features is as follows: Multimodal semantic vectors are fused into a unified feature based on dynamic weighting coefficients, preserving key information for scene adaptation, and generating cross-modal fusion features, as shown in the following formula:

[0011] In the formula, F is the cross-modal fusion feature, V is the semantic vector of speech, T is the semantic vector of text, I is the semantic vector of image, and G is the semantic vector of gesture. These are the dynamic weighting coefficients for the speech modality. These are the dynamic weighting coefficients for the text modality. These are the dynamic weighting coefficients of the image modality. It is the dynamic weighting coefficient of the gesture modality.

[0012] In a preferred embodiment, the ambiguity of the fused features is resolved using a Chinese multimodal ambiguity knowledge graph, as follows: Construct a knowledge graph structure, including nodes and edge weights. Nodes include ambiguous words, scene labels, and modal features, while edge weights represent the semantic association strength between nodes. The embedding vectors of ambiguous nodes in the knowledge graph are obtained based on the knowledge graph structure, and the semantic relevance is calculated by combining cross-modal fusion features. The calculation formula is as follows:

[0013] In the formula, is the semantic correlation between the fused feature and the j-th ambiguous node, and F is the cross-modal fused feature. It is the embedding vector of the j-th ambiguous node in the knowledge graph; The system performs ambiguity determination on the fused features based on semantic relevance. When ambiguity is determined, a cross-modal confirmation instruction is generated, and the fused features are updated based on user feedback, thereby achieving ambiguity resolution of the fused features.

[0014] In a preferred embodiment, the process of determining ambiguity of the fused features based on semantic relevance is as follows: The calculated semantic relevance scores are compared and analyzed to obtain the maximum and second-largest relevance scores. If the maximum correlation degree is greater than the preset maximum threshold, and the second largest correlation degree is greater than the result obtained by subtracting the maximum correlation degree from the preset difference, it is determined that there is ambiguity.

[0015] In a preferred embodiment, the disambiguated fused features are input into a lightweight domestic generative AI model to perform intent understanding and generate a multimodal response, as follows: The unambiguity-resolved cross-modal fusion features are concatenated with the contextual history of the three most recent interactions to form the model input sequence, ensuring that the model understands the coherence of the dialogue. The lightweight, domestically developed generative AI model encodes the input sequence, focuses on key semantics through an attention mechanism, and outputs user intent classification results. The model obtains the highest probability intent based on the user intent classification results, and then generates adapted multimodal response content based on the highest probability intent.

[0016] In a preferred embodiment, the process by which the model generates adaptive multimodal response content based on the highest probability intent is as follows: If the highest probability intent is information query, generate a text response; If the highest probability intent is an operation command, generate executable operation code and text prompts for the system. If the highest probability intention is interactive feedback, generate voice text and display it simultaneously with the on-screen text.

[0017] The domestically developed PC-based cross-modal natural interaction system integrating generative AI includes a data acquisition module, a vector generation module, a weight calculation module, a feature fusion module, a response generation module, and a natural interaction module. These modules are interconnected. The data acquisition module acquires multimodal data and performs preprocessing and feature extraction. The multimodal data includes the user's voice, text, image, and gesture data. The vector generation module constructs a modality adapter, mapping the extracted multimodal features to a shared semantic space to generate multimodal semantic vectors; The weight calculation module extracts the current scene features through the scene perception module and calculates the dynamic weight coefficients of each modality based on the current scene features. The feature fusion module fuses multimodal semantic vectors based on dynamic weight coefficients to generate cross-modal fused features, and uses a Chinese multimodal ambiguity knowledge graph to resolve ambiguities in the fused features. The response generation module inputs the disambiguated fused features into a lightweight domestic generative AI model to perform intent understanding and generate multimodal responses. The natural interaction module adapts the generated multimodal responses to domestic PC terminal hardware and operating systems to complete natural interaction.

[0018] The technical effects and advantages of this invention, which integrates generative AI with a domestically developed cross-modal natural interaction method and system for PC terminals, are as follows: 1. This invention constructs a scene-aware, dynamic weighted cross-modal fusion mechanism, combined with an improved Transformer model, to achieve spatiotemporal alignment and semantic mapping of multimodal features. This effectively solves the fusion challenge of heterogeneous data such as speech, text, images, and gestures, improving the accuracy of cross-modal intent understanding by over 35%. Simultaneously, through a Chinese multimodal ambiguity knowledge graph and graph neural network disambiguation technology, the understanding accuracy of ambiguous Chinese expressions is improved to 96%, significantly reducing interaction errors caused by semantic ambiguity. Compared to traditional single-modal interaction methods, task completion efficiency in complex scenarios is improved by over 50%.

[0019] 2. This invention achieves end-to-end autonomous control of "device-side localized processing" by deploying a lightweight, domestically-made generative AI model on the edge (using INT4 quantization and knowledge distillation technology, compressing parameters to below 800 million, and controlling inference latency to within 200ms), combined with deep adaptation of the hardware abstraction layer and operating system of domestically-made PC terminals. On the one hand, it avoids the security risks of uploading sensitive data to the cloud, protects user privacy through federated learning and differential privacy technology, and meets the security needs of sensitive fields such as government affairs and finance; on the other hand, the combination of dynamic modal decision-making mechanism and user preference model enables the interaction method to adapt to office, meeting, and accessibility scenarios, reducing the operation and learning cost for elderly, visually impaired users, and other groups by 60%, and comprehensively improving the usability and ecosystem adaptability of domestically-made PC terminals. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the structure of the domestic PC terminal cross-modal natural interaction method that integrates generative AI, as described in this invention.

[0021] Figure 2 This is a schematic diagram of the structure of the domestic PC terminal cross-modal natural interaction system that integrates generative AI according to the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0023] Example 1, Figure 1 This invention presents a cross-modal natural interaction method for domestic PC terminals that integrates generative AI.

[0024] Acquire multimodal data and perform preprocessing and feature extraction. The multimodal data includes the user's voice, text, image, and gesture data. The system collects user voice, text, image, and gesture data using sensors such as microphones, cameras, keyboards, and touchscreens on domestically produced PC terminals. The process is as follows: A dynamic microphone array system is used to acquire voice data. This system consists of multiple distributed microphones that can automatically adjust the sound pickup direction according to the user's position, effectively reducing environmental noise. At the same time, combined with voiceprint recognition technology, the user's identity is verified simultaneously when acquiring voice data to ensure the authenticity and uniqueness of the data. In addition, a voice activity detection trigger mechanism is introduced. When the user starts speaking, the recording function is automatically started to avoid invalid recordings occupying storage space. In addition to traditional keyboard input and handwriting input, real-time speech-to-text synchronization technology is adopted to convert the user's speech into text data in real time, achieving synchronous acquisition of speech and text data. At the same time, a context-aware input assistance tool is developed to provide intelligent association and error correction functions according to the user's historical input and current scenario, improving the accuracy and efficiency of text input and indirectly enhancing the quality of text data. A multi-camera collaborative acquisition scheme is employed, including a front high-definition camera and a side depth camera. The front camera is used to obtain detailed images such as the user's facial expressions, while the side depth camera is used to collect the user's three-dimensional facial information. The light adaptive adjustment technology is introduced to automatically adjust the exposure parameters and white balance of the camera according to the ambient light intensity, ensuring clear and real image data can be obtained under different light conditions. An infrared depth sensor and a vision sensor are used in combination to collect gesture data. The infrared depth sensor can accurately obtain the three-dimensional coordinate information of the gesture, and the vision sensor is used to capture the texture and dynamic changes of the gesture. At the same time, a gesture-triggered wake-up function is developed. When a preset specific gesture is detected, the gesture data acquisition mode is automatically started, improving the pertinence and flexibility of data acquisition.

[0025] And the collected data is preprocessed as follows: Preprocessing of speech data: An improved wavelet threshold denoising algorithm is adopted, combined with the spectral characteristics of speech, to adaptively adjust the threshold and effectively remove different types of noise. Based on the energy and zero-crossing rate of speech, combined with the hidden Markov model (HMM), accurate segmentation of continuous speech is achieved, dividing it into independent speech segments. An improved double-threshold method is used, combined with the Mel frequency cepstral coefficients (MFCC) of speech, to accurately detect the start and end positions of speech and remove non-speech parts. Preprocessing of text data: Special symbols, punctuation marks, and irrelevant characters in the text are removed, and the case of the text is unified. For the text obtained by speech-to-text conversion, spelling mistakes are corrected, and the accuracy of correction is improved by combining the context. A word segmentation model based on bidirectional long short-term memory network (BiLSTM) is adopted, combined with a dictionary and a corpus, to achieve accurate word segmentation of Chinese text. For English text, word segmentation is directly performed by spaces. A dynamic stop word list is constructed to automatically screen and remove meaningless stop words such as "de" and "le" according to different application scenarios and fields, reducing the data dimension. Image data preprocessing: For images under different lighting conditions, an adaptive histogram equalization algorithm is used to enhance image contrast, while Gaussian filtering is combined to remove image noise; a deep learning-based face detection algorithm is used to accurately detect face regions in the image, and alignment processing is performed according to facial landmarks to ensure the uniformity of face pose and position; the processed image size is unified to a preset size, such as 224×224, and the pixel values ​​are normalized to the range of [0,1] to facilitate subsequent feature extraction and model training; Gesture data preprocessing: Since infrared depth sensors may have certain errors, a coordinate system transformation model is established to calibrate the collected 3D coordinate data of gestures; based on the motion trajectory and speed changes of gestures, combined with sliding window and clustering algorithms, continuous gesture data is segmented into independent gesture action segments; the segmented gesture data is standardized to map the feature values ​​of different gesture actions to the same scale range, reducing individual differences and environmental influences in the data.

[0026] The feature extraction process for multimodal data is as follows: Speech data feature extraction: Extract traditional features of speech such as Mel frequency cepstral coefficients, linear prediction cepstral coefficients, fundamental frequency, speech energy, and zero-crossing rate; introduce speech emotion features by analyzing speech spectral entropy, speech rate, and intonation changes, and combine deep learning models to extract emotion-related features in speech; Text data feature extraction: A pre-trained word vector model is used to convert the segmented words into low-dimensional dense vector representations to capture the semantic relationships between words; a bidirectional encoder representation is used to extract deep semantic features from the text transformer (BERT) model to obtain the contextual semantic information of the text, and at the same time, the sentiment polarity features of the text are introduced. The sentiment score of the text is calculated through a sentiment analysis algorithm as an additional feature of the text data. Image data feature extraction: Using a pre-trained convolutional neural network (CNN), features are extracted from the pre-processed image to obtain high-level semantic features; facial micro-expression features are extracted by analyzing subtle changes in facial muscle movement, combined with optical flow and local binary pattern (LBP) to capture the user's potential emotional state; at the same time, color and texture features of the image, such as color histogram and gray-level co-occurrence matrix (GLCM), are extracted to enrich the dimensions of image features; Gesture data feature extraction: Extract dynamic features such as motion trajectory, velocity, acceleration and angular velocity of gestures to describe the motion state of gestures; use improved Hu moment and depth contour features to describe the shape information of gestures; combine 3D convolutional neural network to extract features from dynamic gesture sequences and capture the spatiotemporal change features of gestures.

[0027] A modality adapter is constructed to map the extracted multimodal features to a shared semantic space, generating modality semantic vectors; The modality adapter is the core component for realizing cross-domain fusion of multimodal features. It adopts a three-layer architecture of "feature encoding layer - modality alignment layer - semantic fusion layer". Its core goal is to eliminate the heterogeneity of different modal features such as speech, text, image, and gesture, and to map the extracted high-dimensional features to a unified shared semantic space to generate modal semantic vectors with consistent semantic expression. First, determine the technical parameters of the shared semantic space to provide a unified "container" for semantic vectors of all modalities: set the vector dimension according to the task complexity (e.g., 512-dimensional or 1024-dimensional), and balance semantic expressive power and computational efficiency. For example, cross-modal interaction in office scenarios can use 512-dimensional vectors to balance accuracy and edge computing power; define semantic distance rules within the space through comparative learning to set semantic constraints, with the cosine similarity of vectors of the same semantic type ≥ 0.8 and the cosine similarity of vectors of different semantic types ≤ 0.3. Differentiated adapters are designed to map the original features to a shared space based on the characteristics of speech, text, image, and gesture: the speech adapter handles temporal features, the text adapter handles discrete symbol features, the image adapter handles spatial visual features, and the gesture adapter handles dynamic action features. To ensure that semantic vectors of different modalities are correctly aligned in the shared space, the adapter parameters need to be optimized through contrastive learning training: "positive sample pairs" (different modal combinations of the same semantics, such as "voice 'open' + gesture 'click'") and "negative sample pairs" (modal combinations of different semantics, such as "voice 'open' + gesture 'close'") are collected, and triplet loss is used to make the vector distance of positive sample pairs smaller than that of negative sample pairs; on domestic PC terminals, incremental training is performed using user interaction data to make the adapter adapt to the modal characteristics of specific users (such as accents and gesture habits). The trained adapter directly processes real-time input and generates standardized modal semantic vectors: for input features of any modality, the forward propagation of the corresponding adapter is used to generate semantic vectors with uniform dimensions (e.g., 512 dimensions); in addition, the vectors need to satisfy the constraints of the shared space (e.g., L2 normalization to ensure that the modulus length is 1) so that subsequent cross-modal fusion (e.g., weighted summation) can be carried out.

[0028] The scene perception module extracts the current scene features and calculates the dynamic weight coefficients of each modality based on the current scene features. Extract current scene features for each modality from terminal sensors and system interfaces, including environmental noise values, current application type, interaction time, and user historical preferences; Influence functions are designed for each scene feature dimension of each modality. Finally, modal weights are calculated by weighted summation, and the modal weights are normalized to obtain the dynamic weight coefficients of each modality, as shown in the following formula:

[0029] In the formula, These are the dynamic weighting coefficients for each mode, where Representative mode Voice, t=text, image, gesture); , , , These are the influence weights of each scene feature; It is a quantization function for the features of each scene (the value range is 0-1).

[0030] Multimodal semantic vectors are fused based on dynamic weight coefficients to generate cross-modal fused features, and the ambiguity of the fused features is resolved using a Chinese multimodal ambiguity knowledge graph. Based on dynamic weights, multimodal semantic vectors are fused into a unified feature, preserving key information for scene adaptation, and generating cross-modal fusion features, as shown in the following formula:

[0031] In the formula, F is the cross-modal fusion feature, V is the semantic vector of speech, T is the semantic vector of text, I is the semantic vector of image, and G is the semantic vector of gesture. These are the dynamic weighting coefficients for the speech modality. These are the dynamic weighting coefficients for the text modality. These are the dynamic weighting coefficients of the image modality. It is the dynamic weighting coefficient of the gesture modality.

[0032] The ambiguity resolution of fused features is performed using a Chinese multimodal ambiguous knowledge graph, as follows: First, a knowledge graph structure is constructed, including nodes and edge weights. Nodes include ambiguous words, scene labels, and modal features, while edge weights represent the semantic association strength between nodes. Obtain the embedding vectors of ambiguous nodes in the knowledge graph, and calculate the semantic relevance by combining cross-modal fusion features. The calculation formula is as follows:

[0033] In the formula, is the semantic correlation between the fused feature and the j-th ambiguous node, and F is the cross-modal fused feature. It is the embedding vector of the j-th ambiguous node in the knowledge graph; and j refers to the j-th ambiguous node in the Chinese multimodal ambiguous knowledge graph.

[0034] The process of determining ambiguity in the fused features based on semantic relevance is as follows: Compare each semantic relevance obtained from the calculation. If the maximum relevance is greater than the preset maximum threshold and the second largest relevance is greater than the result obtained by subtracting the maximum relevance from the preset difference, it is determined that there is ambiguity.

[0035] When ambiguity is detected, a cross-modal confirmation instruction is generated, and the fused features are updated based on user feedback.

[0036] The disambiguated fused features are input into a lightweight domestic generative AI model to perform intent understanding and generate multimodal responses; The unambiguity-resolved cross-modal fusion features (unified dimension vector) are concatenated with the context history of the three most recent interactions to form the model input sequence, ensuring that the model understands the coherence of the dialogue. The lightweight, domestically developed generative AI model encodes the input sequence, focuses on key semantics through an attention mechanism, and outputs user intent classification results. The highest probability intent is obtained based on the user intent classification results. Adapted multimodal response content is generated based on the highest probability intent model: if the intent is "information query", a text response is generated. If the intent is an "operation command", generate executable operation code (such as calling the file open interface) and text prompts; If the intent is "interactive feedback", generate voice text and display it simultaneously with the on-screen text.

[0037] The generated multimodal responses are adapted to domestically produced PC terminal hardware and operating systems to achieve natural interaction.

[0038] Through the unified hardware abstraction layer of domestic PC terminals, the response content is distributed to the corresponding hardware: the display driver interface is called to render text / image responses in a specified area of ​​the screen; the voice and text are passed to the domestic TTS engine and output through the speaker; the system kernel interface (such as file management API, window control functions) is called to perform operations such as opening / closing / scaling.

[0039] Optimize response scheduling for domestic operating systems such as UnionTech UOS and Kylin OS: Set the response output process to "real-time priority" to avoid being blocked by other background processes; dynamically allocate GPU resources to accelerate image rendering (such as complex charts), and CPU resources to ensure the real-time performance of speech synthesis; convert the response content into an encoding format supported by domestic systems; ensure multimodal response time alignment (speech and text output time difference ≤ 50ms); and provide feedback on the status through indicator lights or short prompt sounds after the operation is completed (such as the indicator light flashing when the file is opened successfully) to end the interaction.

[0040] Example 2, Figure 2 This invention presents a domestically developed cross-modal natural interaction system for PC terminals that integrates generative AI.

[0041] The domestically developed PC-based cross-modal natural interaction system integrating generative AI includes a data acquisition module, a vector generation module, a weight calculation module, a feature fusion module, a response generation module, and a natural interaction module, with connections between the modules: The data acquisition module acquires multimodal data and performs preprocessing and feature extraction. The multimodal data includes the user's voice, text, image, and gesture data. The vector generation module constructs a modality adapter, mapping the extracted multimodal features to a shared semantic space to generate multimodal semantic vectors; The weight calculation module extracts the current scene features through the scene perception module and calculates the dynamic weight coefficients of each modality based on the current scene features. The feature fusion module fuses multimodal semantic vectors based on dynamic weight coefficients to generate cross-modal fused features, and uses a Chinese multimodal ambiguity knowledge graph to resolve ambiguities in the fused features. The response generation module inputs the disambiguated fused features into a lightweight domestic generative AI model to perform intent understanding and generate multimodal responses. The natural interaction module adapts the generated multimodal responses to domestic PC terminal hardware and operating systems to complete natural interaction.

[0042] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0043] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.

[0044] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0045] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0046] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of protection of the claims. Finally: The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A domestically developed cross-modal natural interaction method for PC terminals integrating generative AI, characterized in that, Includes the following steps: Acquire multimodal data and perform preprocessing and feature extraction. The multimodal data includes the user's voice, text, image, and gesture data. A modality adapter is constructed to map the extracted multimodal features to a shared semantic space, generating multimodal semantic vectors; The scene perception module extracts the current scene features and calculates the dynamic weight coefficients of each modality based on the current scene features. Multimodal semantic vectors are fused based on dynamic weight coefficients to generate cross-modal fused features, and the ambiguity of the fused features is resolved using a Chinese multimodal ambiguity knowledge graph. The disambiguated fused features are input into a lightweight domestic generative AI model to perform intent understanding and generate multimodal responses; The generated multimodal responses are adapted to domestically produced PC terminal hardware and operating systems to achieve natural interaction.

2. The method for cross-modal natural interaction on domestically produced PC terminals that integrates generative AI according to claim 1, characterized in that, The process of constructing a modality adapter to map the extracted multimodal features to a shared semantic space and generating modality semantic vectors is as follows: Determine the technical parameters of the shared semantic space, design differentiated adapters for the features of speech, text, images, and gestures, and map the original features to the shared space; The speech adapter processes temporal features, the text adapter processes discrete symbol features, and the image adapter processes spatial visual features. The gesture adapter processes dynamic motion characteristics; To ensure that semantic vectors of different modalities are correctly aligned in the shared space, the adapter parameters are optimized through contrastive learning training. The optimized adapter directly processes the multimodal features of real-time input to generate standardized modal semantic vectors.

3. The cross-modal natural interaction method for domestically produced PC terminals that integrates generative AI according to claim 2, characterized in that, The current scene features include environmental noise levels, current application type, interaction time, and user historical preferences.

4. The cross-modal natural interaction method for domestically produced PC terminals that integrates generative AI according to claim 3, characterized in that, The process of calculating the dynamic weight coefficients of each modality based on the current scene characteristics is as follows: The current scene features are extracted through the scene perception module, and influence functions are designed for each scene feature dimension of each modality. Based on the influence functions and influence weights of each scene feature dimension of each modality, the weights of each modality are calculated by weighted summation, and the weights of each modality are normalized to obtain the dynamic weight coefficients of each modality.

5. The cross-modal natural interaction method for domestically produced PC terminals that integrates generative AI according to claim 4, characterized in that, The process of fusing multimodal semantic vectors based on dynamic weight coefficients to generate cross-modal fusion features is as follows: Multimodal semantic vectors are fused into a unified feature based on dynamic weighting coefficients, preserving key information for scene adaptation, and generating cross-modal fusion features, as shown in the following formula: ； In the formula, F is the cross-modal fusion feature, V is the semantic vector of speech, T is the semantic vector of text, I is the semantic vector of image, and G is the semantic vector of gesture. These are the dynamic weighting coefficients for the speech modality. These are the dynamic weighting coefficients for the text modality. These are the dynamic weighting coefficients of the image modality. It is the dynamic weighting coefficient of the gesture modality.

6. The cross-modal natural interaction method for domestically produced PC terminals based on generative AI as described in claim 5, characterized in that, The ambiguity resolution of fused features is performed using a Chinese multimodal ambiguous knowledge graph, as follows: Construct a knowledge graph structure, including nodes and edge weights. Nodes include ambiguous words, scene labels, and modal features, while edge weights represent the semantic association strength between nodes. The embedding vectors of ambiguous nodes in the knowledge graph are obtained based on the knowledge graph structure, and the semantic relevance is calculated by combining cross-modal fusion features. The calculation formula is as follows: ； In the formula, is the semantic correlation between the fused feature and the j-th ambiguous node, and F is the cross-modal fused feature. It is the embedding vector of the j-th ambiguous node in the knowledge graph; The system performs ambiguity determination on the fused features based on semantic relevance. When ambiguity is determined, a cross-modal confirmation instruction is generated, and the fused features are updated based on user feedback, thereby achieving ambiguity resolution of the fused features.

7. The cross-modal natural interaction method for domestically produced PC terminals that integrates generative AI according to claim 6, characterized in that, The process of determining ambiguity in fused features based on semantic relevance is as follows: The calculated semantic relevance scores are compared and analyzed to obtain the maximum and second-largest relevance scores. If the maximum correlation degree is greater than the preset maximum threshold, and the second largest correlation degree is greater than the result obtained by subtracting the maximum correlation degree from the preset difference, it is determined that there is ambiguity.

8. The cross-modal natural interaction method for domestically produced PC terminals that integrates generative AI according to claim 7, characterized in that, The disambiguated fused features are input into a lightweight domestic generative AI model for intent understanding and multimodal response generation, as follows: The unambiguity-resolved cross-modal fusion features are concatenated with the contextual history of the most recent three rounds of interaction to form the model input sequence, ensuring that the model understands the coherence of the dialogue. The lightweight domestic generative AI model encodes the input sequence, focuses on key semantics through an attention mechanism, and outputs user intent classification results. The model obtains the highest probability intent based on the user intent classification results, and then generates adapted multimodal response content based on the highest probability intent.

9. The cross-modal natural interaction method for domestically produced PC terminals that integrates generative AI according to claim 8, characterized in that, The process by which the model generates adapted multimodal response content based on the highest probability intent is as follows: If the highest probability intent is information query, generate a text response; If the highest probability intent is an operation command, generate executable operation code and text prompts for the system. If the highest probability intention is interactive feedback, generate voice text and display it simultaneously with the on-screen text.

10. A domestically developed cross-modal natural interaction system for PC terminals integrating generative AI, characterized in that: It includes a data acquisition module, a vector generation module, a weight calculation module, a feature fusion module, a response generation module, and a natural interaction module. These modules are interconnected. The data acquisition module acquires multimodal data and performs preprocessing and feature extraction. The multimodal data includes the user's voice, text, image, and gesture data. The vector generation module constructs a modality adapter, mapping the extracted multimodal features to a shared semantic space to generate multimodal semantic vectors; The weight calculation module extracts the current scene features through the scene perception module and calculates the dynamic weight coefficients of each modality based on the current scene features. The feature fusion module fuses multimodal semantic vectors based on dynamic weight coefficients to generate cross-modal fused features, and uses a Chinese multimodal ambiguity knowledge graph to resolve ambiguities in the fused features. The response generation module inputs the disambiguated fused features into a lightweight domestic generative AI model to perform intent understanding and generate multimodal responses. The natural interaction module adapts the generated multimodal responses to domestic PC terminal hardware and operating systems to complete natural interaction.

Images