A semantic representation method and system of multi-modal operation instructions

Abstract

The application provides a kind of multimodal operation instruction semantic representation method and system, it is related to semantic representation technical field, including: according to the characteristics of multimodal instruction data obtains the instruction characteristics of different modalities;According to the abstract description of user intent determines semantic label;According to the instruction characteristics of different modalities and semantic label input to WSABIE model training, to determine the optimal mapping relationship of the instruction characteristics of different modalities and semantic label;Based on optimal mapping relationship, the correlation of the instruction characteristics of different modalities and semantic label is quantitatively evaluated, to accurately analyze multimodal instruction data in semantic level, obtains the semantic representation of multimodal instruction data.The application is extracted by multimodal feature, semantic label generation, semantic mapping based on WSABIE model and correlation quantitative evaluation, successfully realizes the semantic alignment and accurate analysis of multimodal instruction, with the advantages of high precision, high adaptability, high efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of semantic representation, in particular to a semantic representation method and system of multi-modal operation instructions. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, multi-modal interaction has gradually become an important direction of human-computer interaction. Multi-modal instruction input forms include voice, text, images, gestures, touch, and other diversified ways. However, the instruction forms of images, gestures, touch, and other modalities do not directly contain explicit semantic content, which limits their direct application in large language model (LLM) scene understanding.

[0003] Therefore, how to convert these non-directly semantic instructions into semantic information that can be understood by LLM, and realize semantic alignment of multi-modal operation instructions, has become a key problem in current technical development. SUMMARY

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present application is to provide a semantic representation method and system of multi-modal operation instructions, which successfully realizes semantic alignment and accurate analysis of multi-modal instructions through multi-modal feature extraction, semantic label generation, semantic mapping based on WSABIE model, and correlation quantitative evaluation, has the advantages of high precision, high adaptability and high efficiency, provides strong technical support for multi-modal interaction application, and significantly improves user experience and system performance.

[0005] To achieve the above purpose, the present application provides the following scheme:

[0006] A semantic representation method of multi-modal operation instructions, comprising:

[0007] obtaining instruction features of different modalities according to the characteristics of multi-modal instruction data;

[0008] determining semantic labels according to the abstract description of user intent;

[0009] inputting the instruction features of different modalities and the semantic labels into a WSABIE model for training to determine the optimal mapping relationship between the instruction features of different modalities and the semantic labels;

[0010] based on the optimal mapping relationship, quantitatively evaluating the correlation between the instruction features of different modalities and the semantic labels to accurately analyze multi-modal instruction data at the semantic level and obtain the semantic representation of the multi-modal instruction data.

[0011] Preferably, the multi-modal instruction data includes image data, gesture data, and touch data.

[0012] Preferably, the instruction features of different modalities are obtained according to the characteristics of the multi-modal instruction data, including:

[0013] image feature extraction is performed on the image data to obtain instruction features of the image data; the instruction features of the image data include pixel values, color channel distributions, object morphologies, surface textures, and semantic scenes;

[0014] gesture recognition and sensor feature extraction are performed on the gesture data to obtain instruction features of the gesture data; the instruction features of the gesture data include motion trajectories, speeds, accelerations, durations, directions, and static posture features of hand joints.

[0015] feature extraction is performed on the touch data based on a sensor signal processing algorithm to obtain instruction features of the touch data; the instruction features of the touch data include coordinate positions, touch areas, pressures, durations, and execution sequences of touch actions.

[0016] Preferably, the image feature extraction performed on the image data to obtain instruction features of the image data includes:

[0017] filtering preprocessing is performed on the image data to obtain a filtered image;

[0018] segmentation is performed on the filtered image to obtain a target region and a background region;

[0019] pixel value feature extraction, color channel distribution feature extraction, object morphology feature extraction, surface texture feature extraction, and semantic scene feature extraction are performed on the image at a pixel position in the target region to obtain instruction features of the image data.

[0020] Preferably, the filtering preprocessing performed on the image data to obtain a filtered image includes:

[0021] noise points on the image data are detected using a filtering window to obtain to-be-processed noise points;

[0022] when the number of to-be-processed noise points in the filtering window is greater than a preset threshold, the image in the corresponding filtering window is denoised;

[0023] the filtering window is slid until the entire image data is traversed, and the filtered image is obtained.

[0024] Preferably, the expression of the semantic label is:

[0025]

[0026] wherein S is the semantic label, Embed(Intent) is an intent keyword extracted by analyzing the voice or text input of the user through natural language processing technology, C i is the i-th feature of the context, γ i is the weight of the i-th feature, n is the number of features, Action is a specific operation expressed by the multi-modal instruction data, Action is a result obtained by feature extraction through a CNN network, Object is a detection result obtained by an image-based target detection algorithm, β1, β2, and β3 are dynamic weights, and the sum of β1, β2, and β3 is 1.

[0027] Preferably, based on the optimal mapping relationship, the relevance of the instruction features of different modalities and the semantic label is quantitatively evaluated to accurately analyze the multi-modal instruction data at the semantic level and obtain the semantic representation of the multi-modal instruction data, including:

[0028] The relevance of the instruction features and the semantic label is determined according to a mutual information calculation method; the expression of the relevance MI(F, S) is: wherein P(f, s) is the joint probability distribution of the instruction feature F and the semantic label S, P(f) is the marginal probability distribution of the instruction feature F, and P(s) is the marginal probability distribution of the semantic label S.

[0029] The importance weight of different instruction features on the semantic label is automatically learned according to an attention mechanism in deep learning; the expression of the attention mechanism is: wherein e i = Score(F i , S) = W a · (F i ⊙ S) + b a , e i represents the relevance score of the i-th feature F i and the semantic label S; α i is the importance weight of the i-th feature, and after normalization, it satisfies W a and b a represent trainable parameters of the attention mechanism; and ⊙ is an element-wise product.

[0030] The semantic representation of the multi-modal instruction data is determined according to the relevance and the importance weight.

[0031] Preferably, the formula of the semantic representation is:

[0032]

[0033] wherein w i is the optimal weight, wi = λ1·MI(F i ,S) + λ2·α i , λ1 and λ2 are extreme adjustment coefficients, used to balance the contribution of mutual information and attention mechanism.

[0034] A semantic representation system of multi-modal operation instruction, comprising:

[0035] A feature acquisition unit is configured to obtain instruction features of different modalities according to characteristics of multi-modal instruction data;

[0036] A label acquisition unit is configured to determine semantic labels according to abstract descriptions of user intentions;

[0037] A mapping determination unit is configured to input the instruction features of different modalities and the semantic labels into a WSABIE model for training, so as to determine an optimal mapping relationship between the instruction features of different modalities and the semantic labels;

[0038] A semantic representation unit is configured to quantitatively evaluate the correlation between the instruction features of different modalities and the semantic labels based on the optimal mapping relationship, so as to accurately analyze multi-modal instruction data at a semantic level and obtain semantic representation of the multi-modal instruction data.

[0039] According to the embodiments of the present application, the following technical effects are provided:

[0040] The present application provides a semantic representation method and system of multi-modal operation instruction, comprising: obtaining instruction features of different modalities according to characteristics of multi-modal instruction data; determining semantic labels according to abstract descriptions of user intentions; inputting the instruction features of different modalities and the semantic labels into a WSABIE model for training, so as to determine an optimal mapping relationship between the instruction features of different modalities and the semantic labels; quantitatively evaluating the correlation between the instruction features of different modalities and the semantic labels based on the optimal mapping relationship, so as to accurately analyze multi-modal instruction data at a semantic level and obtain semantic representation of the multi-modal instruction data. The present application successfully realizes semantic alignment and accurate analysis of multi-modal instruction through multi-modal feature extraction, semantic label generation, semantic mapping based on the WSABIE model and correlation quantitative evaluation, and has the advantages of high precision, high adaptability and high efficiency, thereby providing strong technical support for multi-modal interaction application and significantly improving user experience and system performance. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below only illustrate some of the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0042] Figure 1 The method flowchart provided for the embodiments of the present application is as shown in

[0043] Figure 2 The system structure schematic diagram provided for the embodiments of the present application is as shown in DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application.

[0045] The purpose of the present application is to provide a semantic representation method and system for multi-modal operation instructions, which successfully realizes semantic alignment and accurate analysis of multi-modal instructions through multi-modal feature extraction, semantic label generation, semantic mapping based on the WSABIE model and correlation quantitative evaluation, has the advantages of high precision, high adaptability and high efficiency, provides strong technical support for multi-modal interaction applications, and significantly improves user experience and system performance.

[0046] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0047] Figure 1 The method flowchart provided for the embodiments of the present application is as shown in Figure 1 The present application provides a semantic representation method for multi-modal operation instructions, which comprises:

[0048] Step 100: obtaining instruction features of different modalities according to the characteristics of multi-modal instruction data;

[0049] Step 200: determining semantic labels according to the abstract description of user intent;

[0050] Step 300: inputting the instruction features of different modalities and the semantic labels into a WSABIE model for training to determine the optimal mapping relationship between the instruction features of different modalities and the semantic labels;

[0051] Step 400: Based on the optimal mapping relationship, the correlation of instruction features of different modalities and semantic labels is quantitatively evaluated to accurately analyze the multi-modal instruction data at the semantic level, and the semantic representation of the multi-modal instruction data is obtained.

[0052] Preferably, the multi-modal instruction data includes image data, gesture data and touch data.

[0053] Specifically, due to the inherent high complexity and diversity of multi-modal data, it presents unique feature representation, where each modality has its irreplaceable characteristics. In terms of image modality, the information contained is extensive, ranging from basic level pixel value and color channel distribution to higher order level object shape, surface texture and overall semantic scene and multiple dimensions. The gesture modality contains rich dynamic features, such as the motion trajectory of the hand joint in space, the motion speed and acceleration, the time length and direction of the gesture, and the features related to the static posture at the beginning and end of the gesture. The touch modality covers multiple aspects of touch interaction, including the specific coordinate position of the touch point, the size of the touch area, the touch pressure, the touch duration and the execution sequence of the touch action. As the core of analyzing multi-modal information, semantic labels play a role in highly abstractly describing user intent. In order to thoroughly understand the deep connection between multi-modal information and semantic labels, the present application needs to reveal the corresponding relationship between different modal features and specific semantics through extensive data analysis and in-depth domain knowledge. For example, the shape and position of an object in the image may be closely related to the semantic label of "activating the application related to the object"; and a specific gesture motion pattern may be mapped to specific semantics such as "confirm" or "cancel".

[0054] Preferably, the instruction features of different modalities are obtained according to the characteristics of the multi-modal instruction data, including:

[0055] The image data is subjected to image feature extraction to obtain the instruction features of the image data; the instruction features of the image data include pixel value, color channel distribution, object shape, surface texture and semantic scene;

[0056] The gesture data is subjected to gesture recognition and sensor feature extraction to obtain the instruction features of the gesture data; the instruction features of the gesture data include motion trajectory, speed, acceleration, duration, direction and static posture features of the hand joint.

[0057] The touch data is subjected to feature extraction based on sensor signal processing algorithm to obtain the instruction features of the touch data; the instruction features of the touch data include coordinate position of the touch point, touch area, pressure, duration and execution sequence of the touch action.

[0058] Optionally, the WSABIE labeling technique endows the present application with powerful capability of processing multi-modal data labeling and semantic mapping. Firstly, in the aspect of feature extraction of multi-modal data, the present application designs a dedicated feature extraction module. For image data, through combining the deep feature extraction capability of convolutional neural network (CNN) with traditional image feature extraction technology, comprehensive representation of image is realized. In the field of gesture recognition, the present application adopts a strategy of fusing sensor data feature extraction algorithm and vision-based gesture recognition technology to ensure the accuracy of gesture description. As for touch feature, the present application effectively obtains the feature expression of touch data through the processing algorithm of touch sensor signal. Projecting multi-modal feature and semantic label into a shared latent embedding space constitutes the core step of the research. When designing this latent embedding space, the dimension difference of different modal features and the multi-element nature of semantics must be fully considered. By training the WSABIE model and optimizing its weight and parameter, the present application can ensure that the multi-modal feature vector and the semantic label vector find the optimal mapping relationship in this space. In the training process, the present application uses a large-scale multi-modal data set containing instruction data generated by users in various scenarios to improve the generalization performance of the model.

[0059] Preferably, the image feature extraction is performed on the image data to obtain instruction features of the image data, including:

[0060] Filtering preprocessing is performed on the image data to obtain a filtered image.

[0061] Segmentation is performed on the filtered image to obtain a target region and a background region.

[0062] Pixel value feature extraction, color channel distribution feature extraction, object shape feature extraction, surface texture feature extraction and semantic scene feature extraction are performed on the image at the pixel position in the target region to obtain the instruction features of the image data.

[0063] Preferably, the filtering preprocessing is performed on the image data to obtain a filtered image, including:

[0064] Noise points on the image data are detected using a filtering window to obtain to-be-processed noise points.

[0065] When the number of to-be-processed noise points in the filtering window is greater than a preset threshold, the image in the corresponding filtering window is denoised.

[0066] The filtering window is slid until the entire image data is traversed to obtain the filtered image.

[0067] Preferably, the expression of the semantic label is:

[0068]

[0069] Where S is the semantic tag, Embed(Intent) is the intent keyword extracted by parsing the user's voice or text input through natural language processing technology; C i For the i-th feature of the context, γ i β1 represents the weight of the i-th feature, n represents the number of features, Action represents the specific operation expressed by the multimodal instruction data, and Action is the result obtained by feature extraction through the CNN network; Object represents the detection result obtained by the image-based object detection algorithm, β1, β2 and β3 are dynamic weights, and the sum of β1, β2 and β3 is 1.

[0070] As an example, the determination of the dynamic weights β1, β2, and β3 in this embodiment is completed automatically during the model training process. During model initialization, weights can be set to equal values ​​(e.g., β1 = β2 = β3) to ensure balanced contributions from each component in the initial state. During training, an objective function (e.g., a loss function based on semantic matching) is defined, and the weights are optimized using backpropagation. The model dynamically adjusts the weight values ​​based on the matching performance between multimodal features (Intent, Action, Object) and semantic labels, ensuring that each weight reflects the actual contribution of its corresponding feature to the semantic label. This dynamic weight adjustment depends on the relevance and importance of the features and semantic labels. The relevance between features and semantic labels is quantified using mutual information calculation, while an attention mechanism captures the dynamic relationship between features and semantic labels. The model dynamically adjusts β1, β2, and β3 based on these quantification results. For example, if a feature (e.g., Intent) contributes significantly to the semantic label, the model automatically increases the weights associated with that feature (e.g., β1). This dynamic adjustment mechanism ensures that the weight distribution adapts to different scenarios and user needs while maintaining the normalization constraint of β1 + β2 + β3 = 1.

[0071] Specifically, the embodiment realizes the generation of semantic labels through the processing and feature extraction of multi-modal data. First, the natural language processing technology (such as BERT or Word2Vec) is used to analyze the user's voice or text input, and the core intent keywords (Intent) of the user are extracted. At the same time, combined with the context information (such as device status, user historical behavior, etc.), the contribution of each feature to the semantic label is dynamically adjusted by weight allocation of multiple features of the context. The weight of the context feature is automatically optimized by model training to ensure that it can accurately reflect the semantic requirements of the current scene. Then, the generation of semantic labels is further improved through the feature extraction of multi-modal data. Specifically, the Action feature extracts key behavior features from the user's gestures, touch or other operations through a convolutional neural network (CNN); the Object feature identifies the target object of the user's operation from the image through a target detection algorithm (such as Faster R-CNN). Finally, the Intent, Action and Object features are combined with dynamic weights for fusion to generate complete semantic labels, ensuring that they can accurately express the user's intent and provide support for subsequent semantic analysis and instruction execution.

[0072] Optionally, step 300 of the embodiment includes:

[0073] 1、First, prepare a multi-modal data set, including instruction features of image, gesture, touch and other modalities and corresponding semantic labels. Preprocess the instruction features of each modality, such as normalizing image data, time series segmentation of gesture data, path feature extraction of touch data, etc., to ensure that the features of all modalities can be uniformly represented. At the same time, the semantic labels are embedded (such as using Word2Vec or BERT), which are converted into high-dimensional vector representation, so as to align with multi-modal features in the same space.

[0074] 2、Next, design and initialize the WSABIE model. The core of the WSABIE model is a shared latent embedding space that maps multi-modal instruction features and semantic labels into the same semantic space. The input of the model includes multi-modal feature vectors and semantic label vectors, and the output is the similarity score between them. By defining an objective function (such as a ranking-based loss function), the model can learn how to maximize the similarity between correct instruction features and semantic labels, while minimizing the similarity of incorrect matches.

[0075] 3. During model training, supervised learning is used to optimize the WSABIE model. Multimodal features and their corresponding semantic labels are used as positive samples, while some incorrectly matched negative samples are randomly generated. Through comparative learning of positive and negative sample pairs, the model can gradually adjust the mapping relationship in the embedding space, so that the similarity score of positive samples is higher than that of negative samples. Gradient descent algorithm can be used to optimize the model parameters during training to ensure that the model converges to the optimal state.

[0076] 4. Finally, validate and fine-tune the model's performance. Evaluate the model's accuracy, recall, and F1 score on the validation set to determine if the WSABIE model can accurately match multimodal features with semantic labels. If performance is unsatisfactory, adjust the model's hyperparameters (such as the dimension of the embedding space, learning rate, etc.) or increase the diversity of the training data. The optimized WSABIE model can generate optimal mapping relationships, providing efficient and accurate support for the semantic parsing of multimodal instructions.

[0077] To accurately parse instructions at the semantic level, this invention requires precise quantification of the correlation between instruction features and semantic labels. In addition to traditional statistical correlation analysis methods, such as mutual information calculation and Pearson correlation coefficient calculation, this invention also introduces an attention mechanism from deep learning. This mechanism can automatically learn the importance weights of different features for semantic labels. For example, in a multimodal instruction that integrates images and gestures, the attention mechanism can identify image regions semantically related to gestures and assign higher weights to these regions, thereby achieving a more accurate measurement of the overall correlation between the instruction and the semantic label. Furthermore, this invention constructs a dynamic correlation evaluation model that can update the correlation measurement between features and semantic labels in real time as new multimodal instruction data is continuously input. This real-time update can be achieved through online learning algorithms, ensuring that the system can adapt to new user behaviors and instruction patterns.

[0078] Preferably, based on the optimal mapping relationship, the correlation between the instruction features of different modalities and semantic tags is quantitatively evaluated to accurately parse the multimodal instruction data at the semantic level and obtain the semantic representation of the multimodal instruction data, including:

[0079] The relevance between the instruction feature and the semantic tag is determined according to the mutual information calculation method; the expression for the relevance MI(F,S) is: Where, P(f,s): the joint probability distribution of instruction feature F and semantic label S, P(f) is the marginal probability distribution of instruction feature F, and P(s) is the marginal probability distribution of semantic label S.

[0080] According to the attention mechanism in deep learning, importance weights of different instruction feature to the semantic label are automatically learned; an expression of the attention mechanism is: wherein e i = Score(F i , S) = W a · (F i ⊙ S) + b a , e i represents a correlation score of the i-th feature F i to the semantic label S; a i is an importance weight of the i-th feature, and after normalization, satisfies W a and b a represent trainable parameters of the attention mechanism; and ⊙ is an element-wise product.

[0081] According to the correlation and the importance weight, a semantic representation of the multi-modal instruction data is determined.

[0082] Preferably, a formula of the semantic representation is:

[0083]

[0084] wherein w i is a maximum weight, w i = λ1· MI(F i , S) + λ2· a i , λ1 and λ2 are extreme weight adjustment coefficients, and are used to balance contributions of mutual information and the attention mechanism.

[0085] Specifically, the step 400 of the embodiment includes:

[0086] 1. Mutual information calculates correlation

[0087] First, the correlation between the instruction feature and the semantic label is quantified by a mutual information calculation method. Mutual information is used to measure the degree of dependence between the instruction feature and the semantic label, and can reflect the contribution size of the feature to the semantic label. In specific implementation, joint probability distribution of the multi-modal instruction feature and the semantic label and marginal probability distribution of each are counted. By analyzing these probability distributions, a correlation score between each modal feature (such as image feature, gesture feature, etc.) and the semantic label can be calculated. The higher the mutual information value, the greater the contribution of the feature to the semantic label, which provides a basis for subsequent weight allocation.

[0088] 2. Attention mechanism learns weight

[0089] Next, the importance weight of different modal features to semantic labels is automatically learned using the attention mechanism in deep learning. The attention mechanism dynamically adjusts the importance of features by calculating the relevance score of each feature to the semantic label. In implementation, a trainable scoring function (such as dot product or multi-layer perceptron) is used to calculate the relevance score of each feature to the semantic label, and a weight distribution is generated by softmax normalization. The trainable parameters of the attention mechanism are continuously optimized through model training, so that the system can focus on the feature area most relevant to the semantic label, thereby improving the accuracy of semantic parsing.

[0090] 3. Generating semantic representation by combining relevance and weight

[0091] After obtaining the mutual information relevance score and the attention mechanism weight, the two are combined to generate the final feature weight. By dynamically adjusting the contribution ratio of mutual information and attention mechanism, it is ensured that the feature weight can fully reflect the relationship between the feature and the semantic label. Mutual information provides a statistical level of relevance quantification, while the attention mechanism captures the dynamic relationship between the feature and the semantic label through deep learning. The combination of the two makes the final feature weight not only statistically meaningful, but also adaptable to complex multi-modal scenarios.

[0092] 4. Generating semantic representation of multi-modal instructions

[0093] Finally, according to the comprehensive feature weight, all modal features are weighted and aggregated to generate the semantic representation of multi-modal instruction data. The semantic representation is a high-dimensional vector that can accurately express the semantic information of multi-modal instructions. By dynamically adjusting the contribution ratio of mutual information and attention mechanism, the semantic representation can adapt to different scenarios and user needs. The generated semantic representation can not only be used for semantic parsing, but also be passed as input to subsequent task modules (such as instruction execution or semantic retrieval), thereby realizing efficient processing and execution of multi-modal instructions.

[0094] Exemplarily, in the model initialization, an initial value (such as uniform distribution or empirical value) is set for each extreme weight adjustment coefficient to ensure that the contribution of different parts is balanced in the initial stage. Subsequently, in the training process, an objective function (such as a loss function of semantic matching or classification accuracy) is defined, and gradient descent and other optimization algorithms are used to iteratively update the extreme weight adjustment coefficients. The model will dynamically adjust the values of these coefficients according to the matching effect of multi-modal features and semantic labels, so that it can balance the contributions of mutual information and attention mechanism, thereby maximizing the accuracy and robustness of the semantic representation. In addition, cross-validation or grid search methods can be used to test different combinations of coefficients on the validation set, and the optimal parameter configuration is selected to ensure that the extreme weight adjustment coefficients can adapt to different scenarios and task requirements.

[0095] The present application aims to construct an integrated technical solution based on an efficient system architecture. This architecture covers key modules such as multi-modal data acquisition, feature extraction, WSABIE-based semantic mapping, correlation measurement, and LLM interface. To optimize system performance, this solution adopts distributed computing technology to handle large multi-modal data sets, significantly improving computing efficiency. In addition, by compressing and quantizing the model, memory requirements and computing resource consumption can be effectively reduced, ensuring that the system can run stably on various hardware platforms. This technical solution aims to achieve fast and accurate processing of multi-modal operation instruction semantics, providing strong technical support for multi-modal interaction applications.

[0096] The following is an implementation case of the present application in an intelligent interaction system, which aims to provide users with convenient and efficient multi-modal interaction experience, and realize accurate understanding and execution of multi-modal operation instruction semantics.

[0097] 1. System environment and data preparation

[0098] The present application implements the developed system on an intelligent terminal integrating various interactive devices, which integrates touch screen, camera, sensor and other devices, aiming to realize the collection of multi-modal data such as touch, image and gesture. To further train and optimize the system, the present application collects a large number of multi-modal instruction samples. These samples reflect the interaction operations of different users in various application scenarios such as games, office work and entertainment. For example, in the game application scenario, users may use gesture instructions to control character movement and touch instructions to execute skill release; in the office application scenario, users may use gesture instructions to switch document pages and image recognition technology to open specified files.

[0099] 2. Multi-modal data acquisition and preprocessing

[0100] During the interaction between the user and the intelligent terminal, the system begins to collect multi-modal data. In the recording of touch data, the sensor accurately records the coordinates, pressure and duration of the touch point and other key information. The camera captures the user's gesture action in real time and extracts the image sequence of the gesture. For image data, the system not only obtains the image of the user on the operation interface, but also includes the image of the surrounding environment (which may be related to the instruction semantics in certain scenarios). The system performs preliminary processing on the collected data, including data cleaning, normalization and other steps. For example, taking touch pressure values as an example, the system normalizes them to a specific range to eliminate the impact of differences between different devices on the data.

[0101] 3. WSABIE-based semantic mapping

[0102] In the feature extraction link, the system extracts the dynamic path characteristics and pressure change characteristics of the touch point for touch data. For gesture images, the system uses a pre-trained convolutional neural network model to extract features such as shape and motion direction. For image data feature extraction, the system uses object recognition and scene classification models to obtain key object and scene information in the image. Next, these multi-modal features and semantic labels are input into the WSABIE-based labeling model. In the trained latent embedding space, the system can accurately map multi-modal features to corresponding semantic labels.

[0103] 4. Correlation measurement and instruction understanding

[0104] The present application quantitatively evaluates the correlation between instruction features and semantic labels by means of attention mechanism and statistical correlation analysis. Taking "open notification bar" as an example, the attention mechanism will focus on the upper sliding action of the gesture and the lower right corner of the touch screen, two key features, and give them higher weights, because these two features have significant correlation with the "open notification bar" semantic in the training data. Further, by calculating the statistical correlation index of these features and semantic labels, the semantic of the instruction is further confirmed.

[0105] 5. Integration and execution with large language model (LLM)

[0106] Once the system accurately masters the semantic content of multi-modal instructions, it is translated into a format that can be recognized by the language model (LLM), and passed to the LLM. Then, the LLM will implement the corresponding command at the application or operating system level according to the received semantic information. For example, if the semantic instruction is "open notification bar", the LLM will issue a specific instruction to the system to activate the display function of the notification bar.

[0107] 6. Continuous optimization and feedback

[0108] During the operation of the system, new multi-modal instruction data is continuously collected and incorporated into the feedback mechanism. This batch of new data will be used to further refine the WSABIE model and the correlation evaluation system. For example, in the face of a novel combination of gestures and touch, the system can update the model based on online learning algorithms to ensure correct interpretation of the semantics of such new instructions. This process continuously improves the accuracy and robustness of the system in understanding the semantics of multi-modal operation instructions, and thus provides users with a more smooth and natural multi-modal interaction experience.

[0109] Corresponding to the above method, the present embodiment also provides a semantic representation system for multi-modal operation instructions, comprising:

[0110] The feature acquisition unit is configured to obtain instruction features of different modalities according to characteristics of the multi-modal instruction data;

[0111] The label acquisition unit is configured to determine semantic labels according to the abstract description of the user intention;

[0112] The mapping determination unit is configured to input the instruction features of the different modalities and the semantic labels into a WSABIE model for training, so as to determine an optimal mapping relationship between the instruction features of the different modalities and the semantic labels;

[0113] The semantic representation unit is configured to quantitatively evaluate the correlation between the instruction features of the different modalities and the semantic labels based on the optimal mapping relationship, so as to accurately analyze the multi-modal instruction data at a semantic level and obtain semantic representations of the multi-modal instruction data.

[0114] The beneficial effects of the present application are as follows:

[0115] (1) The present application solves the problems of feature dimension difference and semantic expression inconsistency between multi-modal data by mapping the features of multi-modal data (such as images, gestures, touch, etc.) and semantic labels to a shared latent embedding space; makes non-directly semantic multi-modal instructions (such as gestures, touch) can be understood by large language models (LLM), thereby realizing semantic alignment of multi-modal instructions.

[0116] (2) The present application quantitatively evaluates the correlation between multi-modal features and semantic labels through mutual information calculation and attention mechanism, which can automatically identify key feature regions related to semantics and give them higher weights; the accurate correlation quantification method of the present application ensures that the semantic analysis of multi-modal instructions is more accurate, reducing the possibility of misunderstanding or incorrect analysis.

[0117] (3) Based on the large-scale training of the WSABIE model, the present application uses diversified instruction samples in the multi-modal data set to improve the adaptability of the model to different scenes and user behaviors; through dynamic correlation evaluation model and online learning algorithm, the system can update the correlation between features and semantic labels in real time, adapt to new user behaviors and instruction modes, and further enhance the generalization ability and robustness of the system.

[0118] (4) Through feature extraction and semantic mapping of multi-modal data such as images, gestures, touch, etc., the present application can support complex multi-modal interaction scenarios, such as games, office work, entertainment, etc.; in these scenarios, users can realize complex operations through multi-modal instructions (such as combination of gestures and touch), and the system can accurately understand and execute the corresponding commands.

[0119] (5) The application adopts distributed computing technology to process large-scale multi-modal data, significantly improves the computing efficiency of the system, reduces the memory requirement and computing resource consumption through model compression and quantization processing, and enables the system to run stably on various hardware platforms.

[0120] (6) The application can be continuously optimized, continuously improves the semantic understanding ability of new instructions through a feedback mechanism and an online learning algorithm, and provides users with a more smooth and natural multi-modal interaction experience; users can realize intuitive and efficient operation through multi-modal instructions (such as gestures, touch, voice, etc.), and reduce the learning cost and operation complexity.

[0121] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the system disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0122] The principles and implementation modes of the application are described by applying specific examples in this paper, and the above embodiment description is only used to help understand the method and core idea of the application; at the same time, for the general technical personnel in the art, according to the idea of the application, the specific implementation mode and application range will be changed. In view of the above, the content of the specification should not be understood as a limitation of the application.

Claims

1. A method of semantic representation of multi-modal operation instructions, characterized in that, The method comprises the following steps: According to the characteristics of the multi-modal instruction data, instruction features of different modalities are obtained; According to the abstract description of the user's intention, a semantic label is determined; The instruction features of different modalities and the semantic label are input into a WSABIE model for training to determine the optimal mapping relationship between the instruction features of different modalities and the semantic label; Based on the optimal mapping relationship, the correlation between the instruction features of different modalities and the semantic label is quantitatively evaluated to accurately analyze the multi-modal instruction data at the semantic level and obtain the semantic representation of the multi-modal instruction data; The expression of the semantic label is: wherein S is the semantic label, Embed(Intent) is an intent keyword extracted by analyzing the voice or text input of the user through a natural language processing technology; C i is the i-th feature of the context, γ i is the weight of the i-th feature, n is the number of features, Action is a specific operation expressed by the multi-modal instruction data, Action is a result obtained by feature extraction through a CNN network; Object is a detection result obtained based on an image-based target detection algorithm, β1, β2 and β3 are dynamic weights, and the sum of β1, β2 and β3 is 1; Based on the optimal mapping relationship, the correlation between the instruction features of different modalities and the semantic label is quantitatively evaluated to accurately analyze the multi-modal instruction data at the semantic level and obtain the semantic representation of the multi-modal instruction data, comprising: The relevance of the instruction feature and the semantic label is determined according to a mutual information calculation method; an expression of the relevance MI(F, S) is: Wherein, P(f, s): joint probability distribution of the instruction feature F and the semantic label S, P(f) is a marginal probability distribution of the instruction feature F, and P(s) is a marginal probability distribution of the semantic label S. According to the attention mechanism in deep learning, the importance weight of different instruction features to the semantic label is automatically learned; the expression of the attention mechanism is: Wherein, e i = Score(F i , S) = W a · (F i ⊙S) + b a , e i represents the correlation score of the ith feature F i and the semantic label S; a i is the importance weight of the ith feature, which satisfies W a and b a represent trainable parameters of the attention mechanism; and ⊙ represents element-wise multiplication. According to the correlation and the importance weight, the semantic representation of the multi-modal instruction data is determined.

2. The method of claim 1, wherein, The multi-modal instruction data comprises image data, gesture data and touch data.

3. The method of claim 2, wherein, According to the characteristics of the multi-modal instruction data, instruction features of different modalities are obtained, comprising: Image feature extraction is performed on the image data to obtain instruction features of the image data; the instruction features of the image data comprise pixel value, color channel distribution, object shape, surface texture and semantic scene; Gesture recognition and sensor feature extraction are performed on the gesture data to obtain instruction features of the gesture data; the instruction features of the gesture data comprise motion trajectory, speed, acceleration, duration, direction and static posture features of hand joints, Based on a sensor signal processing algorithm, feature extraction is performed on the touch data to obtain instruction features of the touch data; the instruction features of the touch data comprise coordinate position, touch area, pressure, duration and execution sequence of touch actions.

4. The method of claim 3, wherein, Image feature extraction is performed on the image data to obtain instruction features of the image data, comprising: Filtering preprocessing is performed on the image data to obtain a filtered image; Segmentation is performed on the filtered image to obtain a target region and a background region; Pixel value feature extraction, color channel distribution feature extraction, object shape feature extraction, surface texture feature extraction and semantic scene feature extraction are performed on the image data at the pixel position in the target region to obtain the instruction features of the image data.

5. The method of claim 4, wherein, Filtering preprocessing is performed on the image data to obtain a filtered image, comprising: Noise points on the image data are detected using a filtering window to obtain to-be-processed noise points; When the number of to-be-processed noise points in the filtering window is greater than a preset threshold, the image in the corresponding filtering window is denoised; The filtering window is slid until the entire image data is traversed, and the filtered image is obtained.

6. The method of claim 1, wherein, The formula of the semantic representation is: where w i is the final weight, w i = λ1·MI(F i ,S) + λ2·α i , λ1and λ2are extreme weight adjustment coefficients, used to balance the contribution of mutual information and attention mechanism.

7. A system for semantic representation of multi-modal operation instructions, characterized by Comprising: a feature acquisition unit configured to obtain instruction features of different modalities according to characteristics of multi-modal instruction data; a label acquisition unit configured to determine a semantic label according to an abstract description of a user's intention; The mapping determination unit is configured to input the instruction features of the different modalities and the semantic labels into a WSABIE model for training, so as to determine an optimal mapping relationship between the instruction features of the different modalities and the semantic labels; The semantic representation unit is configured to quantitatively evaluate the correlation between the instruction features of the different modalities and the semantic labels based on the optimal mapping relationship, so as to accurately analyze the multi-modal instruction data at a semantic level and obtain semantic representation of the multi-modal instruction data. The expression of the semantic label is: wherein S is the semantic label, Embed(Intent) is an intent keyword extracted by analyzing the voice or text input of the user through a natural language processing technology; C i is the i-th feature of the context, γ i is the weight of the i-th feature, n is the number of features, Action is a specific operation expressed by the multi-modal instruction data, Action is a result obtained by feature extraction through a CNN network; Object is a detection result obtained based on an image-based target detection algorithm, β1, β2 and β3 are dynamic weights, and the sum of β1, β2 and β3 is 1; The semantic representation unit is configured to quantitatively evaluate the correlation between the instruction features of the different modalities and the semantic labels based on the optimal mapping relationship, so as to accurately analyze the multi-modal instruction data at a semantic level and obtain semantic representation of the multi-modal instruction data, including: The relevance of the instruction feature and the semantic label is determined according to a mutual information calculation method; an expression of the relevance MI(F, S) is: Wherein, P(f, s): joint probability distribution of the instruction feature F and the semantic label S, P(f) is a marginal probability distribution of the instruction feature F, and P(s) is a marginal probability distribution of the semantic label S. According to the attention mechanism in deep learning, the importance weight of different instruction features to the semantic label is automatically learned; the expression of the attention mechanism is: Wherein, e i = Score(F i , S) = W a · (F i ⊙S) + b a , e i represents the correlation score of the ith feature F i and the semantic label S; α i is the importance weight of the ith feature, which satisfies W a and b a represent trainable parameters of the attention mechanism; and ⊙ represents element-wise multiplication. determining the semantic representation of the multi-modal instruction data according to the correlation and the importance weight.

Images