Method and system for improving intention perception

Abstract

The invention relates to the technical field of natural language processing and machine learning, in particular to an intention perception improving method and system. The method comprises: acquiring image data; inputting the image data into the constructed intention perception model to obtain a corresponding intention perception result; wherein the intention perception model is a model obtained by training with training data, and the training data comprises various types of image data and corresponding feature tag annotations; the intention perception model comprises an LLMTransform module and an intention harmony optimizer which are in communication connection in sequence, the LLMTransform module is used for performing intention prediction according to input image data, and the intention harmony optimizer is used for optimizing a prediction result output by the LLMTransform module according to the probability of co-occurrence between two intention categories. The prediction of the method is more accurate and effective.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing and machine learning technology, and in particular to a method and system for improving intent perception. Background Art

[0002] Intent perception is an important task in the field of natural language processing and computer vision, which aims to predict the user's intention by analyzing image or text data. Intent perception has a wide range of applications in multiple scenarios, such as intelligent assistants, automatic customer service, recommendation systems, and human-computer interaction. Current research mainly focuses on additional modeling mechanisms and multimodal information, but in intent perception, it is often difficult to deal with relationship ambiguity under inaccurate annotations.

[0003] Intent perception has a high degree of content ambiguity, and there is a certain degree of subjectivity in the labeling of intent categories. In addition, some semantically unrelated intent categories often appear together, and the special artistic conception of the situation is not a tangible thing that the model can perceive. The link between pictures and intents can be naturally connected for humans, but for machines, in this perception process, the model is more inclined to discover specific objects, but it is difficult to combine objects and environments to draw similar conclusions as humans. In addition, intent perception has a high degree of relational ambiguity. Conventional fuzzy systems rely on expert experience for definition, but in intent perception tasks, the co-occurrence rules of most categories cannot be simply defined. Previous intent perception methods are not sufficient to meet the above challenges. Summary of the invention

[0004] The present invention provides a method and system for improving intention perception, aiming to solve the problem that current intention perception has high ambiguity and cannot be recognized more accurately.

[0005] The objective of the present invention is achieved by the following technical solutions:

[0006] An intention awareness enhancement method, comprising:

[0007] Acquire image data; input the image data into the constructed intention perception model to obtain corresponding intention perception results;

[0008] The intention perception model is a model trained using training data, wherein the training data includes various types of image data and corresponding feature label annotations;

[0009] The intent perception model includes an LLM Transformer module and an intent harmony optimizer which are communicatively connected in sequence, wherein the LLM Transformer module is used to predict intent based on input image data, and the intent harmony optimizer is used to optimize the prediction result output by the LLM Transformer module based on the probability of co-occurrence between two intent categories.

[0010] As a further improvement of the present invention, the intention harmony optimizer includes an intention harmony matrix module, which is: obtaining intent-aware label annotations based on training data, and obtaining the intention harmony matrix module based on statistical results of co-occurrence frequencies between different intent categories. The intention harmony matrix module is a symmetric matrix, and the elements corresponding to each position represent the co-occurrence probability between two categories.

[0011] As a further improvement of the present invention, the method also includes obtaining co-occurrence weights through the intention harmony matrix, and adjusting the prediction results of other categories based on the co-occurrence weights and corresponding prediction results.

[0012] As a further improvement of the present invention, the LLM Transformer module includes a feature extractor, an encoder, and a decoder connected in sequence, the feature extractor is used to extract a feature vector based on an input image, and input the feature vector into the encoder, the encoder is used to encode the input feature vector and output an output sequence of a hidden state, and the decoder is used to decode the sequence output by the encoder.

[0013] As a further improvement of the present invention, the feature extractor is a ViT module, which is used to divide the input image into a set size, perform linear transformation on each image block after segmentation, convert pixel data into a feature vector, and set a position code for the segmented image block, which represents the relative position of the image block before segmentation.

[0014] As a further improvement of the present invention, the LLM Transformer module also includes a multi-dimensional adapter, which is arranged between the feature extractor and the encoder, and includes a plurality of convolution blocks of different hidden dimensions, and is used for performing visual marking processing and visual prompt processing on the input feature vector.

[0015] As a further improvement of the present invention, the multidimensional adapter is used to project visual markers into different representation spaces at the input stage to adapt to intention perception tasks of different difficulty levels. At the output stage, each embedding vector in the visual marker sequence is converted into a corresponding query vector and key vector, and a unique rotation position code is calculated based on the feature vector corresponding to each marker position. Each pair of query vector and key vector is rotated using the rotation position code, and the inner product between each pair of query vector and key vector is calculated to obtain the result of the self-attention mechanism.

[0016] As a further improvement of the present invention, the encoder includes at least one normalization processing module, and the normalization processing module is used to perform normalization processing on the feature vector extracted by the sub-layer of each encoder.

[0017] As a further improvement of the present invention, the neural architecture adopted by the fully connected layer in the LLM Transformer module is a feedforward neural network, and the activation function in the feedforward neural network is a SωiGLU activation function, and the SωiGLU activation function is:

[0018] FFN SωiGLU (a,W,V,W2)=SωiGLU(a,W,V)W2

[0019]

[0020] Sωish β =aσ(aβ)

[0021] Where a is the feature vector input to the feedforward neural network, FFN SωiGLU is the fully connected layer of the feedforward propagation layer, W is the weight matrix of the first linear transformation, V is the weight matrix of the gating mechanism about the weight matrix of the first linear transformation, W2 is the weight matrix of the second linear transformation, β is the activation parameter, Sωish β is the activation function under parameter β, σ is the Sigmoid activation function and SωiGLU activation function, Represents the digit-by-digit multiplication of the Hadamard product.

[0022] In a second aspect, the present invention further provides an intention perception enhancement system for implementing the above-mentioned intention perception enhancement method, the system comprising:

[0023] A data acquisition module, used for acquiring image data;

[0024] An intention perception module, used to input the image data into the constructed intention perception model to obtain a corresponding intention perception result;

[0025] The intention perception model is a model trained using training data, wherein the training data includes various types of image data and corresponding feature label annotations;

[0026] The intent perception model includes an LLM Transformer module and an intent harmony optimizer which are communicatively connected in sequence, wherein the LLM Transformer module is used to predict intent based on input image data, and the intent harmony optimizer is used to optimize the prediction result output by the LLM Transformer module based on the probability of co-occurrence between two intent categories.

[0027] The beneficial effects of the present invention are as follows: the intent perception enhancement method of the present invention successfully integrates and processes multimodal data such as images and texts mainly through the LLM Transformer module and the intent harmony optimizer. The LLM Transformer module can process feature vectors of different dimensions, and the intent harmony optimizer learns the co-occurrence relationship between matching intent categories, thereby generating more accurate prediction results, which is more suitable for a variety of complex application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1 is an example diagram of the intent harmony optimizer of the present invention;

[0030] Figure 2 It is a flow chart of the intention perception enhancement method of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose and technical solution of the present invention clearer and easier to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. The specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0032] The concept of the present invention is to provide a method and system for improving intention perception, which mainly includes:

[0033] Acquire image data; input the image data into the constructed intention perception model to obtain the corresponding intention perception result;

[0034] The intent perception model is a model trained using training data, which includes various types of image data and corresponding feature label annotations;

[0035] The intent perception model includes an LLM Transformer module and an intent harmony optimizer which are communicatively connected in sequence. The LLM Transformer module is used to predict intent based on input image data, and the intent harmony optimizer is used to optimize the prediction result output by the LLM Transformer module based on the probability of co-occurrence between two intent categories.

[0036] Through the LLM Transformer module and the intent harmony optimizer, the image intent perception prediction results are made more accurate and thus more applicable.

[0037] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings and specific embodiments, wherein the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments.

[0038] Terminology explanation: LLM Transformer: Large Language-Attention Model, a neural network architecture for natural language processing;

[0039] ViT (Vision Transformer): A computer attention vision model used to segment and flatten the original image.

[0040] [CLS]token: category label in ViT;

[0041] MSA: Multi-headed Attention;

[0042] RMS Normalization: Root mean square normalization.

[0043] Example 1

[0044] like Figure 1 , Figure 2 An intention perception enhancement method shown specifically includes the following implementation steps.

[0045] First, training data is obtained. The training data in this embodiment is images with different intent categories. The images of various intent categories are labeled and annotated.

[0046] Construct an intention perception model, and use training data to train the intention perception model. The intention perception model in this embodiment includes an LLM Transformer module and an intention harmony optimizer that are sequentially connected in communication.

[0047] The LLM Transformer module is used to predict intent based on the input image data. The LLM Transformer module includes a feature extractor, an encoder, and a decoder connected in sequence. The feature extractor is located in the input layer and is used to extract a feature vector based on the input image, and input the feature vector into the encoder, which is used to encode the input feature vector and output an output sequence of hidden states, and the decoder is used to decode the sequence output by the encoder.

[0048] The feature extractor in this embodiment is a ViT module, which is used to divide the input image into a set size, perform linear transformation on each image block after segmentation, convert pixel data into a feature vector, and provide a position code for the segmented image block, which indicates the relative position of the image block before segmentation.

[0049] Specifically, Vision Transformer (ViT) is used to extract features. The input image is obtained and divided into small image blocks of a fixed size, and the size of each small image block is a set pixel (for example, 16x16 pixels can be used in this embodiment). This block processing method converts the image into a sequence data format, and each small image block is regarded as an independent input unit. A linear transformation is performed on each image block to convert the pixel data into a feature vector. These feature vectors are low-dimensional representations of the image block and contain the main visual information of the block. In order to retain the relative position information of each image block, position encoding is added to each image block. These position encodings can help the LLM Transformer module understand the spatial relationship between each image block. A learnable classification embedding ([CLS] token) is introduced at the beginning of the sequence to capture global image-level features or class or global information of the image sequence, and generate learnable visual cues.

[0050] The input feature vector is processed by a multi-layer Transformer encoder, which uses a self-attention mechanism to capture the complex relationship between image blocks and further processes and fuses features with a multi-layer neural network. The encoder includes a multi-head self-attention module and a multi-layer perception module, and enhances the global understanding ability of the intent perception model through residual connections and normalization modules. The Transformer encoder is composed of multiple identical layers stacked together, each of which includes at least a self-attention mechanism and a fully connected neural network. Each layer is then processed by residual connections and normalization.

[0051] Specifically, some pixels in the intent image may have different intent classifications or interpretations depending on the global language environment, resulting in a certain degree of semantic ambiguity. To address this challenge, we use a high-quality encoder ViT for feature extraction. For a given intent image, it can be represented as I∈R. We first convert the input into a flat two-dimensional block sequence, represented as:

[0052]

[0053] Where N represents the total number of these flat two-dimensional blocks, then the calculation of N is determined by the following formula:

[0054] N=h*ω / B 2

[0055] Where B is the size of the image block, and the size of each image block is (B 2 ×c).

[0056] A partitioned linear projection is used to obtain the learnable embedding [CLS] in Lin(·) and ViT, where the embedding [CLS] is labeled as e∈R D , for each image block x b , all perform partition linear projection to perform regional projection in the above way. By combining the corresponding position embedding p, the learnable labeling process is encapsulated to obtain the encoder input X t :

[0057]

[0058] Where, X t is the feature vector input to the encoder, The patch of the feature extractor output image is represented as , and p is the position embedding. By adopting a learnable position label tag (i.e., [CLS] tag), it helps to learn the correlation between each patch and the target object. For the embedded patch X t , is forwarded through L consecutive encoder layers. The computation process of each encoder block is as follows:

[0059] x′=MSA(Adapter(LN(x)))+x

[0060] x=MLP(LN(x′))+x′

[0061] Among them, MSA(·) represents multi-head self-attention, Adapter represents adapter, LN represents normalization operation, and MLP is the residual block in the residual network; the adapter usually consists of two linear layers (fully connected layers) and a nonlinear activation function.

[0062] While some types of intent images rely on local parts of the image, others need to consider global information, which leads to a certain ambiguity in the intent perception task. The self-attention mechanism in ViT helps to gain a deeper understanding of the relative importance of various regions in the image. This is achieved by assigning weights to each image patch relative to other image patches, thereby enhancing the model's ability to prioritize and effectively integrate relevant visual information.

[0063] The LLM Transformer module also includes at least one normalization processing module, which is used to be set between the feature extractor and the encoder to normalize the extracted feature vector. By performing layer normalization on the input feature vector, the normalized result is input into the adapter module for further nonlinear transformation, and then processed by multi-head self-attention to obtain the corresponding output data.

[0064] The eigenvectors are normalized to ensure the stability of the input data, specifically:

[0065]

[0066] Among them, RMS(a) is normalized. After normalization, a learnable scaling parameter g is introduced i and the offset parameter b i , where the formula is:

[0067]

[0068] The adapter output Z undergoes three linear transformations to generate the query, key, and value components used in multi-head self-attention. Specifically, the query vector is: Q = WQ·Z, the key vector is: K = WK·Z, and the value vector V = WV·Z, where WQ is the matrix representing the query when transformed, WK is the matrix representing the key when transformed, and WV is the matrix representing the value when transformed. The specific process of multi-head self-attention is:

[0069]

[0070] MSA(Q,K,V)=Concat(head1,head2,...head h )W O

[0071]

[0072] The LLM Transformer module also includes a multidimensional adapter, which is set between the feature extractor and the encoder, and includes several convolution blocks of different hidden dimensions, which are used to perform visual labeling and visual cue processing on the input feature vector. Specifically, the normalized feature vector is passed to the multidimensional adapter. The multidimensional adapter module consists of multiple convolution blocks with different hidden dimensions. Each convolution block projects the feature vector into a different representation space to adapt to intention perception tasks of different difficulties. This design enables the adapter to flexibly handle various types of input features. Finally, the processed feature vector is added to the input feature vector through a residual connection to ensure the stability of the feature vector and the effective transmission of information. Among them, the calculation process expression of the multidimensional adapter is:

[0073] a=Conv R (Droupout(Conv R (a)))+a where R~{α,β,γ}

[0074] Among them, Conv R is the convolutional layer.

[0075] In the multi-dimensional adapter input stage, the multi-dimensional adapter projects visual markers into different representation spaces to adapt to intent perception tasks of different difficulty. The purpose of this is to alleviate the ambiguity problem caused by image diversity. For example, for simple intent images, low-dimensional vector representation is used, while for complex natural landscape combination images, high-dimensional vector representation is required.

[0076] At the output stage, the multi-dimensional adapter converts each embedding vector in the tag sequence into a corresponding query vector and key vector, calculates a unique rotational position code based on the feature vector corresponding to each tag position, uses the rotational position code to rotate each pair of query vectors and key vectors, calculates the inner product between each pair of query vectors and key vectors, and obtains the result of the self-attention mechanism. This embodiment uses rotational position embedding (RoPE) instead of traditional absolute position encoding. RoPE is based on the application of complex numbers and allows the use of mathematical transformations to derive relative position encodings from absolute position encodings.

[0077] The features are further processed and optimized through a feedforward neural network. The output of each submodule is added to the input through a residual connection to ensure the stability of the feature vector and the effective transfer of information. These steps work together to generate high-quality feature representations, enabling the model to effectively understand and process complex visual and textual information. In this embodiment, when improving the LLMTransformer module, the SωiGLU activation function is used instead of the ReLU activation function. This embodiment uses the dynamic gating mechanism of the SωiGLU activation function to process the data. The expression of the SωiGLU activation function is:

[0078] FFN SωiGLU (a,W,V,W2)=SωiGLU(a,W,V)W2

[0079]

[0080] Sωish β =aσ(aβ)

[0081] Where a is the feature vector input to the feedforward neural network (FFN), SωiGLU is the fully connected layer of the feedforward propagation layer, W is the weight matrix of the first linear transformation, which maps the input vector from the original feature space to the hidden layer space, V is the weight matrix of the gating mechanism of the weight matrix of the first linear transformation, W2 is the weight matrix of the second linear transformation, which is mainly used to map the output hidden layer space of the SωiGLU activation function to the output space, β is the activation parameter, Sωish β is the activation function under parameter β, σ is the Sigmoid activation function and SωiGLU activation function, Represents the bit-by-bit multiplication of the Hadamard product. The gating unit is usually combined with the weight matrix of the first linear transformation of the input vector to determine which information should be passed.

[0082] The intent harmony optimizer is used to optimize the prediction results output by the LLMTransformer module based on the probability of co-occurrence between two intent categories.

[0083] Specifically, by collecting intent-aware label annotations from the training data, we count the co-occurrence frequencies between different intent categories. Based on the statistical results, we generate a symmetrical intent harmony matrix. Figure 1As shown in the figure, each element of the matrix represents the probability of two categories appearing at the same time, and the darker the color, the higher the probability. Initialize a symmetric matrix, and the element at each position represents the co-occurrence frequency between two categories. The width and height of the matrix are consistent with the number of categories. The elements in the matrix are normalized to ensure that the co-occurrence probability is between 0 and 1. For the initial intent perception results obtained by the model, the harmony matrix is ​​used to enhance the initial perception results. Furthermore, the co-occurrence weights are obtained through the intent harmony matrix, and the prediction results of other categories are adjusted based on the co-occurrence weights and the corresponding prediction results.

[0084] The specific operation method is to use the average of the probabilities of the co-occurrence categories (i, j) in the label and assign them to the symmetric matrices (i, j) and (j, i). All symmetric matrices are added and the elements in the matrix are normalized to ensure that the co-occurrence probability is between 0 and 1. Generate a harmony matrix and use the harmony matrix to enhance the initial prediction results. For each intent category, according to its initial prediction score, the co-occurrence weights with other categories are obtained from the harmony matrix, and the prediction scores of other categories are adjusted. This adjustment enables the model to more accurately reflect the co-occurrence relationship between categories. For categories with higher scores, the enhanced weights are obtained from the harmony matrix and the prediction scores of other categories are adjusted.

[0085] By combining the powerful representation capabilities of the multi-dimensional adapter, the intent harmony optimizer, and the LLM model, multimodal data (i.e., visual and text) are successfully integrated and processed, thereby significantly improving the accuracy of the intent perception task. The intent perception in this embodiment uses a multi-dimensional adapter to process feature vectors of different dimensions, and learns the co-occurrence relationship between intent categories through the harmony optimizer, ultimately generating more accurate and meaningful prediction results that are applicable to a variety of complex application scenarios.

[0086] Example 2

[0087] An intention perception enhancement system provided in this embodiment is used in the intention perception enhancement method mentioned in Embodiment 1, and the system includes:

[0088] A data acquisition module, used for acquiring image data;

[0089] The intention perception module is used to input image data into the constructed intention perception model to obtain the corresponding intention perception results;

[0090] The intent perception model in the intent perception module is a model trained using training data, where the training data includes various types of image data and corresponding feature label annotations;

[0091] The intent perception model in the intent perception module includes an LLM Transformer module and an intent harmony optimizer that are sequentially connected in communication. The LLM Transformer module is used to predict intent based on input image data, and the intent harmony optimizer is used to optimize the prediction results output by the LLM Transformer module based on the probability of co-occurrence between two intent categories. The implementation steps of each module of the system in this embodiment are described in Example 1 and will not be repeated here.

Claims

1. A method for improving intention perception, characterized in that: include: Get image data; Inputting the image data into the constructed intention perception model to obtain the corresponding intention perception result; The intention perception model is a model trained using training data, wherein the training data includes various types of image data and corresponding feature label annotations; The intent perception model includes an LLM Transformer module and an intent harmony optimizer which are communicatively connected in sequence, wherein the LLM Transformer module is used to predict intent based on input image data, and the intent harmony optimizer is used to optimize the prediction result output by the LLM Transformer module based on the probability of co-occurrence between two intent categories.

2. The method for improving intention perception according to claim 1, characterized in that: The intention harmony optimizer includes an intention harmony matrix module, which is: obtaining intent-aware label annotations based on training data, and obtaining the intention harmony matrix module based on the statistical results of co-occurrence frequencies between different intent categories. The intention harmony matrix module is a symmetric matrix, and the elements corresponding to each position are represented as the co-occurrence probability between two categories.

3. The method for improving intention perception according to claim 2, characterized in that: The method also includes obtaining co-occurrence weights through the intention harmony matrix, and adjusting the prediction results of other categories based on the co-occurrence weights and corresponding prediction results.

4. The method for improving intention perception according to claim 1, characterized in that: The LLM Transformer module includes a feature extractor, an encoder, and a decoder connected in sequence. The feature extractor is used to extract a feature vector according to an input image, and input the feature vector into the encoder. The encoder is used to encode the input feature vector and output an output sequence of a hidden state. The decoder is used to decode the sequence output by the encoder.

5. The method for improving intention perception according to claim 4, characterized in that: The feature extractor is a ViT module, which is used to divide the input image into set sizes, perform linear transformation on each image block after segmentation, convert pixel data into feature vectors, and set position codes on the image blocks after segmentation, which represent the relative positions of the image blocks before segmentation.

6. The method for improving intention perception according to claim 4, characterized in that: The LLM Transformer module also includes a multi-dimensional adapter, which is arranged between the feature extractor and the encoder, and includes a number of convolution blocks of different hidden dimensions, and is used for performing visual labeling and visual cueing processing on the input feature vector.

7. The method for improving intention perception according to claim 6, characterized in that: The multi-dimensional adapter is used to project visual markers into different representation spaces at the input stage to adapt to intention perception tasks of different difficulty levels. At the output stage, each embedding vector in the visual marker sequence is converted into a corresponding query vector and a key vector, a unique rotation position code is calculated according to the feature vector corresponding to each marker position, each pair of query vectors and key vectors is rotated and transformed using the rotation position code, and the inner product between each pair of query vectors and key vectors is calculated to obtain the result of the self-attention mechanism.

8. The method for improving intention perception according to claim 4, characterized in that: The encoder includes at least one normalization processing module, which is used to perform normalization processing on the feature vector extracted by the sub-layer of each encoder.

9. The method for improving intention perception according to claim 4, characterized in that: The neural architecture used in the fully connected layer of the LLM Transformer module is a feedforward neural network, and the activation function in the feedforward neural network is the SωiGLU activation function, which is: FFN SωiGLU (a,W,V,W2)=SωiGLU(a,W,V)W2 Swedish β =aσ(aβ) Where a is the feature vector input to the feedforward neural network, FFN SωiGLU is the fully connected layer of the feedforward propagation layer, W is the weight matrix of the first linear transformation, V is the weight matrix of the gating mechanism about the weight matrix of the first linear transformation, W2 is the weight matrix of the second linear transformation, β is the activation parameter, Sωish β is the activation function under parameter β, σ is the Sigmoid activation function and SωiGLU activation function, Represents the digit-by-digit multiplication of the Hadamard product.

10. An intention perception enhancement system, used to implement the intention perception enhancement method according to any one of claims 1 to 9, characterized in that: The system includes: A data acquisition module, used for acquiring image data; An intention perception module, used to input the image data into the constructed intention perception model to obtain a corresponding intention perception result; The intention perception model is a model trained using training data, wherein the training data includes various types of image data and corresponding feature label annotations; The intent perception model includes an LLM Transformer module and an intent harmony optimizer which are communicatively connected in sequence, wherein the LLM Transformer module is used to predict intent based on input image data, and the intent harmony optimizer is used to optimize the prediction result output by the LLM Transformer module based on the probability of co-occurrence between two intent categories.

Images