Information feedback method, model, discriminator training method, electronic equipment and device

Abstract

The invention provides an information feedback method, a model, a discriminator training method, electronic equipment and a device, and the method comprises the steps: processing an input image and text, and obtaining an image feature and a text feature; carrying out feature fusion on the image features and the text features to obtain fusion features; correcting the fusion feature by using the correction feature to obtain a target feature, the correction feature being used for correcting the fusion feature to align the semantics of the image feature and the text feature included in the fusion feature; determining an output result based on the target feature, wherein the output result is used for responding to the input; and outputting an output result.

Description

Technical Field

[0001] The present application relates to the fields of data processing and deep learning, and in particular to an information feedback method, model, discriminator training method, electronic equipment and device. Background Art

[0002] Currently, users of large language models often have not received training on prompt words and are unable to input professional prompt words. Therefore, even if a large language model contains correct information for a specific question, it is difficult to give an accurate response based on the non-professional prompt words input by the user. Summary of the invention

[0003] The present application provides an information feedback method, a model, a discriminator training method, an electronic device and a device.

[0004] An embodiment of the present application provides, on one hand, an information feedback method, the method comprising:

[0005] Process the input image and text to obtain image features and text features;

[0006] Fusing the image feature and the text feature to obtain a fused feature;

[0007] The fused feature is corrected by using a correction feature to obtain a target feature, wherein the correction feature is used to correct the fused feature to align the semantics of the image feature and the text feature contained in the fused feature;

[0008] Determine an output result based on the target feature, wherein the output result is used to respond to the input;

[0009] The output result is outputted.

[0010] Wherein, the method further comprises:

[0011] Obtaining the modified feature comprises:

[0012] Processing a plurality of images in a correction set and texts corresponding to the images to obtain image features of the plurality of images in the correction set and text features of the corresponding texts, wherein the plurality of images in the correction set and the corresponding texts have consistent semantics;

[0013] Performing feature fusion on the image features of the plurality of images in the correction set and the corresponding text features to obtain a first fused feature;

[0014] Using a vector discriminator to identify semantic consistency between image features and text features included in the first fusion feature, to obtain a first consistency feature;

[0015] The revised feature is determined based on the first fused feature and the first consistent feature.

[0016] Wherein, determining the correction feature based on the first fusion feature and the first consistency feature includes:

[0017] Determining a projection vector of the first fused feature on the first consistent feature;

[0018] The standard deviation of the projection vectors corresponding to the multiple images is determined to obtain the correction feature.

[0019] Wherein, the method further comprises:

[0020] Based on the image features of multiple images in the screening set and the text features of the corresponding texts, multiple attention heads in the cross-attention mechanism are screened to determine at least one target attention head, and the at least one target attention head is used to perform the step of feature fusion of the image features and the text features.

[0021] The step of screening multiple attention heads in the cross-attention mechanism based on the image features of multiple images in the screening set and the text features of the corresponding texts includes:

[0022] Inputting the image features of the plurality of images in the screening set and the text features of the corresponding texts into the plurality of attention heads for feature fusion to obtain a plurality of corresponding second fused features;

[0023] Using a vector discriminator to identify semantic consistency of image features and text features included in a plurality of the second fusion features, to obtain a second consistency feature;

[0024] Determining a loss value based on the second consistency feature and the corresponding label;

[0025] The attention head corresponding to the target number of consistent features with the smallest loss value is determined as the target attention head.

[0026] On the other hand, an embodiment of the present application provides an information feedback model, the model comprising: an encoder, a cross attention layer and a pre-trained model layer;

[0027] The encoder processes the input image and text to obtain image features and text features;

[0028] The cross attention layer performs feature fusion on the image feature and the text feature to obtain a fused feature; and uses a correction feature to correct the fused feature to obtain a target feature, wherein the correction feature is used to correct the fused feature to align the semantics of the image feature and the text feature contained in the fused feature;

[0029] The pre-trained model layer determines an output result based on the target feature, where the output result is used to respond to the input; and outputs the output result.

[0030] Wherein, the model further includes: a vector discriminator;

[0031] The encoder processes a plurality of images in the correction set and texts corresponding to the images to obtain image features of the images and text features of the corresponding texts, wherein the images and the corresponding texts have consistent semantics;

[0032] The cross attention layer performs feature fusion on the image features of the multiple images in the correction set and the corresponding text features to obtain a first fused feature;

[0033] The vector discriminator identifies the semantic consistency of the image feature and the text feature included in the first fusion feature to obtain a first consistency feature; and determines the correction feature based on the first fusion feature and the first consistency feature.

[0034] Another aspect of the present application provides a discriminator training method, the method comprising:

[0035] Acquire a training set, wherein the training set includes a plurality of images, texts corresponding to the images, and labels, wherein the labels indicate whether the semantics of the images and the texts corresponding to the images are consistent;

[0036] Processing a plurality of images in the training set and texts corresponding to the images to obtain image features of the images and text features of the corresponding texts;

[0037] Performing feature fusion on the image features of the plurality of images in the training set and the corresponding text features to obtain a plurality of second fused features;

[0038] Using a vector discriminator to identify semantic consistency of image features and text features included in a plurality of the second fusion features, to obtain a plurality of second consistency features;

[0039] Determining a loss value based on a plurality of the second consistency features and corresponding labels;

[0040] Parameters of the vector discriminator are optimized based on the loss value.

[0041] Another aspect of the present application provides an electronic device, comprising: an input-output system and a processor, wherein the input-output system and the processor are electrically connected;

[0042] The processor processes the input image and text to obtain image features and text features; performs feature fusion on the image features and the text features to obtain fused features; uses a correction feature to correct the fused features to obtain a target feature, wherein the correction feature is used to correct the fused features to align the semantics of the image features and the text features contained in the fused features; and determines an output result based on the target feature, wherein the output result is used to respond to the input;

[0043] The input-output system outputs the output result.

[0044] Another aspect of the present application provides an information feedback device, the device comprising:

[0045] A calculation module is used to process the input image and text to obtain image features and text features; perform feature fusion on the image features and the text features to obtain fused features; modify the fused features using modification features to obtain target features, wherein the modification features are used to modify the fused features to align the semantics of the image features and text features contained in the fused features; and determine an output result based on the target features, wherein the output result is used to respond to the input;

[0046] A processing module is used to output the output result.

[0047] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become readily understood. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, wherein:

[0049] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0050] Figure 1 A flow chart showing an information feedback method according to an embodiment of the present application is shown;

[0051] Figure 2 A flow chart showing an information feedback method according to another embodiment of the present application is shown;

[0052] Figure 3 A flow chart showing an information feedback method according to another embodiment of the present application is shown;

[0053] Figure 4A flow chart showing an information feedback method according to another embodiment of the present application is shown;

[0054] Figure 5 A schematic diagram of the structure of an information feedback model according to an embodiment of the present application is shown;

[0055] Figure 6 A schematic diagram showing the structure of an information feedback model according to another embodiment of the present application is shown;

[0056] Figure 7 A flowchart of a discriminator training method according to an embodiment of the present application is shown;

[0057] Figure 8 A schematic diagram of the structure of an electronic device according to an embodiment of the present application is shown;

[0058] Fig. 9 A schematic structural diagram of an information feedback device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0059] In order to make the purpose, features, and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0060] Most users have not received any training on prompt words, so they may input prompt words that are inconsistent with the image description into the large model. This causes the large model to have reasoning hallucinations and output answers that are inconsistent with the image description. For example, an image shows a cat sitting on a sofa, and the user inputs the text "What is the dog doing?" The large model generates the answer "The dog is sitting on the sofa" based on the image and text.

[0061] In order to ensure that the output distribution of the large model is as consistent as possible with the actual activation distribution, improve the accuracy of the large model generation content and prediction, and eliminate the illusion of large model reasoning, an embodiment of the present application provides an information feedback method, such as Figure 1 As shown, the method includes:

[0062] Step 101, processing the input image and text to obtain image features and text features.

[0063] In this embodiment, CLIP (Contrastive Language-Image Pretraining, a multimodal neural network) can be used to extract features from images and texts. Convolutional neural networks (CNN) or VisionTransformer (large visual model) can also be used to extract features from images, and BERT (a bidirectional Transformer encoder) or Word2Vec (a feature extraction method based on word vectors) can be used to extract features from texts. In other embodiments, any other method that can extract features from images and texts can be used.

[0064] Step 102: Fusing the image feature and the text feature to obtain a fused feature.

[0065] In this embodiment, the image features and text features can be fused using a cross attention mechanism, a bidirectional attention mechanism, or a multimodal self-attention mechanism. In other implementations, any other method that can fuse the image features and text features can be used.

[0066] Step 103: Correct the fused feature using a correction feature to obtain a target feature, wherein the correction feature is used to correct the fused feature to align the semantics of the image feature and the text feature contained in the fused feature.

[0067] Using the correction feature to correct the fused feature can make the semantics of the image feature and the text feature aligned, that is, the description of the image feature and the text feature is consistent. Therefore, it can eliminate the situation where the subsequent large model produces reasoning hallucinations when making predictions based on multimodal features (image features and text features).

[0068] Step 104: determining an output result based on the target feature, wherein the output result is used to respond to the input.

[0069] The corrected target features are input into the large model for prediction to determine the output results in response to the user input.

[0070] In this embodiment, Visual Question Answering (VQA, visual language question answering), GPT-4 (Generative Pre-trained Transformer-4), etc. can be selected as the large model. In other implementations, any other large model that can process multimodal features and give corresponding answers can be selected.

[0071] Step 105: output the output result.

[0072] After the large model determines the output results, it outputs them to the user.

[0073] In the above scheme, the fusion features that contain both image features and text features are corrected by the correction vector to align the semantics of the image features and text features in the fusion features. This can eliminate the situation where the subsequent large model generates reasoning hallucinations when making predictions based on multimodal features. It ensures that the output distribution of the large model is as consistent as possible with the actual activation distribution, which improves the accuracy of the content generated and the prediction of the large model.

[0074] In an example of the present application, an information feedback method is also provided, such as Figure 2 As shown, the method also includes:

[0075] Obtaining the modified feature comprises:

[0076] Step 201 , processing multiple images in a correction set and texts corresponding to the images to obtain image features of the multiple images in the correction set and text features of the corresponding texts, wherein the multiple images in the correction set and the corresponding texts have consistent semantics.

[0077] A plurality of images and texts corresponding to the images are collected to form a correction set, and the semantics of each image in the correction set and the text corresponding to the image are consistent.

[0078] In this embodiment, CLIP (Contrastive Language-Image Pretraining, a multimodal neural network) can also be used to extract features in images and texts. Convolutional neural networks (CNN) or VisionTransformer (large visual model) can also be used to extract features in images, and BERT (a bidirectional Transformer encoder) or Word2Vec (a feature extraction method based on word vectors) can be used to extract features in texts. In other embodiments, any other method that can extract features in images and texts can be used.

[0079] Step 202: Fusing the image features of the multiple images in the correction set with the corresponding text features to obtain a first fused feature.

[0080] In this embodiment, the cross attention mechanism, the bidirectional attention mechanism, and the multimodal self-attention mechanism can also be used to fuse the image features and the text features. In other embodiments, any other method that can fuse the image features and the text features can be used.

[0081] Step 203: using a vector discriminator to identify the semantic consistency of the image feature and the text feature included in the first fusion feature, to obtain a first consistency feature.

[0082] In this embodiment, a fully connected neural network, a convolutional neural network, or a recursive neural network can be selected as the vector discriminator. In other implementations, the vector discriminator can select any other model that can perform binary classification on the vector.

[0083] The first consistency feature includes features that can represent the semantic consistency between the image and the corresponding text. Since the semantics of the image and the corresponding text in the correction set are consistent, the features in the first consistency feature obtained here represent that the semantics of the image and the corresponding text are consistent.

[0084] Step 204: determine the correction feature based on the first fusion feature and the first consistency feature.

[0085] A revised feature is determined based on the first fused feature and the first consistent feature.

[0086] In the above scheme, by collecting multiple semantically consistent images and corresponding texts, the image features and text features in the images and corresponding texts are extracted, and then the image features and text features are fused to obtain the first fused feature. Then the first fused feature is processed using a pre-trained vector discriminator to obtain a first consistency feature. Since the trained vector discriminator can accurately identify the semantic consistency of the image features and text features contained in the first fusion feature, the correction feature determined based on the first fusion feature and the first consistency feature can characterize the difference between the first fusion feature and the first consistency feature, so that the correction feature can align the semantics of the image features and text features contained in the fusion feature. This ensures that the output distribution when the large model is generated can be as consistent as possible with the actual activation distribution, thereby improving the accuracy of the content and prediction generated by the large model.

[0087] In an example of the present application, an information feedback method is also provided, such as Figure 3 As shown, the determining the correction feature based on the first fusion feature and the first consistency feature includes:

[0088] Step 301: determine the projection vector of the first fusion feature on the first consistency feature.

[0089] In this embodiment, the dot product of the first fused feature and the first consistency feature is first determined, then the square of the modulus of the first consistency feature is determined, and finally the projection vector of the first fused feature on the first consistency feature is determined by the dot product of the first fused feature and the first consistency feature, the square of the modulus of the first consistency feature and the first consistency feature.

[0090] Specifically, the projection vector proj of the first fusion feature on the first consistency feature can be determined based on the following formula B (A):

[0091]

[0092] Among them, A is the first fusion feature, B is the first consistency feature, A·B is the dot product of the first fusion feature and the first consistency feature, and B·B is the square of the modulus length of the first consistency feature. The calculation method between B is scalar multiplication.

[0093] In other embodiments, any other method capable of determining the projection vector of the first fused feature on the first consistent feature may be used.

[0094] Step 302, determining the standard deviation of the projection vectors corresponding to the multiple images to obtain the correction feature.

[0095] In this embodiment, the mean vector of the multiple projection vectors is first determined, then the Euclidean distance between each projection vector and the mean vector is determined, and finally the standard deviation of the multiple projection vectors is determined based on the Euclidean distance between each projection vector and the mean vector.

[0096] Specifically, the mean vector of these multiple projection vectors can be determined based on the following formula:

[0097]

[0098] Where n is the total number of projection vectors, A i is the i-th projection vector.

[0099] Then determine the Euclidean distance d between the i-th projection vector and the mean vector based on the following formula: i :

[0100]

[0101] In the above scheme, d is the dimension of the projection vector, A ij is the jth component of the i-th projection vector, is the jth component of the mean vector.

[0102] Finally, the standard deviation σ of these multiple projection vectors is determined based on the following formula:

[0103]

[0104] In other implementations, any other method capable of determining the standard deviation of a plurality of projection vectors may also be used.

[0105] In the above scheme, by determining the projection vector of the first fused feature on the first consistent feature, and then determining the standard deviation of these multiple projection vectors as correction features, the universality of the correction features can be improved. It has a good correction effect on the fused features of image features and text features with different semantic consistency, further improving the accuracy of content generation and prediction of large models.

[0106] In an example of the present application, an information feedback method is also provided, the method further comprising:

[0107] Based on the image features of multiple images in the screening set and the text features of the corresponding texts, multiple attention heads in the cross-attention mechanism are screened to determine at least one target attention head, and the at least one target attention head is used to perform the step of feature fusion of the image features and the text features.

[0108] When there are multiple attention heads in the model selected for feature fusion of image features and text features, it is necessary to filter the multiple attention heads based on the image features of multiple images in the filter set and the text features of the corresponding texts. At least one target attention head filtered out from the multiple attention heads is used to perform feature fusion of image features and text features in the inference stage.

[0109] In an example of the present application, an information feedback method is also provided, such as Figure 4 As shown, the method screens multiple attention heads in the cross attention mechanism based on the image features of multiple images in the screening set and the text features of the corresponding texts, including:

[0110] Step 401: input the image features of the plurality of images in the screening set and the text features of the corresponding texts into the plurality of attention heads for feature fusion to obtain a plurality of corresponding second fused features.

[0111] The image features and the corresponding text features are input into multiple attention heads for feature fusion, and each attention head outputs the corresponding second fused features.

[0112] Step 402: using a vector discriminator to identify the semantic consistency of the image features and text features included in the plurality of the second fusion features, to obtain a second consistency feature.

[0113] In this embodiment, a fully connected neural network, a convolutional neural network, or a recursive neural network can also be used as the vector discriminator. In other implementations, the vector discriminator can use any other model that can perform binary classification on the vector.

[0114] Step 403: determine a loss value based on the second consistency feature and the corresponding label.

[0115] The loss value is determined based on the second consistency feature output by the vector discriminator and the label corresponding thereto.

[0116] Step 404, determine the attention heads corresponding to the target number of consistency features with the smallest loss value as the target attention heads.

[0117] The smaller the loss value, the higher the accuracy of characterizing the second consistency feature, and the better the effect of the corresponding attention head.

[0118] In the above scheme, image features and corresponding text features are obtained by extracting features from the images and corresponding texts in the screening set. Multiple attention heads are then used to perform feature fusion to obtain multiple corresponding second fusion features. The second fusion features are then input into the vector discriminator, and the corresponding second consistency features are output. Since different attention heads have different effects on feature fusion, the accuracy of the second consistency features determined based on the second fusion features is also different. Further, the loss value is determined based on the second consistency feature and the corresponding label, and the attention head corresponding to the target number of consistency features with the smallest loss value is determined as the target attention head, thereby screening out a target number of target attention heads with better effects. This further improves the effect of feature fusion, and further improves the accuracy of content generation and prediction of large models.

[0119] An embodiment of the present application provides an information feedback model, such as Figure 5 As shown, the model includes: an encoder 501, a cross-attention layer 502 and a pre-trained model layer 503.

[0120] The encoder 501 processes the input image and text to obtain image features and text features.

[0121] like Figure 5 As shown, the image input by the user and the corresponding text are input to the encoder 501 for feature extraction to obtain image features and corresponding text features.

[0122] The input image is processed by multiple convolutional layers, pooling layers and fully connected layers in the encoder 501 to obtain image features. Specifically, the input image is convolved by multiple convolutional layers to extract local features. Then, the extracted local features are pooled by the pooling layer to reduce the size of the features and retain important information. Finally, the image features are output through the fully connected layer.

[0123] The cross attention layer 502 performs feature fusion on the image feature and the text feature to obtain a fused feature; and uses a correction feature to correct the fused feature to obtain a target feature, wherein the correction feature is used to correct the fused feature to align the semantics of the image feature and the text feature contained in the fused feature.

[0124] like Figure 5 As shown, the cross attention layer 502 uses multiple attention heads to fuse the image features and the corresponding text features to obtain fused features. Then, the fused features are corrected using a predetermined correction feature to obtain a target feature with semantic consistency between the image features and the text features.

[0125] After the cross-attention layer 502 is selected to perform feature fusion on the image features and the corresponding text features, each attention head will output a sub-fusion feature and the attention weight of the sub-fusion feature. Before weighted summing up the multiple sub-fusion features using the attention weights, it is necessary to correct the sub-fusion features using the correction features to obtain multiple corrected sub-fusion features, and then weighted summation of the multiple corrected sub-fusion features is performed based on the corresponding attention weights to obtain the target feature.

[0126] Specifically, the sub-fusion feature can be corrected based on the correction feature according to the following formula to determine the corrected sub-fusion feature x′:

[0127] x′=x+av

[0128] Among them, x is the fusion feature, a is the sub-fusion feature, and v is the correction feature.

[0129] It should be pointed out that after performing weighted summation on multiple sub-fusion features using corresponding attention weights, the weighted summed fusion features can be corrected to obtain the target features.

[0130] The pre-trained model layer 503 determines an output result based on the target feature, where the output result is used to respond to the input; and outputs the output result.

[0131] The obtained target features are input into the pre-trained model layer 503 for prediction, and the output results are obtained and output to the user.

[0132] In the above scheme, accurate image features and text features can be extracted from the image and the corresponding text through the encoder 501. The cross-attention layer 502 is then used to fuse the image features and text features, effectively capturing the relationship between the image features and the text features, thereby improving the effect of the multimodal task. The cross-attention layer 502 also uses the correction features to correct the fused features, so that the semantics of the image features and text features contained in the corrected target features are consistent. This improves the accuracy of the output results obtained after prediction by the pre-trained model layer 503.

[0133] In an example of the present application, an information feedback model is also provided, such as Figure 6 As shown, the model further includes: a vector discriminator 504.

[0134] The encoder 501 processes multiple images in the correction set and texts corresponding to the images to obtain image features of the images and text features of the corresponding texts, and the semantics of the images and the corresponding texts are consistent.

[0135] Similarly, encoder 501 is used to perform feature extraction on multiple images in the correction set and texts corresponding to the images to obtain image features of the images and text features of the corresponding texts.

[0136] The cross attention layer 502 performs feature fusion on the image features of the multiple images in the correction set and the corresponding text features to obtain a first fused feature.

[0137] The cross attention layer 502 is used to perform feature fusion on the image features and the corresponding text features of the image to obtain a first fused feature.

[0138] It should be noted that when the model is used to determine the correction feature, the cross attention layer 502 will not correct the first fused feature based on the correction vector.

[0139] The vector discriminator 504 identifies the semantic consistency of the image feature and the text feature included in the first fused feature to obtain a first consistency feature; and determines the correction feature based on the first fused feature and the first consistency feature.

[0140] The first fused feature is input into the vector discriminator 504, and the corresponding first consistent feature is outputted. Then, a modified feature is determined based on the first fused feature and the first consistent feature.

[0141] In the above scheme, the first fusion feature is processed by the pre-trained vector discriminator 504 to obtain the first consistency feature. Since the trained vector discriminator 504 can accurately identify the semantic consistency of the image features and text features contained in the first fusion feature, the correction feature determined based on the first fusion feature and the first consistency feature can characterize the difference between the first fusion feature and the first consistency feature, so that the correction feature can align the semantics of the image features and text features contained in the fusion feature. This ensures that the output distribution when the large model is generated can be as consistent as possible with the real activation distribution, thereby improving the accuracy of the content and prediction generated by the large model.

[0142] In an example of the present application, a discriminator training method is also provided, such as Figure 7 As shown, the method includes:

[0143] Step 701 : obtaining a training set, wherein the training set includes a plurality of images, texts corresponding to the images, and labels, wherein the labels indicate whether the semantics of the images and the texts corresponding to the images are consistent.

[0144] Collect multiple images and the texts corresponding to the images, and annotate them based on whether the semantics of the images and the texts corresponding to them are consistent to obtain labels corresponding to the images. Multiple images, texts corresponding to the images, and labels form a training set.

[0145] Step 702: Process the multiple images in the training set and the texts corresponding to the images to obtain image features of the images and text features of the corresponding texts.

[0146] In this embodiment, CLIP (Contrastive Language-Image Pretraining, a multimodal neural network) can also be used to extract features in images and texts. Convolutional neural networks (CNN) or VisionTransformer (large visual model) can also be used to extract features in images, and BERT (a bidirectional Transformer encoder) or Word2Vec (a feature extraction method based on word vectors) can be used to extract features in texts. In other embodiments, any other method that can extract features in images and texts can be used.

[0147] Step 703: Fusing the image features of the multiple images in the training set with the corresponding text features to obtain multiple second fused features.

[0148] In this embodiment, the cross attention mechanism, the bidirectional attention mechanism, and the multimodal self-attention mechanism can also be used to fuse the image features and the text features. In other embodiments, any other method that can fuse the image features and the text features can be used.

[0149] Step 704: using a vector discriminator to identify the semantic consistency of the image features and text features included in the plurality of second fusion features, to obtain a plurality of second consistency features.

[0150] In this embodiment, a fully connected neural network, a convolutional neural network, or a recursive neural network can also be used as the vector discriminator. In other implementations, the vector discriminator can use any other model that can perform binary classification on the vector.

[0151] Step 705: determine a loss value based on a plurality of the second consistency features and corresponding labels.

[0152] The loss value is determined based on the second consistency feature output by the vector discriminator and the label corresponding thereto. The loss function may be a binary cross entropy loss.

[0153] Step 706: Optimize the parameters of the vector discriminator based on the loss value.

[0154] The parameters of the vector discriminator are optimized based on the loss value.

[0155] In the above scheme, the vector discriminator is trained using the training set, so that the trained vector discriminator can accurately identify the semantic consistency of the image features and text features contained in the fusion features. Therefore, the correction features subsequently determined based on the first fusion features and the first consistency features can characterize the difference between the first fusion features and the first consistency features, so that the correction features can align the semantics of the image features and text features contained in the fusion features. This ensures that the output distribution when the large model is generated can be as consistent as possible with the real activation distribution, improving the accuracy of the content and prediction generated by the large model.

[0156] An example of the present application provides an electronic device, Figure 8A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0157] like Figure 8 As shown, the electronic device 800 includes a processor 801 and an input-output system 802, and the processor 801 and the input-output system 802 are electrically connected;

[0158] The processor 801 processes the input image and text to obtain image features and text features; performs feature fusion on the image features and the text features to obtain fused features; uses a correction feature to correct the fused features to obtain a target feature, wherein the correction feature is used to correct the fused features to align the semantics of the image features and the text features contained in the fused features; and determines an output result based on the target feature, wherein the output result is used to respond to the input;

[0159] The input-output system 802 outputs the output result.

[0160] Among them, the processor 801 processes multiple images in the correction set and texts corresponding to the images to obtain image features of the multiple images in the correction set and text features of the corresponding texts, and the multiple images in the correction set and the corresponding texts are semantically consistent; the image features of the multiple images in the correction set and the corresponding text features are feature-fused to obtain a first fused feature; a vector discriminator is used to identify the semantic consistency of the image features and text features contained in the first fused feature to obtain a first consistency feature; and the correction feature is determined based on the first fused feature and the first consistency feature.

[0161] The processor 801 determines a projection vector of the first fusion feature on the first consistency feature; and determines a standard deviation of the projection vectors corresponding to the multiple images to obtain the correction feature.

[0162] Among them, the processor 801 screens multiple attention heads in the cross-attention mechanism based on the image features of multiple images in the screening set and the text features of the corresponding texts, and determines at least one target attention head, and the at least one target attention head is used to execute the step of feature fusion of the image features and the text features.

[0163] Among them, the processor 801 inputs the image features of multiple images in the screening set and the text features of the corresponding texts into the multiple attention heads for feature fusion to obtain the corresponding multiple second fusion features; uses a vector discriminator to identify the semantic consistency of the image features and text features contained in the multiple second fusion features to obtain a second consistency feature; determines the loss value based on the second consistency feature and the corresponding label; and determines the attention head corresponding to the target number of consistency features with the smallest loss value as the target attention head.

[0164] In another embodiment of the present application, the input-output system 802 obtains a training set, wherein the training set includes a plurality of images, texts corresponding to the images, and labels, wherein the labels indicate whether the semantics of the images and the texts corresponding to the images are consistent;

[0165] The processor 801 processes multiple images in the training set and texts corresponding to the images to obtain image features of the images and text features of the corresponding texts; performs feature fusion on the image features of the multiple images in the training set and the corresponding text features to obtain multiple second fused features; uses a vector discriminator to identify the semantic consistency of the image features and text features contained in the multiple second fused features to obtain multiple second consistency features; determines a loss value based on the multiple second consistency features and the corresponding labels; and optimizes the parameters of the vector discriminator based on the loss value.

[0166] This application provides an example of an information feedback device, such as Fig. 9 As shown, the device comprises:

[0167] The computing module 10 is used to process the input image and text to obtain image features and text features; perform feature fusion on the image features and the text features to obtain fused features; modify the fused features using modification features to obtain target features, wherein the modification features are used to modify the fused features to align the semantics of the image features and text features contained in the fused features; and determine an output result based on the target features, wherein the output result is used to respond to the input;

[0168] The processing module 20 is used to output the output result.

[0169] Among them, the calculation module 10 is also used to process multiple images in the correction set and texts corresponding to the images to obtain image features of the multiple images in the correction set and text features of the corresponding texts, and the semantics of the multiple images in the correction set and the corresponding texts are consistent; perform feature fusion on the image features of the multiple images in the correction set and the corresponding text features to obtain a first fused feature; use a vector discriminator to identify the semantic consistency of the image features and text features contained in the first fused feature to obtain a first consistency feature; and determine the correction feature based on the first fused feature and the first consistency feature.

[0170] The calculation module 10 is further used to determine the projection vector of the first fusion feature on the first consistency feature; and determine the standard deviation of the projection vectors corresponding to the multiple images to obtain the modified feature.

[0171] Among them, the computing module 10 is also used to screen multiple attention heads in the cross-attention mechanism based on the image features of multiple images in the screening set and the text features of the corresponding texts, and determine at least one target attention head, and the at least one target attention head is used to execute the step of feature fusion of the image features and the text features.

[0172] Among them, the calculation module 10 is also used to input the image features of multiple images in the screening set and the text features of the corresponding texts into the multiple attention heads for feature fusion to obtain the corresponding multiple second fusion features; use a vector discriminator to identify the semantic consistency of the image features and text features contained in the multiple second fusion features to obtain a second consistency feature; determine the loss value based on the second consistency feature and the corresponding label; and determine the attention head corresponding to the target number of consistency features with the smallest loss value as the target attention head.

[0173] In another embodiment of the present application, the processing module 20 is further used to obtain a training set, wherein the training set includes a plurality of images, texts corresponding to the images, and labels, wherein the labels indicate whether the semantics of the images and the texts corresponding to the images are consistent;

[0174] The computing module 10 is also used to process multiple images in the training set and texts corresponding to the images to obtain image features of the images and text features of the corresponding texts; perform feature fusion on the image features of the multiple images in the training set and the corresponding text features to obtain multiple second fused features; use a vector discriminator to identify the semantic consistency of the image features and text features contained in the multiple second fused features to obtain multiple second consistency features; determine a loss value based on the multiple second consistency features and the corresponding labels; and optimize the parameters of the vector discriminator based on the loss value.

[0175] An example of the present application provides a computer-readable storage medium, which stores a computer program, wherein the computer program is used to process an input image and text to obtain image features and text features; perform feature fusion on the image features and the text features to obtain fused features; use a correction feature to correct the fused features to obtain a target feature, wherein the correction feature is used to correct the fused features to align the semantics of the image features and text features contained in the fused features; and determine an output result based on the target feature, wherein the output result is used to respond to the input; and output the output result.

[0176] Among them, the computer program is also used to process multiple images in the correction set and texts corresponding to the images to obtain image features of the multiple images in the correction set and text features of the corresponding texts, and the semantics of the multiple images in the correction set and the corresponding texts are consistent; perform feature fusion on the image features of the multiple images in the correction set and the corresponding text features to obtain a first fused feature; use a vector discriminator to identify the semantic consistency of the image features and text features contained in the first fused feature to obtain a first consistency feature; and determine the correction feature based on the first fused feature and the first consistency feature.

[0177] The computer program is further used to determine a projection vector of the first fusion feature on the first consistency feature; and to determine a standard deviation of the projection vectors corresponding to the multiple images to obtain the correction feature.

[0178] The computer program is also used to screen multiple attention heads in the cross-attention mechanism based on image features of multiple images in the screening set and text features of corresponding texts, and determine at least one target attention head, and the at least one target attention head is used to execute the step of feature fusion of the image features and the text features.

[0179] Among them, the computer program is also used to input the image features of multiple images in the screening set and the text features of the corresponding texts into the multiple attention heads for feature fusion to obtain the corresponding multiple second fusion features; use a vector discriminator to identify the semantic consistency of the image features and text features contained in the multiple second fusion features to obtain a second consistency feature; determine the loss value based on the second consistency feature and the corresponding label; and determine the attention head corresponding to the target number of consistency features with the smallest loss value as the target attention head.

[0180] In another embodiment of the present application, the computer program is used to obtain a training set, which includes multiple images, texts corresponding to the images, and labels, wherein the labels indicate whether the semantics of the images and the texts corresponding to the images are consistent; the multiple images in the training set and the texts corresponding to the images are processed to obtain image features of the images and text features of the corresponding texts; the image features and the corresponding text features of the multiple images in the training set are feature-fused to obtain multiple second fused features; a vector discriminator is used to identify the semantic consistency of the image features and the text features contained in the multiple second fused features to obtain multiple second consistency features; a loss value is determined based on the multiple second consistency features and the corresponding labels; and the parameters of the vector discriminator are optimized based on the loss value.

[0181] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), integrated systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0182] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0183] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0184] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0185] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0186] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0187] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0188] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0189] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. An information feedback method, the method comprising: Process the input image and text to obtain image features and text features; Fusing the image feature and the text feature to obtain a fused feature; The fused feature is corrected by using a correction feature to obtain a target feature, wherein the correction feature is used to correct the fused feature to align the semantics of the image feature and the text feature contained in the fused feature; Determine an output result based on the target feature, wherein the output result is used to respond to the input; The output result is output.

2. The method according to claim 1, further comprising: Obtaining the correction feature comprises: Processing a plurality of images in a correction set and texts corresponding to the images to obtain image features of the plurality of images in the correction set and text features of the corresponding texts, wherein the plurality of images in the correction set and the corresponding texts have consistent semantics; Performing feature fusion on the image features of the plurality of images in the correction set and the corresponding text features to obtain a first fused feature; Using a vector discriminator to identify semantic consistency between image features and text features included in the first fusion feature, to obtain a first consistency feature; The revised feature is determined based on the first fused feature and the first consistent feature.

3. The method according to claim 2, wherein determining the correction feature based on the first fusion feature and the first consistency feature comprises: Determining a projection vector of the first fused feature on the first consistent feature; The standard deviation of the projection vectors corresponding to the multiple images is determined to obtain the correction feature.

4. The method according to claim 3, further comprising: Based on the image features of multiple images in the screening set and the text features of the corresponding texts, multiple attention heads in the cross-attention mechanism are screened to determine at least one target attention head, and the at least one target attention head is used to perform the step of feature fusion of the image features and the text features.

5. The method according to claim 4, wherein the step of screening multiple attention heads in the cross-attention mechanism based on the image features of the multiple images in the screening set and the text features of the corresponding texts comprises: Inputting the image features of the plurality of images in the screening set and the text features of the corresponding texts into the plurality of attention heads for feature fusion to obtain a plurality of corresponding second fused features; Using a vector discriminator to identify semantic consistency of image features and text features included in a plurality of the second fusion features, to obtain a second consistency feature; Determining a loss value based on the second consistency feature and the corresponding label; The attention head corresponding to the target number of consistent features with the smallest loss value is determined as the target attention head.

6. An information feedback model, the model comprising: Encoder, criss-cross attention layer, and pre-trained model layer; The encoder processes the input image and text to obtain image features and text features; The cross attention layer performs feature fusion on the image feature and the text feature to obtain a fused feature; and using a correction feature to correct the fused feature to obtain a target feature, wherein the correction feature is used to correct the fused feature to align the semantics of the image feature and the text feature contained in the fused feature; The pre-trained model layer determines an output result based on the target feature, where the output result is used to respond to the input; and outputs the output result.

7. The model according to claim 6, further comprising: Vector Discriminator; The encoder processes a plurality of images in the correction set and texts corresponding to the images to obtain image features of the images and text features of the corresponding texts, wherein the images and the corresponding texts have consistent semantics; The cross attention layer performs feature fusion on the image features of the multiple images in the correction set and the corresponding text features to obtain a first fused feature; The vector discriminator identifies the semantic consistency of the image feature and the text feature included in the first fusion feature to obtain a first consistency feature; And determining the correction feature based on the first fusion feature and the first consistency feature.

8. A discriminator training method, the method comprising: Acquire a training set, wherein the training set includes a plurality of images, texts corresponding to the images, and labels, wherein the labels indicate whether the semantics of the images and the texts corresponding to the images are consistent; Processing a plurality of images in the training set and texts corresponding to the images to obtain image features of the images and text features of the corresponding texts; Performing feature fusion on the image features of the plurality of images in the training set and the corresponding text features to obtain a plurality of second fused features; Using a vector discriminator to identify semantic consistency of image features and text features included in a plurality of the second fusion features, to obtain a plurality of second consistency features; Determining a loss value based on a plurality of the second consistency features and corresponding labels; Parameters of the vector discriminator are optimized based on the loss value.

9. An electronic device, comprising: An input-output system and a processor, wherein the input-output system and the processor are electrically connected; The processor processes the input image and text to obtain image features and text features; Fusing the image feature and the text feature to obtain a fused feature; The fused feature is corrected by using a correction feature to obtain a target feature, wherein the correction feature is used to correct the fused feature to align the semantics of the image feature and the text feature contained in the fused feature; and determining an output result based on the target feature, the output result being used in response to the input; The input-output system outputs the output result.

10. An information feedback device, comprising: A computing module is used to process the input image and text to obtain image features and text features; Fusing the image feature and the text feature to obtain a fused feature; The fused feature is corrected by using a correction feature to obtain a target feature, wherein the correction feature is used to correct the fused feature to align the semantics of the image feature and the text feature contained in the fused feature; and determining an output result based on the target feature, the output result being used in response to the input; A processing module is used to output the output result.

Images