Model training method, prediction method, device, storage medium, and program product

Abstract

The application provides a model training method, a prediction method, equipment, a storage medium and a program product. The method comprises the following steps: determining visual representation features corresponding to an image by a visual coding module according to the image and text to be processed, and determining language representation features corresponding to the text by a language coding module; determining attention values corresponding to each image block in the image and / or each character in the text according to the visual representation features and the language representation features, and determining an attention loss according to the attention values, wherein the attention value of the image block is used to represent the contribution of the image block to text prediction, and the attention value of the character is used to represent the contribution of the character to image prediction; determining a prediction result corresponding to the image and / or the text by a fusion module according to the visual representation features and the language representation features, and determining a prediction loss according to the prediction result; and adjusting parameters of a model according to the attention loss and the prediction loss, so as to improve the accuracy of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a model training method, a prediction method, a device, a storage medium and a program product. BACKGROUND

[0002] With the continuous development of artificial intelligence technology, the data modalities that can be processed by artificial intelligence models are also expanding. In some technologies, a multi-modal interaction model can be used to process information of multiple modalities, thereby improving the prediction effect of the model.

[0003] For example, for a piece of merchandise, there can be multiple modalities of data such as images and text. By using a multi-modal interaction model to comprehensively process these data, interaction between modalities such as images and text can be achieved, which helps to improve the prediction effect of the merchandise. However, the accuracy of the current multi-modal interaction model still needs to be improved. SUMMARY

[0004] The main purpose of the embodiments of the present application is to provide a model training method, a prediction method, a device, a storage medium and a program product to improve the accuracy of a multi-modal interaction model.

[0005] In a first aspect, the embodiments of the present application provide a model training method, the model comprising a visual encoding module, a language encoding module and a fusion module; the method comprising:

[0006] According to the image and the text to be processed, the visual representation features corresponding to the image are determined by the visual encoding module, and the language representation features corresponding to the text are determined by the language encoding module;

[0007] According to the visual representation features and the language representation features, the attention values corresponding to each image block in the image and / or each character in the text are determined, and an attention loss is determined according to the attention values; wherein the attention value of an image block is used to represent the contribution of the image block to the text prediction, and the attention value of a character is used to represent the contribution of the character to the image prediction;

[0008] According to the visual representation features and the language representation features, a prediction result corresponding to the image and / or the text is determined by the fusion module, and a prediction loss is determined according to the prediction result;

[0009] According to the attention loss and the prediction loss, the parameters of the model are adjusted.

[0010] Optionally, according to the visual representation features and the language representation features, the attention values corresponding to each image block in the image and / or each character in the text are determined, comprising:

[0011] According to the visual representation features of the plurality of image blocks in the image and the language representation features of the plurality of characters in the text, a cross-attention matrix is calculated, and an element in the cross-attention matrix is used to represent the contribution of an image block to a character and / or the contribution of a character to an image block.

[0012] For any image block, the contributions of the image block to each character are added to obtain an attention value corresponding to the image block; and / or, for any character, the contributions of the character to each image block are added to obtain an attention value corresponding to the character.

[0013] Optionally, the attention loss is determined according to the attention values, comprising:

[0014] The attention loss is determined according to the attention values of each image block and corresponding labels, and / or the attention values of each character and corresponding labels.

[0015] The labels used to determine the attention loss match the labels used to determine the prediction loss.

[0016] Optionally, the attention loss is determined according to the attention values of each image block and corresponding labels, and / or the attention values of each character and corresponding labels, comprising:

[0017] A first cross-entropy loss is calculated according to the attention values of each image block and corresponding labels.

[0018] A second cross-entropy loss is calculated according to the attention values of each character and corresponding labels.

[0019] The corresponding attention loss is determined according to the first cross-entropy loss and the second cross-entropy loss.

[0020] Optionally, the model further comprises a visual prediction module and / or a language prediction module; according to the visual representation features and the language representation features, a prediction result corresponding to the image and / or the text is determined through a fusion module, comprising:

[0021] The visual representation features and the language representation features are input into the fusion module to obtain a multi-modal representation feature.

[0022] According to the multi-modal representation feature, a prediction result of each image block is obtained through a visual prediction module, and / or a prediction result of each character is obtained through a language prediction module.

[0023] Optionally, the parameters of the model are adjusted according to the attention loss and the prediction loss, comprising:

[0024] According to the attention loss, the parameters of the visual encoding module and the language encoding module are adjusted.

[0025] According to the prediction loss, parameters of each module in the model are adjusted.

[0026] In a second aspect, the embodiments of the present application provide a model training method, the model comprising a visual encoding module, a language encoding module and a fusion module; the method comprising:

[0027] According to the corresponding product image and product title of the product, the visual representation feature corresponding to the product image is determined through the visual encoding module, and the language representation feature corresponding to the product title is determined through the language encoding module;

[0028] According to the visual representation feature and the language representation feature, the attention value corresponding to each image block in the product image and / or each character in the product title is determined, and the attention loss is determined according to the attention value; wherein the attention value of the image block is used to represent the contribution of the image block to the text prediction, and the attention value of the character is used to represent the contribution of the character to the image prediction;

[0029] According to the visual representation feature and the language representation feature, the prediction result corresponding to the product image and / or the product title is determined through the fusion module, and the prediction loss is determined according to the prediction result; wherein the prediction result corresponding to the product image is used to locate the product subject in the product image, and the prediction result corresponding to the product title is used to locate the center word of the product title;

[0030] According to the attention loss and the prediction loss, the parameters of the model are adjusted.

[0031] In a third aspect, the embodiments of the present application provide a model training method, the model comprising a first modality encoding module, a second modality encoding module and a fusion module; the method comprising:

[0032] The first modality representation feature corresponding to the first modality information is determined through the first modality encoding module, and the second modality representation feature corresponding to the second modality information is determined through the second modality encoding module;

[0033] According to the first modality representation feature and the second modality representation feature, the attention value corresponding to each first sub-modality information in the first modality information and / or each second sub-modality information in the second modality information is determined, and the attention loss is determined according to the attention value; wherein the attention value of the first sub-modality information is used to represent the contribution of the first sub-modality information to the prediction of the second modality information, and the attention value of the second sub-modality information is used to represent the contribution of the second sub-modality information to the prediction of the first modality information;

[0034] According to the first modality representation feature and the second modality representation feature, a prediction result of the first modality information and / or the second modality information and a corresponding prediction loss are determined by a fusion module.

[0035] According to the attention loss and the prediction loss, parameters of the model are adjusted.

[0036] In a fourth aspect, an embodiment of the present application provides a prediction method, comprising:

[0037] Obtaining first modality information and second modality information to be processed; wherein the first modality information and the second modality information comprise any two of the following information: image, text, audio, video, and sensing information.

[0038] According to the first modality information and the second modality information to be processed, a prediction result corresponding to the first modality information and / or the second modality information is obtained by a multi-modal interaction model.

[0039] The multi-modal interaction model is trained based on the model training method in any of the above aspects.

[0040] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising:

[0041] at least one processor; and

[0042] a memory in communication connection with the at least one processor;

[0043] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the electronic device to perform the method in any of the above aspects.

[0044] In a sixth aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer execution instructions, and when a processor executes the computer execution instructions, the method in any of the above aspects is implemented.

[0045] In a seventh aspect, an embodiment of the present application provides a computer program product, comprising a computer program, which is executed by a processor to implement the method in any of the above aspects.

[0046] The model training method, the prediction method, the device, the storage medium and the program product provided by the embodiments of the present application can determine the visual representation feature corresponding to the image through a visual coding module according to the image and the text to be processed, and determine the language representation feature corresponding to the text through a language coding module, determine the attention value corresponding to each image block in the image and / or each character in the text according to the visual representation feature and the language representation feature, and determine the attention loss according to the attention value, wherein the attention value of the image block is used to represent the contribution of the image block to the text prediction, and the attention value of the character is used to represent the contribution of the character to the image prediction, the prediction result corresponding to the image and / or the text is determined through a fusion module according to the visual representation feature and the language representation feature, the prediction loss is determined according to the prediction result, the parameters of the model are adjusted according to the attention loss and the prediction loss, so that the model can be trained through the loss corresponding to the prediction result and the intermediate attention loss, not only the final prediction result can be supervised, but also the intermediate representation feature can be supervised, so that the extracted representation feature can more accurately reflect the importance of the image block or the character, thereby better performing fusion in the multi-modal fusion module, giving more attention to important image blocks or characters, improving the accuracy of the final prediction result, and improving the model effect. BRIEF DESCRIPTION OF DRAWINGS

[0047] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application.

[0048] Figure 1 An application scenario diagram provided by an embodiment of the present application;

[0049] Figure 2 A training principle diagram provided by an embodiment of the present application;

[0050] Figure 3 A flow diagram of a model training method provided by an embodiment of the present application;

[0051] Figure 4 A principle diagram of calculating the attention value of an image block and a character provided by an embodiment of the present application;

[0052] Figure 5 A model training process diagram provided by an embodiment of the present application;

[0053] Figure 6 A flow diagram of another model training method provided by an embodiment of the present application;

[0054] Figure 7 An existing graph search diagram;

[0055] Figure 8 A flowchart of a method for providing a map search according to an embodiment of the present application;

[0056] Figure 9 A flowchart of a method for training a model according to an embodiment of the present application;

[0057] Figure 10 A flowchart of a method for providing a prediction according to an embodiment of the present application;

[0058] Figure 11 A structural diagram of an electronic device according to an embodiment of the present application.

[0059] The above-described drawings show specific embodiments of the present application, which will be described in more detail below. These drawings and written descriptions are not intended to limit the scope of the present application concept in any way, but to illustrate the present application concept to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0060] The exemplary embodiments will be described in detail herein with reference to the accompanying drawings. The same reference numbers in different drawings denote the same or similar elements throughout the following description with reference to the drawings.

[0061] The term "and / or" is used herein to describe the association relationship of associated objects, and specifically represents that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone.

[0062] First, the terms related to the present application are explained:

[0063] Attention mechanism: a data processing method in machine learning, derived from human vision which can focus on key areas in the global image, and invest more attention in key areas, so as to quickly filter out high-value information from a large amount of information, reduce attention to other information, and even filter out irrelevant information, improve the efficiency and accuracy of task processing. A conventional attention mechanism includes query (query condition), key (key), and value (value), which scores value through the relationship between query and key, and finally takes out the most relevant value through soft (soft) or hard (hard) way.

[0064] Product picture subject selection: multiple objects are detected in a product picture, only one of which is a product for sale, and subject selection is to select the subject frame of the product for sale from multiple objects in the product picture.

[0065] Central word recognition of commodity title: the most core word in the commodity title is recognized, such as the title "2022 spring and summer women's clothing port style retro shirt top", the central word is "shirt".

[0066] The embodiments of the present application can be applied to any field that needs to perform cross-modal interaction. For image-text interaction, the embodiments of the present application provide a multi-modal fusion scheme of image and text, and the obtained multi-modal features can be used to process the text and image respectively. Exemplarily, it can be used as an important module of commodity understanding in the e-commerce scene, and is used to improve the precision of downstream image search and text search.

[0067] In the e-commerce scene, the understanding and cognition of commodities are the basic capabilities of e-commerce platforms. For commodities, the corresponding data is usually composed of titles and pictures, and the title and the picture can assist each other in prediction. In actual application, merchants usually pile up a large number of redundant words in the commodity title, and the main body of the commodity is usually placed in the core part in the commodity picture, and the main body of the commodity can determine the central word of the title. At the same time, when multiple items are included in the commodity picture, the on-sale commodity can be determined through the commodity title. Therefore, through the interaction of the picture and the text, the understanding ability of the commodity itself can be effectively improved.

[0068] At present, for text recognition and image recognition, two different schemes are usually used to construct models for two tasks respectively. For example, when a text recognition model needs to be trained, the text and the image can be used as input, and the central word can be used as the prediction result for training; when an image recognition model needs to be trained, the text and the image can be used as input, and the main body box can be used as the prediction result for training. Although this training scheme uses the features of image-text interaction, only a single task is used as a supervision signal, and the training result may not be ideal.

[0069] To solve this problem, a multi-task learning scheme can be used, the input of the model is the picture and the title of the commodity, and the output is the main body box of the commodity image and the central word of the title. The multi-task learning method is used to jointly train the main body detection and the title recognition, and the two tasks share the same multi-modal encoder, and the classification loss (Loss) of the title and the regression Loss of the main body selection are constructed at the same time.

[0070] Figure 1 An application scenario diagram is provided for the embodiments of the present application. As shown in Figure 1As shown, after extracting the embedding features of each image patch in the image and each character in the text through the embedding feature extraction module, the image patch embedding and the character embedding can be input into the visual-language encoder. In addition to the embedding features, some special tokens, such as REG (regression), CLS (classification), and SEP (separation), can be added during input to adapt the input information to the upper-level task.

[0071] Visual-language encoders can achieve cross-modal fusion to obtain multimodal features after visual and text interaction. These multimodal features can be used to achieve higher-level image prediction and text prediction tasks.

[0072] In image prediction tasks, REG can be used to represent the multimodal features after visual and textual interaction. These multimodal features are input into the coordinates regression module to obtain four data points: x, y, w, and h, which represent the horizontal and vertical coordinates and the width and height, respectively. These four data points can be used to determine the main bounding box of the product.

[0073] In text prediction tasks, multimodal features resulting from visual and textual interactions can be input into the text prediction module to obtain the prediction result for each character. The prediction result includes three types: O, B, and I, representing non-central words, the starting character of the central word, and non-starting characters of the central word, respectively. If a character belongs to B or I, it means that the character belongs to the central word. For example, in "patchwork work shorts dress for women", "shorts" is the central word.

[0074] In the multi-task learning scheme described above, the training process of the model can be supervised simultaneously through the central words and subject boxes of the product, which effectively improves the performance of the model. However, there is still room for improvement in the accuracy of the model.

[0075] In view of this, embodiments of this application propose a multimodal attention loss function that can effectively utilize head words and subject bounding boxes to guide cross-modal interaction. Since product text and images contain a great deal of redundant information, direct text-image fusion is difficult. However, text head words and image subject bounding boxes provide excellent summarization capabilities for multimodal interactions. Therefore, by using head words and subject bounding boxes as supervisory signals, cross-modal interaction between images and text can be guided, thereby enabling more effective fusion of key information.

[0076] Figure 2 This is a schematic diagram illustrating a training principle provided in an embodiment of this application. Figure 2As shown, the input of the model can be the embedding features of the image blocks and the embedding features of the characters in the text, in the modal representation layer, the visual representation features of the image blocks and the language representation features of the characters can be extracted through the visual encoder and the language encoder respectively, in the multi-modal fusion layer, the visual representation features and the language representation features can be fused, and the fused features can be input into the upper application layer to obtain the prediction result.

[0077] After the modal representation layer, an attention loss function can be designed to constrain the representation features output by the modal representation layer. The attention loss function can be determined by the attention values of the image blocks and the characters. In the multi-modal fusion layer and the upper application layer, the visual representation features of the image blocks will play a role in the text prediction process, and the language representation features of the characters will play a role in the image prediction process. When constructing the attention loss function, the attention value of each image block can be used to measure the contribution of the image block in the text prediction, and the attention value of each character can be used to measure the contribution of the character in the image prediction.

[0078] If a character in the text belongs to a central word or a block region in the image belongs to a main frame, a higher weight can be given during multi-modal fusion, and vice versa. Through the design of the multi-modal attention loss function, the fusion process of the picture information and the text information is designed as a training target, and parameter updating and learning are performed based on the supervised signal.

[0079] In the upper application layer, after obtaining the prediction result, the loss function corresponding to the prediction result can be determined according to the prediction result and the true label. Between the modal representation layer and the multi-modal fusion layer, the output of the modal representation layer can be constrained through the multi-modal loss function. The attention loss function can align the outputs of the visual encoding module and the language encoding module to a space, and each other knows which part is important, so as to reduce the burden of the fusion module. Through the loss function corresponding to the prediction result and the intermediate multi-modal loss function, the model is trained, which not only supervises the final prediction result, but also supervises the intermediate single-modal representation features, so that the extracted single-modal representation features can more accurately reflect the importance of the image blocks or characters, thereby better performing fusion in the multi-modal fusion layer, giving more attention to important image blocks or characters, improving the accuracy of the final prediction result, and improving the model effect.

[0080] In addition to the above e-commerce scenarios, the scheme of the embodiments of the present application can also be applied to any scene that can perform image-text interaction. Among them, the image and the text in the image-text interaction process can be matched image and text.

[0081] Optionally, in a social platform scenario, for each user, the image and text of any content published by the user on the social platform can be taken as the matched image and text. Through the scheme provided in the embodiments of the present application, the key positions in the image are extracted as the prediction result of the image, and the key words in the text are extracted as the prediction result of the text, so that the corresponding label of the content can be determined according to the key positions of the image and the key words of the text. For a target user browsing the social platform, the content published by other users can be recommended to the target user according to the portrait or real-time preference of the target user.

[0082] In a sports platform scenario, after a coach or a user publishes a sports video, a cover image and recommended text can be set for the video. On this basis, the cover image and the recommended text can be taken as the matched image and text according to the scheme provided in the embodiments of the present application, and the training positions in the cover image and the training parts in the text are extracted. For example, the main training part is the waist, so "waist" can be highlighted in the text or taken as a key word, and the waist can be highlighted in the cover image, so that other users can more intuitively determine the main training part of the sports video.

[0083] Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The embodiments described below and the features in the embodiments can be combined with each other without conflict. In addition, the sequence of steps in each method embodiment described below is only an example and is not strictly limited.

[0084] Figure 3 A flowchart of a model training method provided in an embodiment of the present application is shown. The execution subject of the method in the embodiment can be applied to any device with data processing function, such as a terminal device or a server. The method can be used to train a model, especially a multi-modal interaction model, which can include a visual encoding module, a language encoding module and a fusion module. As shown in Figure 3 The method can include:

[0085] Step 301: According to the image and text to be processed, the visual representation features corresponding to the image are determined by the visual encoding module, and the language representation features corresponding to the text are determined by the language encoding module.

[0086] The image and text to be processed can refer to the matched image and text, for example, the image and text corresponding to the same commodity, or the image and text in a piece of content published by a user on a social platform.

[0087] The image and text can be extracted by the visual encoding module and the language encoding module to obtain the corresponding visual representation features and language representation features.

[0088] In an example, the image can be directly input into the visual encoding module to obtain visual representation features, and the text can be input into the language encoding module to obtain language representation features.

[0089] In another example, a corresponding embedding feature extraction module can be arranged before the visual encoding module and the language encoding module. The image can be input into the visual embedding feature extraction module to obtain corresponding embedding features of the image, and then the embedding features of the image are input into the visual encoding module to obtain corresponding visual representation features. The text can be input into the language embedding feature extraction module to obtain corresponding embedding features of the text, and then the embedding features of the text are input into the language encoding module to obtain corresponding language representation features.

[0090] Optionally, the image and the text can be divided into multiple parts, for example, the image can be divided into multiple image blocks, and the text can be divided into multiple characters. For each image block or each character, corresponding visual representation features or language representation features can be obtained through the visual encoding module or the language encoding module.

[0091] For example, the image can be divided into 10*10 image blocks in the height and width directions, and the 10*10 image blocks can be input into the visual encoding module to obtain corresponding 10*10 visual representation features. If the text contains 20 characters, the 20 characters can be input into the language encoding module to obtain corresponding 20 language representation features.

[0092] Optionally, the visual encoding module and the language encoding module can be implemented through any encoder, for example, a Transformer, a Resnet, or other CNN (Convolutional Neural Networks) models.

[0093] In step 302, according to the visual representation features and the language representation features, attention values corresponding to each image block in the image and / or each character in the text are determined, and an attention loss is determined according to the attention values.

[0094] The attention value of the image block is used to represent the contribution of the image block to the text prediction, and the attention value of the character is used to represent the contribution of the character to the image prediction.

[0095] In the text prediction, the image plays a certain role, and in the image prediction, the text also plays a certain role. Therefore, the embodiment introduces the attention value to measure the contribution of each image block and each character, which can refer to the importance of the image block or the character in the subsequent prediction process.

[0096] Specifically, the contribution of the image block can be used to represent the importance of the image block in the text prediction process. If a certain image block belongs to a positive result (for example, belongs to the subject frame; usually labeled as 1) in the image prediction result, it means that it belongs to an important area, and its importance in the text prediction process is higher than that of other non-positive result image blocks. Therefore, more attention should be paid to it, and therefore its corresponding attention value can be higher.

[0097] Similarly, the contribution of the character can be used to represent the importance of the character in the image prediction process. If a certain character belongs to a positive result (for example, belongs to the center word; usually labeled as 1) in the text prediction result, it means that it belongs to an important character, and its importance in the image prediction process is higher than that of other non-positive result character blocks. Therefore, more attention should be paid to it, and therefore its corresponding attention value can be higher.

[0098] In the model training process, the attention values of the image block and the character can be determined by the visual representation features and the language representation features of the image block and the character, so as to constrain the representation features output by the encoding module, so that the important image block or character corresponds to the representation feature with higher importance in the representation features input to the fusion module.

[0099] After determining the contribution of the image block and the character through the visual representation features and the language representation features, the attention loss value can be determined according to the contribution, so that the contribution of the image block or the character can approximate its own importance. Optionally, the importance can be determined by the label of the image block or the character.

[0100] Step 303, determining the prediction result corresponding to the image and / or the text through the fusion module according to the visual representation features and the language representation features, and determining the prediction loss according to the prediction result.

[0101] Among them, the fusion module can fuse the visual representation features and the language representation features to obtain the multi-modal features after visual-linguistic interaction. The multi-modal features can be used to predict the image or the text to obtain the prediction result.

[0102] In an example, the visual representation features and the language representation features can be input into the fusion module to directly obtain the prediction result of the image and the prediction result of the text.

[0103] In another example, the visual representation features and the language representation features can be input into the fusion module to obtain the fused features, which are then input into the prediction module to obtain the prediction result.

[0104] Optionally, the prediction result can be set according to actual needs. For example, the prediction result of the image can be whether each image block belongs to the subject frame, so as to determine the subject frame in the image; and the prediction result of the text can be whether each character belongs to the center word, so as to determine the center word in the text.

[0105] After obtaining the prediction result, a corresponding prediction loss can be obtained according to the labels of the image and the text. The label of the image can be used to indicate whether each image block belongs to the subject frame. If any image block belongs to the subject frame, the corresponding label is 1, otherwise it is 0. Similarly, the label of the text can be used to indicate whether each character belongs to the center word. If any character belongs to the center word, the corresponding label can be 1, otherwise it is 0.

[0106] In step 304, the parameters of the model are adjusted according to the attention loss and the prediction loss.

[0107] The prediction loss can be used to represent the difference between the prediction result and the label, and the attention loss can be used to represent the difference between the attention value corresponding to the representation feature of the encoded module output character or image block and the importance of the character or image block.

[0108] The parameters of the model can be adjusted through the attention loss and the prediction loss. Optionally, gradient descent or other methods can be used to adjust the parameters of the model. After adjustment, steps 301 to 304 can be re-executed until the iteration number or the loss value meets the preset condition, and the model training is completed.

[0109] It should be noted that the present embodiment defines that the image block can be processed alone, the text can be processed alone, and the image and the text can be processed simultaneously through "and / or". The following will be described respectively.

[0110] In an optional implementation, the model can be used to predict the image. During the training process, the contribution of each character in the text to the image prediction can be constructed, and the corresponding attention loss can be obtained according to the contribution. In addition, according to the visual representation feature and the language representation feature, the prediction result corresponding to the image can be obtained, and the prediction loss can be determined according to the prediction result. The parameters of the model are adjusted according to the attention loss and the prediction loss.

[0111] In another optional implementation, the model can be used to predict the text. During the training process, the attention loss and the prediction loss can be used to serve the text prediction. The contribution of each image block in the image to the text prediction can be constructed, and the corresponding attention loss can be obtained according to the contribution. In addition, according to the visual representation feature and the language representation feature, the prediction result corresponding to the text can be obtained, and the prediction loss can be determined according to the prediction result.

[0112] In yet another optional implementation, the model can be used to predict text and images, and during the training process, the contribution of each image block in the image to the text prediction and the contribution of each character in the text to the image prediction can be constructed, and the corresponding attention loss is determined according to the constructed contribution. In addition, according to the visual representation features and the language representation features, the prediction result corresponding to the image can be obtained, and the prediction result corresponding to the text can also be obtained, and the prediction loss is determined according to the prediction result, so that the prediction result and the attention value of the text and the image can be supervised at the same time.

[0113] To sum up, the model training method provided in the embodiment can train the model through the loss corresponding to the prediction result and the intermediate attention loss, which can not only supervise the final prediction result, but also supervise the intermediate representation features, so that the extracted representation features can more accurately reflect the importance of the image blocks or characters, thereby better performing fusion in the multi-modal fusion module, giving more attention to important image blocks or characters, improving the accuracy of the final prediction result, and improving the model effect.

[0114] In one or more embodiments of the present application, optionally, determining the attention value corresponding to each image block in the image and / or each character in the text according to the visual representation features and the language representation features can include: calculating the cross-attention matrix according to the visual representation features of the multiple image blocks in the image and the language representation features of the multiple characters in the text, wherein the elements in the cross-attention matrix are used to represent the contribution of the image block to the character and / or the contribution of the character to the image block; for any image block, the contributions of the image block to each character are added to obtain the attention value corresponding to the image block, and / or for any character, the contributions of the character to each image block are added to obtain the attention value corresponding to the character.

[0115] Optionally, for each image block, the contribution of the image block to each character in the text, i.e., the importance of the image block in calculating the prediction result corresponding to each character, can be calculated, and then the sum of the contributions of the image block to all characters is calculated to obtain the attention value of the image block in the entire text prediction process. Similarly, for each character, the contribution of the character to each image block in the image, i.e., the importance of the character in calculating the prediction result corresponding to each image block, can be calculated, and then the sum of the contributions of the character to all image blocks is calculated to obtain the attention value of the character in the entire image prediction process.

[0116] Optionally, the visual representation features of multiple image patches and the linguistic representation features of multiple characters output by the encoding module can be multiplied by a dot product to obtain a cross attention matrix. Each element in the matrix can be used to represent the contribution of the corresponding image patch to the character, or the contribution of the character to the image patch.

[0117] Figure 4 This is a schematic diagram illustrating the principle of calculating attention values ​​for image blocks and characters, provided as an embodiment of this application. For example... Figure 4 As shown, assuming an image consists of M image patches and text consists of N characters, the visual encoding module can output visual representation features corresponding to the M image patches, and the language encoding module can output language representation features corresponding to the N characters. Performing a dot product operation on the M visual representation features and the N language representation features yields an M*N cross-attention matrix. In this matrix, each row corresponds to an image patch, and each column corresponds to a character. More specifically, the element in the i-th row and j-th column represents the contribution of the i-th image patch to the j-th character. Here, it can be assumed that the contribution of the image patch to the character is the same as the contribution of the character to the image patch; that is, the contribution of the i-th image patch to the j-th character can be equal to the contribution of the j-th character to the i-th image patch.

[0118] Optional, Figure 4 In the visual representation features, each small square can represent the visual representation features corresponding to an image patch, and the visual representation features corresponding to each image patch can be a D-dimensional vector. The visual representation features of the entire image can be N×D in dimension. Similarly, in the language representation features, each small square can represent the visual representation features corresponding to a character, and the visual representation features corresponding to each character can also be a D-dimensional vector. The visual representation features of the entire text can be M×D in dimension.

[0119] The visual representation features of an image patch can be operated on with the linguistic representation features of a character to obtain the elements in the cross-attention matrix. For example, the visual representation features of the i-th image patch are operated on with the linguistic representation features of the j-th character to obtain the element in the i-th row and j-th column of the matrix.

[0120] After obtaining the cross-attention matrix, for each row of the matrix, the elements of that row can be summed to obtain the contribution of the corresponding image patch to the entire text, i.e., the attention value of the image patch; for each column of the matrix, the elements of that column can be summed to obtain the contribution of the corresponding character to the entire image, i.e., the attention value of the character.

[0121] Optionally, before the step of "for any image block, adding the contribution of the image block to each character to obtain the attention value corresponding to the image block, and / or for any character, adding the contribution of the character to each image block to obtain the attention value corresponding to the character", normalization operation can be performed on the matrix, and after the normalization operation, the adding operation is performed to obtain the attention value, or the adding operation is performed first, and then the normalization operation is performed to obtain the normalized attention value. The normalization operation can limit the final obtained attention value to a predetermined range, and improve the accuracy of the attention loss.

[0122] Optionally, the elements in each row or each column of the matrix can be added by the pooling operation to obtain the corresponding attention value.

[0123] In other optional implementations, the cross-attention matrix can also be obtained by other operations other than dot product, such as linear transformation, etc.

[0124] The "and / or" defined in the embodiment includes various implementation manners. When the model is used for image prediction, the attention value of the character in the text to the image can be calculated and the attention loss can be constructed. When the model is used for text prediction, the attention value of the image block in the image to the text can be calculated and the attention loss can be constructed. When the model can be used for image prediction and text prediction at the same time, the obtained matrix can represent the contribution of the character to the image block and the contribution of the image block to the character at the same time, so as to determine the attention value of the image block and the attention value of the character.

[0125] In summary, by performing dot product operation on the representation features of the image block and the character, the cross-attention matrix is obtained, and then the attention value is determined according to the matrix, which can directly determine the contribution of each other through the interaction of the visual representation features and the language representation features, and effectively improve the efficiency and accuracy.

[0126] In one or more embodiments of the present application, the model further includes a visual prediction module and / or a language prediction module; according to the visual representation features and the language representation features, the prediction result corresponding to the image and / or the text is determined by the fusion module, including: inputting the visual representation features and the language representation features into the fusion module to obtain multi-modal representation features; according to the multi-modal representation features, the prediction result of each image block is obtained by the visual prediction module, and / or the prediction result of each character is obtained by the language prediction module.

[0127] In the embodiments of the present application, vision corresponds to images, language corresponds to text, and the modules related to vision can be used to process images to obtain vision-related features, and the modules related to language can be used to process text to obtain language-related features. The prediction operations on images and text can be performed by the vision prediction module and the language prediction module respectively.

[0128] Specifically, if the model is used to implement both text prediction and image prediction functions, the vision prediction module and the language prediction module can be included, and if only one of the functions is implemented, only one corresponding prediction module can be included.

[0129] The information output by the fusion module can be a multi-modal representation feature, which is obtained by interacting the vision representation feature and the language representation feature. The features output to the vision prediction module and the features output to the language prediction module can be the same or different, which is not limited in the embodiments. Optionally, the fusion module can be implemented based on an attention mechanism to improve the feature fusion effect.

[0130] In summary, in the embodiments, the features of vision and language are first fused by the fusion module to obtain a multi-modal feature, and then the corresponding result is determined by the prediction module of the business layer. The fusion module only needs to focus on the fusion effect of vision and language, and the prediction module can be set according to the actual business needs, which improves the feature fusion effect, further improves the model accuracy, and facilitates the decoupling of feature fusion and business prediction to meet the use requirements of different scenarios.

[0131] In one or more embodiments of the present application, optionally, the attention loss is determined according to the attention values, which can include: determining the attention loss according to the attention values of each image block and the corresponding label, and / or the attention values of each character and the corresponding label; wherein the labels used when determining the attention loss match the labels used when determining the prediction loss.

[0132] Optionally, the labels of the image and / or text are used to represent whether each image block in the image and / or each character in the text meets a preset requirement; and the attention loss is used to represent the difference between the attention value corresponding to the image block and / or the character and the label corresponding to the image block and / or the character.

[0133] For example, the label of the image can be used to represent whether each image block in the image belongs to the main frame, and if it belongs to the main frame, it is 1, otherwise it is 0. Therefore, the label of the image can be regarded as an N-dimensional vector, and N is the number of image blocks. Similarly, the label of the text can be regarded as an M-dimensional vector, and M is the number of characters. The label here can specifically refer to the label corresponding to the prediction result.

[0134] Figure 5A model training process schematic diagram provided by an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the embedding features of the image and the embedding features of the text are respectively input into corresponding encoding modules to obtain visual representation features and language representation features, the visual representation features and the language representation features are jointly calculated to obtain attention values of the image and the text, and the visual representation features and the language representation features are input into a fusion module to obtain prediction results of the image and the text through a visual prediction module and a language prediction module. Figure 5

[0135] Exemplarily, in the input layer, for the image, a patch-based method can be used to divide the image into M image blocks, and a linear layer is used to obtain the embedding features of the image blocks; for the text, the embedding features of N characters in the text can be obtained, and the two parts are spliced together through special tokens such as REG and CLS as inputs. The embedding features of each image block or character can be a 256-dimensional vector.

[0136] In the encoding layer, the visual encoding module and the language encoding module can obtain M visual representation features of the image blocks and N language representation features of the characters, and the fusion module can obtain M+N multi-modal features. Each visual representation feature, language representation feature or multi-modal feature can be a 768-dimensional vector.

[0137] Optionally, the visual encoding module, the language encoding module and the fusion module can be implemented through a self-attention mechanism. Exemplarily, in the visual encoding module, for each image block, the influence of the remaining M-1 image blocks on it can be calculated, and the output features are determined according to the attention scores; similarly, in the language encoding module, for each character, the influence of the remaining N-1 characters on it can be calculated; in the fusion module, for each image block or character, the influence of the remaining M+N-1 image blocks and characters on it can be calculated. Through the self-attention mechanism, the mutual influence between the characters and the image blocks can be combined to determine the final output features, thereby improving the accuracy of the feature representation.

[0138] In the prediction layer, two parts can be used for prediction. For the commodity subject selection, a regression loss is used as the optimization target, and for the center word recognition, a classification loss is used as the optimization target. Specifically, the multi-modal features corresponding to the M image blocks can be input into the visual prediction module to obtain the prediction results of the image, and the multi-modal features corresponding to the N characters can be input into the language prediction module to obtain the prediction results of the text.

[0139] ​Optionally, the prediction result of the image, the attention value of the image, and the label of the image can be regarded as N-dimensional vectors, and the prediction result of the text, the attention value of the text, and the label of the text can be regarded as M-dimensional vectors. The prediction loss of the image can be determined by the prediction result of the image and the label of the image. The attention loss of the image can be determined by the attention value of the image and the label of the image. The prediction loss of the text can be determined by the prediction result of the text and the label of the text. The attention loss of the text can be determined by the attention value of the text and the label of the text. In this way, the label can be used to supervise not only the prediction result but also the attention value.

[0140] Optionally, the label used to determine the attention loss matches the label used to determine the prediction loss, i.e., the above-mentioned prediction result label.

[0141] In an example, the label used to determine the attention loss can be the same as the label used to determine the prediction loss. For any image block, if the image block belongs to the subject frame, the label corresponding to the prediction result and the label corresponding to the attention value can both be 1, so that the prediction result and the attention value are both close to 1.

[0142] In another example, the label used to determine the attention loss can have a certain proportional relationship with the label used to determine the prediction loss. For example, the label corresponding to the attention value can be equal to the label corresponding to the prediction result divided by K, where K is the number of image blocks or characters with positive results. For example, if there are four image blocks in the image that belong to the subject frame, the label corresponding to the prediction result of the four image blocks can all be 1, and the label corresponding to the attention value can be 1 / 4, so that the attention value is close to 1 / 4.

[0143] In summary, by setting the label used to determine the attention loss to match the label used to determine the prediction loss, the label corresponding to the prediction result can be used to supervise not only the prediction loss but also the attention loss, guiding the cross-modal interaction between the image and the text, thereby more effectively fusing the key information, and the attention value of the image block and the character with a positive result is greater, thereby giving them a higher weight in the prediction process, making the training effect of each stage of the model consistent, improving the performance of the model, and without introducing too many features, simplifying the algorithm, and reducing the computational overhead.

[0144] In other optional embodiments, the model can be used for classification prediction, for example, given the image and text of a commodity, the category of the commodity is predicted, which can include clothing, home appliances, food, cosmetics, etc. The label used to determine the prediction result corresponding to the image and text can be used to represent the category of the commodity, and the label used to determine the attention loss can be different from the label used to determine the prediction result, and the label used to determine the attention loss can be used to represent the key position of the text or image, which can be determined by manual labeling or other means.

[0145] In one or more embodiments of the present application, optionally, according to the attention value of each image block and the corresponding label, and / or the attention value of each character and the corresponding label, the attention loss can be determined, which can include: calculating a first cross-entropy loss according to the attention value of each image block and the corresponding label; calculating a second cross-entropy loss according to the attention value of each character and the corresponding label; and determining the corresponding attention loss according to the first cross-entropy loss and the second cross-entropy loss.

[0146] The following gives an example of the calculation formula of each stage of the model training process in this embodiment.

[0147] Optionally, the multi-modal features can be calculated by the following formula (1) to formula (4), and the prediction result can be determined according to the multi-modal features.

[0148]

[0149] P′ v =Visual-Transformer(P v ) (2)

[0150] P′ l =Linguistic-Transformer(P l ) (3)

[0151] P″ v ,P″ l =Visual-Linguistic-Transformer(P′ v ,P′ l ) (4)

[0152] Wherein, X is the information input to the encoding module, v represents vision, l represents language, visual tokens P v are used to represent the embedding features corresponding to the image, linguistic tokens P l are used to represent the embedding features corresponding to the text, is the embedding feature of REGtoken, is the embedding feature corresponding to the CLS token, N v is the number of image patches in the image, N l is the number of characters in the text. P' v is the visual representation feature, Visual-Transformer is the visual encoding module; P' l is the linguistic representation feature, Linguistic-Transformer is the linguistic encoding module, P" v , P" l is the multimodal feature, Visual-Linguistic-Transformer is the fusion module for realizing visual and linguistic interaction. P" v , P" l are input into the visual prediction module and the linguistic prediction module respectively, and the corresponding prediction results can be obtained.

[0153] Further, the cross-attention matrix can be calculated by formula (5):

[0154]

[0155] The attention value corresponding to the image is calculated by formula (6):

[0156]

[0157] The attention value corresponding to the text is calculated by formula (7):

[0158]

[0159] The attention loss is calculated by formula (8):

[0160]

[0161] where Cross-Entropy represents the cross-entropy loss, y v and y l are the label corresponding to the image and the label corresponding to the text respectively.

[0162] In summary, through the above formulas, the cross-entropy loss corresponding to the image and the text can be calculated, and the cross-entropy losses of the two are added to obtain the corresponding attention loss. According to the attention loss, the model parameters are adjusted, which can simultaneously make the attention value of the image patch approximate the label of the image patch, and make the attention value of the character approximate the label of the character, improve the accuracy of the representation features of the image and the text, and further improve the accuracy of the model prediction.

[0163] In one or more embodiments of the present application, adjusting the parameters of the model according to the attention loss and the prediction loss can comprise: adjusting the parameters of the visual encoding module and the language encoding module according to the attention loss; and adjusting the parameters of each module in the model according to the prediction loss.

[0164] Specifically, the attention loss is used to measure the performance of the representation features output by the encoding modules, and therefore the visual encoding module and the language encoding module can be adjusted according to the attention loss, without adjusting other modules in the model. The prediction loss is used to measure the performance of the prediction results output by the model, and therefore the parameters of all modules in the model can be adjusted according to the prediction loss.

[0165] Optionally, each module can be implemented by a neural network or the like.

[0166] It should be noted that the order of calculating the loss values and adjusting the parameters is not limited in the embodiments of the present application. For example, the attention loss and the prediction loss can be calculated simultaneously, or the attention loss can be calculated first and then the prediction loss, or the prediction loss can be calculated first and then the attention loss. The order of adjusting the parameters of each module is also not limited.

[0167] In summary, the visual encoding module and the language encoding module are constrained by the attention loss, which has no effect on the subsequent other modules, so that the attention loss can focus on the adjustment of the encoding modules, improve the accuracy of the modal representation output by the encoding modules, and improve the overall training efficiency of the model.

[0168] Figure 6 Another flowchart of a model training method provided by the embodiments of the present application is shown. The embodiments of the present application can be applied to the e-commerce field, for example. Figure 6 As shown in the figure, the method can comprise:

[0169] Step 601: determining a visual representation feature corresponding to a product image of a product by a visual encoding module according to the product image and a product title of the product, and determining a language representation feature corresponding to the product title by a language encoding module.

[0170] Optionally, there are a large number of products on an e-commerce platform, and each product has a corresponding product detail page. On the product detail page, there is an image corresponding to the product, such as a product main image, and a title corresponding to the product. For any product, the image main body box and the title center word corresponding to the product can be extracted by the method provided by the embodiments of the present application.

[0171] Specifically, in this step, the image and the title corresponding to the product can be input into the visual encoding module and the language encoding module respectively to obtain the visual representation feature and the language representation feature.

[0172] In step 602, attention values corresponding to each image block in the product image and / or each character in the product title are determined according to the visual representation features and the language representation features, and an attention loss is determined according to the attention values.

[0173] The attention value of an image block is used to represent the contribution of the image block to text prediction, and the attention value of a character is used to represent the contribution of the character to image prediction.

[0174] In step 603, a prediction result corresponding to the product image and / or the product title is determined by a fusion module according to the visual representation features and the language representation features, and a prediction loss is determined according to the prediction result.

[0175] The prediction result corresponding to the product image is used to locate a product subject in the product image, and the prediction result corresponding to the product title is used to locate a center word in the product title.

[0176] Optionally, the product subject can be a subject frame in the product image, which can be a frame in which the target product is located. In the image corresponding to the product, there can be other items in addition to the product. By the method in this embodiment, the target product in the image can be located.

[0177] In step 604, the parameters of the model are adjusted according to the attention loss and the prediction loss.

[0178] The implementation process and principles of the method in this embodiment can be referred to the foregoing embodiments, which will not be described here.

[0179] By the above method, the center word corresponding to each product and the subject frame of the product image in the e-commerce platform can be constructed.

[0180] Optionally, a keyword or an image input by a user to be queried can also be obtained, and according to the keyword or the image, a product matching the keyword or the image to be queried is selected from a plurality of products in the e-commerce platform and displayed. The matching product can include a product whose center word is the same as the keyword, or a product whose subject frame matches the image.

[0181] For example, if the keyword input by the user is "shorts", the product whose center word is "shorts" can be found and displayed to the user. The center word or the subject frame matching can more accurately determine the product that the user expects to query, and the method in this embodiment can assist in improving the accuracy of determining the center word or the subject frame.

[0182] Figure 7 An existing graph search diagram is shown in FIG. 1. Figure 7As shown, the image to be queried input by the user is bottled drinking water. If the image to be queried is directly matched with the image of the commodity, some commodities that are not bottled water but contain bottled water in the image may be presented to the user. For example, a beer merchant sells beer, and in order to show the capacity of the beer, a 500ml standard bottled water is used as a comparison. In this case, the user wants to buy bottled water, but the beer commodity is recommended to the user, and the recommendation accuracy is poor.

[0183] Figure 8 A flowchart of a method for searching an image according to an embodiment of the present application is shown in FIG. 6. As shown, the method includes the following steps. Figure 8 As shown, after the method provided by the present application is used, the subject frame of each commodity image can be located. For example, for the beer commodity, the subject in the commodity image is beer rather than bottled water, which can be understood in combination with the title of the commodity. Therefore, when searching the image, if the user searches for bottled water, the beer will not be shown to the user. The image of the bottled water sold by a merchant may contain decorations such as a bouquet. The subject in the image is bottled water, which can be understood in combination with the title of the commodity. Therefore, the commodity meets the needs of the user, and the commodity can be pushed to the user, thereby improving the user experience.

[0184] The embodiment can realize the output of the core capability of e-commerce multi-modal, in addition to image search and text search, there can be other applications.

[0185] In an example, after the subject frame of the commodity image is determined, the subject frame can be input to the image recognition model to more accurately recognize the core product in the subject frame.

[0186] In another example, the category of the commodity can be predicted according to the subject frame or the center word of the title of the commodity, thereby realizing the classification operation of the commodity, wherein the category can include clothing, food, household appliances, etc.

[0187] In summary, the model training method provided by the embodiment can train the model by the loss corresponding to the prediction result and the attention loss in the middle. Not only can the final prediction result be supervised, but also the intermediate representation feature can be supervised, so that the extracted representation feature can more accurately reflect the importance of the image block or the character, thereby better performing fusion in the multi-modal fusion module, giving more attention to important image blocks or characters, improving the accuracy of the final prediction result, more quickly and accurately determining the center word in the commodity title and the subject frame in the commodity image, improving the accuracy of the text search and the image search, and improving the user experience.

[0188] Figure 9 A flowchart of another model training method provided by an embodiment of the present application is shown in FIG. 8. The model includes a first modal encoding module, a second modal encoding module, and a fusion module. As shown, the method includes the following steps.Figure 9 The method comprises:

[0189] In step 901, a first modality encoding module is used to determine first modality representation features corresponding to first modality information, and a second modality encoding module is used to determine second modality representation features corresponding to second modality information.

[0190] The modality can refer to a form in which data exists, such as text, image, audio, video, and sensor data, which belong to different modalities. The first modality information and the second modality information can be first modality information and second modality information corresponding to a target object, and the target object can be a commodity.

[0191] Optionally, the first modality information and the second modality information include any two of the following information: image, text, audio, video, and sensor information. For example, the first modality information can include any one of image, text, audio, video, and sensor information, and the second modality information can include any one of audio, video, and sensor information and be different from the first modality information.

[0192] In step 902, according to the first modality representation features and the second modality representation features, attention values corresponding to each first sub-modality information in the first modality information and / or each second sub-modality information in the second modality information are determined, and an attention loss is determined according to the attention values.

[0193] The attention value of the first sub-modality information is used to represent the contribution of the first sub-modality information to the prediction of the second modality information, and the attention value of the second sub-modality information is used to represent the contribution of the second sub-modality information to the prediction of the first modality information.

[0194] Optionally, the first modality information can be divided into a plurality of first sub-modality information, and the second modality information can be divided into a plurality of second sub-modality information.

[0195] For example, the first modality information can be video, and the first sub-modality information can be each frame image in the video or a video segment. Alternatively, the first modality information can be audio, and the first sub-modality information can be an audio segment. Alternatively, the first modality information can be sensor information, which can be specifically divided into a plurality of groups, each group serving as a first sub-modality information. The relationship between the second modality information and the second sub-modality information is similar, and will not be described here.

[0196] In step 903, according to the first modality representation features and the second modality representation features, a fusion module is used to determine a prediction result of the first modality information and / or the second modality information and a corresponding prediction loss.

[0197] Optionally, the first modality representation feature and the second modality representation feature can be input into a fusion module to obtain a multi-modal feature, and the multi-modal feature can be used to represent the corresponding feature of the target object in the multi-modal space. Then, the multi-modal feature is input into the first modality information prediction module and the second modality information prediction module to obtain the corresponding prediction result.

[0198] The multi-modal space can refer to a common space to which data of different modalities is mapped. In the multi-modal space, the more important a certain sub-modality information is, the higher the corresponding weight is. Through the multi-modal space, high-level features of data of various modalities can be abstracted to realize the prediction function of the model.

[0199] At step 904, the parameters of the model are adjusted according to the attention loss and the prediction loss.

[0200] In this embodiment, the model can be trained according to the first modality information and the second modality information, and the obtained model can be used to predict the first modality information and / or the second modality information.

[0201] The implementation process and principles of the method in this embodiment can be referred to the foregoing embodiments. Only the image in the foregoing embodiments is replaced by the first modality information, the text is replaced by the second modality information, the image block is replaced by the first sub-modality information, and the character is replaced by the second sub-modality information, and the vision and the language are replaced by the first modality and the second modality, respectively.

[0202] Exemplarily, the first modality information and the second modality information can be video and text respectively, for example, can be the video and the text of a commodity, and specifically can be the video of a commodity detail page and the title of the commodity. The corresponding prediction results are used to represent the video segment in which the commodity is located and the center word of the title. By processing the attention values of each segment in the video and each character in the title, the attention loss is constructed, which can assist in improving the accuracy of the model, so as to more quickly and accurately locate the segment in which the commodity appears in the video and the center word of the title for subsequent text search, image search or video search.

[0203] In another example, the first modality information and the second modality information can be image and audio respectively, for example, can be the image and the audio of a commodity, and the corresponding prediction results are used to represent the main body frame of the commodity in the image and the audio segment in which the commodity is mentioned in the audio. Therefore, the method provided in this embodiment can be used to more quickly and accurately locate the main body frame of the commodity in the image and the segment in which the commodity is mentioned in the audio for subsequent image search or audio search.

[0204] In yet another example, the first modality information and the second modality information can be sensor information and images, for example, sensor information and images collected by a vehicle or a roadside device. The sensor information can be infrared sensor data, point cloud data, etc. Through the method provided in this embodiment, the sensor information and the images can be processed, and the trained model can be used to determine the position of an obstacle in the sensor information or the images, thereby providing a basis for vehicle driving and improving driving safety.

[0205] In summary, the model training method provided in this embodiment can train the model through the loss corresponding to the prediction result and the intermediate attention loss, which can not only supervise the final prediction result, but also supervise the intermediate representation feature, so that the extracted representation feature can more accurately reflect the importance of each first sub-modality information and second sub-modality information, thereby better performing fusion in the multi-modal fusion module, giving more attention to important first sub-modality information or second sub-modality information, improving the accuracy of the final prediction result, and improving the model effect.

[0206] Figure 10 A flowchart of a prediction method provided in an embodiment of the present application is shown in FIG. 10. Figure 10 The method comprises the following steps:

[0207] In step 1001, first modality information and second modality information to be processed are obtained.

[0208] The first modality information and the second modality information include any two of the following information: images, texts, audios, videos, and sensor information.

[0209] In step 1002, a multi-modal interaction model is used to obtain a prediction result corresponding to the first modality information and / or the second modality information according to the first modality information and the second modality information to be processed.

[0210] The multi-modal interaction model is trained based on the method of any of the above embodiments.

[0211] Optionally, when the model trained by the embodiment shown in FIG. 10 is used, the first modality information and the second modality information can be images and texts. Figures 1 to 8

[0212] For example, the image and the text of a commodity can be input into the model, and through the interaction between the image and the text, a prediction result corresponding to the image, i.e., a bounding box of the commodity in the image, can be obtained, or a prediction result corresponding to the text, i.e., a center word of the text, can be obtained, or the prediction results of the image and the text can be obtained at the same time.

[0213] ​In summary, the prediction method provided in the embodiment adopts a model trained by a loss corresponding to a prediction result and an intermediate attention loss, can supervise the final prediction result and the intermediate representation feature, so that the extracted representation feature can more accurately reflect the importance of the first sub-modal information and the second sub-modal information, thereby better performing fusion in the multi-modal fusion module, giving more attention to important first sub-modal information or second sub-modal information, and improving the accuracy of the prediction result.

[0214] Corresponding to the above model training method, the embodiment of the present application further provides a model training device, the model comprising a visual encoding module, a language encoding module and a fusion module; the device comprises:

[0215] a first input unit configured to determine visual representation features corresponding to the image by the visual encoding module and determine language representation features corresponding to the text by the language encoding module according to the image and the text to be processed;

[0216] a first attention processing unit configured to determine attention values corresponding to each image block in the image and / or each character in the text according to the visual representation features and the language representation features, and determine an attention loss according to the attention values; wherein the attention value of the image block is used to represent the contribution of the image block to the text prediction, and the attention value of the character is used to represent the contribution of the character to the image prediction;

[0217] a first fusion unit configured to determine a prediction result corresponding to the image and / or the text by the fusion module according to the visual representation features and the language representation features, and determine a prediction loss according to the prediction result;

[0218] a first adjustment unit configured to adjust the parameters of the model according to the attention loss and the prediction loss.

[0219] In one or more embodiments of the present application, optionally, when the first attention processing unit determines the attention values corresponding to each image block in the image and / or each character in the text according to the visual representation features and the language representation features, it is specifically configured to:

[0220] calculate the visual representation features of a plurality of image blocks in the image and the language representation features of a plurality of characters in the text to obtain a cross-attention matrix, and the elements in the cross-attention matrix are used to represent the contribution of the image block to the character and / or the contribution of the character to the image block;

[0221] The attention value corresponding to each image block is obtained by adding the contribution of each character to the image block; and / or the attention value corresponding to each character is obtained by adding the contribution of each image block to the character.

[0222] In one or more embodiments of the present application, the first attention processing unit is specifically configured to:

[0223] determine the attention loss according to the attention value of each image block and the corresponding label, and / or the attention value of each character and the corresponding label.

[0224] The label used to determine the attention loss matches the label used to determine the prediction loss.

[0225] In one or more embodiments of the present application, the first attention processing unit is specifically configured to:

[0226] calculate the first cross-entropy loss according to the attention value of each image block and the corresponding label;

[0227] calculate the second cross-entropy loss according to the attention value of each character and the corresponding label;

[0228] determine the corresponding attention loss according to the first cross-entropy loss and the second cross-entropy loss.

[0229] In one or more embodiments of the present application, the model further comprises a visual prediction module and / or a language prediction module; and the first fusion unit is specifically configured to:

[0230] input the visual representation feature and the language representation feature into a fusion module to obtain a multi-modal representation feature;

[0231] obtain the prediction result of each image block through the visual prediction module and / or the prediction result of each character through the language prediction module according to the multi-modal representation feature.

[0232] In one or more embodiments of the present application, the first adjustment unit is specifically configured to:

[0233] adjust the parameters of the visual encoding module and the language encoding module according to the attention loss;

[0234] adjust the parameters of each module in the model according to the prediction loss.

[0235] The embodiment of the application further provides another model training device, the model comprising a visual encoding module, a language encoding module and a fusion module; the device comprising:

[0236] a second input unit configured to determine visual representation features corresponding to the product images by the visual encoding module according to the product images and the product titles corresponding to the products, and determine language representation features corresponding to the product titles by the language encoding module;

[0237] a second attention processing unit configured to determine attention values corresponding to each image block in the product images and / or each character in the product titles according to the visual representation features and the language representation features, and determine an attention loss according to the attention values; wherein the attention value of the image block is used to represent the contribution of the image block to the text prediction, and the attention value of the character is used to represent the contribution of the character to the image prediction;

[0238] a second fusion unit configured to determine prediction results corresponding to the product images and / or the product titles by the fusion module according to the visual representation features and the language representation features, and determine a prediction loss according to the prediction results; wherein the prediction result corresponding to the product image is used to locate the product subject in the product image, and the prediction result corresponding to the product title is used to locate the center word of the product title;

[0239] a second adjustment unit configured to adjust the parameters of the model according to the attention loss and the prediction loss.

[0240] The embodiment of the application further provides another model training device, the model comprising a visual encoding module, a language encoding module and a fusion module; the device comprising:

[0241] a third input unit configured to determine first modality representation features corresponding to the first modality information by the first modality encoding module, and determine second modality representation features corresponding to the second modality information by the second modality encoding module;

[0242] a third attention processing unit configured to determine attention values corresponding to each first sub-modality information in the first modality information and / or each second sub-modality information in the second modality information according to the first modality representation features and the second modality representation features, and determine an attention loss according to the attention values; wherein the attention value of the first sub-modality information is used to represent the contribution of the first sub-modality information to the prediction of the second modality information, and the attention value of the second sub-modality information is used to represent the contribution of the second sub-modality information to the prediction of the first modality information;

[0243] a third fusion unit configured to determine, by a fusion module, a prediction result of the first modality information and / or the second modality information and a corresponding prediction loss according to the first modality representation feature and the second modality representation feature.

[0244] a third adjustment unit configured to adjust parameters of the model according to the attention loss and the prediction loss.

[0245] The embodiment of the present application further provides a prediction device, comprising:

[0246] a first acquisition unit configured to acquire first modality information and second modality information to be processed; wherein the first modality information and the second modality information comprise any two of the following information: image, text, audio, video and sensor information;

[0247] a prediction unit configured to obtain a prediction result corresponding to the first modality information and / or the second modality information by a multi-modal interaction model according to the first modality information and the second modality information to be processed;

[0248] The multi-modal interaction model is trained based on the method in any of the foregoing embodiments.

[0249] The model training device and the prediction device provided by the embodiment of the present application can be used to execute the method in any of the foregoing Figures 1 to 10 The technical solutions of the embodiment shown in the foregoing

[0250] Figure 11 A structural schematic diagram of an electronic device is provided in the embodiment of the present application. As shown in the figure, Figure 11 The electronic device in the embodiment of the present application can comprise:

[0251] at least one processor 1101; and

[0252] a memory 1102 in communication connection with the at least one processor;

[0253] The memory 1102 stores instructions executable by the at least one processor 1101, and the instructions are executed by the at least one processor 1101 to enable the electronic device to execute the method in any of the foregoing embodiments.

[0254] Optionally, the memory 1102 can be independent or integrated with the processor 1101.

[0255] The implementation principle and technical effects of the electronic device provided by the embodiment of the present application can be referred to the foregoing embodiments, and will not be repeated here.

[0256] The embodiment of the present application further provides a computer readable storage medium, wherein computer execution instructions are stored in the computer readable storage medium, and when a processor executes the computer execution instructions, the method in any of the foregoing embodiments is implemented.

[0257] The embodiment of the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method in any of the foregoing embodiments.

[0258] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other manners. For example, the above-described device embodiments are merely illustrative. For example, the division of the modules is merely logical function division. There can be another division manner for actual implementation. For example, a plurality of modules or features can be combined or integrated into another system, or some features can be ignored or not executed.

[0259] The integrated modules in the form of software function modules described above can be stored in a computer readable storage medium. The software function modules stored in the storage medium can include a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods described in the embodiments of the present application.

[0260] It should be understood that the processor described above can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor and the like. The steps of the methods disclosed in the present application can be directly embodied as the execution of the hardware processor, or the combination of hardware and software modules in the processor. The memory can include a high-speed RAM memory, and can also include a non-volatile storage NVM, such as at least one disk memory, and can also be a U disk, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0261] The storage medium described above can be implemented by any type of volatile or non-volatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0262] An exemplary storage medium is coupled to the processor such that the processor can read information from, and can write information to, the storage medium. Of course, the storage medium can be a part of the processor. Consistent with the teachings provided herein, the processor can execute instructions embodied by computer code stored on the storage medium, and the storage medium can store computer code which, when executed by the processor, causes the processor to perform methods as described herein. The processor and the storage medium can be supplemented by, or incorporated in, ASICs (application- specific integrated circuits).

[0263] It should be noted that, as used in this document, the terms "includes," "including," "has," "having," "contains," "containing," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element preceded by "comprises a... " does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0264] The above-mentioned sequence numbers of the embodiments of the present application are only for description, and do not represent advantages or disadvantages of the embodiments.

[0265] Those skilled in the art can clearly understand the above-mentioned embodiment methods by means of software and necessary general hardware platforms, of course, they can also be implemented by hardware, but in many cases, the former is a better implementation. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for causing a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device) to execute the methods described in the various embodiments of the present application.

[0266] In the technical solutions of the present application, the collection, storage, use, processing, transmission, provision and disclosure of user data and other information comply with relevant laws and regulations and do not violate public order and good customs.

[0267] The above is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation based on the content of the specification and drawings, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A model training method, characterized in that, The model comprises a visual encoding module, a language encoding module and a fusion module; the method comprises: According to the image to be processed and the text, the visual representation features corresponding to the image are determined by the visual encoding module, and the language representation features corresponding to the text are determined by the language encoding module; According to the visual representation features of the plurality of image blocks in the image and the language representation features of the plurality of characters in the text, a cross-attention matrix is obtained, and the elements in the cross-attention matrix are used to represent the contribution of the image block to the character and / or the contribution of the character to the image block; For any image block, the contributions of the image block to each character are added to obtain the attention value corresponding to the image block; and / or, for any character, the contributions of the character to each image block are added to obtain the attention value corresponding to the character, and an attention loss is determined according to the attention value; wherein the attention value of the image block is used to represent the contribution of the image block to the text prediction, and the attention value of the character is used to represent the contribution of the character to the image prediction; According to the visual representation features and the language representation features, the prediction result corresponding to the image and / or the text is determined by the fusion module, and a prediction loss is determined according to the prediction result; According to the attention loss and the prediction loss, the parameters of the model are adjusted.

2. The method of claim 1, wherein, According to the attention value, the attention loss is determined, comprising: According to the attention value of each image block and the corresponding label, and / or the attention value of each character and the corresponding label, the attention loss is determined; Wherein, the label used for determining the attention loss matches the label used for determining the prediction loss.

3. The method of claim 2, wherein, According to the attention value of each image block and the corresponding label, and / or the attention value of each character and the corresponding label, the attention loss is determined, comprising: According to the attention value of each image block and the corresponding label, a first cross-entropy loss is calculated; According to the attention value of each character and the corresponding label, a second cross-entropy loss is calculated; According to the first cross-entropy loss and the second cross-entropy loss, the corresponding attention loss is determined.

4. The method according to any one of claims 1 to 3, characterized in that, The model further comprises a visual prediction module and / or a language prediction module; according to the visual representation features and the language representation features, the prediction result corresponding to the image and / or the text is determined by the fusion module, comprising: The visual representation features and the language representation features are input into the fusion module to obtain a multi-modal representation feature; According to the multi-modal representation feature, the prediction result of each image block is obtained by the visual prediction module, and / or the prediction result of each character is obtained by the language prediction module.

5. The method according to any one of claims 1 to 3, characterized in that, According to the attention loss and the prediction loss, the parameters of the model are adjusted, comprising: According to the attention loss, the parameters of the visual encoding module and the language encoding module are adjusted; According to the prediction loss, the parameters of each module in the model are adjusted.

6. A model training method, comprising: The model comprises a visual encoding module, a language encoding module and a fusion module; the method comprises: According to the image and the title corresponding to the commodity, the visual representation features corresponding to the image are determined by the visual encoding module, and the language representation features corresponding to the title are determined by the language encoding module; According to the visual representation features of the plurality of image blocks in the commodity image and the language representation features of the plurality of characters in the commodity title, a cross-attention matrix is calculated, and an element in the cross-attention matrix is used to represent the contribution of an image block to a character and / or the contribution of a character to an image block; For any image block, the contributions of the image block to each character are added to obtain an attention value corresponding to the image block; and / or, for any character, the contributions of the character to each image block are added to obtain an attention value corresponding to the character, and an attention loss is determined according to the attention value; wherein the attention value of an image block is used to represent the contribution of the image block to text prediction, and the attention value of a character is used to represent the contribution of the character to image prediction; According to the visual representation features and the language representation features, a prediction result corresponding to the commodity image and / or the commodity title is determined by a fusion module, and a prediction loss is determined according to the prediction result; wherein the prediction result corresponding to the commodity image is used to locate a commodity subject in the commodity image, and the prediction result corresponding to the commodity title is used to locate a central word of the commodity title; According to the attention loss and the prediction loss, the parameters of the model are adjusted.

7. A method of prediction, characterized by, Comprising: Obtaining first modality information and second modality information to be processed; wherein the first modality information and the second modality information include any two of the following information: image, text, audio, video, and sensor information; According to the first modality information and the second modality information to be processed, a prediction result corresponding to the first modality information and / or the second modality information is obtained by a multi-modal interaction model; The multi-modal interaction model is trained based on the method of any one of claims 1-6.

8. An electronic device, comprising: Comprising: At least one processor; And A memory connected in communication with the at least one processor; Wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to cause the electronic device to perform the method of any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer execution instructions, and when the processor executes the computer execution instructions, the method of any one of claims 1-7 is implemented.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1-7.

Images