Cross-modal processing for vision and language

Abstract

According to the implementation of the present disclosure, a scheme for cross-modal processing is proposed. In this scheme, a set of visual features of a training image is extracted based on a visual feature extraction sub-model in a target model. Each visual feature corresponds to a pixel block in the training image. Based on a visual semantic dictionary, a set of visual semantic features corresponding to the set of visual features is determined. Based on a text feature extraction sub-model in the target model, a set of text features of a training text corresponding to the training image is extracted. Each text feature corresponds to at least one word in the training text. Based on the set of visual semantic features and the set of text features, a target model is trained to determine the association information between the input text and the input image. In this way, the trained target model can be promoted to provide results accurately and quickly in various visual-language tasks.

Description

Background Art

[0001] In the field of artificial intelligence, image processing and natural language processing have made significant progress and have found widespread application. Consequently, there is a growing demand for cross-modal processing of vision and language for various vision-language tasks. Examples of such tasks include image retrieval, text retrieval, visual question answering (VQA), and natural language visual reasoning (NLVR). Currently, several cross-modal processing techniques have been proposed for these vision-language tasks. Summary of the Invention

[0002] According to the implementation of the present disclosure, a scheme for cross-modal processing is proposed. In this scheme, a set of visual features of a training image is extracted according to a visual feature extraction sub-model in a target model. Each visual feature corresponds to a pixel block in the training image. Based on a visual semantic dictionary, a set of visual semantic features corresponding to the set of visual features is determined. According to a text feature extraction sub-model in the target model, a set of text features of a training text corresponding to the training image is extracted. Each text feature corresponds to at least one word in the training text. Based on the set of visual semantic features and the set of text features, a target model is trained to determine the association information between the input text and the input image. In this way, the trained target model can be promoted to provide results accurately and quickly in various visual-language tasks.

[0003] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. It is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 A block diagram illustrating a computing device capable of implementing various implementations of the present disclosure is shown;

[0005] Figure 2 An architectural diagram illustrating training a vision-language model according to some implementations of the present disclosure is shown;

[0006] Figure 3 A schematic diagram illustrating a visual semantic embedding layer according to some implementations of the present disclosure is shown;

[0007] Figure 4 A schematic diagram illustrating pre-training based on multiple general tasks according to some implementations of the present disclosure is shown;

[0008] Figure 5 shows an example of a visual semantic dictionary obtained by training a vision-language model according to some implementations of the present disclosure;

[0009] Figure 6 An architectural diagram illustrating an application vision-language model according to some implementations of the present disclosure is shown;

[0010] Figure 7 A flowchart illustrating a method of training a model according to some implementations of the present disclosure; and

[0011] Figure 8 A flowchart illustrating a method of applying a model according to some implementations of the present disclosure is shown.

[0012] In these drawings, the same or similar reference symbols are used to designate the same or similar elements. DETAILED DESCRIPTION

[0013] The present disclosure will now be discussed with reference to several example implementations. It should be understood that these implementations are discussed only to enable those skilled in the art to better understand and implement the present disclosure, rather than to imply any limitation on the scope of the present disclosure.

[0014] As used herein, the term "including" and its variations are to be interpreted as open-ended terms meaning "including but not limited to." The term "based on" is to be interpreted as "based, at least in part, on." The terms "an implementation" and "an implementation" are to be interpreted as "at least one implementation." The term "another implementation" is to be interpreted as "at least one other implementation." The terms "first," "second," and so on may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0015] As used herein, a "neural network" is capable of processing inputs and providing corresponding outputs, and typically includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer. The neural networks used in deep learning applications typically include many hidden layers, thereby extending the depth of the network. The various layers of the neural network are connected in sequence so that the output of the previous layer is provided as the input of the next layer, wherein the input layer receives the input of the neural network, and the output of the output layer serves as the final output of the neural network. Each layer of the neural network includes one or more nodes (also referred to as processing nodes or neurons), each of which processes the input from the previous layer. In this article, the terms "neural network", "network" and "neural network model" are used interchangeably.

[0016] As mentioned above, some cross-modal learning schemes for vision and language have been proposed for various vision-language tasks. An important goal of vision-language learning is to bridge the semantic gap between the visual and language domains and align vision and language in a common space. Attention mechanisms have been proposed to establish a connection between the visual and language domains, including for model pre-training and downstream task-specific training. For visual input, these existing solutions utilize object detection models to extract region-level visual features. However, object detection models are not always able to detect regions well, and the detected regions are limited by the predefined categories used by the object detection model.

[0017] There are some problems with using region-level visual features for visual-language understanding. First, the detected regions focus on objects within the bounding box and ignore contextual information outside the bounding box. The global context of an image is very important for relational understanding and reasoning in the language domain. However, region-level visual features only represent a part of the image without the global context of the image, which makes it difficult for existing solutions to fully understand the semantics of the image. For example, without contextual information, existing solutions may not correctly understand the relationship between the detected objects, resulting in incorrect answers for text retrieval or VQA. Second, in existing solutions, visual understanding of images is limited to predefined object categories (e.g., 1,600 objects) and attribute categories (e.g., 400 attributes). Other semantics outside the predefined object categories and attribute categories will be ignored.

[0018] Furthermore, region-level visual features are extracted by object detection models, which are subject to low quality, noise, and oversampling. The use of object detection models also makes end-to-end training difficult. The extracted region-level visual features have a semantic gap with the features in the language domain, and existing solutions struggle to bridge this gap with only one or a few fully connected layers.

[0019] According to an implementation of the present disclosure, a solution for cross-modal processing is provided, which aims to solve one or more of the above-mentioned problems and other potential problems. In this solution, a set of visual features of a training image is extracted according to a visual feature extraction sub-model in a visual-language model. Each visual feature corresponds to a pixel block in the training image. Based on a visual semantic dictionary, a set of visual semantic features corresponding to the set of visual features is determined. According to a text feature extraction sub-model in the visual-language model, a set of text features of a training text corresponding to the training image is extracted. Each text feature corresponds to at least one word in the training text. Based on the set of visual semantic features and the set of text features, a visual-language model is trained.

[0020] In this way, the trained vision-language model can extract comprehensive image features for cross-modal understanding. The trained vision-language model can provide more accurate results in handling various vision-language tasks. Furthermore, because it does not require object detection, the trained vision-language model can accelerate inference time in vision-language tasks by a factor of 10, for example, compared to existing solutions. Therefore, the trained vision-language model can better support online vision-language applications.

[0021] Various example implementations of this solution are described in detail below in conjunction with the accompanying drawings.

[0022] Sample Environment

[0023] Figure 1 FIG1 shows a block diagram of a computing device 100 capable of implementing various implementations of the present disclosure. It should be understood that Figure 1 The computing device 100 shown is merely exemplary and should not be construed as limiting the functionality and scope of the implementations described herein. Figure 1 As shown, computing device 100 comprises a computing device in the form of a general-purpose computing device 100. Components of computing device 100 may include, but are not limited to, one or more processors or processing units 110, memory 120, storage device 130, one or more communication units 140, one or more input devices 150, and one or more output devices 160.

[0024] In some implementations, the computing device 100 can be implemented as various user terminals or service terminals with computing capabilities. The service terminal can be a server, a large computing device, etc. provided by various service providers. The user terminal is such as a mobile terminal, a fixed terminal, or a portable terminal of any type, including a mobile phone, a site, a unit, a device, a multimedia computer, a multimedia tablet, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof. It is also foreseeable that the computing device 100 can support any type of interface for the user (such as a "wearable" circuit, etc.).

[0025] Processing unit 110 may be a real or virtual processor and is capable of performing various processes according to a program stored in memory 120. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to increase the parallel processing capabilities of computing device 100. Processing unit 110 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0026] The computing device 100 typically includes a plurality of computer storage media. Such media can be any available media accessible to the computing device 100, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 120 can be a volatile memory (e.g., registers, cache, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The memory 120 can include a multimodal processing module 122, which are program modules configured to perform the functions of the various implementations described herein. The multimodal processing module 122 can be accessed and executed by the processing unit 110 to implement the corresponding functions.

[0027] Storage device 130 may be removable or non-removable media and may include machine-readable media that can be used to store information and / or data and can be accessed within computing device 100. Computing device 100 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not described in Figure 1 As shown in FIG, a magnetic disk drive for reading or writing from a removable, nonvolatile magnetic disk and an optical disk drive for reading or writing from a removable, nonvolatile optical disk can be provided. In these cases, each drive can be connected to a bus (not shown) by one or more data media interfaces.

[0028] The communication unit 140 enables communication with other computing devices via a communication medium. Additionally, the functionality of the components of the computing device 100 can be implemented as a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the computing device 100 can operate in a networked environment using logical connections to one or more other servers, personal computers (PCs), or another general network node.

[0029] Input device 150 may be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. Output device 160 may be one or more output devices, such as a display, speaker, printer, etc. Computing device 100 may also communicate with one or more external devices (not shown) via communication unit 140 as needed, such as storage devices, display devices, etc., with one or more devices that allow a user to interact with computing device 100, or with any device that allows computing device 100 to communicate with one or more other computing devices (e.g., a network card, modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0030] In some implementations, in addition to being integrated on a single device, some or all of the various components of computing device 100 may be configured in the form of a cloud computing architecture. In a cloud computing architecture, these components may be remotely located and work together to implement the functionality described herein. In some implementations, cloud computing provides computing, software, data access, and storage services that do not require the end user to be aware of the physical location or configuration of the systems or hardware providing these services. In various implementations, cloud computing provides services over a wide area network (such as the Internet) using appropriate protocols. For example, a cloud computing provider provides applications over a wide area network, and these applications can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture and the corresponding data may be stored on servers at remote locations. Computing resources in a cloud computing environment may be consolidated at remote data center locations or they may be dispersed. Cloud computing infrastructure can provide services through shared data centers, even though they appear to be a single access point for users. Therefore, the components and functionality described herein can be provided by a service provider at a remote location using a cloud computing architecture. Alternatively, they can be provided from a conventional server, or they can be installed directly or otherwise on the client device.

[0031] The computing device 100 may be used to implement multimodal processing in various implementations of the present disclosure. Figure 1 As shown, the computing device 100 may receive a training dataset 170 through the input device 150. The training dataset 170 includes a plurality of image-text pairs, each image-text pair including a training image and training text corresponding to the training image. Figure 1 An example of an image-text pair is shown in , namely, a training image 171 and a training text 172 corresponding to the training image 171 .

[0032] In some implementations, the training image and training text in each image-text pair may be matched or aligned. For example, training text 172 describes training image 171. In some implementations, the training image and training text in some image-text pairs may not match, while the training image and training text in other image-text pairs may match. In such implementations, training dataset 170 also includes a label indicating whether the training image and training text match.

[0033] Although the training text 172 is shown as English in the examples herein, it should be understood that this is merely illustrative and is not intended to limit the scope of protection of the present disclosure. The present disclosure is applicable to texts in any language.

[0034] The computing device 100 uses the training dataset 170 to train a visual-language model 180. Accordingly, in this document, the visual-language model 180 may also be referred to as a "target model." The trained visual-language model 180 may be used to determine association information between images and text in visual-language tasks. In some implementations, the training of the visual-language model 180 at the computing device 100 may be pre-training for general tasks. The obtained visual-language model 180 may subsequently be fine-tuned for the specific task to which the visual-language model 180 will be applied.

[0035] The specific tasks to which the visual-language model 180 is applied are also referred to herein as “downstream tasks.” Downstream tasks may include, but are not limited to, image retrieval, text retrieval, VQA, NLVR, visual entailment (VE), generating text from images, generating images from text, and the like.

[0036] In some implementations, training the vision-language model 180 at the computing device 100 can be fine-tuning a pre-trained model for a downstream task. In some implementations, training the vision-language model 180 at the computing device 100 can be directly training an initial version of the vision-language model 180 for a downstream task.

[0037] The computing device 100 uses the image-text pairs in the training dataset 170 to train the visual-language model 180 end-to-end. Through training, the visual-language model 180 can learn the visual representation, language representation, and the association between vision and language from the image-text pairs. Figures 2 to 5 To describe how the vision-language model 180 learns visual representation, language representation, and the relationship between vision and language through training.

[0038] System Architecture

[0039] Figure 2FIG2 shows an architecture diagram 200 for training a vision-language model 180 according to some implementations of the present disclosure. Figure 2 As shown, the visual-language model 180 may generally include a text feature extraction sub-model 210, a visual feature extraction sub-model 220, a visual semantic embedding layer 230, and a fusion sub-model 240. It should be understood that the structure and function of the visual-language model 180 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure. Implementations of the present disclosure may also be implemented in different structures and / or functions.

[0040] The text feature extraction submodel 210 is configured to extract a set of text features from a text (training text in the training phase and input text in the inference phase). Each text feature corresponds to at least one word in the text. For example, the text feature extraction submodel 210 can tokenize the text to obtain tokens of the text. The text feature extraction submodel 210 can be implemented using any suitable framework or network. As an example, the text feature extraction submodel 210 can be implemented using the Bidirectional Encoder Representation from Transformer (BERT).

[0041] like Figure 2 As shown, during the training phase, the text feature extraction sub-model 210 extracts a set of text features 215 of the input text 172. Each text feature in the set of text features 215 corresponds to at least one word in the training text 172. To represent the set of text features 215, and use w i express The i-th embedding vector in . That is, each embedding vector w i Corresponds to at least one word in the training text 172. Figure 2 In the example, Embedding vectors w1 to w in 10 can correspond to the words "a", "yellow", "dog", "meets", "a", "car", "coming", "down", "the", and "road" respectively. It should be understood that the embedding vector w i The number of depends on the length of the training text 172 and the granularity of the text division. In this paper, it is assumed that each embedding vector w i Has dimension c.

[0042] The above describes how the visual-language model 180 processes text in the language domain. In the visual domain, the visual-language model 180 includes a visual feature extraction sub-model 220. The visual feature extraction sub-model 220 is configured to extract a set of visual features from an image (training images during the training phase and input images during the inference phase). Each visual feature corresponds to a pixel block in the image.

[0043] like Figure 2 As shown, during the training phase, the visual feature extraction sub-model 220 extracts a set of visual features 260 of the training image 171 . Each visual feature in the set of visual features 260 corresponds to a pixel block in the training image 171 . Figure 2 , which corresponds to the upper left pixel block of the training image 171, and the visual feature 262 which corresponds to the upper right pixel block of the training image 171. To represent the set of visual features 260, and to To represent the training image 171. Then, the set of visual features 260 can be obtained by the following formula:

[0044]

[0045] where E(·; θ) is the visual feature extraction submodel 220 with parameter θ; i express The i-th eigenvector of The feature vector v included in i The number of eigenvectors; c represents each eigenvector v i The dimension of , which has the same dimension as the text feature 215. In addition, the value of l, i.e. The feature vector v included in i The number of pixels depends on the number of pixel blocks into which the training image 171 is divided.

[0046] The visual feature extraction sub-model 220 can be implemented as a trainable visual feature encoder. During the training of the vision-language model 180, the visual feature extraction sub-model 220 is trained together. Any suitable framework and network can be used to implement the visual feature extraction sub-model 220. For example, a residual network followed by a 1×1 convolutional layer and a 2×2 maximum pooling layer can be used as the structure of the visual feature extraction sub-model 220.

[0047] A trainable visual feature extraction sub-model 220, such as a convolutional neural network (CNN) encoder, is used in the vision-language model 180. The visual feature extraction sub-model 220 takes the entire image as input and generates image-level visual features, rather than the region-level visual features extracted in existing solutions. In this way, the complete visual information in the training image can be preserved. Compared to the existing solutions mentioned above, without the constraints of bounding boxes, the visual feature extraction sub-model 220 can be trained and updated end-to-end, thereby optimizing cross-modal learning.

[0048] The visual features 260 extracted by the visual feature extraction sub-model 220 are more diverse than the text features 215, which will bring difficulties to cross-modal learning. In view of this, the vision-language model 180 also includes a visual semantic dictionary 250, which can also be simply referred to as a visual dictionary (VD). In general, the visual features 260 extracted by the visual feature extraction sub-model 220 are low-level image features, such as color features, shape features, and pattern features. Using the visual semantic dictionary 250, low-level image features with similar visual semantics can be aggregated into the same visual semantics, that is, represented by the same visual semantic features.

[0049] The visual semantic dictionary 250 includes a plurality of candidate semantic features with indices. Figure 2 Schematically shows a candidate semantic feature 251 with an index of 2, a candidate semantic feature 252 with an index of 3, a candidate semantic feature 253 with an index of 4, a candidate semantic feature 254 with an index of 5, and a candidate semantic feature 255 with an index of 9. Figure 3 As described, during the training phase, the visual semantic dictionary 250 is updated as the vision-language model 180 is trained.

[0050] like Figure 2 As shown, a set of visual features 260 extracted by the visual feature extraction sub-model 220 is input to the visual semantic embedding layer 230. The visual semantic embedding layer 230 determines a set of visual semantic features 235 corresponding to the set of visual features 260 based on the visual semantic dictionary 250. It can be understood that since each visual feature in the set of visual features 260 corresponds to a pixel block in the training image 171, each visual semantic feature in the set of visual semantic features 235 also corresponds to a pixel block in the training image 171. In this article, the embedding of the visual semantic dictionary 250 implemented by the visual semantic embedding layer 230 is defined as a mapping function f. Accordingly, it can be used To represent the set of visual semantic features 235. Figure 3 The operation of the visual semantic embedding layer 230 is described in detail.

[0051] Next, the visual-language model 180 is trained based on the text features 215 of the language domain and the visual semantic features 235 of the visual domain. Figure 2 In the example of , for cross-modal processing, the fusion sub-model 240 in the vision-language model 180 is configured to generate a set of fusion features 245 of the training text 172 and the training image 171 based on the visual semantic features 235 and the text features 215. Any suitable framework or network can be used to implement the fusion sub-model 240. For example, a multi-layer transformer can be used to implement the fusion sub-model 240. Such a multi-layer transformer can learn cross-modal representations by fusion of features in the visual domain and features in the language domain. It should be understood that Figure 2 The fused sub-model 240 and fused features 245 shown are merely exemplary. In the implementation of the present disclosure, any suitable sub-model may be utilized to combine text features and visual semantic features.

[0052] During the training phase, an objective function may be determined based on the fused features 245, and the visual-language model 180 may be trained by minimizing the objective function. In some implementations, the training of the visual-language model 180 may be pre-training. In such implementations, the objective function may be determined for one or more general tasks used for pre-training. The general tasks may include determining whether an image and text match, predicting text features to which a mask is applied, predicting visual semantic features to which a mask is applied, etc., as will be referenced below. Figure 4 Descriptive.

[0053] In some implementations, the training of the visual-language model 180 can be fine-tuning or training for a specific downstream task. In such an implementation, the objective function can be determined for the specific downstream task.

[0054] Embedding of visual semantic features

[0055] Figure 3 Schematic diagram of the visual semantic embedding layer 230 according to some implementations of the present disclosure is shown. Figure 3 As shown, the visual semantic embedding layer includes at least a query operation 301 and a mapping operation 302. The query operation 301 determines from the visual semantic dictionary 250 at least one candidate semantic feature to which a set of visual features 260 of the training image 171 is mapped. Specifically, for the visual feature v i , the query operation 301 can determine the visual feature v from the visual semantic dictionary 250 i The most similar candidate semantic features.

[0056] The candidate semantic features to which the set of visual features 260 are mapped can be represented by the index of the candidate semantic features in the visual semantic dictionary 250. In this paper, the visual semantic dictionary 250 is defined as a matrix It contains k c-dimensional embedding vectors. The j-th embedding vector d j represents the jth candidate semantic feature. Then, for the visual feature v i , can be obtained by Search for v i The nearest neighbor is used to calculate the visual feature v i The index of the candidate semantic feature to which it is mapped (also called the "mapping index"):

[0057] h i =argmin j ||v i -d j ||2 (2)

[0058] where h i Represents the i-th visual feature v i The mapping index of .

[0059] exist Figure 3 In the example of , visual features in a set of visual features 260 are respectively mapped to a candidate semantic feature 251 with an index of 2, a candidate semantic feature 252 with an index of 3, a candidate semantic feature 253 with an index of 4, a candidate semantic feature 254 with an index of 5, and a candidate semantic feature 255 with an index of 9. For example, visual feature 261 is mapped to candidate semantic representation 252 with an index of 3; visual feature 362 is mapped to candidate semantic representation 251 with an index of 2; and visual feature 363 is mapped to candidate semantic representation 253 with an index of 5.

[0060] Since each visual feature corresponds to a pixel block in the training image 171 , the candidate semantic feature to which each visual feature is mapped corresponds to the corresponding pixel block. Figure 3 The index of the candidate semantic feature corresponding to each pixel block is schematically shown in FIG. For example, pixel block 321 corresponding to visual feature 261 corresponds to a candidate semantic feature with an index of 3; pixel block 322 corresponding to visual feature 362 corresponds to a candidate semantic feature with an index of 2; and pixel block 323 corresponding to visual feature 363 corresponds to a candidate semantic feature with an index of 5.

[0061] Next, mapping operation 302 generates a set of visual semantic features 235 corresponding to the set of visual features 260 based on the candidate semantic features to which the set of visual features 260 is mapped. Accordingly, each visual semantic feature in the set of visual semantic features 235 is mapped to a candidate semantic feature in the visual semantic dictionary 250. The index of the candidate semantic feature can be considered as the mapping index of the visual semantic feature. For example, visual semantic features 333, 334, and 335 have a mapping index of 5; visual semantic feature 332 has a mapping index of 2; and visual semantic feature 331 has a mapping index of 3.

[0062] In some implementations, for the visual feature v i , the mapping operation 302 can transform the visual feature v i The candidate semantic features mapped to are determined to be the same as the visual features v i Corresponding visual semantic features. For example, the mapping operation 302 may determine the candidate semantic feature 251 with an index of 3 as the visual semantic feature corresponding to the visual feature 261 .

[0063] As mentioned above, the embedding of the visual semantic dictionary 250 implemented by the visual semantic embedding layer 230 can be defined as a mapping function f, and To represent a set of visual semantic features 235. Then, in this implementation, with the visual feature v i The corresponding visual semantic feature f(v i ) is represented by the following formula:

[0064]

[0065] Among them, for the visual feature v i , using the same i The closest embedding vector

[0066] In the implementation of training the vision-language model 180 for fine-tuning for downstream tasks, the visual semantic feature f(v i ). This is due to the fact that in the pre-training stage before fine-tuning, the visual semantic dictionary 250 has learned the representation of visual semantic features.

[0067] In some implementations, mapping operation 302 may update visual semantic dictionary 250 based on set of visual features 260, and determine set of visual semantic features 235 based on updated visual semantic dictionary 250. Specifically, for a candidate semantic feature in visual semantic dictionary 250, mapping operation 302 may determine one or more visual features mapped to the candidate semantic feature from set of visual features 260, and update the candidate semantic feature based on the determined one or more visual features. The updated candidate semantic feature is determined to be the visual semantic feature corresponding to the one or more visual features.

[0068] exist Figure 3 In the example of FIG, the visual features corresponding to the pixel blocks 323, 324, and 325 are mapped to the candidate semantic feature 253 with an index of 5. Accordingly, the candidate semantic feature 253 with an index of 5 is updated based on the visual features corresponding to the pixel blocks 323, 324, and 325. In the set of visual semantic features 235, the visual semantic features 333, 334, and 335 corresponding to the pixel blocks 323, 324, and 325 are determined as the updated candidate semantic feature with an index of 5.

[0069] The following uses f -1 (j) represents the inverse mapping function of f, which maps the index j back to one or more visual features in the set of visual features 260. -1 (j)| represents the size of the inverse mapping group, that is, the number of one or more visual features mapped back by index j. Figure 3 In, |f -1 (2)|、|f -1 (3)|、|f -1 (4)|、|f -1 (5)| and|f -1 The values ​​of (9)| are 4, 8, 2, 3, and 3 respectively.

[0070] As an example, a momentum method can be used to update the visual semantic dictionary 250. The visual semantic dictionary 250 represented by can be randomly initialized and further updated by a moving average operation during a batch or a mini-batch of training. The embedding vector d can be updated according to the following formula j Candidate semantic features represented by:

[0071]

[0072] in represents the updated embedding vector d j ; γ is the momentum coefficient, which can be viewed as the embedding vector d j The value range of γ is [0,1], for example, γ=0.99.

[0073] As shown in formula (4), the embedding vector d j The candidate semantic feature represented by is updated based on the visual features mapped to the candidate semantic feature and their number, as well as the weight for the candidate semantic feature. Note that formula (4) is only valid when |f -1 It can only be applied when |(j)≠ 0. In other words, in a batch or a mini-batch of training, only the candidate semantic features that are mapped to visual features are updated.

[0074] In this implementation, the mapping operation 302 may determine a set of visual semantic features 235 based on the updated candidate semantic features. i The corresponding visual semantic feature f(v i ) is represented by the following formula:

[0075]

[0076] Among them, for the visual feature v i , using the updated embedding vector

[0077] exist Figure 3 In the example of , the visual semantic feature 331 corresponding to the pixel block 321 is determined as the updated candidate semantic feature 252 with an index of 3. The visual semantic feature 332 corresponding to the pixel block 322 is determined as the updated candidate semantic feature 251 with an index of 2.

[0078] During the training of the visual-language model 180 using different batches of image-text pairs, the visual semantic dictionary 250 is also updated. Considering that the argmin operation in equation (2) is not differentiable, the gradient backpropagation will be stopped by the visual semantic dictionary 250. In order to enable the visual feature extraction sub-model 220 to be trained, f(v i ):

[0079]

[0080] where sg[·] is the stopping gradient operator.

[0081] The above describes the operation of the visual semantic embedding layer and the updating of the visual semantic dictionary. The role of updating the visual semantic dictionary 250 can be understood as follows: the visual semantic dictionary 250 performs clustering on visual features based on the similarity of visual features; visual features with similar semantics will be aggregated into the same cluster. In other words, visual features with similar semantics are mapped to the same visual semantic feature, and the index of the visual semantic feature can be regarded as a visual semantic label. The visual semantic dictionary 250 is updated as the visual-language model 180 is trained, and therefore will be affected by the visual-language task used for training (described below). In this way, the candidate semantic features in the visual semantic dictionary 250 (i.e., The embedding vector d in j ) is more suitable for cross-modal understanding and alignment.

[0082] Training mission

[0083] As mentioned above, in some implementations, the training of the visual-language model 180 can be pre-training. In such implementations, the visual-language model 180 can be pre-trained for one or more general tasks. The general tasks can include determining whether an image and text match, predicting masked text features, predicting masked visual semantic features, etc. Figure 4 A schematic diagram 400 illustrating pre-training based on multiple general tasks according to some implementations of the present disclosure is shown. Figure 4 Examples include a masked visual modeling (MVM) task 441 for predicting visual semantic features to which a mask is applied, a masked language modeling (MLM) task 442 for predicting text features to which a mask is applied, and an image-text matching (ITM) task 443 for determining whether an image and text match.

[0084] Before applying the visual semantic features 235 of the visual domain and the text features 215 of the language domain to the fusion sub-model 240, the visual semantic features 235 and the text features 215 are preprocessed. Figure 4 As shown, the concatenation operation 401 will be performed by The text features 215 represented by The visual semantic features 235 represented are concatenated together to form a concatenated feature 405, which serves as the input sequence for cross-modal learning. In addition to the text features 215 and the visual semantic features 235, the concatenation operation 401 adds two special tags to the concatenated feature 405: a tag [CLS] 402 and a tag [SEP] 403. The tag [CLS] 402 indicates a classification position, the output of which can be used for subsequent classification tasks. The tag [SEP] 403 indicates the end of the text feature, that is, it is used to separate the language domain features from the visual domain features.

[0085] For the MLM task 442, the masking layer 412 takes the concatenated features 405 as input and applies a mask (e.g., Figure 4 ) to output the masked concatenation feature 422. If the text feature w i is masked, the masked concatenated feature 422 includes other text features that are not masked And all visual semantic features The fusion sub-model 240 generates a fused feature 432 based on the masked concatenated feature 422. In the case where the fusion sub-model 240 is implemented by a transformer, the fused feature 432 can be regarded as a feature to which attention is applied.

[0086] The MLM task 442 can determine the objective function for the task based on the fused features 432. The MLM task 442 is used to enable the visual-language model 108 to infer language knowledge from visual information and contextual language information. In view of this, the goal of the MLM task 442 is to minimize the log-likelihood based on other text features that are not masked. And all visual semantic features Predict the masked text feature w i Accordingly, the objective function of the MLM task can be expressed as follows:

[0087]

[0088] Where D represents the entire training data set 170 below. That is, the objective function of the MLM task is based on the predicted text features and the actual text features w i Any suitable masking strategy can be adopted for the MLM task, such as the same masking strategy as in BERT.

[0089] For the MVM task 441, the mask layer 411 takes the concatenated features 405 as input and applies a mask (e.g., Figure 4 ) to output the masked concatenated features 421. If the visual semantic feature f(v j ) is masked, the masked concatenated feature 421 includes other visual semantic features that are not masked and all text features The fusion sub-model 240 generates a fused feature 431 based on the masked concatenated feature 421. In the case where the fusion sub-model 240 is implemented by a transformer, the fused feature 431 can be regarded as a feature to which attention is applied.

[0090] The MVM task 441 can determine the objective function for the task based on the fused features 431. The MVM task 441 is used to enable the visual-language model 108 to infer visual knowledge from language information and contextual visual information. In view of this, the goal of the MVM task 441 is to minimize the log-likelihood based on other visual semantic features that are not masked. and all text features To predict the visual semantic features f(v j ). Accordingly, the objective function of the MVM task can be expressed as follows:

[0091]

[0092] Where D represents the entire training dataset 170 below. That is, the objective function of the MVM task is based on the predicted visual semantic features and the actual visual semantic features f(v j ) is determined by the difference between them.

[0093] When the visual semantic feature f(v j ) is masked, the visual semantic feature f(v j ) in the visual semantic dictionary 250. i can be considered as its label. Figure 3 As shown, adjacent visual semantic features may have similar values ​​and therefore share the same mapping index. For example, both visual semantic features 333 and 334 have a mapping index of 5, corresponding to the candidate semantic feature with index 5. This may result in the visual-language model 180 directly selecting the candidate semantic feature f(v j ) for prediction. To prevent this, the visual semantic features to which the mask is applied may correspond to the same candidate semantic feature in the visual semantic dictionary 250. For example, the mask layer 411 may randomly select the mapping index j of any visual semantic feature in the set of visual semantic features 235, and then replace all visual semantic features with the mapping index j with a special tag [MASK]. That is, the mask layer 411 may replace f with the embedding vector corresponding to the tag [MASK]. -1 All embedding vectors in (j).

[0094] For the ITM task 443, the fusion sub-model 240 takes the concatenated features 405 as input and generates fused features 433 based on the concatenated features 405. In the case where the fusion sub-model 240 is implemented by a transformer, the fused features 433 can be regarded as features to which attention is applied.

[0095] ITM task 443 can determine the objective function for the task based on fused features 433. ITM task 443 is used to enable visual-language model 180 to establish a mapping between language domain features and visual domain features to enhance cross-modal matching. In view of this, ITM task 443 can apply a binary classifier φ(·) to the label [CLS] in fused features 433 to predict whether the training image and training text match. Accordingly, the objective function of the ITM task can be expressed as follows:

[0096]

[0097] Where y∈{0,1} indicates whether the image and text match, for example, y=1 indicates a match, and y=0 indicates a mismatch.

[0098] Based on the general tasks described above, the visual feature extraction sub-model 220, the visual semantic embedding layer 230 (especially the visual semantic dictionary 250), the text feature extraction sub-model 210, and the fusion sub-model 240 can be jointly trained end-to-end. Assuming that the objective functions of the above three tasks are assigned equal weights, the complete pre-training objective function can be expressed as follows:

[0099]

[0100] A large dataset can be used to pre-train the vision-language model 180. A vision-language model 180 pre-trained for a general task can learn general representations for vision-language tasks. Therefore, the vision-language model 180 obtained in this way can have broad applicability.

[0101] It should be understood that reference Figure 4 The general tasks described are merely exemplary. In some implementations, the vision-language model 180 may be pre-trained based on more or fewer general tasks. Alternatively or additionally, in some implementations, the weights assigned to the objective functions of different general tasks may be different. For example, the size of the assigned weights may depend on the possible application scenarios of the vision-language model 180.

[0102] The above reference Figure 4 The training tasks in the implementation of pre-training are described. In some implementations, the training of the vision-language model 180 can be fine-tuned or trained for a specific downstream task. In such an implementation, the objective function can be determined for the specific downstream task. The downstream tasks can include but are not limited to image retrieval, text retrieval, VQA, NLVR, VE, text generation from images, image generation from text, etc. In fine-tuning or training for downstream tasks, Figure 4The general tasks shown in can be replaced by downstream tasks. In addition, the preprocessing before inputting the visual semantic features 235 and the text features 215 into the fusion sub-model 240 can also be different depending on the specific downstream tasks.

[0103] Image retrieval and text retrieval are collectively referred to as image-text retrieval below. In fine-tuning or training for image-text retrieval, the training dataset includes both matching image-text pairs and non-matching image-text pairs. In order to enable the visual-language model 180 to predict the correct classification for matching image-text pairs and non-matching image-text pairs, fine-tuning or training for image-text retrieval can be regarded as a binary classification problem. For example, the label [CLS] in the fused features from the fusion sub-model 240 can be used to predict whether the training image and training text match. This is similar to the ITM task described above.

[0104] VQA requires the vision-language model 180 to take an image and a question as input and output an answer. To this end, fine-tuning or training for VQA can be considered a multi-classification problem. For example, the [CLS] tag from the fused features from the fusion sub-model 240 can be used for multi-classification.

[0105] When fine-tuning or training for generating text from images, the training task can be adjusted as follows: Masking is not applied to the visual semantic features of the visual domain, but rather to the text features of the language domain, sequentially according to word order. The masked text features are predicted based on the visual semantic features and the preceding text features to fine-tune or train the visual-language model 180. Similarly, when fine-tuning or training for generating images from text, the text features of the language domain are not masked. Visual features corresponding to each pixel block of the image are predicted one by one based on the text features to fine-tune or train the visual-language model 180.

[0106] The above describes fine-tuning or training for downstream tasks using image-text retrieval, VQA, text generation, and image generation as examples. Figure 4 As described above, in implementations of the present disclosure, the vision-language model 180 may be fine-tuned or trained for various specific downstream tasks.

[0107] By training the visual-language model 180, such as pre-training for general tasks, fine-tuning for downstream tasks, and training for downstream tasks, the visual-language model 180 can learn representations of image features, representations of text features, and alignment between image features and text features. In particular, the visual semantic dictionary 250, which is updated through training, can convert low-level image features with similar visual semantics into visual semantics.

[0108] Figure 5An example of a visual semantic dictionary 250 obtained by training the vision-language model 180 according to some implementations of the present disclosure is shown. As described above, a pixel block of an image may correspond to a candidate semantic feature in the visual semantic dictionary 250. Figure 5 2 shows a plurality of pixel blocks 510 corresponding to candidate semantic features 501 in the visual semantic dictionary 250 obtained through training, and a plurality of pixel blocks 520 corresponding to candidate semantic features 502. Figure 5 It can be seen that the candidate semantic feature 501 reflects the semantic "person", while the candidate semantic feature 502 reflects the semantic "building". It should be understood that Figure 5 The various pixel blocks shown are only for illustrating the visual semantic dictionary 250 and are not intended to limit the scope of the present disclosure.

[0109] Example application scenario

[0110] Figure 6 An architectural diagram 600 illustrating the application of a visual-language model 180 according to some implementations of the present disclosure is shown. The trained visual-language model 180 can be applied to various visual-language tasks for reasoning. Such visual-language tasks may include, but are not limited to, image retrieval, text retrieval, VQA, NLVR, image generation, or text generation.

[0111] like Figure 6 As shown, in the language domain, the trained visual-language model 180 includes a trained text feature extraction sub-model 210. The text feature extraction sub-model 210 extracts a set of text features 615 of the input text 602. Each text feature in the set of text features 615 corresponds to at least one word in the input text 602.

[0112] In the visual domain, the trained vision-language model 180 includes a trained visual feature extraction sub-model 220 and a trained visual semantic dictionary 250. The visual feature extraction sub-model 220 extracts a set of visual features 660 of the input image 601. Each visual feature in the set of visual features 660 corresponds to a pixel block in the input image 601.

[0113] The set of visual features 660 extracted by the trained visual feature extraction sub-model 220 is input to the visual semantic embedding layer 230. The visual semantic embedding layer 230 determines a set of visual semantic features 635 corresponding to the set of visual features 660 based on the visual semantic dictionary 250. The visual semantic dictionary 250 includes a plurality of candidate semantic features with indexes. Specifically, for each visual feature in the set of visual features 660, the visual semantic embedding layer 230 determines the candidate semantic feature that is most similar to the visual feature from the visual semantic dictionary 250 and uses the determined candidate semantic feature as the visual semantic feature corresponding to the visual feature.

[0114] Next, association information 645 between the input text 602 and the input image 601 is determined based on a set of visual semantic features 635 and a set of text features 615. For example, Figure 6 As shown, the trained fusion sub-model 240 can determine association information 645 between the input text 602 and the input image 601 based on a set of visual semantic features 635 and a set of text features 615. The association information can be, for example, visual semantic features and text features to which attention is applied.

[0115] In the image retrieval scenario, input text 602 may be text input or otherwise specified by a user initiating image retrieval, and input image 601 may be a candidate image in an image repository. Association information 645 may be used to determine whether input image 601 matches input text 602.

[0116] In the text search scenario, input image 601 may be an image input or otherwise specified by a user initiating text search, and input text 602 may be a candidate text in a text repository. Association information 645 may be used to determine whether input text 602 matches input image 604.

[0117] In the context of VQA, the input text 602 may be a question, and the input image 601 may be the image to which the question is directed. The associated information 645 may be used to determine the answer to the question from the input image 601 .

[0118] In the context of NLVR, the input text 602 may be a description, and the input image 601 may include a pair of images. The association information 645 may be used to determine whether the description is related to or consistent with the pair of images.

[0119] In the image generation scenario, the input text 602 may be a description of a target image to be generated, and the input image 601 may be a reference image. The association information may be used to determine pixel blocks or regions in the reference image that are related to the description for use in generating the target image.

[0120] In the text generation scenario, the input image 601 may be an image that matches the target text to be generated, and the input text 602 may be a reference text. The association information may be used to determine the text in the reference text that can describe the input image 601 for generating the target text.

[0121] Example Method

[0122] Figure 7Flowchart of method 700 for training a target model according to some implementations of the present disclosure is shown. Method 700 may be implemented by computing device 100, for example, at multimodal processing module 122 in memory 120 of computing device 100.

[0123] like Figure 7 As shown, at box 710, the computing device 100 extracts a set of visual features of the training image based on the visual feature extraction sub-model in the target model. Each visual feature corresponds to a pixel block in the training image. At box 720, the computing device 100 determines a set of visual semantic features corresponding to the set of visual features based on the visual semantic dictionary. At box 730, the computing device 100 extracts a set of text features of the training text corresponding to the training image based on the text feature extraction sub-model in the target model. Each text feature corresponds to at least one word in the training text. At box 740, the computing device 100 trains the target model based on the set of visual semantic features and the set of text features to determine the association information between the input text and the input image.

[0124] In some implementations, determining a set of visual semantic features based on a visual semantic dictionary includes: determining at least one candidate semantic feature to which a set of visual features are mapped from the visual semantic dictionary; updating at least one candidate semantic feature based on the set of visual features; and determining a set of visual semantic features based on the updated at least one candidate semantic feature.

[0125] In some implementations, updating the at least one candidate semantic feature includes: determining, for a specified candidate semantic feature among the at least one candidate semantic feature, at least one visual feature from a set of visual features that is mapped to the specified candidate semantic feature; and updating the specified candidate semantic feature based on the at least one visual feature and a weight for the specified candidate semantic feature. Determining a set of visual semantic features based on the updated at least one candidate semantic feature includes determining the updated specified candidate semantic feature as the visual semantic feature from the set of visual semantic features that corresponds to the at least one visual feature.

[0126] In some implementations, training the target model includes: generating a set of fused features for training text and training images based on a set of visual semantic features and a set of text features according to a fusion sub-model in the target model; determining an objective function based on the set of fused features; and training the target model by minimizing the objective function.

[0127] In some implementations, generating a set of fused features includes: applying a mask to at least one visual semantic feature from the set of visual semantic features; generating the set of fused features according to a fusion sub-model based on the set of textual features and the masked set of visual semantic features. Determining an objective function includes: generating, based on the set of fused features, a predicted feature corresponding to the at least one masked visual semantic feature; and determining the objective function based on a difference between the predicted feature and the at least one visual semantic feature.

[0128] In some implementations, the at least one visual semantic feature corresponds to the same candidate semantic feature in the visual semantic dictionary.

[0129] In some implementations, determining the objective function includes determining the objective function based on a set of fused features and a specific task to which the objective model is to be applied.

[0130] In some implementations, the specific task includes at least one of: image retrieval, text retrieval, visual question answering, visual reasoning of natural language, image generation, or text generation.

[0131] Figure 8 A flowchart of a method 800 for applying a target model according to some implementations of the present disclosure is shown. The method 800 may be implemented by the computing device 100, for example, at the multimodal processing module 122 in the memory 120 of the computing device 100. The method 800 may also be implemented by another computing device different from the computing device 100.

[0132] like Figure 8 As shown, at box 810, the computing device 100 extracts a set of visual features of the input image based on the visual feature extraction submodel in the trained target model. Each visual feature corresponds to a pixel block in the input image. At box 820, the computing device 100 determines a set of visual semantic features corresponding to the set of visual features based on the visual semantic dictionary. At box 830, the computing device 100 extracts a set of text features of the input text corresponding to the input image based on the text feature extraction submodel in the target model. Each text feature corresponds to at least one word in the input text. At box 840, the computing device 100 determines the association information between the input text and the input image based on the set of visual semantic features and the set of text features.

[0133] In some implementations, the association information is used for at least one of the following tasks: image retrieval based on input text, text retrieval based on input image, VQA related to input text and input image, NLVR related to input text and input image, image generation based on input text, or text generation based on input image.

[0134] In an implementation where the association information is used for image retrieval, the input text may be text entered or otherwise specified by a user initiating the image retrieval, and the input image may be a candidate image in an image repository. The association information may be used to determine whether the input image matches the input text.

[0135] In an implementation where the association information is used for text retrieval, the input image may be an image input by a user initiating text retrieval or otherwise specified, and the input text may be a candidate text in a text repository. The association information may be used to determine whether the input text matches the input image.

[0136] In an implementation where context information is used for VQA, the input text may be a question, and the input image may be the image to which the question is directed. The context information may be used to determine the answer to the question from the input image.

[0137] In an implementation where the association information is used for NLVR, the input text may be a description, and the input image may include a pair of images. The association information may be used to determine whether the description is related to or consistent with the pair of images.

[0138] In an implementation where the association information is used for image generation, the input text may be a description of a desired target image, and the input image may be a reference image. The association information may be used to determine pixel blocks or regions in the reference image that are relevant to the description for use in generating the target image.

[0139] In an implementation where association information is used for text generation, the input image may be an image that matches the target text to be generated, and the input text may be a reference text. The association information may be used to determine words in the reference text that can describe the input image for use in generating the target text.

[0140] Example Implementation

[0141] Some example implementations of the present disclosure are listed below.

[0142] In a first aspect, the present disclosure provides a computer-implemented method. The method comprises: extracting a set of visual features of a training image based on a visual feature extraction sub-model in a target model, wherein each visual feature corresponds to a pixel block in the training image; determining a set of visual semantic features corresponding to the set of visual features based on a visual semantic dictionary; extracting a set of text features of a training text corresponding to the training image based on a text feature extraction sub-model in the target model, wherein each text feature corresponds to at least one word in the training text; and training the target model based on the set of visual semantic features and the set of text features to determine association information between input text and an input image.

[0143] In some implementations, determining the set of visual semantic features based on the visual semantic dictionary includes: determining at least one candidate semantic feature to which the set of visual features is mapped from the visual semantic dictionary; updating the at least one candidate semantic feature based on the set of visual features; and determining the set of visual semantic features based on the at least one updated candidate semantic feature.

[0144] In some implementations, updating the at least one candidate semantic feature includes: determining, for a specified candidate semantic feature in the at least one candidate semantic feature, at least one visual feature in the set of visual features that is mapped to the specified candidate semantic feature; and updating the specified candidate semantic feature based on the at least one visual feature and a weight for the specified candidate semantic feature, and wherein determining the set of visual semantic features based on the updated at least one candidate semantic feature includes: determining the updated specified candidate semantic feature as the visual semantic feature in the set of visual semantic features that corresponds to the at least one visual feature.

[0145] In some implementations, training the target model includes: generating a set of fused features for the training text and the training image based on the set of visual semantic features and the set of text features according to a fusion sub-model in the target model; determining an objective function based on the set of fused features; and training the target model by minimizing the objective function.

[0146] In some implementations, generating the set of fused features includes: applying a mask to at least one visual semantic feature in the set of visual semantic features; generating the set of fused features according to the fusion sub-model based on the set of textual features and the masked set of visual semantic features, and wherein determining the objective function includes: generating a predicted feature corresponding to the at least one visual semantic feature to which the mask is applied based on the set of fused features; and determining the objective function based on a difference between the predicted feature and the at least one visual semantic feature.

[0147] In some implementations, the at least one visual semantic feature corresponds to a same candidate semantic feature in the visual semantic lexicon.

[0148] In some implementations, determining the objective function includes determining the objective function based on the set of fused features and a specific task to which the objective model is to be applied.

[0149] In some implementations, the specific task includes at least one of: image retrieval, text retrieval, visual question answering, visual reasoning of natural language, image generation, or text generation.

[0150] In a second aspect, the present disclosure provides a computer-implemented method. The method comprises: extracting a set of visual features of an input image based on a visual feature extraction submodel in a trained target model, wherein each visual feature corresponds to a pixel block in the input image; determining a set of visual semantic features corresponding to the set of visual features based on a visual semantic dictionary; extracting a set of text features of an input text corresponding to the input image based on a text feature extraction submodel in the target model, wherein each text feature corresponds to at least one word in the input text; and determining association information between the input text and the input image based on the set of visual semantic features and the set of text features.

[0151] In some implementations, the association information is used for at least one of the following tasks: image retrieval based on the input text, text retrieval based on the input image, visual question answering related to the input text and the input image, visual reasoning of natural language related to the input text and the input image, image generation based on the input text, or text generation based on the input image.

[0152] In a third aspect, the present disclosure provides an electronic device. The electronic device includes: a processing unit; and a memory coupled to the processing unit and containing instructions stored thereon, wherein when the processing unit executes the instructions, the device performs actions, the actions including: extracting a set of visual features of a training image according to a visual feature extraction submodel in a target model, wherein each visual feature corresponds to a pixel block in the training image; determining a set of visual semantic features corresponding to the set of visual features based on a visual semantic dictionary; extracting a set of text features of a training text corresponding to the training image according to a text feature extraction submodel in the target model, wherein each text feature corresponds to at least one word in the training text; and training the target model based on the set of visual semantic features and the set of text features to determine association information between input text and an input image.

[0153] In some implementations, determining the set of visual semantic features based on the visual semantic dictionary includes: determining at least one candidate semantic feature to which the set of visual features is mapped from the visual semantic dictionary; updating the at least one candidate semantic feature based on the set of visual features; and determining the set of visual semantic features based on the at least one updated candidate semantic feature.

[0154] In some implementations, updating the at least one candidate semantic feature includes: determining, for a specified candidate semantic feature in the at least one candidate semantic feature, at least one visual feature in the set of visual features that is mapped to the specified candidate semantic feature; and updating the specified candidate semantic feature based on the at least one visual feature and a weight for the specified candidate semantic feature, and wherein determining the set of visual semantic features based on the updated at least one candidate semantic feature includes: determining the updated specified candidate semantic feature as the visual semantic feature in the set of visual semantic features that corresponds to the at least one visual feature.

[0155] In some implementations, training the target model includes: generating a set of fused features for the training text and the training image based on the set of visual semantic features and the set of text features according to a fusion sub-model in the target model; determining an objective function based on the set of fused features; and training the target model by minimizing the objective function.

[0156] In some implementations, generating the set of fused features includes: applying a mask to at least one visual semantic feature in the set of visual semantic features; generating the set of fused features according to the fusion sub-model based on the set of textual features and the masked set of visual semantic features, and wherein determining the objective function includes: generating a predicted feature corresponding to the at least one visual semantic feature to which the mask is applied based on the set of fused features; and determining the objective function based on a difference between the predicted feature and the at least one visual semantic feature.

[0157] In some implementations, the at least one visual semantic feature corresponds to a same candidate semantic feature in the visual semantic lexicon.

[0158] In some implementations, determining the objective function includes determining the objective function based on the set of fused features and a specific task to which the objective model is to be applied.

[0159] In some implementations, the specific task includes at least one of: image retrieval, text retrieval, visual question answering, visual reasoning of natural language, image generation, or text generation.

[0160] In a fourth aspect, the present disclosure provides an electronic device. The electronic device includes: a processing unit; and a memory coupled to the processing unit and containing instructions stored thereon, wherein when the instructions are executed by the processing unit, the device performs actions, the actions including: extracting a set of visual features of an input image according to a visual feature extraction submodel in a trained target model, wherein each visual feature corresponds to a pixel block in the input image; determining a set of visual semantic features corresponding to the set of visual features based on a visual semantic dictionary; extracting a set of text features of an input text corresponding to the input image according to a text feature extraction submodel in the target model, wherein each text feature corresponds to at least one word in the input text; and determining association information between the input text and the input image based on the set of visual semantic features and the set of text features.

[0161] In some implementations, the association information is used for at least one of the following tasks: image retrieval based on the input text, text retrieval based on the input image, visual question answering related to the input text and the input image, visual reasoning of natural language related to the input text and the input image, image generation based on the input text, or text generation based on the input image.

[0162] In a fifth aspect, the present disclosure provides a computer program product, which is tangibly stored in a non-transitory computer storage medium and includes machine-executable instructions, which, when executed by a device, cause the device to perform the method of the first aspect.

[0163] In a sixth aspect, the present disclosure provides a computer program product, which is tangibly stored in a non-transitory computer storage medium and includes machine-executable instructions, which, when executed by a device, cause the device to perform the method of the second aspect above.

[0164] In a seventh aspect, the present disclosure provides a computer-readable medium having machine-executable instructions stored thereon, which, when executed by a device, causes the device to perform the method of the first aspect described above.

[0165] In an eighth aspect, the present disclosure provides a computer-readable medium having machine-executable instructions stored thereon, which, when executed by a device, causes the device to perform the method of the second aspect described above.

[0166] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), and the like.

[0167] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0168] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0169] In addition, although each operation is described in a specific order, this should be understood as requiring such operation to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain the desired result. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate implementation can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination mode.

[0170] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A computer-implemented method comprising: extracting a set of visual features of the training image according to the visual feature extraction sub-model in the target model, wherein each visual feature corresponds to a pixel block in the training image; determining, based on a visual semantic dictionary, a set of visual semantic features corresponding to the set of visual features; extracting a set of text features of the training text corresponding to the training image according to the text feature extraction sub-model in the target model, wherein each text feature corresponds to at least one word in the training text; as well as Based on the set of visual semantic features and the set of text features, the target model is trained to determine association information between input text and input image, Wherein training the target model comprises: Based on the set of visual semantic features and the set of text features, generating a set of fused features for the training text and the training image according to a fusion sub-model in the target model; determining an objective function based on the set of fused features; and The target model is trained by minimizing the target function.

2. The method according to claim 1, wherein determining the set of visual semantic features based on the visual semantic dictionary comprises: determining, from the visual semantic dictionary, at least one candidate semantic feature to which the set of visual features is mapped; updating the at least one candidate semantic feature based on the set of visual features; as well as The set of visual semantic features is determined based on the updated at least one candidate semantic feature.

3. The method according to claim 2, Wherein updating the at least one candidate semantic feature comprises: For a specified candidate semantic feature among the at least one candidate semantic feature, determining at least one visual feature among the set of visual features that is mapped to the specified candidate semantic feature; as well as updating the designated candidate semantic feature based on the at least one visual feature and a weight for the designated candidate semantic feature, and Wherein, determining the set of visual semantic features based on the updated at least one candidate semantic feature comprises: The updated designated candidate semantic feature is determined as a visual semantic feature in the set of visual semantic features corresponding to the at least one visual feature.

4. The method according to claim 1, Generating the set of fusion features includes: applying a mask to at least one visual semantic feature of the set of visual semantic features; Based on the set of text features and the masked set of visual semantic features, generating the set of fused features according to the fusion sub-model, and Wherein determining the objective function comprises: generating, based on the set of fused features, a predicted feature corresponding to the at least one visual semantic feature to which the mask is applied; as well as The objective function is determined based on a difference between the predicted feature and the at least one visual semantic feature. The method according to claim 4 , wherein the at least one visual semantic feature corresponds to a same candidate semantic feature in the visual semantic dictionary.

6. The method of claim 1 , wherein determining the objective function comprises: The objective function is determined based on the set of fused features and the specific task to which the objective model is to be applied.

7. The method according to claim 6, wherein the specific task comprises at least one of the following: Image retrieval, Text retrieval, Visual Question Answering, Visual reasoning for natural language, Image generation, or Text generation.

8. A computer-implemented method comprising: Extracting a set of visual features of an input image according to a visual feature extraction sub-model in the trained target model, wherein each visual feature corresponds to a pixel block in the input image; determining, based on a visual semantic dictionary, a set of visual semantic features corresponding to the set of visual features; extracting a set of text features of an input text corresponding to the input image according to a text feature extraction sub-model in the target model, wherein each text feature corresponds to at least one word in the input text; and determining association information between the input text and the input image based on the set of visual semantic features and the set of text features, The target model is trained as follows: Based on a set of visual semantic features and a set of text features, generating a set of fusion features for training text and training images according to a fusion sub-model in the target model; determining an objective function based on the set of fused features; as well as The target model is trained by minimizing the target function.

9. The method according to claim 8, wherein the association information is used for at least one of the following tasks: Image retrieval based on the input text, Text retrieval based on the input image, visual question answering related to the input text and the input image, visual reasoning of natural language associated with the input text and the input image, generating an image based on the input text, or Text is generated based on the input image.

10. An electronic device comprising: processing unit; as well as a memory coupled to the processing unit and containing instructions stored thereon, the instructions, when executed by the processing unit, causing the apparatus to perform the following actions: extracting a set of visual features of the training image according to the visual feature extraction sub-model in the target model, wherein each visual feature corresponds to a pixel block in the training image; determining, based on a visual semantic dictionary, a set of visual semantic features corresponding to the set of visual features; extracting a set of text features of the training text corresponding to the training image according to the text feature extraction sub-model in the target model, wherein each text feature corresponds to at least one word in the training text; as well as Based on the set of visual semantic features and the set of text features, the target model is trained to determine association information between input text and input image, Wherein training the target model comprises: Based on the set of visual semantic features and the set of text features, generating a set of fused features for the training text and the training image according to a fusion sub-model in the target model; determining an objective function based on the set of fused features; and The target model is trained by minimizing the target function.

11. The apparatus of claim 10, wherein determining the set of visual semantic features based on the visual semantic dictionary comprises: determining, from the visual semantic dictionary, at least one candidate semantic feature to which the set of visual features is mapped; updating the at least one candidate semantic feature based on the set of visual features; as well as The set of visual semantic features is determined based on the updated at least one candidate semantic feature.

12. The device according to claim 11, Wherein updating the at least one candidate semantic feature comprises: For a specified candidate semantic feature among the at least one candidate semantic feature, determining at least one visual feature among the set of visual features that is mapped to the specified candidate semantic feature; as well as updating the designated candidate semantic feature based on the at least one visual feature and a weight for the designated candidate semantic feature, and Wherein, determining the set of visual semantic features based on the updated at least one candidate semantic feature comprises: The updated designated candidate semantic feature is determined as a visual semantic feature in the set of visual semantic features corresponding to the at least one visual feature.

13. The apparatus according to claim 10, Generating the set of fusion features includes: applying a mask to at least one visual semantic feature of the set of visual semantic features; Based on the set of text features and the masked set of visual semantic features, generating the set of fused features according to the fusion sub-model, and Wherein determining the objective function comprises: generating, based on the set of fused features, a predicted feature corresponding to the at least one visual semantic feature to which the mask is applied; as well as The objective function is determined based on a difference between the predicted feature and the at least one visual semantic feature. The apparatus of claim 13 , wherein the at least one visual semantic feature corresponds to a same candidate semantic feature in the visual semantic dictionary.

15. The apparatus of claim 10, wherein determining the objective function comprises: The objective function is determined based on the set of fused features and the specific task to which the objective model is to be applied.

16. The apparatus according to claim 15, wherein the specific task comprises at least one of the following: Image retrieval, Text retrieval, Visual Question Answering, Visual reasoning for natural language, Image generation, or Text generation.

17. An electronic device comprising: processing unit; as well as a memory coupled to the processing unit and containing instructions stored thereon, the instructions, when executed by the processing unit, causing the apparatus to perform the following actions: Extracting a set of visual features of an input image according to a visual feature extraction sub-model in the trained target model, wherein each visual feature corresponds to a pixel block in the input image; determining, based on a visual semantic dictionary, a set of visual semantic features corresponding to the set of visual features; extracting a set of text features of an input text corresponding to the input image according to a text feature extraction sub-model in the target model, wherein each text feature corresponds to at least one word in the input text; and determining association information between the input text and the input image based on the set of visual semantic features and the set of text features, The target model is trained as follows: Based on a set of visual semantic features and a set of text features, generating a set of fusion features for training text and training images according to a fusion sub-model in the target model; determining an objective function based on the set of fused features; as well as The target model is trained by minimizing the target function.

18. The apparatus according to claim 17, wherein the association information is used for at least one of the following tasks: Image retrieval based on the input text, Text retrieval based on the input image, visual question answering related to the input text and the input image, visual reasoning of natural language associated with the input text and the input image, generating an image based on the input text, or Text is generated based on the input image.

19. A computer program product comprising machine-executable instructions that, when executed by a device, cause the device to perform actions comprising: extracting a set of visual features of the training image according to the visual feature extraction sub-model in the target model, wherein each visual feature corresponds to a pixel block in the training image; determining, based on a visual semantic dictionary, a set of visual semantic features corresponding to the set of visual features; extracting a set of text features of the training text corresponding to the training image according to the text feature extraction sub-model in the target model, wherein each text feature corresponds to at least one word in the training text; as well as Based on the set of visual semantic features and the set of text features, the target model is trained to determine association information between input text and input image, Wherein training the target model comprises: Based on the set of visual semantic features and the set of text features, generating a set of fused features for the training text and the training image according to a fusion sub-model in the target model; determining an objective function based on the set of fused features; and The target model is trained by minimizing the target function.

20. The computer program product of claim 19, wherein determining the set of visual semantic features based on the visual semantic dictionary comprises: determining, from the visual semantic dictionary, at least one candidate semantic feature to which the set of visual features is mapped; updating the at least one candidate semantic feature based on the set of visual features; as well as The set of visual semantic features is determined based on the updated at least one candidate semantic feature.

21. The computer program product according to claim 20, Wherein updating the at least one candidate semantic feature comprises: For a specified candidate semantic feature among the at least one candidate semantic feature, determining at least one visual feature among the set of visual features that is mapped to the specified candidate semantic feature; as well as updating the designated candidate semantic feature based on the at least one visual feature and a weight for the designated candidate semantic feature, and Wherein, determining the set of visual semantic features based on the updated at least one candidate semantic feature comprises: The updated designated candidate semantic feature is determined as a visual semantic feature in the set of visual semantic features corresponding to the at least one visual feature.

22. The computer program product according to claim 19, Generating the set of fusion features includes: applying a mask to at least one visual semantic feature of the set of visual semantic features; Based on the set of text features and the masked set of visual semantic features, generating the set of fused features according to the fusion sub-model, and Wherein determining the objective function comprises: generating, based on the set of fused features, a predicted feature corresponding to the at least one visual semantic feature to which the mask is applied; as well as The objective function is determined based on a difference between the predicted feature and the at least one visual semantic feature.

23. A computer program product comprising machine-executable instructions that, when executed by a device, cause the device to perform actions comprising: Extracting a set of visual features of an input image according to a visual feature extraction sub-model in the trained target model, wherein each visual feature corresponds to a pixel block in the input image; determining, based on a visual semantic dictionary, a set of visual semantic features corresponding to the set of visual features; extracting a set of text features of an input text corresponding to the input image according to a text feature extraction sub-model in the target model, wherein each text feature corresponds to at least one word in the input text; and determining association information between the input text and the input image based on the set of visual semantic features and the set of text features, The target model is trained as follows: Based on a set of visual semantic features and a set of text features, generating a set of fusion features for training text and training images according to a fusion sub-model in the target model; determining an objective function based on the set of fused features; as well as The target model is trained by minimizing the target function.

24. The computer program product of claim 23, wherein the association information is used for at least one of the following tasks: Image retrieval based on the input text, Text retrieval based on the input image, visual question answering related to the input text and the input image, visual reasoning of natural language associated with the input text and the input image, generating an image based on the input text, or Text is generated based on the input image.

Images