Cross-modal feature fusion model training method, fusion method, device and equipment

Abstract

The present disclosure relates to a cross-modal feature fusion model training method, a fusion method, an apparatus and a device. Through training the cross-modal feature fusion model to be trained, the cross-modal feature fusion model is obtained, and then when the image-text pair is aligned and fused based on the cross-modal feature fusion model, the accuracy of the image-text alignment and fusion is improved, the inference speed of the image-text alignment and fusion is improved, that is, the visual and language task accuracy is improved, and the visual and language task inference speed is improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a cross-modal feature fusion model training method, fusion method, apparatus and device. Background Technology

[0002] The vision and language (VL) task tests a system's ability to understand and reason about the semantics of the visual world with the help of natural language. Vision is part of human perception, and language is part of human communication; therefore, the efficient and effective use of information from both visual and auditory modalities is a crucial factor in model optimization.

[0003] In existing technologies, dual encoders and fusion encoders are commonly used to process visual and language tasks. However, dual encoders result in low accuracy for visual and language tasks due to the shallow interaction between images and text; fusion encoders require joint encoding of all image-text pairs, which slows down the inference speed for visual and language tasks. Summary of the Invention

[0004] To address the aforementioned technical problems, or at least partially address them, this disclosure provides a cross-modal feature fusion model training method, fusion method, apparatus, and device to improve the accuracy of visual and language tasks and enhance the inference speed of visual and language tasks.

[0005] In a first aspect, embodiments of this disclosure provide a method for training a cross-modal feature fusion model, including:

[0006] Obtain multiple image-text pairs from the target dataset, wherein each image-text pair contains an image and text;

[0007] For each image-text pair, the first vector of the image and the second vector of the text are encoded using a cross-modal feature fusion model to be trained. The image and the text are then masked according to a masking strategy, and the third vector of the masked image and the fourth vector of the masked text are encoded.

[0008] The fifth vector is encoded based on the first vector and the fourth vector, and the sixth vector is encoded based on the third vector and the second vector;

[0009] Calculate the loss value based on the first vector, the second vector, the third vector, the fourth vector, the fifth vector, the sixth vector, and a preset loss function;

[0010] The cross-modal feature fusion model to be trained is trained based on the loss value.

[0011] Secondly, embodiments of this disclosure provide a fusion method, including:

[0012] Acquire multiple image-text pairs, wherein each image-text pair contains an image and text;

[0013] The multiple image-text pairs are input into a cross-modal feature fusion model, which outputs a first vector of the image and a second vector of the text. The cross-modal feature fusion model is trained using the training method described in the first aspect above.

[0014] The image and the text are determined to be fused based on the fact that the cosine similarity between the first vector and the second vector is greater than a fourth preset value.

[0015] Thirdly, embodiments of this disclosure provide a cross-modal feature fusion model training apparatus, comprising:

[0016] The first acquisition module is used to acquire multiple image-text pairs in the target dataset, wherein the image-text pairs contain images and text;

[0017] The first encoding module is used to encode a first vector of the image and a second vector of the text for each image-text pair using a cross-modal feature fusion model to be trained, to mask the image and the text according to a masking strategy, and to encode a third vector of the masked image and a fourth vector of the masked text.

[0018] The second encoding module is used to encode a fifth vector based on the first vector and the fourth vector, and to encode a sixth vector based on the third vector and the second vector.

[0019] The calculation module is used to calculate the loss value based on the first vector, the second vector, the third vector, the fourth vector, the fifth vector, the sixth vector, and a preset loss function;

[0020] The training module is used to train the cross-modal feature fusion model to be trained based on the loss value.

[0021] Fourthly, embodiments of this disclosure provide a fusion apparatus, including:

[0022] The second acquisition module is used to acquire multiple image-text pairs, wherein the image-text pairs contain images and text;

[0023] The output module is used to input the multiple image-text pairs into a cross-modal feature fusion model, so that the cross-modal feature fusion model outputs a first vector of the image and a second vector of the text, wherein the cross-modal feature fusion model is trained by the training method described in the first aspect above;

[0024] The determination module is used to determine the fusion of the image and the text based on the fact that the cosine similarity between the first vector and the second vector is greater than a fourth preset value.

[0025] Fifthly, embodiments of this disclosure provide an electronic device, including:

[0026] Memory;

[0027] Processor; and

[0028] Computer programs;

[0029] The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in the first aspect.

[0030] In a sixth aspect, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method described in the first aspect.

[0031] In a seventh aspect, embodiments of this disclosure also provide a computer program product comprising a computer program or instructions that, when executed by a processor, implement the method described in the first aspect.

[0032] The cross-modal feature fusion model training method, fusion method, apparatus, and device provided in this disclosure acquire multiple image-text pairs from a target dataset, where each image-text pair contains an image and text. For each image-text pair, the cross-modal feature fusion model to be trained encodes a first vector of the image and a second vector of the text. The image and text are then masked according to a masking strategy, and a third vector of the masked image and a fourth vector of the masked text are encoded. A fifth vector is encoded based on the first and fourth vectors, and a sixth vector is encoded based on the third and second vectors. A loss value is calculated based on the first, second, third, fourth, fifth, and sixth vectors and a preset loss function. The cross-modal feature fusion model to be trained is then trained based on the loss value. Because the image-text-image-text encoding within the image-text pair is considered during the training process, the accuracy of image-text alignment and fusion is improved, and the inference speed of image-text alignment and fusion is increased when performing alignment and fusion of image-text pairs based on the cross-modal feature fusion model. This improves the accuracy and inference speed of visual and language tasks. Attached Figure Description

[0033] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0034] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0035] Figure 1 A flowchart of the cross-modal feature fusion model training method provided in this embodiment of the disclosure;

[0036] Figure 2 A schematic diagram of a cross-modal feature fusion model provided in an embodiment of this disclosure;

[0037] Figure 3 A schematic diagram of the occlusion network model and the reconstruction network model provided in the embodiments of this disclosure;

[0038] Figure 4 A flowchart of the fusion method provided in this embodiment of the disclosure;

[0039] Figure 5 This is a schematic diagram of the structure of the cross-modal feature fusion model training device provided in the embodiments of this disclosure;

[0040] Figure 6 A schematic diagram of the fusion device provided in the embodiments of this disclosure;

[0041] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0042] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0043] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0044] Visual and language (VL) tasks test a system's ability to understand and reason about the semantics of the visual world with the help of natural language. Since vision is part of human perception and language is part of human communication, the efficient and effective use of information from both visual and auditory modalities is a crucial factor in model optimization.

[0045] In existing technologies, dual encoders and fusion encoders are commonly used to process visual and language tasks. However, dual encoders suffer from low accuracy in visual and language tasks due to shallow interactions between images and text; fusion encoders require joint encoding of all image-text pairs, resulting in slow inference speeds for visual and language tasks. To address this issue, this disclosure provides a cross-modal feature fusion model training method, which is described below with reference to specific embodiments.

[0046] Figure 1 This is a flowchart illustrating a cross-modal feature fusion model training method provided in this embodiment. The method can be executed by a cross-modal feature fusion model training device, which can be implemented in software and / or hardware. This device can be configured in an electronic device, such as a server or terminal, where the terminal specifically includes a mobile phone, computer, or tablet computer. Furthermore, this method can be applied to various scenarios involving cross-modal feature fusion model training. It is understood that the cross-modal feature fusion model training method provided in this embodiment can also be applied in other scenarios.

[0047] The following is about Figure 1 The training method for the cross-modal feature fusion model is described below, and the specific steps of this method are as follows:

[0048] S101. Obtain multiple image-text pairs from the target dataset, wherein the image-text pairs contain images and text.

[0049] Retrieve multiple image-text pairs from the target dataset. Each image-text pair contains an image (view, V) and text (test, T). For example, the target dataset has a size of C, and C contains multiple image-text pairs. The following explanation will use the target dataset size of C as an example.

[0050] S102. For each image-text pair, the first vector of the image and the second vector of the text are encoded using the cross-modal feature fusion model to be trained. The image and the text are masked according to the masking strategy, and the third vector of the masked image and the fourth vector of the masked text are encoded.

[0051] For each image-text pair mentioned above, the first vector V of the image is encoded using the cross-modal feature fusion model to be trained. e The second vector T of the encoded text e The image and text are masked according to the masking strategy to obtain the masked image and the masked text. The third vector V of the masked image is encoded. m The fourth vector T of the encoded masked text m .

[0052] Optionally, the cross-modal feature fusion model to be trained can be an image-text processing model or other model that can be used to process images and text and fuse them. This embodiment does not impose any restrictions on this.

[0053] Specifically, the image-text pair (V, T) is input into the cross-modal feature fusion model to be trained, and the image V is encoded to obtain the first vector V. e The image V is masked according to the masking strategy to obtain the masked image. The masked image is then encoded to obtain the third vector V. m It's understandable that an image is a complete sample, while a masked image is an incomplete sample. Similarly, inputting the image-text pair (V, T) into the cross-modal feature fusion model to be trained encodes the text T, resulting in the second vector T. e The text T is masked according to the masking strategy to obtain the masked text. The masked text is then encoded to obtain the fourth vector T. m It is understandable that the text is a complete sample, while the masked text is an incomplete sample.

[0054] S103. Encode the fifth vector according to the first vector and the fourth vector, and encode the sixth vector according to the third vector and the second vector.

[0055] According to the first vector V e and the fourth vector T m Encoding the fifth vector M Vm The fifth vector represents the image-masked text, based on the third vector V. m Second vector T e Encoding the sixth vector M tm The sixth vector represents the masked image-text.

[0056] Specifically, complete samples and incomplete samples are matched to obtain (V m ,T e ) and (V e ,T m According to the first vector V e and the fourth vector T m Encoding the fifth vector, i.e., the pair (V) m ,T e Encode the vector to obtain the fifth vector M. Vm The sixth vector is encoded based on the third and second vectors, that is, for (V) e ,T m Encode the vector to obtain the sixth vector M. tm In the fifth vector M Vm The segmentation flag for image encoding and masked text encoding is M. Vm_cls Similarly, in the sixth vector M... tmThe segmentation flag for image encoding and text encoding after occlusion is M. tm_cls .

[0057] S104. Calculate the loss value based on the first vector, the second vector, the third vector, the fourth vector, the fifth vector, the sixth vector, and a preset loss function.

[0058] Calculate the loss value based on the first vector, second vector, third vector, fourth vector, fifth vector, sixth vector and the preset loss function.

[0059] Optionally, the preset loss function is obtained by adding the image fusion loss function, text fusion loss function, similarity loss function and reconstruction loss function.

[0060] Specifically, the image fusion loss function is L V =L v +L v′ The purpose of this image fusion loss function is to obtain the loss and target values ​​for aligning the three different modules of visual-text-visual-text fusion. v′ The goal is to ensure a consistent fusion pattern between images and text. Among these, L... v L represents the image-to-text loss function. v′ This represents the text-image loss function.

[0061] L v The calculation formula is as follows:

[0062]

[0063]

[0064] Here, τ is a temperature scalar, and the s(,) function represents the dot product similarity, used to measure the similarity between different modes. For example, The middle represents Transformation

[0065] L v′ The calculation formula is as follows:

[0066]

[0067] Similarly, the text fusion loss function is L I =L i +L i′ The purpose of this text fusion loss function is to obtain the loss value and target value for aligning the three different modules of text-visual-text-visual fusion. i′ The goal is to ensure a consistent fusion pattern between text and images. Among them, L iL represents the text-image loss function. i′ This represents the image-text loss function.

[0068] L i The calculation formula is as follows:

[0069]

[0070]

[0071] L i′ The calculation formula is as follows:

[0072]

[0073] Here, τ is a temperature scalar, and s(,) represents the dot product similarity function, which is used to measure the similarity between different modes.

[0074] Specifically, the similarity loss function L rank This constraint and expression, due to the gap in semantic consistency between the masked image and the masked text, enables the cross-modal feature fusion model to maintain fine-grained perceptual capabilities and improve its generalizability. Because (V m ,T e ) and (V e ,T m ) is the corresponding positive sample, therefore, V m and T m It is considered a weakened version of the positive sample.

[0075] L rank The calculation formula is as follows:

[0076]

[0077] Where τ is a temperature scalar, λ is a constant, and the sim(,) function is used to calculate the similarity between two vectors.

[0078] Specifically, the reconstruction loss function L mlm It is a constraint on the loss of image and text reconstruction.

[0079] L mlm The calculation formula is as follows:

[0080]

[0081] in, Let γ represent the masked prediction probability distribution in the i-th sentence of the target dataset C, where γ is a hyperparameter.

[0082] In summary, the preset loss function L = L V +LI +L rank +L mlm According to the first vector V e The second vector T e The third vector V m The fourth vector T m Fifth vector M Vm The sixth vector M tm Calculate the loss value using the preset loss function L.

[0083] S105. Train the cross-modal feature fusion model to be trained based on the loss value.

[0084] After calculating the loss value, the parameters of the cross-modal feature fusion model to be trained are adjusted based on the loss value. The cross-modal feature fusion model to be trained is then trained until it converges, resulting in a well-trained cross-modal feature fusion model.

[0085] This embodiment of the disclosure acquires multiple image-text pairs from a target dataset, where each image-text pair contains an image and text. For each image-text pair, a cross-modal feature fusion model to be trained encodes a first vector of the image and a second vector of the text. The image and text are then masked according to a masking strategy, and a third vector of the masked image and a fourth vector of the masked text are encoded. A fifth vector is encoded based on the first and fourth vectors, and a sixth vector is encoded based on the third and second vectors. A loss value is calculated based on the first, second, third, fourth, fifth, and sixth vectors and a preset loss function. The cross-modal feature fusion model to be trained is then trained based on the loss value. Because the image-text-image-text encoding within the image-text pair is considered during the training process, the accuracy of image-text alignment and fusion is improved, and the inference speed of image-text alignment and fusion is increased when performing alignment and fusion on image-text pairs based on the cross-modal feature fusion model. This improves the accuracy and inference speed of visual and language tasks.

[0086] Based on the above embodiments, obtaining multiple image-text pairs in a target dataset includes: obtaining an original dataset; dividing the original dataset to obtain multiple divided datasets; determining a target dataset based on the multiple divided datasets; and obtaining multiple image-text pairs in the target dataset.

[0087] Obtain the original dataset. For example, the original dataset could be LAION-400M. LAION stands for Large-scale Artificial Intelligence Open Network, and LAION-400M refers to the image-text pair dataset within Large-scale Artificial Intelligence Open Network, which contains 400 million data points. Train a cross-modal feature fusion model using the 400 million data points from the original dataset. Obtain multiple image-text pairs from the target dataset, where each image-text pair contains an image (view, V) and text (test, T).

[0088] This embodiment of the disclosure obtains an original dataset; divides the original dataset into multiple partitioned datasets; determines a target dataset based on the multiple partitioned datasets; and obtains multiple image-text pairs from the target dataset. This enables the entire target dataset to be input into the cross-modal feature fusion model to be trained, and determines the target dataset in multiple rounds. This allows the original dataset to be used to train the cross-modal feature fusion model to be trained, thereby improving the accuracy of the cross-modal feature fusion model.

[0089] In some embodiments, encoding a first vector of the image and a second vector of the text using a cross-modal feature fusion model to be trained includes: encoding the image and the text respectively using a single-modal encoder in the cross-modal feature fusion model to be trained, to obtain a first vector of the image and a second vector of the text.

[0090] Figure 2 This is a schematic diagram of a cross-modal feature fusion model provided in an embodiment of the present disclosure, as shown below. Figure 2 As shown, the image is input into the image unimodal encoder of the cross-modal feature fusion model to be trained. The image is encoded by the image unimodal encoder to obtain the first vector V. e The text is input into the text unimodal encoder of the cross-modal feature fusion model to be trained. The text is then encoded by the text unimodal encoder to obtain the second vector T. e .

[0091] This embodiment of the disclosure encodes images and text separately using a single-modal encoder in a cross-modal feature fusion model to be trained, obtaining a first vector of the image and a second vector of the text. The origin of the first and second vectors is clarified, providing a data foundation for the training of the cross-modal feature fusion model.

[0092] In some embodiments, masking the image and the text according to a masking strategy, and encoding a third vector of the masked image and a fourth vector of the masked text, includes: segmenting the image to obtain multiple image blocks of the same size; masking a first preset number of image blocks in the image according to the masking strategy to obtain a masked image; encoding the masked image to obtain a third vector; masking a second preset number of characters in the text to obtain masked text, wherein the masking strategy includes the second preset number of characters having a text ratio of less than a first preset value; and encoding the masked text to obtain a fourth vector.

[0093] The image is segmented to obtain multiple image blocks. For example, if an image is m*n in length and width, and you want to obtain a*a image blocks, where m and n are multiples of a (i.e., a is divisible by m or n), then the number of image blocks is (m / a)*(n / a). These multiple image blocks are of the same size. A first preset number of image blocks in the image are then masked according to a masking strategy to obtain the masked image. Figure 2 As shown, the masked image is encoded using a single-modal image encoder to obtain the third vector V. m Masking a second preset number of characters in the text yields the masked text. This masking strategy includes ensuring that the percentage of the second preset number of characters relative to the total text is less than a first preset value, which can be 20%. Figure 2 As shown, the masked text is encoded using a text-only encoder to obtain the fourth vector T. m For example, the text could be "A man wearing an earring is cutting another man's hair", and the masked text could be "A *** man is cutting another man's hair". The second preset quantity is 3, and the proportion of the second preset quantity of text to the total text is approximately 16.7%. 16.7% < 20%, that is, 16.7% < the first preset value.

[0094] This disclosure provides a data foundation for training a cross-modal feature fusion model by describing how to obtain the third and fourth vectors.

[0095] In some embodiments, the occlusion strategy includes at least a first stage, a second stage, and a third stage; in the first stage, any image patch in the image is occluded to train the cross-modal feature fusion model to be trained, wherein the area ratio of the any image patch to the image is less than or equal to a second preset value; in the second stage, a third preset number of image patches in the image are occluded to train the cross-modal feature fusion model to be trained, wherein the area ratio of the third preset number of image patches to the image is greater than the second preset value and less than the third preset value; in the third stage, a fourth preset number of image patches in the image are occluded to train the cross-modal feature fusion model to be trained, wherein the area ratio of the fourth preset number of image patches to the image is equal to the third preset value.

[0096] In the early stages of training, the cross-modal feature fusion model has not yet learned the feature encoding well. High-difficulty data can easily cause the cross-modal feature fusion model to learn in the wrong direction. Therefore, in the first stage, it is not necessary to use the most difficult image patches to train the cross-modal feature fusion model.

[0097] The masking strategy includes at least three stages: a first stage, a second stage, and a third stage. These three stages represent a progressively more complex mask generation method, guiding the cross-modal feature fusion model to reconstruct the mask step by step. It can be understood that the masked portion corresponds to the masked image patch in the aforementioned image.

[0098] Specifically, in the first stage, any image patch in the image is masked, and the cross-modal feature fusion model to be trained is trained to obtain the loss value corresponding to each image patch. The area ratio of any image patch to the image is less than or equal to a second preset value. For example, the second preset value can be 25%, that is, the area ratio of the image patch to the image is less than or equal to 25%.

[0099] Specifically, in the second stage, a third preset number of image blocks in the occluded image are masked. This third preset number is greater than 1, and the third preset number of image blocks includes the image block α with the largest loss value. t The image patch with the largest loss value is Train the cross-modal feature fusion model to be trained to obtain the loss value of a third preset number of image patches. The area ratio of the third preset number of image patches to the image is greater than the second preset value but less than the third preset value. For example, the second preset value can be 25% and the third preset value can be 75%, that is, 25% < the area ratio of the third preset number of image patches to the image < 75%.

[0100] Specifically, in the third stage, a fourth preset number of image patches in the image are masked. This fourth preset number is greater than the third preset number. The cross-modal feature fusion model to be trained is trained to obtain the loss value of the fourth preset number of image patches. The area ratio of the fourth preset number of image patches to the image is equal to the third preset value. For example, the third preset value can be 75%, that is, the area ratio of the fourth preset number of image patches to the image is equal to 75%.

[0101] This disclosure, through a detailed description of the masking strategy, clarifies the phased training process of the cross-modal feature fusion model to be trained, making the image masking method more intelligent and improving the accuracy of cross-modal feature fusion model training.

[0102] In some embodiments, encoding a fifth vector based on the first vector and the fourth vector includes: encoding the first vector by a first multimodal encoder in a cross-modal feature fusion model to be trained, and encoding the fourth vector by a second multimodal encoder in a cross-modal feature fusion model to be trained, to obtain the fifth vector;

[0103] Specifically, such as Figure 2 As shown, the first vector V is processed by the encoder of the first multimodal mode in the cross-modal feature fusion model to be trained. e Encoding is performed, and the fourth vector T is encoded by the second multimodal encoder in the cross-modal feature fusion model to be trained. m Encode the vector to obtain the fifth vector M. Vm .

[0104] Accordingly, encoding the sixth vector based on the third vector and the second vector includes: encoding the third vector using the first multimodal encoder in the cross-modal feature fusion model to be trained, and encoding the second vector using the second multimodal encoder in the cross-modal feature fusion model to be trained, to obtain the sixth vector.

[0105] Specifically, such as Figure 2 As shown, the third vector V is processed by the encoder of the first multimodal mode in the cross-modal feature fusion model to be trained. , Encoding is performed, and the second vector T is encoded by the second multimodal encoder in the cross-modal feature fusion model to be trained. e Encode the vector to obtain the sixth vector M. tm .

[0106] Optionally, the first multimodal encoder includes a first self-attention algorithm module, a cross-attention algorithm module, and a first fully connected layer module; the second multimodal encoder includes a second self-attention algorithm module and a second fully connected layer module; wherein the first multimodal encoder and the second multimodal encoder have the same type and number of parameters.

[0107] Specifically, such as Figure 2 As shown, the first multimodal encoder includes a first self-attention algorithm module, a cross-attention algorithm module, and a first fully connected layer module; the second multimodal encoder includes a second self-attention algorithm module and a second fully connected layer module; wherein, the parameters of the first multimodal encoder and the second multimodal encoder are shared, that is, the first multimodal encoder and the second multimodal encoder have the same types and number of parameters, but different parameter values.

[0108] This disclosure provides a data foundation for training a cross-modal feature fusion model by specifically describing the origin of the fifth and sixth vectors.

[0109] Figure 2 This is a schematic diagram of a cross-modal feature fusion model provided in an embodiment of this disclosure, as shown below. Figure 2 The cross-modal feature fusion model shown can be a cross-modal feature fusion model to be trained. This model includes an image input layer for inputting the image from the image-text pair; a text input layer for inputting the text from the image-text pair; a first self-attention algorithm module for extracting features from the first and third vectors; a second self-attention algorithm module for extracting features from the fourth and second vectors; a cross-attention algorithm module for fusing the image and text features obtained by the first and second self-attention algorithm modules respectively; a first fully connected layer module for further feature extraction from the fused features; and a second fully connected layer module for further feature extraction from the text features. The first self-attention algorithm module, the cross-attention algorithm module, and the first fully connected layer module constitute a transformer, i.e., a first multimodal encoder; the second self-attention algorithm module and the second fully connected layer module constitute a transformer, i.e., a second multimodal encoder. The first and second multimodal encoders can share parameters; that is, the first and second multimodal encoders have the same types and number of parameters, but different parameter values.

[0110] Figure 3 This is a schematic diagram of the occlusion network model and the reconstruction network model provided in the embodiments of this disclosure, as shown below. Figure 3As shown, the occlusion network model and the reconstruction network model have the same structure. For example, the occlusion network model could be a teacher network model, and the reconstruction network model could be a student network model. Both the occlusion network model and the reconstruction network model have cross-modal feature fusion models. For an image-text pair, the occlusion rules of the occlusion network model are used to allow the reconstruction network model to reconstruct the occluded image-text. The large loss in the cross-modal feature fusion model indicates that this occlusion method is more challenging.

[0111] Specifically, the above occlusion strategy can be used as occlusion rules for occlusion reconstruction. The overall model includes an occlusion network model and a reconstruction network model. The occlusion network model and the reconstruction network model share the network results. The parameter updates of the occlusion network model are obtained by using the exponential moving average (EMA) of the reconstruction network model. The specific calculation method is as follows:

[0112] v t =β·v t-1 +(1-β)·θ t

[0113] For each iteration of occlusion reconstruction, the image is first segmented into image patches. The loss of each image patch is calculated through the occlusion network model. Then, a portion of the image patches are selected and combined to form an occlusion rule, which is then fed into the reconstruction network model for learning and reconstruction.

[0114] Figure 4 A flowchart of the fusion method provided in the embodiments of this disclosure is shown below. Figure 4 As shown, the method includes the following steps:

[0115] S401. Obtain multiple image-text pairs, wherein the image-text pairs contain images and text.

[0116] Retrieve multiple image-text pairs, each containing an image (view, V) and text (test, T).

[0117] S402. Input the multiple image-text pairs into the cross-modal feature fusion model, so that the cross-modal feature fusion model outputs a first vector of the image and a second vector of the text.

[0118] Multiple image-text pairs are input into a cross-modal feature fusion model, which outputs a first vector of the image and a second vector of the text. This cross-modal feature fusion model is trained using the cross-modal feature fusion model training method described in the above embodiment.

[0119] S403. Based on the fact that the cosine similarity between the first vector and the second vector is greater than a fourth preset value, determine that the image and the text are fused.

[0120] The image and text fusion is determined based on the fact that the cosine similarity between the first and second vectors is greater than the fourth preset value.

[0121] This embodiment of the disclosure determines the fusion of image and text by specifically describing the first vector of the image and the second vector of the text as having a cosine similarity greater than a fourth preset value, thereby achieving the purpose of image and text retrieval.

[0122] Figure 5 This is a schematic diagram of the structure of a cross-modal feature fusion model training device provided in an embodiment of this disclosure. The cross-modal feature fusion model training device can be a terminal as described in the above embodiment, or it can be a component or assembly within the terminal. The cross-modal feature fusion model training device provided in this embodiment can execute the processing flow provided in the embodiments of the cross-modal feature fusion model training method, such as... Figure 5 As shown, the cross-modal feature fusion model training device 50 includes: a first acquisition module 51, a first encoding module 52, a second encoding module 53, a calculation module 54, and a training module 55; wherein, the first acquisition module 51 is used to acquire multiple image-text pairs in the target dataset, the image-text pairs containing images and text; the first encoding module 52 is used to encode a first vector of the image and a second vector of the text for each image-text pair using the cross-modal feature fusion model to be trained, and to encode a third vector of the image and a fourth vector of the text after occlusion according to an occlusion strategy; the second encoding module 53 is used to encode a fifth vector based on the first vector and the fourth vector, and to encode a sixth vector based on the third vector and the second vector; the calculation module 54 is used to calculate a loss value based on the first vector, the second vector, the third vector, the fourth vector, the fifth vector, the sixth vector, and a preset loss function; the training module 55 is used to train the cross-modal feature fusion model to be trained based on the loss value.

[0123] Optionally, the first acquisition module 51 is further configured to acquire an original dataset; divide the original dataset to obtain multiple divided datasets; determine a target dataset based on the multiple divided datasets; and acquire multiple image-text pairs in the target dataset.

[0124] Optionally, the first encoding module 52 is further configured to encode the image and the text respectively using a single-modal encoder in the cross-modal feature fusion model to be trained, to obtain a first vector of the image and a second vector of the text.

[0125] Optionally, the first encoding module 52 is further configured to segment the image to obtain multiple image blocks of the same size; mask a first preset number of image blocks in the image according to a masking strategy to obtain a masked image; encode the masked image to obtain a third vector; mask a second preset number of characters in the text to obtain masked text, wherein the masking strategy includes the second preset number of characters having a character ratio of less than a first preset value; and encode the masked text to obtain a fourth vector.

[0126] Optionally, the occlusion strategy includes at least a first stage, a second stage, and a third stage;

[0127] In the first stage, any image patch in the image is occluded, and the cross-modal feature fusion model to be trained is trained, wherein the area ratio of the any image patch to the image is less than or equal to a second preset value.

[0128] In the second stage, a third preset number of image patches in the image are masked, and the cross-modal feature fusion model to be trained is trained. The area ratio of the third preset number of image patches to the image is greater than a second preset value and less than a third preset value.

[0129] In the third stage, a fourth preset number of image patches in the image are masked, and the cross-modal feature fusion model to be trained is trained. The area ratio of the fourth preset number of image patches to the image is equal to a third preset value.

[0130] Optionally, the second encoding module 53 is further configured to encode the first vector using the first multimodal encoder in the cross-modal feature fusion model to be trained, and to encode the fourth vector using the second multimodal encoder in the cross-modal feature fusion model to be trained, to obtain a fifth vector; and to encode the third vector using the first multimodal encoder in the cross-modal feature fusion model to be trained, and to encode the second vector using the second multimodal encoder in the cross-modal feature fusion model to be trained, to obtain a sixth vector.

[0131] Optionally, the first multimodal encoder includes a first self-attention algorithm module, a cross-attention algorithm module, and a first fully connected layer module; the second multimodal encoder includes a second self-attention algorithm module and a second fully connected layer module; wherein the first multimodal encoder and the second multimodal encoder have the same type and number of parameters.

[0132] Figure 5The cross-modal feature fusion model training device of the embodiment shown can be used to execute the technical solution of the above-described cross-modal feature fusion model training method embodiment. Its implementation principle and technical effect are similar, and will not be repeated here.

[0133] Figure 6 This is a schematic diagram of the structure of the fusion device provided in the embodiments of this disclosure. The fusion device can be a terminal as described in the above embodiments, or it can be a component or assembly within the terminal. The fusion device provided in the embodiments of this disclosure can execute the processing flow provided in the fusion method embodiments, such as... Figure 6 As shown, the fusion device 60 includes: a second acquisition module 61, an output module 62, and a determination module 63; wherein, the second acquisition module 61 is used to acquire multiple image-text pairs, the image-text pairs including images and text; the output module 62 is used to input the multiple image-text pairs into a cross-modal feature fusion model, such that the cross-modal feature fusion model outputs a first vector of the image and a second vector of the text, the cross-modal feature fusion model being trained by the cross-modal feature fusion model training method in the above embodiment; the determination module 63 is used to determine the fusion of the image and the text based on the fact that the cosine similarity between the first vector and the second vector is greater than a fourth preset value.

[0134] Figure 6 The fusion apparatus shown in the embodiment can be used to execute the technical solution of the above-described fusion method embodiment. Its implementation principle and technical effect are similar, and will not be repeated here.

[0135] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. The electronic device can be a terminal as described in the above embodiments. The electronic device provided in this disclosure can execute the processing flow provided in the embodiments of the cross-modal feature fusion model training method, such as... Figure 7 As shown, the electronic device 70 includes: a memory 71, a processor 72, a computer program, and a communication interface 73; wherein the computer program is stored in the memory 71 and is configured to be executed by the processor 72 as described above for training the cross-modal feature fusion model.

[0136] In addition, this disclosure also provides a computer-readable storage medium storing a computer program thereon, which is executed by a processor to implement the cross-modal feature fusion model training method described in the above embodiments.

[0137] Furthermore, this disclosure also provides a computer program product, which includes a computer program or instructions that, when executed by a processor, implement the cross-modal feature fusion model training method described above.

[0138] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0139] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0140] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0141] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0142] Obtain multiple image-text pairs from the target dataset, wherein each image-text pair contains an image and text;

[0143] For each image-text pair, the first vector of the image and the second vector of the text are encoded using a cross-modal feature fusion model to be trained. The image and the text are then masked according to a masking strategy, and the third vector of the masked image and the fourth vector of the masked text are encoded.

[0144] The fifth vector is encoded based on the first vector and the fourth vector, and the sixth vector is encoded based on the third vector and the second vector;

[0145] Calculate the loss value based on the first vector, the second vector, the third vector, the fourth vector, the fifth vector, the sixth vector, and a preset loss function;

[0146] The cross-modal feature fusion model to be trained is trained based on the loss value.

[0147] In addition, the electronic device can also perform other steps in the cross-modal feature fusion model training method described above.

[0148] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0149] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0150] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0151] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0152] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0153] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0154] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for training a cross-modal feature fusion model, characterized in that, The method includes: Obtain multiple image-text pairs from the target dataset, wherein each image-text pair contains an image and text; For each image-text pair, the first vector of the image and the second vector of the text are encoded using a cross-modal feature fusion model to be trained. The image and the text are then masked according to a masking strategy, and the third vector of the masked image and the fourth vector of the masked text are encoded. The fifth vector is encoded based on the first vector and the fourth vector, and the sixth vector is encoded based on the third vector and the second vector; The loss value is calculated based on the first vector, the second vector, the third vector, the fourth vector, the fifth vector, the sixth vector, and a preset loss function, wherein the preset loss function is obtained by adding the image fusion loss function, the text fusion loss function, the similarity loss function, and the reconstruction loss function; The cross-modal feature fusion model to be trained is trained based on the loss value. The formula for calculating the image fusion loss function is as follows: ， in, The image-to-text loss function is represented by... The loss function representing the text-image transition; The calculation formula is as follows: ， The calculation formula is as follows: ， The calculation formula is as follows: ， in, It is a temperature scalar. The function represents the similarity of the dot product, used to measure the similarity between different modalities. For the fifth vector Segmentation markers for image encoding and masked text encoding; The formula for calculating the text fusion loss function is as follows: ， in, The text-image loss function is represented by... The loss function representing the image-text transition; The calculation formula is as follows: ， The calculation formula is as follows: ， The calculation formula is as follows: in, It is a temperature scalar. This represents the dot product similarity function, used to measure the similarity between different modalities. For the sixth vector Segmentation markers for image and text encoding after occlusion; The formula for calculating the similarity loss function is as follows: ， in, It is a temperature scalar. For a constant value, sim The function is used to calculate the similarity between two vectors; The formula for calculating the reconstruction loss function is as follows: ， in, Represents the first element in the target dataset C. The predicted probability distribution that is masked in each sentence. It is a hyperparameter; in, Represents the first vector, Represents the second vector, Represents the third vector, Represents the fourth vector, Represents the fifth vector, This represents the sixth vector.

2. The method according to claim 1, characterized in that, Retrieve multiple image-text pairs from the target dataset, including: Obtain the original dataset; The original dataset is divided into multiple partitioned datasets; Based on the multiple partitioned datasets, the target dataset is determined; Obtain multiple image-text pairs from the target dataset.

3. The method according to claim 1, characterized in that, The first vector of the image and the second vector of the text are encoded using a cross-modal feature fusion model to be trained, including: The image and the text are encoded by the single-modal encoder in the cross-modal feature fusion model to be trained, respectively, to obtain the first vector of the image and the second vector of the text.

4. The method according to claim 1, characterized in that, The image and text are masked according to a masking strategy, and the third vector of the masked image and the fourth vector of the masked text are encoded, including: The image is segmented to obtain multiple image blocks, and the multiple image blocks are of the same size; The first preset number of image blocks in the image are masked according to the masking strategy to obtain the masked image; The masked image is encoded to obtain a third vector; The text is masked by a second preset number of characters to obtain the masked text. The masking strategy includes a ratio of the second preset number of characters to the total text of the text that is less than a first preset value. The masked text is encoded to obtain a fourth vector.

5. The method according to claim 1, characterized in that, The masking strategy includes at least a first phase, a second phase, and a third phase; In the first stage, any image patch in the image is occluded, and the cross-modal feature fusion model to be trained is trained, wherein the area ratio of the any image patch to the image is less than or equal to a second preset value. In the second stage, a third preset number of image patches in the image are masked, and the cross-modal feature fusion model to be trained is trained. The area ratio of the third preset number of image patches to the image is greater than a second preset value and less than a third preset value. In the third stage, a fourth preset number of image patches in the image are masked, and the cross-modal feature fusion model to be trained is trained. The area ratio of the fourth preset number of image patches to the image is equal to a third preset value.

6. The method according to claim 1, characterized in that, Encoding a fifth vector based on the first vector and the fourth vector includes: The first vector is encoded by the first multimodal encoder in the cross-modal feature fusion model to be trained, and the fourth vector is encoded by the second multimodal encoder in the cross-modal feature fusion model to be trained, to obtain the fifth vector; Accordingly, encoding the sixth vector based on the third vector and the second vector includes: The third vector is encoded by the first multimodal encoder in the cross-modal feature fusion model to be trained, and the second vector is encoded by the second multimodal encoder in the cross-modal feature fusion model to be trained, to obtain the sixth vector.

7. The method according to claim 6, characterized in that, The first multimodal encoder includes a first self-attention algorithm module, a cross-attention algorithm module, and a first fully connected layer module; The second multimodal encoder includes a second self-attention algorithm module and a second fully connected layer module; The first multimodal encoder and the second multimodal encoder have the same type and number of parameters.

8. A fusion method, characterized in that, The method includes: Acquire multiple image-text pairs, wherein each image-text pair contains an image and text; The plurality of image-text pairs are input into a cross-modal feature fusion model, such that the cross-modal feature fusion model outputs a first vector of the image and a second vector of the text, wherein the cross-modal feature fusion model is trained by the training method as described in any one of claims 1-7; The image and the text are determined to be fused based on the fact that the cosine similarity between the first vector and the second vector is greater than a fourth preset value.

9. A cross-modal feature fusion model training device, characterized in that, The device includes: The first acquisition module is used to acquire multiple image-text pairs in the target dataset, wherein the image-text pairs contain images and text; The first encoding module is used to encode a first vector of the image and a second vector of the text for each image-text pair using a cross-modal feature fusion model to be trained, to mask the image and the text according to a masking strategy, and to encode a third vector of the masked image and a fourth vector of the masked text. The second encoding module is used to encode a fifth vector based on the first vector and the fourth vector, and to encode a sixth vector based on the third vector and the second vector. The calculation module is used to calculate a loss value based on the first vector, the second vector, the third vector, the fourth vector, the fifth vector, the sixth vector, and a preset loss function, wherein the preset loss function is obtained by adding an image fusion loss function, a text fusion loss function, a similarity loss function, and a reconstruction loss function; The training module is used to train the cross-modal feature fusion model to be trained based on the loss value. The formula for calculating the image fusion loss function is as follows: ， in, The image-to-text loss function is represented by... The loss function representing the text-image transition; The calculation formula is as follows: ， The calculation formula is as follows: ， The calculation formula is as follows: ， in, It is a temperature scalar. The function represents the similarity of the dot product, used to measure the similarity between different modalities. For the fifth vector Segmentation markers for image encoding and masked text encoding; The formula for calculating the text fusion loss function is as follows: ， in, The text-image loss function is represented by... The loss function representing the image-text transition; The calculation formula is as follows: ， The calculation formula is as follows: ， The calculation formula is as follows: in, It is a temperature scalar. This represents the dot product similarity function, used to measure the similarity between different modalities. For the sixth vector Segmentation markers for image and text encoding after occlusion; The formula for calculating the similarity loss function is as follows: ， in, It is a temperature scalar. For a constant value, sim The function is used to calculate the similarity between two vectors; The formula for calculating the reconstruction loss function is as follows: ， in, Represents the first element in the target dataset C. The predicted probability distribution that is masked in each sentence. It is a hyperparameter; in, Represents the first vector, Represents the second vector, Represents the third vector, Represents the fourth vector, Represents the fifth vector, This represents the sixth vector.

10. A fusion device, characterized in that, The device includes: The second acquisition module is used to acquire multiple image-text pairs, wherein the image-text pairs contain images and text; An output module is used to input the plurality of image-text pairs into a cross-modal feature fusion model, such that the cross-modal feature fusion model outputs a first vector of the image and a second vector of the text, wherein the cross-modal feature fusion model is trained by the training method as described in any one of claims 1-7; The determination module is used to determine the fusion of the image and the text based on the fact that the cosine similarity between the first vector and the second vector is greater than a fourth preset value.

11. An electronic device, characterized in that, include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in any one of claims 1-8.

Images