Cross-modal matching method, system and equipment assisted by fused semantic generation

Abstract

The invention discloses a cross-modal matching method, a cross-modal matching system and cross-modal matching equipment assisted by fusion semantic generation, and particularly relates to the field of cross-modal matching, and the cross-modal matching method comprises the following steps: calculating a cross-modal similarity between an initial image and an initial text according to a text feature, an image feature, a text attention feature and an image attention feature; inputting the image features into a text decoder of a Transform model to obtain text generation features corresponding to the image features, and inputting the text features into an image decoder of the Transform model to obtain image generation features corresponding to the text features; utilizing a cross attention mechanism to calculate text modal internal similarity between the text generation features and the text features and image modal internal similarity between the image generation features and the image features; and constructing a matching judgment score based on the cross-modal similarity, the image intra-modal similarity and the text intra-modal similarity, and matching the initial image and the initial text based on the matching judgment score.

Description

Technical Field

[0001] The present invention relates to the field of cross-modal matching, and in particular to a cross-modal matching method, system and device integrated with semantic generation assistance. Background Art

[0002] Image-text matching can jointly analyze and match image and text information, assist decision-making and analysis, and improve information retrieval efficiency, which has important application value in many real-world scenarios. Moreover, the research technology involves multiple fields such as computer vision and natural language processing, which can promote the intersection and integration between these fields and promote the research and application of cross-modal learning.

[0003] With the development of deep learning, researchers are exploring more powerful methods for representing image and text features to improve the performance of image-text matching. For example, methods such as deep convolutional neural networks (CNN) and recurrent neural networks (RNN) have been widely used in the extraction of image and text features and have achieved good results.

[0004] However, this deep learning network based on large-scale training also brings its own problems. First, it lacks interpretability, which means it cannot ensure its effectiveness on specific tasks. Second, image-text matching requires a certain logical reasoning ability, and the model needs to have a more fine-grained understanding of image and text information. Single visual or natural language models often do not work. Third, with the rapid growth of large-scale image and text data, how to efficiently process this data has also become an important issue. In addition, as the complementarity between images and texts is increasingly valued, how to better utilize the semantic association information between images and texts has also become an important direction for image-text matching research. Summary of the invention

[0005] The technical problem to be solved by the present invention is that the existing image-text matching model has a poor understanding of image and text information, the utilization rate of semantic association information between image and text is low, and it is difficult to complete efficient image-text matching. The purpose is to provide a cross-modal matching method, system and device assisted by integrated semantic generation. First, the initial text and the initial image are feature extracted, and the encoding is embedded in the vector space to form text features and image features that can calculate similarity; then, the text-focused features that focus on the image features in the text features, and the image-focused features that focus on the text features in the image features are extracted, and then according to the text features, image features, The text attention features and image attention features calculate the cross-modal similarity between the initial image and the initial text; then the text generation features corresponding to the image features and the image generation features corresponding to the text features are obtained through the text decoder and the image decoder, and the cross-attention mechanism is used to calculate the intra-text modal similarity between the text generation features and the text features, as well as the intra-image modal similarity between the image generation features and the image features, thereby obtaining the similarity in two dimensions, intra-modal and cross-modal, to increase the comprehension of the image and text, thereby assisting the matching process of the image and text, performing data enhancement and data fusion enhancement on the matching mechanism, and improving the matching accuracy.

[0006] The present invention is achieved through the following technical solutions:

[0007] A cross-modal matching method fused with semantic generation assistance, the method comprising:

[0008] Input the initial image and initial text into the Transformer model to obtain feature features and text features;

[0009] Extracting text attention features that focus on image features from text features, and image attention features that focus on text features from image features;

[0010] Calculate the cross-modal similarity between the initial image and the initial text based on the text features, image features, text attention features and image attention features;

[0011] Input the image features into the text decoder of the Transformer model to obtain the text generation features corresponding to the image features, and input the text features into the image decoder of the Transformer model to obtain the image generation features corresponding to the text features;

[0012] The cross-attention mechanism is used to calculate the intra-text modality similarity between text-generated features and text features, and the intra-image modality similarity between image-generated features and image features;

[0013] A matching judgment score is constructed based on cross-modal similarity, intra-image modality similarity and intra-text modality similarity, and the initial image and the initial text are matched based on the matching judgment score.

[0014] Furthermore, the initial image and initial text are input into the Transformer model to obtain feature features and text features, specifically:

[0015] The pre-trained Faster-RCNN model is used to recognize the initial image, and a regional image set constructed by multiple regional images is obtained. The regional image set is input into the encoder of the Transformer model for feature extraction to obtain image features.

[0016] The pre-trained Word2Ve model is used to recognize the initial text to obtain a text vector set represented by multiple text vectors, and the text vector set is input into the encoder in the Transformer model for feature extraction to obtain text features.

[0017] Furthermore, the text attention features that focus on the image features in the text features and the image attention features that focus on the text features in the image features are extracted, specifically:

[0018] Based on the matrix product calculation of text features and image features, the semantic similarity matrix is ​​obtained;

[0019] The semantic similarity matrix and the text feature are multiplied to obtain the text attention feature that pays attention to the image feature;

[0020] The image attention features that focus on text features are obtained by multiplying the semantic similarity matrix and image features.

[0021] Furthermore, a cross-modal similarity between the initial image and the initial text is calculated based on the text features, the image features, the text attention features and the image attention features, wherein the cross-modal similarity includes local similarity, global similarity and global-local similarity.

[0022] Furthermore, the local similarity between the initial image and the initial text is calculated, specifically:

[0023] Calculate the cosine similarity between the image features and the image focus features that focus on the text features to obtain the local similarity of the image;

[0024] Calculate the cosine similarity between the text feature and the text focus feature that focuses on the image feature to obtain the text local similarity;

[0025] The local similarity between the image and the text is combined to obtain the local similarity between the initial image and the initial text.

[0026] Furthermore, the global similarity between the initial image and the initial text is calculated, specifically:

[0027] The image features and the text attention features are concatenated to obtain the global vector representation of the image, and the text features and the image attention features are concatenated to obtain the global vector representation of the text;

[0028] Based on the multi-head self-attention mechanism, the global vector representation of the image and the global vector representation of the text are calculated to obtain the global image features and the global text features, and the global similarity scores of the global image features and the global text features are calculated to obtain the global similarity between the initial image and the initial text.

[0029] Furthermore, the global local similarity between the initial image and the initial text is calculated, specifically:

[0030] The global image features and text features are calculated using the attention mechanism to obtain the global text attention features that pay attention to the image features; the global text features and image features are calculated using the attention mechanism to obtain the global image attention features that pay attention to the text features;

[0031] Calculate the cosine similarity between the global text feature and the global image feature that focuses on the text feature to obtain the global local similarity of the text;

[0032] Calculate the cosine similarity between the global image features and the global text attention features that pay attention to the image features to obtain the global local similarity of the image;

[0033] The global local similarity between the text and the image is combined to obtain the global local similarity between the initial image and the initial text.

[0034] The present invention also provides a cross-modal matching system integrated with semantic generation assistance, which is used in any one of the cross-modal matching methods integrated with semantic generation assistance described above, and the system includes:

[0035] The feature extraction module is used to input the initial image and the initial text into the Transformer model to obtain characteristic features and text features;

[0036] An attention feature extraction module is used to extract text attention features that pay attention to image features in text features, and image attention features that pay attention to text features in image features;

[0037] A cross-modal similarity calculation module, used to calculate the cross-modal similarity between the initial image and the initial text according to the text features, the image features, the text attention features and the image attention features;

[0038] A generation feature extraction module is used to input image features into the text decoder of the Transformer model to obtain text generation features corresponding to the image features, and input text features into the image decoder of the Transformer model to obtain image generation features corresponding to the text features;

[0039] The intra-modality similarity calculation module is used to calculate the intra-modality similarity between text generation features and text features, and the intra-modality similarity between image generation features and image features using a cross-attention mechanism;

[0040] The image-text matching module is used to construct a matching judgment score based on cross-modal similarity, image-modal similarity and text-modal similarity, and to match the initial image and the initial text based on the matching judgment score.

[0041] The present invention also provides a computer device, comprising a system memory and a processor, wherein the system memory stores a computer program, and the processor implements the steps of any one of the above-mentioned methods when executing the computer program.

[0042] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any one of the methods described above are implemented.

[0043] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0044] In the present invention, firstly, feature extraction is performed on the initial text and the initial image, and the encoding is embedded in the vector space to form text features and image features whose similarity can be calculated; then, text attention features that pay attention to image features in the text features, and image attention features that pay attention to text features in the image features are extracted, and then the cross-modal similarity between the initial image and the initial text is calculated based on the text features, image features, text attention features and image attention features; then, the text generation features corresponding to the image features and the image generation features corresponding to the text features are obtained through a text decoder and an image decoder, and the text-modal similarity between the text generation features and the text features, and the image-modal similarity between the image generation features and the image features are calculated using a cross-attention mechanism, thereby obtaining the similarity in two dimensions, intra-modal and cross-modal, to increase the comprehension of the image and text, thereby assisting the matching process of the image and text, performing data enhancement and data fusion enhancement on the matching mechanism, and improving the matching accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without creative work. In the drawings:

[0046] Figure 1 This is a method flow chart of a cross-modal matching method assisted by integrated semantic generation in this embodiment;

[0047] Figure 2 Schematic diagram of the structure of an improved Transformer block in a cross-modal matching method assisted by fusion semantic generation in this embodiment;

[0048] Figure 3 This is a schematic diagram of module connections of a cross-modal matching system assisted by integrated semantic generation in this embodiment;

[0049] Figure 4 It is a structural schematic diagram of a computer device in this embodiment. DETAILED DESCRIPTION

[0050] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0051] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0052] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0053] Example 1

[0054] See also Figure 1 As shown, this embodiment provides a cross-modal matching method assisted by fusion semantic generation, the method comprising:

[0055] S1: Input the initial image and initial text into the Transformer model to obtain feature features and text features, specifically:

[0056] S101: Use the pre-trained Faster-RCNN model to recognize the initial image, obtain a regional image set constructed by multiple regional images, and input the regional image set into the encoder of the Transformer model for feature extraction to obtain image features;

[0057] It should be noted that, in this embodiment, the Faster-RCNN model is an existing image recognition model in the art, and the training process of the Faster-RCNN model is a conventional technical means, which will not be described in detail here.

[0058] Specifically, in this embodiment, for each initial image, the pre-trained bottom-up attention model Faster-RCNN model is first used to identify the initial image, obtain K significant region images, and construct a region image set I = {o 1 ,…,o K}∈R K×2048 , and then a fully connected layer is used to downsample the region image set (F v : R 2048 →R 1024 ), and then the downsampled regional image set is input into the encoder of the Transformer model for feature extraction to obtain image features, which are expressed as:

[0059]

[0060] Among them, R v Represents image features; represents the downsampling operation of the fully connected layer; Represents the feature extraction operations of the image encoder of the Transformer model.

[0061] S102: using the pre-trained Word2Ve model to recognize the initial text, obtaining a text vector set represented by multiple text vectors, and inputting the text vector set into the image encoder in the Transformer model for feature extraction to obtain text features;

[0062] It should be noted that in this embodiment, the Word2Ve model is an existing text model in the art, and the training process of the Word2Ve model is a conventional technical means, which will not be described in detail here. Figure 2 As shown in FIG. 1 , the encoder in the Transformer model adopts an improved Transformer block, which has a structure similar to the Transformer block structure in the prior art, except that the original input is directly connected to the output of each layer of the conventional Transformer block, and a redundant accumulation calculation is completed with each output.

[0063] Specifically, in this embodiment, the initial text is input into the pre-trained Word2Ve model to obtain L text vector representations, and a text vector set T = {w 1 ,…,w L}∈R L×300 , and then use a fully connected layer to upsample the text vector set (F t : R 300 →R 1024 ), the initial text vector is upsampled to a larger vector dimension, and then the upsampled text vector set is input into the text encoder of the Transformer model for feature extraction to obtain text features, which are expressed as:

[0064]

[0065] Among them, R t Represents text features, represents the upsampling operation of the fully connected layer; Represents the feature extraction operations of the text encoder of the Transformer model.

[0066] S2: Extracting text attention features that focus on image features from text features, and image attention features that focus on text features from image features, specifically:

[0067] S201: Obtaining a semantic similarity matrix based on matrix product calculation of text features and image features;

[0068] Specifically, in this embodiment, the attention mechanism is used to perform segment-level matching calculations between image features and text features. First, a semantic similarity matrix is ​​obtained through a matrix multiplication, specifically:

[0069] A=R v ×(R t ) T

[0070] Where A represents the semantic similarity matrix; (R t ) T Represents the transposed text features;

[0071] S202: Calculate the product of the semantic similarity matrix and the text feature to obtain the text focus feature that focuses on the image feature;

[0072] Specifically, in this embodiment, the semantic similarity matrix is ​​calculated through a softmax function and then multiplied by the text feature to obtain the text focus feature (the shape is the same as the image feature) that focuses on the image feature, specifically:

[0073]

[0074] S203: Calculating the product of the semantic similarity matrix and the image feature to obtain the image attention feature that pays attention to the text feature;

[0075] Specifically, in this embodiment, similar to the above text attention feature calculation process, specifically:

[0076]

[0077] Among them, A T represents the transposed semantic similarity matrix.

[0078] S3: Calculate the cross-modal similarity between the initial image and the initial text based on the text features, image features, text attention features, and image attention features, where the cross-modal similarity includes local similarity, global similarity, and global-local similarity, as follows:

[0079] S301: Calculate the local similarity between the initial image and the initial text, specifically:

[0080] The cosine similarity between the image features and the image features that focus on the text features is calculated to obtain the local image similarity; the cosine similarity between the text features and the text features that focus on the image features is calculated to obtain the local text similarity; the image local similarity and the text local similarity are combined to obtain the local similarity between the initial image and the initial text, specifically:

[0081]

[0082] in, Represents the local similarity between the initial image and the initial text; represents the i-th image region attention feature in the image attention feature; Represents the i-th image region feature in the image feature; represents the jth text vector attention feature in the text attention feature; Represents the jth text vector feature in the text feature; Indicates that each image region that pays attention to the text feature pays attention to the feature Both are consistent with the original image region features A cosine similarity is calculated, and then the sum is taken to get the local similarity of the image; Represents the text vector of each image feature that is focused on Both are consistent with the original text vector features A cosine similarity is calculated, and then the sum is taken to get the local similarity of the text.

[0083] S302: Calculate the global similarity between the initial image and the initial text, specifically:

[0084] The image features and the text attention features are concatenated to obtain the global vector representation of the image, and the text features and the image attention features are concatenated to obtain the global vector representation of the text;

[0085] Specifically, in this embodiment, a 2048-dimensional vector is obtained by concatenating image features with text-focused features that focus on image features, and then downsampled to a 1024-dimensional image global vector representation through a fully connected layer; a 2048-dimensional vector is obtained by concatenating text features with image-focused features that focus on text features, and then scaled to a 1024-dimensional text global vector representation through a fully connected layer.

[0086] Based on the multi-head self-attention mechanism, the global vector representation of the image and the global vector representation of the text are calculated to obtain the global image features and the global text features, and the global similarity scores of the global image features and the global text features are calculated to obtain the global similarity between the initial image and the initial text, specifically:

[0087]

[0088] in, Represents the global similarity between the initial image and the initial text; Represents global image features; Represents global text features.

[0089] S303: Calculate the global and local similarity between the initial image and the initial text, specifically:

[0090] The global image features and text features are calculated using the attention mechanism to obtain the global text attention features that pay attention to the image features; the global text features and image features are calculated using the attention mechanism to obtain the global image attention features that pay attention to the text features;

[0091] Specifically, in this embodiment, the image and text are aligned from a global-local perspective. First, the global image features and text features are calculated through the attention mechanism to obtain the global text attention features that focus on the image features. Similarly, the global image attention features that focus on the text features are obtained, which are specifically:

[0092]

[0093] Among them, C t Indicates the global text attention feature that pays attention to the image features; C v Represents the global image attention feature that pays attention to text features; Represents global image features; Represents global text features.

[0094] Calculate the cosine similarity between the global text feature and the global image feature that focuses on the text feature to obtain the global local similarity of the text; calculate the cosine similarity between the global image feature and the global text feature that focuses on the image feature to obtain the global local similarity of the image; combine the global local similarity of the text with the global local similarity of the image to obtain the global local similarity between the initial image and the initial text, specifically:

[0095]

[0096] in, represents the global local similarity between the initial image and the initial text; Indicates the calculation of the cosine similarity between the global text feature and the global image focus feature that focuses on the text feature; It represents the calculation of the cosine similarity between the global image features and the global text attention features that focus on the image features.

[0097] Specifically, the cross-modal similarity between the initial image and the initial text is:

[0098]

[0099] S4: Input the image features into the text decoder of the Transformer model to obtain the text generation features corresponding to the image features, and input the text features into the image decoder of the Transformer model to obtain the image generation features corresponding to the text features;

[0100] It should be noted that, in this embodiment, the text decoder and the image decoder include three layers of conventional Transformer structures, and each sublayer is used for residual connection; the image decoder adopts a three-layer reconstructed Transformer structure. The above structures are all conventional technical means and will not be elaborated here.

[0101] Specifically, in this embodiment, the image features are input into the text decoder of the Transformer model to obtain the text generation features corresponding to the image features, specifically:

[0102]

[0103] Among them, D T Represents the decoding operation of the text decoder of the Transformer model; Represents text generation features;

[0104] Input the text features into the image decoder of the Transformer model to obtain the image generation features corresponding to the text features, specifically:

[0105]

[0106] Among them, D I Represents the decoding operation of the image decoder of the Transformer model; Represents image generation features.

[0107] S5: Use the cross-attention mechanism to calculate the intra-text modality similarity between text-generated features and text features, and the intra-image modality similarity between image-generated features and image features;

[0108] Specifically, in this embodiment, the interaction between the text features and the text generation features is enhanced by N cross attention calculations in sequence, specifically:

[0109]

[0110] Where n represents the number of attention calculations; Both represent the output of the nth attention calculation; represents the fully connected layer;

[0111] Then, average pooling is applied to the output of the final nth attention calculation to produce an abstract text feature and Using two abstract text features and The intra-text modality similarity between the text generation feature and the text feature is calculated, specifically:

[0112]

[0113] Among them, S T represents the intra-modal similarity of text; σ represents the sigmoid function; [;] represents the connection operation; G represents a multi-layer feedforward network; ⊙ represents the Hadamard product.

[0114] Similarly, the image generation features and image features are calculated through n cross-attention calculations to obtain the abstract image features. and Using two abstract image features and The intra-image modality similarity between the image generation features and the image features is calculated, specifically:

[0115]

[0116] Among them, S I Represents the intra-modality similarity of images.

[0117] S6: Construct a matching judgment score based on the cross-modal similarity, the image modality similarity, and the text modality similarity, and match the initial image and the initial text based on the matching judgment score.

[0118] Specifically, in this embodiment, a matching judgment score is constructed based on cross-modal similarity, image modality similarity, and text modality similarity, specifically:

[0119] S=S X +w·(S T +S I )

[0120] Among them, w represents a hyperparameter that balances the two similarity scores;

[0121] Specifically, in this embodiment, first, feature extraction is performed on the initial text and the initial image, and the encoding is embedded in the vector space to form text features and image features whose similarity can be calculated; then, text attention features that focus on image features in the text features, and image attention features that focus on text features in the image features are extracted, and then the cross-modal similarity between the initial image and the initial text is calculated based on the text features, image features, text attention features and image attention features; then, the text generation features corresponding to the image features and the image generation features corresponding to the text features are obtained through the text decoder and the image decoder, and the text-modal similarity between the text generation features and the text features, as well as the image-modal similarity between the image generation features and the image features are calculated using the cross-attention mechanism, thereby obtaining the similarity in two dimensions, intra-modal and cross-modal, to assist the matching process of the image and the text, thereby performing data enhancement and data fusion enhancement on the matching mechanism, and improving the matching accuracy.

[0122] Example 2

[0123] See also Figure 3As shown, this embodiment also provides a cross-modal matching system fused with semantic generation assistance, which is used in any one of the cross-modal matching methods fused with semantic generation assistance described above, and the system includes:

[0124] A feature extraction module 100 is used to input the initial image and the initial text into the Transformer model to obtain characteristic features and text features;

[0125] A focus feature extraction module 200 is used to extract text focus features that focus on image features from text features, and image focus features that focus on text features from image features;

[0126] A cross-modal similarity calculation module 300, used to calculate the cross-modal similarity between the initial image and the initial text according to the text features, the image features, the text focus features and the image focus features;

[0127] A generating feature extraction module 400 is used to input the image features into the text decoder of the Transformer model to obtain the text generating features corresponding to the image features, and input the text features into the image decoder of the Transformer model to obtain the image generating features corresponding to the text features;

[0128] An intra-modality similarity calculation module 500, for calculating the intra-text modality similarity between text generation features and text features, and the intra-image modality similarity between image generation features and image features using a cross-attention mechanism;

[0129] The image-text matching module 600 is used to construct a matching judgment score based on cross-modal similarity, image-modal similarity and text-modal similarity, and to match the initial image and the initial text based on the matching judgment score.

[0130] It should be noted that the modules in the system of Example 2 correspond to the steps in the method of Example 1. The steps in the method of Example 1 have been described in detail in Example 1. In this Example 2, the contents of the modules in the system will not be described in detail.

[0131] Example 3

[0132] See also Figure 4 As shown, this embodiment further provides a computer device, including a system memory 1005 and a processor 1001, wherein the system memory 1005 stores a computer program, and the processor 1001 implements the steps of any of the above methods when executing the computer program.

[0133] It should be noted that the processor 1001 is used to execute the steps in the above method embodiments according to the instructions in the program code. Alternatively, the processor 1001 implements the functions of each module / unit in the above system / device embodiments when executing the computer program.

[0134] Specifically, in this embodiment, the computer program may be divided into one or more modules / units, one or more modules / units are stored in the system memory 1005, and are executed by the processor 1001 to complete the present application. One or more modules / units may be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal device.

[0135] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor 1001 and a system memory 1005. Those skilled in the art will appreciate that this does not constitute a limitation on the terminal device, and may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device may also include an input / output device 1003, a network access device 1002, a bus 1006, etc.

[0136] The processor 1001 may be a central processing unit (CPU), or other general-purpose processors 1001, digital signal processors 1001 (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor 1001 may be a microprocessor or any conventional processor, etc.

[0137] The system memory 1005 may be an internal storage unit of the terminal device, such as a hard disk or memory of the terminal device. The system memory 1005 may also be a storage device 1004 of the terminal device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the terminal device. Further, the system memory 1005 may also include both the internal storage unit of the terminal device and the storage device 1004. The system memory 1005 is used to store computer programs and other programs and data required by the terminal device. The system memory 1005 may also be used to temporarily store data that has been output or is to be output.

[0138] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0139] Example 4

[0140] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the above methods are implemented.

[0141] Among them, the computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, system or device, or any combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk. Random Access Memory (RAM), Read-Only Memory (ROM), Erasable Programmable Read Only Memory (EPROM), registers, hard disks, optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above, or any other form of computer-readable storage medium known in the art.

[0142] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application-specific integrated circuit (ASIC). In an embodiment of the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system, or device.

[0143] Example 5

[0144] This embodiment also provides a computer program product including instructions. When the instructions are executed by a computer device cluster, the computer device cluster executes the method described in Embodiment 1.

[0145] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A cross-modal matching method fused with semantic generation assistance, characterized in that: Methods include: Input the initial image and initial text into the Transformer model to obtain feature features and text features; Extracting text attention features that focus on image features from text features, and image attention features that focus on text features from image features; Calculate the cross-modal similarity between the initial image and the initial text based on the text features, image features, text attention features and image attention features; Input the image features into the text decoder of the Transformer model to obtain the text generation features corresponding to the image features, and input the text features into the image decoder of the Transformer model to obtain the image generation features corresponding to the text features; The cross-attention mechanism is used to calculate the intra-text modality similarity between text-generated features and text features, and the intra-image modality similarity between image-generated features and image features; A matching judgment score is constructed based on cross-modal similarity, intra-image modality similarity and intra-text modality similarity, and the initial image and the initial text are matched based on the matching judgment score.

2. A cross-modal matching method assisted by semantic generation, characterized in that: Input the initial image and initial text into the Transformer model to obtain feature features and text features, specifically: The pre-trained Faster-RCNN model is used to recognize the initial image, and a regional image set constructed by multiple regional images is obtained. The regional image set is input into the encoder of the Transformer model for feature extraction to obtain image features. The pre-trained Word2Ve model is used to recognize the initial text to obtain a text vector set represented by multiple text vectors, and the text vector set is input into the encoder in the Transformer model for feature extraction to obtain text features.

3. According to claim 1, a cross-modal matching method assisted by fusion semantic generation is characterized in that: Extract text attention features that focus on image features in text features, and image attention features that focus on text features in image features, specifically: Based on the matrix product calculation of text features and image features, the semantic similarity matrix is ​​obtained; The semantic similarity matrix and the text feature are multiplied to obtain the text attention feature that pays attention to the image feature; The image attention features that focus on text features are obtained by multiplying the semantic similarity matrix and image features.

4. According to claim 3, a cross-modal matching method assisted by fusion semantic generation is characterized in that: The cross-modal similarity between the initial image and the initial text is calculated based on the text features, the image features, the text attention features and the image attention features, wherein the cross-modal similarity includes local similarity, global similarity and global-local similarity.

5. According to claim 4, a cross-modal matching method assisted by fusion semantic generation is characterized in that: Calculate the local similarity between the initial image and the initial text, specifically: Calculate the cosine similarity between the image features and the image focus features that focus on the text features to obtain the local similarity of the image; Calculate the cosine similarity between the text feature and the text focus feature that focuses on the image feature to obtain the text local similarity; The local similarity between the image and the text is combined to obtain the local similarity between the initial image and the initial text.

6. According to claim 4, a cross-modal matching method assisted by fusion semantic generation is characterized in that: Calculate the global similarity between the initial image and the initial text, specifically: The image features and the text attention features are concatenated to obtain the global vector representation of the image, and the text features and the image attention features are concatenated to obtain the global vector representation of the text; Based on the multi-head self-attention mechanism, the global vector representation of the image and the global vector representation of the text are calculated to obtain the global image features and the global text features, and the global similarity scores of the global image features and the global text features are calculated to obtain the global similarity between the initial image and the initial text.

7. The cross-modal matching method assisted by fusion semantic generation according to claim 6 is characterized in that: Calculate the global local similarity between the initial image and the initial text, specifically: The global image features and text features are calculated using the attention mechanism to obtain the global text attention features that pay attention to the image features; the global text features and image features are calculated using the attention mechanism to obtain the global image attention features that pay attention to the text features; Calculate the cosine similarity between the global text feature and the global image feature that focuses on the text feature to obtain the global local similarity of the text; Calculate the cosine similarity between the global image features and the global text attention features that pay attention to the image features to obtain the global local similarity of the image; The global local similarity between the text and the image is combined to obtain the global local similarity between the initial image and the initial text.

8. A cross-modal matching system integrating semantic generation assistance, characterized in that: The system is used in a cross-modal matching method assisted by fusion semantic generation as described in any one of claims 1 to 7, and the system comprises: The feature extraction module is used to input the initial image and the initial text into the Transformer model to obtain characteristic features and text features; An attention feature extraction module is used to extract text attention features that pay attention to image features in text features, and image attention features that pay attention to text features in image features; A cross-modal similarity calculation module, used to calculate the cross-modal similarity between the initial image and the initial text according to the text features, the image features, the text attention features and the image attention features; A generation feature extraction module is used to input image features into the text decoder of the Transformer model to obtain text generation features corresponding to the image features, and input text features into the image decoder of the Transformer model to obtain image generation features corresponding to the text features; The intra-modality similarity calculation module is used to calculate the intra-modality similarity between text generation features and text features, and the intra-modality similarity between image generation features and image features using a cross-attention mechanism; The image-text matching module is used to construct a matching judgment score based on cross-modal similarity, image-modal similarity and text-modal similarity, and to match the initial image and the initial text based on the matching judgment score.

9. A computer device comprising a system memory and a processor, wherein the system memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images