Image-text matching methods, devices, and storage media

Abstract

This application discloses an image-text matching method, device, and storage medium. The image-text matching method includes: extracting global visual features using the visual coding features of the target image, and extracting global text features using the text coding features of the target text; using the text coding features, generating importance parameters corresponding to each visual feature channel in the global visual features; removing features corresponding to visual feature channels whose importance parameters do not meet the requirements from the global visual features to obtain enhanced visual features; and performing feature matching on the enhanced visual features and the global text features to obtain a matching result for the target image and the target text. This method can improve the accuracy of image-text matching.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, device and storage medium for image-text matching. Background Technology

[0002] With the rapid development of the internet and multimedia technologies, image and text data are exploding in size. Achieving accurate understanding and retrieval of cross-modal information has become a research hotspot in both academia and industry. Image-text matching, as a key technology connecting vision and language, aims to measure the semantic relevance between given image-text pairs to determine whether they match. Image-text matching technology is the cornerstone of cross-modal intelligent applications, demonstrating irreplaceable application value in scenarios such as intelligent search engines (e.g., image-to-text search, text-to-image search), personalized content recommendation, and visual question answering.

[0003] Therefore, how to improve the accuracy of image-text matching has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a method, apparatus, and computer-readable storage medium for image-text matching, which can improve the accuracy of image-text matching.

[0005] To address the aforementioned technical problems, this application provides a method for image-text matching, comprising: extracting global visual features using the visual encoding features of the target image, and extracting global text features using the text encoding features of the target text; generating importance parameters corresponding to each visual feature channel in the global visual features using the text encoding features; removing features corresponding to each visual feature channel in the global visual features whose importance parameters do not meet the requirements, thereby obtaining enhanced visual features; and performing feature matching on the enhanced visual features and the global text features to obtain a matching result for the target image and the target text.

[0006] In one embodiment, the text encoding features include a plurality of first text feature vectors. Using the text encoding features, importance parameters corresponding to each visual feature channel in the global visual features are generated, including: determining the feature weights corresponding to the plurality of first text feature vectors; for each text feature channel of the text encoding features, using the feature weights of the plurality of first text feature vectors to perform a weighted summation of the features of the text feature channels to obtain the global text encoding features; and determining the importance parameters corresponding to each text feature channel in the global text encoding features as the importance parameters corresponding to each visual feature channel.

[0007] In one embodiment, determining the importance parameter corresponding to each text feature channel in the global text encoding feature includes: performing a first normalization operation on the features of each text feature channel in the global text encoding feature; and using the normalized value corresponding to each text feature channel in the global text encoding feature as the importance parameter corresponding to each text feature channel in the global text encoding feature.

[0008] In one embodiment, removing features corresponding to each visual feature channel whose importance parameters do not meet the requirements in the global visual features to obtain enhanced visual features includes: for each visual feature channel, determining the target mask value corresponding to the visual feature channel using the importance parameters of the visual feature channel, and multiplying the feature corresponding to the visual feature channel with the target mask value corresponding to the visual feature channel to obtain enhanced visual features; wherein the target mask value is a first mask value or a second mask value, the first mask value is used to remove features, and the second mask value is used to retain features.

[0009] In one embodiment, determining the target mask value corresponding to a visual feature channel using the importance parameter of the visual feature channel includes: determining the target mask value corresponding to the visual feature channel as a first mask value in response to the importance parameter of the visual feature channel being less than a preset threshold; and determining the target mask value corresponding to the visual feature channel as a second mask value in response to the importance parameter of the visual feature channel being greater than or equal to the preset threshold.

[0010] In one embodiment, the visual coding features and text coding features are obtained by encoding the target image and the target text respectively using an encoder. The encoder includes at least one first coding unit and at least one second coding unit. The first coding unit includes a visual coding network and a text coding network, and the second coding unit includes a hybrid expert network.

[0011] In one embodiment, the encoder includes an n-layer coding unit, which includes two second coding units, and the two second coding units are located at the i-th layer and the n-th layer of the n-layer coding unit, respectively, where i is greater than 1 and less than n, and n is a positive integer.

[0012] And / or, the second coding unit includes a first hybrid expert network and a second hybrid expert network, wherein the first hybrid expert network is used to process the visual features output by the previous coding unit, the second hybrid expert network is used to process the text features output by the previous coding unit, and the network parameters of the first hybrid expert network and the second hybrid expert network are the same.

[0013] In one embodiment, the hybrid expert network includes a routing network, a general expert network, and several specialized expert networks; the routing network is used to determine the specialized expert network that is compatible with the input feature vector from among the several specialized expert networks, wherein the input feature vector is a feature vector of visual features output by the previous encoding unit or a feature vector of text features output by the previous encoding unit; the general expert network is used to extract cross-modal general features of images and text from the input feature vector; and the specialized expert networks are used to extract modality-specific features from the input feature vector.

[0014] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide an electronic device, including a memory and a processor coupled to each other, wherein the memory stores program instructions; and the processor is used to execute the program instructions stored in the memory to implement the above-mentioned image-text matching method.

[0015] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a computer-readable storage medium for storing program instructions that can be executed by a processor to implement the above-mentioned image-text matching method.

[0016] The above scheme addresses the issue that the global visual features of the target image may contain redundant information unrelated to the target text. By utilizing the text encoding features of the target image to generate importance parameters for each visual feature channel in the global visual features, and removing the features corresponding to visual feature channels whose importance parameters do not meet the requirements, the obtained enhanced visual features remove redundant channel information that the target text does not need to focus on, while retaining the effective channel information that the target text needs to focus on. In other words, the obtained enhanced visual features can focus more on the information that the target text needs to focus on, and better fit the semantic requirements of the target text. As a result, the matching results obtained by feature matching using the enhanced visual features of the target image and the global text features of the target text are more accurate, thus improving the accuracy of image-text matching. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating an embodiment of the image-text matching method provided in this application; Figure 2 This is a flowchart illustrating another embodiment of the image-text matching method provided in this application; Figure 3 This is a schematic diagram of the network structure of the encoder provided in this application; Figure 4 This is a schematic diagram of the model structure of the image-text matching model provided in this application; Figure 5 This is a schematic diagram of the framework of an embodiment of the image-text matching device provided in this application; Figure 6This is a schematic diagram of the framework of an embodiment of the electronic device provided in this application; Figure 7 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium provided in this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0019] It should be noted that the term "and / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The terms "first" and "second" in this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. The term "multiple" in this application means at least two, such as two, three, etc. The term "several" in this application means at least two. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0020] In related technologies, the image-text matching stage typically involves first acquiring the global visual features of the target image and the global textual features of the target text, and then directly performing feature matching on these global visual and textual features to obtain the matching result for the target image and target text. However, since the visual features of the target image usually contain redundant information unrelated to the target text, this redundant information can interfere with the calculation of feature matching, reducing the accuracy of image-text matching.

[0021] To address this, this application provides a text-image matching method that removes redundant channel information from global visual features that the target text does not need to focus on before feature matching, thereby improving the accuracy of text-image matching.

[0022] The image-text matching method provided in this application can be executed by a terminal device. For example, the terminal device can be a mobile phone, tablet, computer, in-vehicle terminal, wearable device, etc. Furthermore, the image-text matching method provided in this application can be applied to scenarios such as text-to-image search, image-to-text search, personalized content recommendation, and visual question answering.

[0023] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the image-text matching method provided in this application. Figure 1 As shown, the method includes the following steps: S11: Global visual features are extracted using the visual encoding features of the target image, and global text features are extracted using the text encoding features of the target text.

[0024] In some implementations, in a text-to-image search scenario, the target text is the text entered by the user on the display interface of the terminal device, and the target image is an image obtained from an image library; in an image-to-text search scenario, the target text is the image entered by the user on the display interface of the terminal device, and the target text is the text obtained from a text library.

[0025] The visual encoding features of the target image and the text encoding features of the target text are obtained by encoding the target image and target text using an encoder. Specifically, the encoder includes a visual encoding module and a text encoding module. The visual encoding module encodes the target image to obtain visual encoding features. The text encoding module encodes the target text to obtain text encoding features. Global visual features can represent the global information of the visual encoding features, and global text features can represent the global information of the text encoding features.

[0026] In one embodiment, the visual encoding features of the target image are directly used as global visual features, and the text encoding features of the target text are directly used as global text features.

[0027] In another implementation, both the visual encoding module and the text encoding module are constructed using the Transformer architecture. The cls token (located at the beginning of the visual encoding feature sequence) can be extracted from the visual encoding features as a global visual feature, and the EOS token (located at the end of the text encoding feature sequence) can be extracted from the text encoding features as a global text feature. Since the Transformer architecture aggregates information through an attention mechanism, the cls token and EOS token interact with all other tokens through this mechanism, thereby capturing the semantic information of the entire sequence. Therefore, the cls token and EOS token can represent global information.

[0028] It is understandable that a token refers to the basic unit for information processing in a network model.

[0029] It is understandable that the acquired global visual features include features from several visual feature channels, and the acquired global text features include features from several text feature channels. The number of visual feature channels is the same as the number of text feature channels, meaning that the feature dimensions of the global visual features and the feature dimensions of the global text features are the same in number.

[0030] S12: Utilize text encoding features to generate importance parameters corresponding to each visual feature channel in the global visual features.

[0031] The importance parameters of the visual feature channels reflect the degree of importance of the visual feature channels. In step S12, the importance parameters corresponding to each text feature channel in the text encoding features can be obtained as the importance parameters corresponding to each visual feature channel.

[0032] S13: Remove the features corresponding to each visual feature channel in the global visual features whose importance parameters do not meet the requirements, and obtain the enhanced visual features.

[0033] If the importance parameter of a visual feature channel does not meet the requirements, it indicates that the visual feature channel is not important and is a redundant feature channel, and the feature corresponding to the visual feature channel can be removed; if the importance parameter of a visual feature channel meets the requirements, it indicates that the visual feature channel is relatively important and is a valid feature channel, and the feature corresponding to the visual feature channel can be retained.

[0034] It should be noted that removing the features corresponding to each visual feature channel whose importance parameters do not meet the requirements does not mean erasing each visual feature channel whose importance parameters do not meet the requirements from the global visual features. Each visual feature channel whose importance parameters do not meet the requirements still exists in the global visual features, but it no longer contains feature information.

[0035] In one embodiment, an importance condition can be set, whereby an importance parameter is greater than or equal to an importance threshold. The importance threshold can be set according to actual needs. In this embodiment, for each visual feature channel, if the importance parameter of the visual feature channel meets the importance condition, the feature of that visual feature channel is retained; if the importance parameter of the visual feature channel does not meet the importance condition, the feature of that visual feature channel is removed.

[0036] In another embodiment, a target mask value for each visual feature channel can be determined based on the importance parameter of each visual feature channel. The target mask value is either a first mask value or a second mask value. The first mask value is used to remove features, and the second mask value is used to retain features. Further, for each visual feature channel, the feature corresponding to the visual feature channel is multiplied by the target mask value corresponding to the visual feature channel. This achieves the removal of features corresponding to visual feature channels whose importance parameters do not meet the requirements in the global visual features, and the retention of features corresponding to visual feature channels whose importance parameters meet the requirements in the global visual features.

[0037] It should be noted that in steps S12 and S13, since the importance parameters corresponding to each visual feature channel of the global visual features are generated based on the text encoding features of the target text, the process of filtering redundant feature channels (i.e., visual feature channels whose importance parameters do not meet the requirements) of the global visual features is not a fixed and indiscriminate process, but is dynamically dominated or guided by the semantic information of the target text. That is, when the input target text is different, the importance parameters corresponding to each generated visual feature channel will change accordingly, thereby filtering out different redundant feature channels from the global visual features and removing features from each redundant feature channel.

[0038] S14: Perform feature matching on enhanced visual features and global text features to obtain matching results for the target image and target text.

[0039] Step S14, the step of performing feature matching between the enhanced visual features and the global text features, includes: determining the semantic matching score between the enhanced visual features and the global text features. For example, the cosine similarity between the enhanced visual features and the global text features is determined as the semantic matching score between the enhanced visual features and the global text features.

[0040] Furthermore, the semantic matching score between enhanced visual features and global text features is used as the matching result for the target image and the target text.

[0041] The semantic matching score between enhanced visual features and global text features can be used to determine whether a target image and target text match. For example, if the semantic matching score between enhanced visual features and global text features is greater than or equal to a score threshold, then the target image and target text match; if the semantic matching score between enhanced visual features and global text features is less than the score threshold, then the target image and target text do not match. The similarity threshold can be set according to actual needs.

[0042] In this embodiment, since the global visual features corresponding to the target image may contain redundant information unrelated to the target text, the importance parameters corresponding to each visual feature channel in the global visual features are generated using the text encoding features of the target image. The features corresponding to each visual feature channel whose importance parameters do not meet the requirements are removed. This allows the obtained enhanced visual features to remove redundant channel information that the target text does not need to pay attention to, while retaining the effective channel information that the target text needs to pay attention to. In other words, the obtained enhanced visual features can focus more on the information that the target text needs to pay attention to, and better fit the semantic requirements of the target text. As a result, the matching results obtained by feature matching using the enhanced visual features of the target image and the global text features of the target text are more accurate, thus improving the accuracy of image-text matching.

[0043] Please see Figure 2 , Figure 2 This is a schematic flowchart of another embodiment of the image-text matching method provided in this application. Figure 2 As shown, the method includes the following steps: S21: Obtain the visual encoding features of the target image and the text encoding features of the target text.

[0044] In step S21, the visual encoding features and text encoding features are obtained by encoding the target image and target text respectively using an encoder.

[0045] In one embodiment, the encoder includes at least one first encoding unit, which includes a visual encoding network and a text encoding network.

[0046] In one example, both the visual encoding network and the text encoding network use Transformer networks.

[0047] In other examples, visual coding networks may also use other coding networks that can process images, and text coding networks may also use other coding networks that process text; no specific limitations are made here.

[0048] In this embodiment, the encoder may include only one layer of first coding units. Alternatively, the encoder may include multiple layers of first coding units. It is understood that by designing multiple layers of first coding units, more in-depth information can be extracted, enhancing the expressive power of the output features.

[0049] In this embodiment, the visual coding networks together constitute the visual coding module, and the text coding networks together constitute the text coding module.

[0050] In another embodiment, the encoder, in addition to including at least one first coding unit as described above, further includes at least one second coding unit, the second coding unit comprising a hybrid expert network. In this embodiment, the encoder may include one or more first coding units and one or more second coding units, the number of layers in the first coding unit and the number of layers in the second coding unit can be specifically designed according to actual needs. Furthermore, the first coding unit of the encoder (i.e., the coding unit that processes the target image and target text first) is the first coding unit, and the remaining coding units are either first coding units or second coding units.

[0051] In one example, the network structure of each second coding unit of the encoder is the same.

[0052] In one example, the second coding unit includes only one hybrid expert network, which processes both the visual features output by the previous coding unit and the text features output by the previous coding unit.

[0053] In another example, the second coding unit includes a first hybrid expert network and a second hybrid expert network. The first hybrid expert network processes the visual features output by the previous coding unit, and the second hybrid expert network processes the text features output by the previous coding unit. In this example, the network parameters of the first and second hybrid expert networks in the second coding unit are the same.

[0054] In the two examples above, the preceding coding unit of the second coding unit is the coding unit at the level preceding the second coding unit. Specifically, the preceding coding unit of the second coding unit can be either the first coding unit or the second coding unit.

[0055] To facilitate understanding of the technical effects of embedding a hybrid expert network in the encoder in this embodiment, the hybrid expert network and its principles will be introduced below.

[0056] Hybrid expert networks comprise a routing network and several expert networks configured in parallel. These expert networks are all feedforward neural networks, and while their network structures are identical, their parameters are independent. The expert networks may include a general expert network and several specialized expert networks.

[0057] The routing network is used to determine the appropriate specialized expert network (SERN) from several SERN networks that matches the input feature vector. The visual or textual features output by the pre-encoding unit each consist of multiple feature vectors; the feature vector corresponding to a visual feature can be called a visual token, and the feature vector corresponding to a textual feature can be called a text token. The input feature vector is either the feature vector of the visual feature output by the pre-encoding unit or the feature vector of the textual feature output by the pre-encoding unit.

[0058] General expert networks are used to extract cross-modal general features between images and text from the input feature vector. Specialized expert networks are used to extract modality-specific features from the input feature vector. Specifically, if the input feature vector is a visual token, then visual modality-specific features are extracted, such as spatial layout and target shape; if the input feature vector is a text token, then text modality-specific features are extracted, such as semantic logic and contextual relationships.

[0059] The process of using a hybrid expert network to perform feature processing on the visual features (including several visual tokens) and text features (including several text tokens) output by the pre-encoding unit may include the following steps: Step 1: For each visual token, use the routing network to determine the fit score between the visual token and each dedicated expert network; and for each text token, use the routing network to determine the fit score between the text token and each dedicated expert network.

[0060] Specifically, the routing network first maps each visual token to an adaptation vector through a fully connected layer, and then performs a softmax operation on the adaptation vector of each visual token to obtain the adaptation score of each visual token with each dedicated expert network. Similarly, the adaptation score of each text token with each dedicated expert network can be obtained.

[0061] Step 2: For each visual token, select the highest fit score from the fit scores between the visual token and each dedicated expert network, and assign the visual token to the dedicated expert network corresponding to the highest fit score to extract the modality-specific features of the visual token; and for each text token, select the highest fit score from the fit scores between the text token and each dedicated expert network, and assign the text token to the dedicated expert network corresponding to the highest fit score to extract the modality-specific features of the text token.

[0062] In step two, the top-1 allocation strategy can dynamically assign each visual token and each text token to the most suitable dedicated expert network, achieving adaptive allocation of dedicated computing resources. Furthermore, it enables each dedicated expert network to focus on in-depth optimization of the corresponding type of detailed features, thereby improving the representation accuracy of the modality-specific features of visual tokens and the modality-specific features of text tokens.

[0063] Step 3: For each visual token, input the visual token into a general expert network to extract the cross-modal general features of the visual token; and for each text token, input the text token into a general expert network to extract the cross-modal general features of the text token.

[0064] In step three, the general features of text and images across modalities are extracted through a general expert network, which can make up for the shortcomings of dedicated expert networks in extracting general features.

[0065] After step three, each visual token has a corresponding modality-specific feature and a cross-modal general feature for text and images, and each text token has a corresponding modality-specific feature and a cross-modal general feature for text and images.

[0066] It should be noted that steps two and three can be performed simultaneously.

[0067] Step four: fuse the modality-specific features and the cross-modal features of each visual token to obtain visual features; and fuse the modality-specific features and the cross-modal features of each text token to obtain text features.

[0068] For example, the modality-specific features and the cross-modal features of each visual token are weighted according to their respective weighting coefficients to obtain visual features; and the modality-specific features and the cross-modal features of each text token are weighted according to their respective weighting coefficients to obtain text features.

[0069] In this example, for each visual token, the weighting coefficient corresponding to the modality-specific features of the visual token is the highest fit score among the fit scores between the visual token and each specialized expert network, and the weighting coefficient corresponding to the cross-modal general features of the visual token is a preset value; for each text token, the weighting coefficient corresponding to the modality-specific features of the text token is the highest fit score among the fit scores between the text token and each specialized expert network, and the weighting coefficient corresponding to the cross-modal general features of the text token is a preset value. For example, the preset value is 1, or the preset value can be any other set value.

[0070] The visual and text features obtained through fusion processing contain both accurate modality-specific information and cross-modal universality and generalization capabilities, laying the foundation for subsequent cross-modal alignment of visual and text-encoded features, as well as global visual and global text features. Cross-modal alignment refers to alignment within the same embedding feature space (semantic space).

[0071] Furthermore, after obtaining visual and textual features, the hybrid expert network can output these visual and textual features to the next level encoding unit.

[0072] In this embodiment, the visual features (including several visual tokens) and text features (including several text tokens) output by the pre-encoding unit share the same second encoding unit (a hybrid expert network, or two hybrid expert networks with the same parameters). Since each specialized expert network is good at processing different types of information, according to the feature processing process described in steps one to four above, the routing network can learn the semantic commonalities between visual tokens and text tokens across modalities. Semantically similar visual tokens (such as vectors describing red areas) and text tokens (such as vectors describing the word "red") will be assigned by the routing network to the same specialized expert network for processing. This dynamic routing mechanism enables the visual features (including several visual tokens) and text features (including several text tokens) output by the pre-encoding unit to interact at the token level within the shared second encoding unit, achieving deep alignment of the output visual features and text features in the embedded feature space.

[0073] Therefore, embedding one or more second coding units in the encoder enables the encoder to extract deeply aligned visual coding features and text coding features, thereby obtaining deeply aligned global visual features and global text features, which further improves the accuracy of subsequent image-text matching.

[0074] Meanwhile, the addition of one or more second coding units in the encoder can further enhance the feature representation capabilities of visual and text coding features, thereby improving the feature representation capabilities of global visual and global text features.

[0075] Figure 3 This is a schematic diagram of the network structure of the encoder provided in this application. For example... Figure 3 As shown, the encoder includes n layers of coding units, where n is a positive integer. Each n-layer coding unit includes two layers of second coding units, located at the i-th and n-th layers of the n-layer coding unit, respectively. Here, i is greater than 1 and less than n. For example, if n is 6, i can be 2, 3, 4, or 5. Or, if n is 5, i can be 2, 3, or 4.

[0076] In the n-layer coding unit, all layers except the i-th and n-th layers are first coding units. Furthermore, since the n-th layer is the last layer of the encoder, the visual features output by the second coding unit of the n-th layer correspond to the visual coding features of the target image, and the text features output by the second coding unit of the n-th layer correspond to the text coding features of the target text.

[0077] Since the second coding unit is implemented using a hybrid expert network, embedding the second coding unit in the i-th layer of the encoder enables preliminary cross-modal alignment of the visual and textual features output by the i-th layer second coding unit. Furthermore, by embedding the second coding unit in the n-th layer of the encoder, further deep cross-modal alignment of the visual and textual features output by the n-th layer second coding unit is achieved. In other words, during the extraction of visual and textual coding features using the encoder, two-stage deep alignment is achieved through two layers of second coding units.

[0078] It should be noted that, Figure 3 The encoder structure shown is for illustrative purposes only. In other examples, the encoder may have other structures, which will not be illustrated here.

[0079] S22: Global visual features are obtained by extracting visual encoding features, and global text features are obtained by extracting text encoding features.

[0080] In one embodiment, the cls token is extracted from the visual encoding features as a global visual feature, and the EOS token is extracted from the text encoding features. The cls token and EOS token are described above and will not be repeated here.

[0081] S23: Determine the feature weights corresponding to several first text feature vectors in the text encoding features.

[0082] The text encoding features include several first text feature vectors. The feature weight of each first text feature vector represents its importance; the larger the feature weight, the more important it is to the overall semantics of the target text. Each first text feature vector corresponds to a text token in the text encoding features, and the feature weight of each first text feature vector reflects which information in the target text is critical and which is secondary.

[0083] In one embodiment, each first text feature vector corresponds to several text feature channels, meaning each first text feature vector is multi-dimensional. In this embodiment, determining the feature weights corresponding to the several first text feature vectors in the text encoding features further includes the following steps: Step 1: Perform dimensionality reduction on several first text feature vectors to obtain several second text feature vectors after dimensionality reduction.

[0084] The number of text feature channels in each second text feature vector is 1, that is, the dimension of each second text feature vector is 1.

[0085] For example, several first text feature vectors can be simultaneously input into a linear layer for dimensionality reduction.

[0086] Step 2: Perform a second normalization operation on several second text feature vectors to obtain the feature weights corresponding to each second text feature vector.

[0087] The sum of the feature weights of each second text feature vector is 1.

[0088] For example, the second normalization operation is implemented through the softmax operation. The specific details of the softmax operation can be found in known techniques and will not be explained further here.

[0089] Step 3: Use the feature weights corresponding to each second text feature vector as the feature weights corresponding to each second text feature vector.

[0090] For example, the text encoding features of the acquired target text are represented as follows: Here, L represents the number of first text feature vectors in the text encoding features, and C represents the number of text feature channels in the text encoding features. Each first text feature vector corresponds to C text feature channels, meaning each first text feature channel includes C feature dimensions. After dimensionality reduction processing on the L first text feature vectors, L second text feature vectors are obtained, each with 1 text feature channel. After performing a softmax operation on the L second text feature vectors, L feature weights are obtained, which can be represented as weight vectors. .

[0091] S24: For each text feature channel of the text encoding features, the features of the text feature channels are weighted and summed using the feature weights of several first text feature vectors to obtain the global text encoding features.

[0092] In step S24, the global text encoding features are obtained by weighted summation of the text encoding features along the feature dimension. It can be understood that the calculation process for obtaining the global text encoding features corresponds to the aforementioned weight vector... The transpose is multiplied by the text encoding features.

[0093] The obtained global text encoding features can be represented as It integrates information from all text tokens in the text encoding features, but highlights the contribution of key text tokens, ensuring that the global text encoding features can focus on the core semantic information of the target text, and providing an accurate semantic representation for the subsequent generation of importance parameters corresponding to each text feature channel.

[0094] S25: Determine the importance parameters corresponding to each text feature channel in the global text encoding features, and use them as the importance parameters corresponding to each visual feature channel in the global visual features.

[0095] In step S25, the importance parameters corresponding to each text feature channel in the global text encoding features are determined, which further includes the following steps: Step 1: Perform the first normalization operation on the features of each text feature channel in the global text encoding features.

[0096] For example, the first normalization operation is implemented by the sigmoid operation. The specific details of the sigmoid operation can be found in known techniques and will not be explained further here.

[0097] Step 2: The normalized value corresponding to each text feature channel in the global text encoding feature is used as the importance parameter corresponding to each text feature channel in the global text encoding feature.

[0098] In the global text features, the importance parameters corresponding to each text feature channel are all located in the range (0,1). The closer the importance parameter of a text feature channel is to 1, the higher the importance of the text feature channel; the closer the importance parameter of a text feature channel is to 0, the lower the importance of the text feature channel.

[0099] For example, the importance parameters corresponding to several text feature channels can be expressed as follows: , This is called a soft mask.

[0100] By performing a first normalization operation on the features of each text feature channel in the global text encoding features, and using the normalized values ​​corresponding to each text feature channel in the global text encoding features as the importance parameters corresponding to each text feature channel in the global text encoding features, the numerical scale differences of different text feature channels can be eliminated, so that the importance parameters of each text feature channel can be under the same standard and have comparability.

[0101] Furthermore, after determining the importance parameters corresponding to each text feature channel in the global text encoding features, the importance parameter of the j-th text feature channel is used as the importance parameter corresponding to the j-th visual feature channel in the global visual features.

[0102] S26: For each visual feature channel in the global visual features, the target mask value corresponding to the visual feature channel is determined by using the importance parameter of the visual feature channel.

[0103] Step S26, which involves determining the target mask value corresponding to a visual feature channel using the importance parameter of the visual feature channel, includes: when the importance parameter of the visual feature channel is less than a preset threshold, determining the target mask value corresponding to the visual feature channel as a first mask value; and when the importance parameter of the visual feature channel is greater than or equal to the preset threshold, determining the target mask value corresponding to the visual feature channel as a second mask value. The first mask value is used to remove features, and the second mask value is used to retain features.

[0104] For example, the first mask value is 0 and the second mask value is 1.

[0105] For example, the target mask values ​​of several visual feature channels can be represented as , This is called a hard mask.

[0106] For example, the preset threshold is 0.5. The preset threshold can also be set to other values, which can be set according to actual needs.

[0107] The target mask value corresponding to the visual feature channel can identify whether the visual feature channel is a redundant feature channel or a valid feature channel. Specifically, if the target mask value corresponding to the visual feature channel is the first mask value, it means that the visual feature channel is a redundant feature channel; if the target mask value corresponding to the visual feature channel is the second mask value, it means that the visual feature channel is a valid feature channel.

[0108] S27: For each visual feature channel, multiply the feature corresponding to the visual feature channel with the target mask value corresponding to the visual feature channel to obtain the enhanced visual feature.

[0109] When the target mask value corresponding to the visual feature channel is the first mask value, the feature corresponding to the visual feature channel will be cleared after multiplying the feature corresponding to the visual feature channel with the target mask value corresponding to the visual feature channel.

[0110] When the target mask value corresponding to the visual feature channel is the second mask value, the feature corresponding to the visual feature channel will be preserved after multiplying the feature corresponding to the visual feature channel with the target mask value corresponding to the visual feature channel.

[0111] S28: Perform feature matching on enhanced visual features and global text features to obtain matching results for the target image and target text.

[0112] The relevant content of step S28 can be referred to the aforementioned step S14, and will not be repeated here.

[0113] In this embodiment, the importance parameters corresponding to each visual feature channel in the global visual features are generated using the text encoding features of the target image. For each visual feature channel, the target mask value corresponding to the visual feature channel is determined based on the importance parameters of the visual feature channel. The feature corresponding to the visual feature channel is multiplied with the target mask value corresponding to the visual feature channel. In this way, redundant channel information that does not need to be concerned in the target text in the global visual features is removed, which improves the targeting of the enhanced visual features as equivalent to the target text and improves the effectiveness of the enhanced visual features. This makes the matching results obtained by feature matching using enhanced visual features and global text features more accurate, that is, it can improve the accuracy of image-text matching.

[0114] In this embodiment, the aforementioned image-text matching method can be implemented using a trained image-text matching model.

[0115] Please see Figure 4 , Figure 4 This is a schematic diagram of the model structure of the image-text matching model provided in this application. For example... Figure 4 As shown, the image-text matching model includes an encoder, a mask generation module, and a matching calculation module. The encoder specifically includes a visual encoding module and a text encoding module.

[0116] The visual encoding module encodes the input target image to obtain visual encoded features. The text encoding module encodes the input target text to obtain text encoded features. The mask generation module uses the text encoded features to generate importance parameters for each visual feature channel in the global visual features corresponding to the visual encoded features. For each visual feature channel in the global visual features, the importance parameters of the visual feature channel are used to determine the target mask value corresponding to that visual feature channel. The features corresponding to each visual feature channel are multiplied channel-by-channel with the target mask value corresponding to that visual feature channel to obtain the enhanced visual features. The matching calculation module calculates the semantic matching score of the global text features corresponding to the enhanced visual features and the text encoded features, which serves as the matching result for the target image and the target text.

[0117] The processing details of each module in the above image-text matching model can be found in the previous embodiment, and will not be repeated here.

[0118] The above image-text matching model can be trained using a sample dataset, which includes several image-text sample pairs. Each image-text sample pair includes a sample image and a sample text.

[0119] Specifically, on the one hand, web crawling technology can be used to selectively crawl several compliant first images from the publicly available internet domain. These first images should cover as many different scenes, object categories, and lighting conditions as possible. On the other hand, several second images of real-world scenes can be collected through on-site photography. This multi-source data acquisition strategy ensures data diversity and adaptability to actual application needs.

[0120] Furthermore, all the acquired images (including the aforementioned first images and second images) can be preliminarily processed to remove invalid images such as duplicates, blurriness, and damage, thereby obtaining several third images.

[0121] Furthermore, several third images are input into the multimodal large model, and preset prompt text is used to instruct the multimodal large model. Utilizing the image understanding capabilities of the multimodal large model, key information such as target objects and scene information in each third image is identified, and image description text is generated for each third image. For each third image, the image description text directly corresponds to the visual information presented by the third image, accurately reflecting the core content of the third image. For example, the multimodal large model can be Qwen3-VL, InternVL, etc.

[0122] Furthermore, each third image is used as a sample image, and the image description text corresponding to each third image is used as the sample text corresponding to each third image, thus obtaining several image-text sample pairs.

[0123] Furthermore, the aforementioned image-text sample pairs can be manually verified to correct any obvious errors in the annotations, thereby obtaining the final sample dataset used to train the image-text matching model.

[0124] In one embodiment, several positive sample pairs and several negative sample pairs can be constructed based on the aforementioned image-text sample pairs. Positive sample pairs correspond to sample images and sample text that match, while negative sample pairs correspond to sample images and sample text that do not match. Then, the constructed positive and negative sample pairs are used to train an image-text matching model.

[0125] During training, positive and negative sample pairs are input into the image-text matching model, and then... Figure 2 The same processing logic is used to obtain the semantic matching scores corresponding to positive sample pairs and negative sample pairs.

[0126] During training, the contrastive loss is calculated as a supervision signal to adjust the model parameters of the image-text matching model, thereby maximizing the semantic matching score of positive sample pairs and minimizing the semantic matching score of negative sample pairs. For example, the contrastive loss can be calculated using the InfoNCE loss function. The specific details of the InfoNCE loss function can be found in known techniques and will not be elaborated upon here.

[0127] Training an image-text matching model by calculating contrastive loss enables the model to align extracted visual and textual encoded features into the same semantic space. The model autonomously learns the correlation between the feature representation attributes of different visual feature channels of the input image and the text tokens of the input text, achieving cross-modal alignment of the acquired visual and textual encoded features. This, in turn, achieves cross-modal alignment of the acquired global visual and textual features, thereby improving the accuracy of subsequent image-text matching. The hard mask generated on this basis can accurately determine the importance or effectiveness of each visual feature channel in the global visual features, enabling targeted removal of features from redundant (or invalid) visual feature channels, further improving the accuracy of subsequent image-text matching.

[0128] In one embodiment, for each second encoding unit of the image-text matching model: during the training phase, the second encoding unit adopts a hybrid expert network to ensure the stability of cross-modal alignment; during the inference phase, the network parameters of the hybrid expert network of the second encoding unit obtained during the training phase are solidified and copied to obtain two hybrid expert networks corresponding to the second encoding unit (i.e., the first hybrid expert network and the second hybrid expert network with the same network parameters). These two hybrid expert networks can independently complete the visual encoding task and text encoding task of the second encoding unit, thereby improving the flexibility of model deployment.

[0129] Please see Figure 5 , Figure 5 This is a schematic diagram of the framework of an embodiment of the image-text matching device provided in this application. In this embodiment, the image-text matching device 50 includes a feature encoding module 51, an importance parameter generation module 52, a visual feature enhancement module 53, and a feature matching module 54.

[0130] The feature encoding module 51 is used to extract global visual features using the visual encoding features of the target image, and to extract global text features using the text encoding features of the target text; the importance parameter generation module 52 is used to generate importance parameters corresponding to each visual feature channel in the global visual features using the text encoding features; the visual feature enhancement module 53 is used to remove the features corresponding to each visual feature channel in the global visual features that do not meet the requirements of the importance parameters, and to obtain enhanced visual features; the feature matching module 54 is used to perform feature matching on the enhanced visual features and the global text features to obtain the matching results for the target image and the target text.

[0131] In one embodiment, the text encoding features include a plurality of first text feature vectors. The importance parameter generation module 52 is used to determine the feature weights corresponding to the plurality of first text feature vectors respectively; for each text feature channel of the text encoding features, the features of the text feature channels are weighted and summed using the feature weights of the plurality of first text feature vectors to obtain global text encoding features; and the importance parameters corresponding to each text feature channel in the global text encoding features are determined as the importance parameters corresponding to each visual feature channel respectively.

[0132] In one embodiment, the importance parameter generation module 52 is used to perform a first normalization operation on the features of each text feature channel in the global text encoding features; and to use the normalized value corresponding to each text feature channel in the global text encoding features as the importance parameter corresponding to each text feature channel in the global text encoding features.

[0133] In one embodiment, for each visual feature channel, the visual feature enhancement module 53 is used to determine the target mask value corresponding to the visual feature channel using the importance parameter of the visual feature channel, and to multiply the feature corresponding to the visual feature channel with the target mask value corresponding to the visual feature channel to obtain enhanced visual features; wherein, the target mask value is a first mask value or a second mask value, the first mask value is used to remove features, and the second mask value is used to retain features.

[0134] In one embodiment, the visual feature enhancement module 53 is used to determine the target mask value corresponding to the visual feature channel as a first mask value in response to the importance parameter of the visual feature channel being less than a preset threshold; and to determine the target mask value corresponding to the visual feature channel as a second mask value in response to the importance parameter of the visual feature channel being greater than or equal to the preset threshold.

[0135] In one embodiment, the visual coding features and text coding features are obtained by encoding the target image and the target text respectively using an encoder. The encoder includes at least one first coding unit and at least one second coding unit. The first coding unit includes a visual coding network and a text coding network, and the second coding unit includes a hybrid expert network.

[0136] In one embodiment, the encoder includes an n-layer coding unit, which includes two second coding units, and the two second coding units are located at the i-th layer and the n-th layer of the n-layer coding unit, respectively, where i is greater than 1 and less than n, and n is a positive integer.

[0137] And / or, the second coding unit includes a first hybrid expert network and a second hybrid expert network, wherein the first hybrid expert network is used to process the visual features output by the previous coding unit, the second hybrid expert network is used to process the text features output by the previous coding unit, and the network parameters of the first hybrid expert network and the second hybrid expert network are the same.

[0138] In one embodiment, the hybrid expert network includes a routing network, a general expert network, and several specialized expert networks; the routing network is used to determine the specialized expert network that is compatible with the input feature vector from among the several specialized expert networks, wherein the input feature vector is a feature vector of visual features output by the previous encoding unit or a feature vector of text features output by the previous encoding unit; the general expert network is used to extract cross-modal general features of images and text from the input feature vector; and the specialized expert networks are used to extract modality-specific features from the input feature vector.

[0139] It should be noted that the apparatus of this embodiment can perform the steps in the above method. For detailed descriptions of the relevant content, please refer to the method section above, which will not be repeated here.

[0140] Please see Figure 6 , Figure 6 This is a schematic diagram of a framework of an embodiment of the electronic device provided in this application. In this embodiment, the electronic device 60 includes a memory 61 and a processor 62.

[0141] Processor 62 can also be referred to as a CPU (Central Processing Unit). Processor 62 may be an integrated circuit chip with signal processing capabilities. Processor 62 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor, or processor 62 can be any conventional processor 62, etc.

[0142] The memory 61 in the electronic device 60 is used to store the program instructions required for the processor 62 to run.

[0143] The processor 62 is used to execute program instructions to implement the image-text matching method in this application.

[0144] Please see Figure 7 , Figure 7 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium provided in this application. The computer-readable storage medium 70 of this application embodiment stores program instructions 71, which, when executed, implement the image-text matching method provided in this application. The program instructions 71 can form a program file and be stored in the aforementioned computer-readable storage medium 70 in the form of a software product, so that a computer device (which may be a personal computer, server, or network device, etc.) can execute all or part of the steps of the methods of various embodiments of this application. The aforementioned computer-readable storage medium 70 includes various media capable of storing program code, such as a USB flash drive, mobile hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or terminal devices such as computers, servers, mobile phones, and tablets.

[0145] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0146] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0147] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatuses, and systems can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or units may be electrical, mechanical, or other forms.

[0148] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0149] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0150] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0151] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method of image-text matching, characterized by, The method comprises: extracting global visual features by using visual coding features of a target image, and extracting global text features by using text coding features of a target text; generating importance parameters corresponding to each visual feature channel in the global visual features by using the text coding features; removing features corresponding to each visual feature channel with an importance parameter not meeting a requirement in the global visual features to obtain enhanced visual features; performing feature matching on the enhanced visual features and the global text features to obtain a matching result about the target image and the target text.

2. The method of claim 1, wherein, The text coding features comprise a plurality of first text feature vectors, and the generating of the importance parameters corresponding to each visual feature channel in the global visual features by using the text coding features comprises: determining feature weights corresponding to the plurality of first text feature vectors; performing weighted summation on features of each text feature channel of the text coding features by using the feature weights of the plurality of first text feature vectors to obtain global text coding features; determining importance parameters corresponding to each text feature channel in the global text coding features as importance parameters corresponding to each visual feature channel.

3. The method of claim 2, wherein, The determining of the importance parameters corresponding to each text feature channel in the global text coding features comprises: performing first normalization operation on features of each text feature channel in the global text coding features; taking normalized values corresponding to each text feature channel in the global text coding features as the importance parameters corresponding to each text feature channel in the global text coding features.

4. The method of claim 1, wherein, The removing of the features corresponding to each visual feature channel with an importance parameter not meeting a requirement in the global visual features to obtain enhanced visual features comprises: for each visual feature channel, determining a target mask value corresponding to the visual feature channel by using the importance parameter of the visual feature channel, and multiplying features corresponding to the visual feature channel with the target mask value corresponding to the visual feature channel to obtain the enhanced visual features; wherein the target mask value is a first mask value or a second mask value, the first mask value is used for clearing features, and the second mask value is used for retaining features.

5. The method of claim 4, wherein, The determining of the target mask value corresponding to the visual feature channel by using the importance parameter of the visual feature channel comprises: in response to the importance parameter of the visual feature channel being less than a preset threshold, determining the target mask value corresponding to the visual feature channel as the first mask value; in response to the importance parameter of the visual feature channel being greater than or equal to the preset threshold, determining the target mask value corresponding to the visual feature channel as the second mask value.

6. The method of claim 1, wherein, The visual coding feature and the text coding feature are obtained by respectively coding the target image and the target text by using an encoder, the encoder comprising at least one first coding unit and at least one second coding unit, the first coding unit comprising a visual coding network and a text coding network, and the second coding unit comprising a hybrid expert network.

7. The method of claim 6, wherein, the encoder comprises n layers of coding units, two layers of the second coding units are included in the n layers of coding units, and the two layers of the second coding units are respectively located at an i-th layer and an n-th layer in the n layers of coding units, where i is greater than 1 and less than n, and n is a positive integer; and / or, the second coding unit comprises a first hybrid expert network and a second hybrid expert network, the first hybrid expert network is used for processing a visual feature output by a previous coding unit, the second hybrid expert network is used for processing a text feature output by the previous coding unit, and network parameters of the first hybrid expert network and the second hybrid expert network are the same.

8. The method of claim 6, wherein, the hybrid expert network comprises a routing network, a general expert network, and a plurality of special expert networks; the routing network is used for determining, from the plurality of special expert networks, a special expert network that is adapted to an input feature vector, where the input feature vector is a feature vector of a visual feature output by a previous coding unit or a feature vector of a text feature output by the previous coding unit; the general expert network is used for extracting a cross-modal general feature in the input feature vector; the special expert network is used for extracting a modal-specific feature in the input feature vector.

9. An electronic device, comprising: comprise a memory and a processor coupled to each other, the memory stores program instructions; the processor is configured to execute the program instructions stored in the memory to implement the method of any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, the computer-readable storage medium is configured to store program instructions executable by a processor to implement the method of any one of claims 1-8. the computer-readable storage medium is configured to store program instructions executable by a processor to implement the method of any one of claims 1-8.

Images