Content matching method and device, electronic equipment and readable medium

Abstract

The invention provides a content matching method and device, electronic equipment and a readable medium. The method comprises the steps that a to-be-processed image and a to-be-processed text are acquired, the to-be-processed image comprises a to-be-recognized target, and the to-be-processed text comprises a target description text of the to-be-recognized target in the to-be-processed image; performing image coding on the to-be-processed image through the content matching model to obtain a global image feature for the to-be-processed image and a regional image feature for the to-be-recognized target; performing text coding on the to-be-processed text through the content matching model to obtain global text features for the to-be-processed text and regional text features for the target description text; and determining a matching result between the to-be-processed image and the to-be-processed text according to the comparison loss between the global image feature and the global text feature and the comparison loss between the regional image feature and the regional text feature. According to the method, the integrity of correlation between the image and the text can be improved, so that the accuracy of image-text matching is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more particularly to a content matching method, apparatus, electronic device, and readable medium. Background Technology

[0002] With the development of artificial intelligence (AI) technology, AI has been applied to fields such as image and text data retrieval. Correspondingly, cross-modal models for image-text matching have begun to emerge. Image-text matching technology is an important research direction within cross-modal models, and the degree of matching between images and text is a key factor influencing the results of subsequent tasks.

[0003] In related technologies, the model extracts features from the input image and text, calculates the similarity between the obtained text features and image features by inner product calculation, and thus determines the matching image and text.

[0004] However, in the above scheme, the determination of the matching degree between images and text mainly relies on the global information of images and text, which leads to the loss of local feature information in image features and text features, and thus cannot accurately express the information of specific targets in the image. This affects the integrity of the correlation between images and text, thereby reducing the accuracy of image-text matching. Summary of the Invention

[0005] To address the aforementioned technical issues, this application provides a content matching method, apparatus, electronic device, and readable medium to improve the completeness of the correlation between images and text, thereby enhancing the accuracy of image-text matching.

[0006] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0007] According to one aspect of the embodiments of this application, a content matching method is provided, including:

[0008] Obtain an image to be processed and text to be processed, wherein the image to be processed contains a target to be identified, and the text to be processed contains target description text of the target to be identified in the image to be processed;

[0009] The image to be processed is encoded using a content matching model to obtain global image features for the image to be processed and regional image features for the target to be identified.

[0010] The text to be processed is encoded using the content matching model to obtain global text features for the text to be processed and regional text features for the target descriptive text.

[0011] The matching result between the image to be processed and the text to be processed is determined based on the contrast loss between the global image features and the global text features and the contrast loss between the regional image features and the regional text features.

[0012] According to one aspect of the embodiments of this application, a content matching apparatus is provided, comprising:

[0013] The acquisition module is configured to acquire an image to be processed and text to be processed, wherein the image to be processed contains a target to be identified, and the text to be processed contains target description text of the target to be identified in the image to be processed;

[0014] The image encoding module is configured to encode the image to be processed using a content matching model to obtain global image features for the image to be processed and regional image features for the target to be identified.

[0015] The text encoding module is configured to encode the text to be processed using the content matching model to obtain global text features for the text to be processed and regional text features for the target description text.

[0016] The result determination module is configured to determine the matching result between the image to be processed and the text to be processed based on the contrast loss between the global image features and the global text features and the contrast loss between the regional image features and the regional text features.

[0017] In some embodiments of this application, based on the above technical solutions, the content matching device further includes:

[0018] The training data acquisition module is configured to acquire a training dataset, which includes training images and corresponding training text. The training images include training targets and target location information of the training targets in the training images. The training text includes training description text for each training target in the training images.

[0019] The training module is configured to perform image encoding on the training image and the target location information through a first pre-trained model to obtain first global training image features and first region training image features, and to perform text encoding on the training text and the training description text to obtain first global training text features and first region training text features.

[0020] The comparison module is configured to compare the first global training image features and the first global training text features to obtain the global training loss, and to compare the first region training image features and the first region training text features to obtain the target training loss.

[0021] The parameter adjustment module is configured to adjust the parameters of the first pre-trained model based on the global training loss and the target training loss to obtain the content matching model.

[0022] In some embodiments of this application, based on the above technical solutions, the parameter adjustment module is specifically configured to: perform image encoding on the training image and the target location information using a second pre-trained model to obtain second global training image features and second region training image features; and perform text encoding on the training text and the training description text to obtain second global training text features and second region training text features, wherein the number of parameters of the second pre-trained model is higher than the number of parameters of the first pre-trained model; perform correlation calculation based on the first global training image features and the first global training text features to determine the first global correlation distribution of the first pre-trained model; and perform correlation calculation based on the second global training image features and the second global training text features. The process involves: calculating and determining the second global relevance distribution of the second pre-trained model; performing relevance calculations based on the training image features and training text features of the first region to determine the first region relevance distribution of the first pre-trained model; performing relevance calculations based on the training image features and training text features of the second region to determine the second region relevance distribution of the second pre-trained model; obtaining a relevance loss based on the divergence between the first and second global relevance distributions and the divergence between the first and second region relevance distributions; and adjusting the parameters of the first pre-trained model based on the global training loss, the target training loss, and the relevance loss to obtain the content matching model.

[0023] In some embodiments of this application, based on the above technical solutions, the training data acquisition module is specifically configured to: acquire a set of resource images, each resource image in the set containing a training target; perform image segmentation on the resource images to obtain mask information of the training target, the mask information serving as target location information indicating the training target in the training image; perform target recognition based on the mask information of the training target and the corresponding resource image to obtain descriptive text of the training target; merge each resource image and the mask information of the training target in each resource image to obtain a training image, and merge the descriptive text of the training target in each resource image to obtain the training text corresponding to the training image, which serves as the training dataset.

[0024] In some embodiments of this application, based on the above technical solutions, the resource image set further includes labeling information of the training target, which is used to indicate the position of the training target in the resource image; the training data acquisition module is specifically configured to: perform image encoding on the resource image to obtain resource image encoding; generate resource hint encoding corresponding to the training target based on the preset embedding corresponding to the position encoding of the labeling information of the training target in the resource image and the category of the labeling information; and perform mask prediction based on the resource image encoding and the resource hint encoding to obtain the mask information of the training target.

[0025] In some embodiments of this application, based on the above technical solutions, the first training module is further configured as follows:

[0026] The training images in the training dataset are encoded using the image encoder of the second model to be trained to obtain second pre-trained image features, wherein the pre-trained image features are determined based on regional feature constraints for the training target in the training images; the training text in the training dataset is encoded using the text encoder of the second pre-trained model to obtain second pre-trained text features; and the parameters of the second model to be trained are adjusted according to the matching results between the second pre-trained image features and the second pre-trained text features to obtain the second pre-trained model.

[0027] In some embodiments of this application, based on the above technical solutions, the first training module is further configured to: merge the prompt information parameters of the first model to be trained with the training images in the training dataset to obtain training input data; encode the training input data through the image encoder of the first model to be trained to obtain first pre-trained image features; and adjust the prompt information parameters according to the matching result between the first pre-trained image features and the second pre-trained text features to obtain the first pre-trained model.

[0028] In some embodiments of this application, based on the above technical solutions, the first training module is specifically configured to: perform dimensional mapping on the first pre-trained image features according to the feature dimension information of the second pre-trained text features, and obtain a feature mapping result by using the mapping parameters of the first model to be trained; adjust the prompt information parameters and the mapping parameters according to the matching result between the feature mapping result and the second pre-trained text features, and obtain the first pre-trained model.

[0029] In some embodiments of this application, based on the above technical solutions, the result determination module is further configured to: determine the target description text corresponding to each target to be identified in the image to be processed according to the matching result; and classify each target to be identified in the image to be processed according to the target description text.

[0030] In some embodiments of this application, based on the above technical solutions, the result determination module is further configured to: input the text to be processed into an image generation model for text encoding to obtain a generated image embedding; adjust the parameters of the image generation model based on the matching result between the generated image embedding and the image features of the image to be processed that match the text to be processed; perform embedding prediction based on the input text through the image generation model to obtain a predicted image embedding; and decode the predicted image embedding through the image generation model to obtain a generated image corresponding to the input text.

[0031] According to one aspect of the embodiments of this application, an electronic device is provided, the electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the content matching method as described above by executing the executable instructions.

[0032] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the content matching method as described in the above technical solutions.

[0033] According to one aspect of the embodiments of this application, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the content matching method provided in the various optional implementations described above.

[0034] In the embodiments of this application, the content matching device acquires an image to be processed and text to be processed. The image to be processed contains a target to be identified, and the text to be processed contains target description text of the target to be identified in the image to be processed. Subsequently, the content matching device performs image encoding on the image to be processed using a content matching model to obtain global image features for the image to be processed and regional image features for the target to be identified. Similarly, it performs text encoding on the text to be processed using the content matching model to obtain global text features for the text to be processed and regional text features for the target description text. Finally, the content matching device determines the matching result between the image to be processed and the text to be processed based on the contrast loss between the global image features and the global text features, and the contrast loss between the regional image features and the regional text features. In this way, the influence of regional features of the image and text on the matching result is further considered during the content matching process, which helps to improve the completeness of the correlation between the image and text, thereby improving the accuracy of image-text matching.

[0035] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0036] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0037] Figure 1 The content matching method of this application is applied to the system architecture of a cloud host platform.

[0038] Figure 2 This is a flowchart of a content matching method according to an embodiment of this application.

[0039] Figure 3 This is a flowchart of a content matching method according to an embodiment of this application.

[0040] Figure 4 This is a schematic diagram illustrating the automatic text data annotation process in the embodiments of this application.

[0041] Figure 5 This is a schematic flowchart illustrating the model training process in the embodiments of this application.

[0042] Figure 6 A schematic block diagram illustrating the composition of the content matching device in an embodiment of this application is shown.

[0043] Figure 7 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0044] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0045] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0046] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve the predetermined function, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0047] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0048] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0049] It should be understood that the solutions in this application can be applied to the field of basic visual language models, specifically in systems or internet applications that provide content processing. Such systems or applications can provide functions such as artificial intelligence assistants, and can perform services such as target or image recognition, image or video generation, and multimodal data processing. For example, in image generation services, the system can generate images containing specified content based on text input by the user. In target recognition services, the system can retrieve images containing relevant content from a vast number of images or videos based on descriptive information input by the user. For example, if the user inputs "green bus with mooncake advertisements on the body," the system can search for corresponding images or video clips from a vast number of videos based on this description.

[0050] With the development of artificial intelligence (AI) technology, AI has been applied to fields such as image and text data retrieval. Correspondingly, cross-modal models for image-text matching have emerged. Image-text matching technology is an important research direction within cross-modal models, and the degree of matching between images and text is a key factor affecting the results of subsequent tasks. In related technologies, the model extracts features from the input image and text, calculates the similarity between the obtained text features and image features using inner product calculation, and thus determines the matching image and text. However, in the above schemes, determining the degree of matching between images and text mainly relies on the global information of the image and text, leading to the loss of local feature information in the image and text features, which cannot accurately express the information of specific targets in the image. This affects the completeness of the correlation between the image and text, thereby reducing the accuracy of image-text matching.

[0051] Based on this, the technical solution of this application embodiment proposes a content matching scheme. Specifically, please refer to... Figure 1 The system architecture of the content matching method applied to a blockchain system according to the embodiments of this application can mainly include a terminal device 110, a network 120, and a server 130. The terminal device 110 can include smartphones, tablets, laptops, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. The server 130 can be a server providing various services; it can be an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The network 120 can be a communication medium of various connection types capable of providing a communication link between the terminal device 110 and the server 130, such as a wired communication link or a wireless communication link.

[0052] Depending on the implementation requirements, the system architecture in this application embodiment can have any number of terminal devices, networks, and servers. For example, server 130 can be a server group composed of multiple server devices. In addition, the technical solutions provided in this application embodiment can be applied to terminal device 110, or to server 130, or can be implemented jointly by terminal device 110 and server 130. This application does not impose any special limitations on this.

[0053] like Figure 1As shown, the image processing model in this application can be deployed in server 130. Server 130 provides an artificial intelligence server based on the image processing model to terminal device 110. Users send text or images as input data to the server through terminal device 110. Server 130 obtains the image to be processed and the text to be processed from terminal device 110 via network 120. The image to be processed contains a target to be identified, and the text to be processed contains target description text of the target to be identified in the image. Subsequently, server 130 performs image encoding on the image to be processed using a content matching model to obtain global image features and regional image features for the target to be identified. Similarly, server 130 performs text encoding on the text to be processed using a content matching model to obtain global text features and regional text features for the target description text. Finally, server 130 determines the matching result between the image to be processed and the text to be processed based on the contrast loss between global image features and global text features, and the contrast loss between regional image features and regional text features. Server 130 can then send the matching result to terminal device 110. In some cases, terminal device 110 may simultaneously propose downstream tasks to be executed, and server 130 may execute the tasks specified by terminal device 110 based on the matching results and send the task execution results to terminal device 110.

[0054] The solution in this application can be applied to multimodal large language model tasks. Multimodality refers to using multiple different modalities (such as images, audio, and text) to process and understand complex multimodal data. This involves using data from multiple perceptual modalities as input to the model and outputting corresponding recognition results. In the content matching task provided in this application's embodiments, the model can consider not only the pixel information of the image but also the textual description information related to the image, and the textual description information can further include information from different languages. A large language model refers to a deep learning model trained using a large amount of text data, capable of generating natural language text or understanding the meaning of language text. Large language models can handle various natural language tasks, such as text classification, question answering, and dialogue, and are an important pathway to artificial intelligence.

[0055] The implementation details of the technical solutions in the embodiments of this application are described in detail below: Figure 2 A flowchart of a content matching method according to an embodiment of this application is shown. This content matching method can be executed by a device with computing processing capabilities, such as a server or a terminal device. The content matching method of this application will now be described from the perspective of a server. (Refer to...) Figure 2 As shown, this content matching method includes at least steps S210 to S240, which are described in detail below:

[0056] Step S210: Obtain the image to be processed and the text to be processed, wherein the image to be processed contains a target to be identified, and the text to be processed contains a target description text of the target to be identified in the image to be processed.

[0057] The images and text to be processed are typically data output from a client. In this application, the images to be processed are typically high-resolution images, such as 512x512 and 1024x1024 or higher. The text to be processed is typically long text. It is understood that the images to be processed can contain multiple images, and the text to be processed can contain multiple texts, and the number of images and texts to be processed can be different. The images to be processed contain targets to be identified, which typically refer to objects, such as objects or regions, within the images. Generally, one image to be processed contains multiple targets to be identified. The text to be processed contains descriptive text about the targets to be identified in the images. For example, the text to be processed could be a descriptive text about a person or animal. It is important to note that although the text to be processed contains descriptive text about the targets to be identified, the input data does not contain a direct correlation with the images to be processed. For example, the image to be processed might be an image of a dog, while the text to be processed could contain descriptions of different breeds of dogs or descriptions of multiple animals.

[0058] Step S220: Image encoding is performed on the image to be processed using a content matching model to obtain global image features for the image to be processed and regional image features for the target to be identified.

[0059] The content matching model is a model trained according to the embodiments of this application. The content matching model encodes the image to be processed, typically using an image encoder to process the image and obtain global and regional image features. Specifically, global image features are obtained by encoding the image to be processed. Regional image features are obtained by encoding all possible objects in the image to be identified using the content matching model. In some embodiments, the image to be processed contains markers or bounding boxes representing the target to be identified; the content matching model can perform region-of-interest pooling based on this marker information to obtain the corresponding regional image features.

[0060] Step S230: The text to be processed is encoded using the content matching model to obtain global text features for the text to be processed and regional text features for the target descriptive text.

[0061] Similarly, text encoding of the text to be processed using a content matching model typically involves processing the text using a text encoder within the content matching model to obtain global and regional text features. Regional text features can be understood as follows: the text to be processed is semantically divided into target description texts for different objectives, which are then encoded using a text encoder. For example, the text can be split into sentences and encoded separately. In some embodiments, a piece of text to be processed is itself a collection or combination of multiple target description texts, and the text to be processed contains information about how these target description texts are combined, such as being separated by specific symbols or containing sequence numbers. In such cases, the individual target description texts can be directly separated for processing, while global text features are obtained by concatenating the individual target description texts before inputting them into the text encoder.

[0062] Step S240: Determine the matching result between the image to be processed and the text to be processed based on the contrast loss between the global image features and the global text features and the contrast loss between the regional image features and the regional text features.

[0063] Specifically, the contrast loss is calculated using the contrast loss function configured in the content matching model. When there are multiple images and texts to be processed, they are paired in a permutation and combination manner, and the contrast loss is calculated accordingly. Similarly, the contrast loss between region image features and region text features is also calculated using the contrast loss function configured in the content matching model, and is also paired in a permutation and combination manner, with the contrast loss calculated as well. Based on the contrast loss, the relevance between the image and the text can be determined, and the content matching device selects the image and text with the highest relevance as the matching result.

[0064] In the embodiments of this application, the content matching device acquires an image to be processed and text to be processed. The image to be processed contains a target to be identified, and the text to be processed contains target description text of the target to be identified in the image to be processed. Subsequently, the content matching device performs image encoding on the image to be processed using a content matching model to obtain global image features for the image to be processed and regional image features for the target to be identified. Similarly, it performs text encoding on the text to be processed using the content matching model to obtain global text features for the text to be processed and regional text features for the target description text. Finally, the content matching device determines the matching result between the image to be processed and the text to be processed based on the contrast loss between the global image features and the global text features, and the contrast loss between the regional image features and the regional text features. In this way, the influence of regional features of the image and text on the matching result is further considered during the content matching process, which helps to improve the completeness of the correlation between the image and text, thereby improving the accuracy of image-text matching.

[0065] In some embodiments of this application, based on the technical solutions of this application, the content matching device further determines the target description text corresponding to each target to be identified in the image to be processed according to the matching result, and then classifies each target to be identified in the image to be processed according to the target description text. In this embodiment, the content matching model can be applied to semantic segmentation and target recognition tasks. Specifically, the semantic segmentation task requires semantic annotation of each pixel in the image to be processed. Through the matching result, the content matching device can determine the target description text corresponding to each target to be identified in the image to be processed, thereby determining the target description text corresponding to each pixel in the image to be processed, that is, the pixels in the area where the target to be identified is located are all marked with the target description text. Subsequently, the target description text is used to classify each target to be identified in the image to be processed, and the pixels of each target to be identified can also be marked, thereby completing the semantic segmentation or target recognition task. In this way, regional features are fully considered in tasks such as target detection and semantic segmentation, thereby improving the accuracy of the model output results.

[0066] In some embodiments of this application, based on the technical solutions of this application, the content matching device further encodes the text to be processed into an image generation model to obtain a generated image embedding. Then, based on the matching result between the generated image embedding and the image features of the image to be processed that matches the text to be processed, the device adjusts the parameters of the image generation model. Subsequently, the content matching device performs embedding prediction based on the input text using the image generation model to obtain a predicted image embedding. Then, the device decodes the predicted image embedding using the image generation model to obtain a generated image corresponding to the input text. In this embodiment, the content matching model can be applied to image generation tasks. Specifically, in this embodiment, the image to be processed and the text to be processed in the matching result are used as training data and input into the image generation model. The text to be processed is used as training input data, while the image to be processed is used as a benchmark value to be compared with the image output during training, thereby adjusting the parameters of the image generation model based on the comparison result. When the parameter-adjusted image generation model receives text input, it generates text features based on the text encoder of the content matching model, predicts image features in the content matching model that will match these text features, and finally converts the image features into an image using an inverse encoder, thus obtaining the image generated from the input text. This approach introduces region-specific text and image features into the image generation task, making the targets in the generated images more accurate and improving the controllable editing and understanding capabilities of the image generation task.

[0067] In embodiments of this application, a method for... Figure 2 Other detailed embodiments of the technical solution shown in the example are as follows: Figure 3 As shown, the content matching method in one embodiment of this application may include the following steps:

[0068] Step S310: Obtain a training dataset. The training dataset contains training images and corresponding training text. The training images contain training targets and target location information of the training targets in the training images. The training text contains training description text for each training target in the training images.

[0069] Specifically, the content configuration device first acquires a training dataset, which includes training images and corresponding training text. The training images contain training targets and their location information within the images. The training text contains training descriptions of each training target within the images. Specifically, the training text can be obtained by annotating the training images and their training targets and then merging the annotation information. The target location information of each training target within the images can be a mask corresponding to that target. The training descriptions of each training target can be labels, or text obtained by expanding upon the labels or through paradigm processing.

[0070] Step S320: Image encoding is performed on the training image and the target location information using the first pre-trained model to obtain the first global training image features and the first region training image features. Text encoding is also performed on the training text and the training description text to obtain the first global training text features and the first region training text features.

[0071] Specifically, the first global training image features can be obtained by processing the training image using the first pre-trained model, while the first region training image features are obtained by masking the image features output by the model using target location information and then pooling the result. The first global training text features are obtained by encoding the training text as a whole, while the first region training text features can be obtained by splitting the training description text and then encoding it.

[0072] Step S330: Compare the first global training image features and the first global training text features to obtain the global training loss, and compare the first region training image features and the first region training text features to obtain the target training loss.

[0073] Step S340: Adjust the parameters of the first pre-trained model according to the global training loss and the target training loss to obtain the content matching model.

[0074] The global training loss indicates the degree of matching between the training image and the training text, while the target training loss includes the degree of matching between the target to be trained and the training description text. By comparing the loss parameters, the degree of matching indicated by the global training loss and the target training loss can be increased, thereby obtaining an accurate content matching model.

[0075] Step S350: Obtain the image to be processed and the text to be processed, wherein the image to be processed contains the target to be identified, and the text to be processed contains the target description text of the target to be identified in the image to be processed.

[0076] Optionally, the implementation details of step S350 are the same as... Figure 2 The steps S210 shown are the same and will not be repeated here.

[0077] Step S360: Image encoding is performed on the image to be processed using a content matching model to obtain global image features for the image to be processed and regional image features for the target to be identified.

[0078] Optionally, the implementation details of step S360 are the same as... Figure 2 The steps S220 shown are the same and will not be repeated here.

[0079] Step S370: The text to be processed is encoded using the content matching model to obtain global text features for the text to be processed and regional text features for the target descriptive text.

[0080] Optionally, the implementation details of step S370 are the same as those of... Figure 2 The steps S230 shown are the same and will not be repeated here.

[0081] Step S380: Determine the matching result between the image to be processed and the text to be processed based on the contrast loss between the global image features and the global text features and the contrast loss between the regional image features and the regional text features.

[0082] Optionally, the implementation details of step S380 are the same as those of... Figure 2 The steps S240 shown are the same and will not be repeated here.

[0083] In the embodiments of this application, the content matching device acquires an image to be processed and text to be processed. The image to be processed contains a target to be identified, and the text to be processed contains target description text of the target to be identified in the image to be processed. Subsequently, the content matching device performs image encoding on the image to be processed using a content matching model to obtain global image features for the image to be processed and regional image features for the target to be identified. Similarly, it performs text encoding on the text to be processed using the content matching model to obtain global text features for the text to be processed and regional text features for the target description text. Finally, the content matching device determines the matching result between the image to be processed and the text to be processed based on the contrast loss between the global image features and the global text features, and the contrast loss between the regional image features and the regional text features. In this way, the influence of regional features of the image and text on the matching result is further considered during the content matching process, which helps to improve the completeness of the correlation between the image and text, thereby improving the accuracy of image-text matching.

[0084] In some optional embodiments of this application, during the process of adjusting the parameters of the second pre-trained model according to the global training loss and the target training loss to obtain the content matching model, the content matching device performs image encoding on the training image and the target location information through the second pre-trained model to obtain second global training image features and second region training image features, and performs text encoding on the training text and the training description text to obtain second global training text features and second region training text features. The number of parameters in the second pre-trained model is higher than the number of parameters in the first pre-trained model. Then, the content matching device performs correlation calculation based on the first global training image features and the first global training text features to determine the first global correlation distribution of the first pre-trained model, and performs correlation calculation based on the second global training image features and the second global training text features to determine the second global correlation distribution of the second pre-trained model. The content matching device performs correlation calculation based on the first region training image features and the first region training text features to determine the first region correlation distribution of the first pre-trained model, and performs correlation calculation based on the second region training image features and the second region training text features to determine the second region correlation distribution of the second pre-trained model. Finally, the content matching device obtains the relevance loss based on the divergence between the first global relevance distribution and the second global relevance distribution, and based on the divergence between the first regional relevance distribution and the second regional relevance distribution. Then, based on the training global loss, the training target loss, and the relevance loss, it adjusts the parameters of the second pre-trained model to obtain the content matching model.

[0085] In this embodiment, during the training of the content matching model, a second pre-trained model guides the first pre-trained model. The parameters of the first pre-trained model are adjusted using global training loss and target training loss to obtain the content matching model. The second pre-trained model, being a pre-trained model, acts as a teacher model to guide the training of the first pre-trained model. Typically, the second pre-trained model has a higher number of parameters than the first pre-trained model; therefore, its model complexity and processing power are higher. The first pre-trained model acts as a student model, learning from the output of the second pre-trained model to acquire the association information between the image and text output by the first pre-trained model. The divergence between the first and second global correlation distributions reflects the difference between the student model's output on training images and training text and the teacher model's output on training images and training text. Similarly, the divergence between the first and second regional correlation distributions represents the difference between the student model's and the teacher model's output on training targets and training description text. By adjusting parameters to reduce global and regional differences, the first pre-trained model learns the second pre-trained model's ability to match images and text. The first pre-trained model has relatively fewer parameters, which reduces the computational power required for image and text processing, thereby improving the processing efficiency of the prediction or inference process.

[0086] In some optional embodiments of this application, during the acquisition of the training dataset, the content matching device acquires a set of resource images, each of which contains a training target. Then, it performs image segmentation on each resource image to obtain mask information of the training target. This mask information serves as information indicating the target location of the training target within the training image. Subsequently, the content matching device performs target recognition based on the mask information of the training target and the corresponding resource image to obtain descriptive text of the training target. Then, it merges each resource image and the mask information of the training target within each resource image to obtain a training image, and merges the descriptive text of the training target within each resource image to obtain the training text corresponding to the training image, which serves as the training dataset. In this embodiment, the training dataset is automatically generated by labeling the resource image set. Specifically, the resource image set is typically an existing set of images, which may not contain descriptive information for the images. The subsequent scheme performs image segmentation on each resource image to obtain mask information of the training target. The mask information indicates the target location of the training target within the training image; therefore, the mask information serves as the target location information of the training target. Image segmentation can be performed using a suitable machine learning model. Based on the mask information of the training target and the corresponding resource image, target recognition is performed to obtain the descriptive text of the training target. Specifically, the mask information can be used to determine the region where the training target is located from the resource image. Based on the image of that region, a pre-trained multimodal large model or target recognition model can be used to generate descriptive text of the object within that region. In this way, images can be automatically labeled, and a training dataset containing images and text can be generated based on the labeling results, which helps to improve the efficiency of training data generation.

[0087] In some optional embodiments of this application, the resource image set further includes labeling information of the training target, which is used to indicate the location of the training target in the resource images. During image segmentation of each resource image to obtain the mask information of the training target, the content matching device performs image encoding on the resource image to obtain resource image encoding. Then, based on the preset embedding corresponding to the position encoding and category of the labeling information of the training target in the resource image, a resource hint encoding corresponding to the training target is generated. Finally, mask prediction is performed based on the resource image encoding and the resource hint encoding to obtain the mask information of the training target. Resource image encoding is an embedding representation obtained by convolutional sampling of the resource training. The labeling information of the training target can typically be marker points or bounding boxes, which are usually provided with the resource image, for example, they can be labeled by the user. The position encoding of the labeling information can be the coordinate position of the marker point or bounding box in the resource image. The category of the labeling information generally refers to the classification of the location of the target marked by the labeling information in the image, for example, it can be divided into foreground and background categories. Image segmentation models pre-learn different preset embeddings for different categories. Combining location information with these preset embeddings yields resource cue codes, enabling more accurate representation of labeled points. When using an image segmentation model, the model predicts a mask based on the resource image code and resource cue code, typically generating multiple mask candidates. Finally, the most suitable mask is selected based on the matching of the labeled information. This approach provides a concrete method for image segmentation, improving the feasibility of the solution.

[0088] In some optional embodiments of this application, after acquiring the training dataset, the content matching device further encodes the training images in the training dataset using the image encoder of the second model to be trained, obtaining second pre-trained image features. These pre-trained image features are determined based on regional feature constraints for the training target in the training images. Then, the training text in the training dataset is encoded using the text encoder of the second pre-trained model, obtaining second pre-trained text features. Finally, based on the matching result between the second pre-trained image features and the second pre-trained text features, the parameters of the second model to be trained are adjusted to obtain the second pre-trained model. By adding regional feature constraints during model training, finer-grained feature training is achieved, improving the model's recognition ability.

[0089] In some optional embodiments of this application, the content matching device further merges the prompt information parameters of the first model to be trained with the training images in the training dataset to obtain training input data. Then, the training input data is encoded by the image encoder of the first model to be trained to obtain first pre-trained image features. Finally, based on the matching result between the first pre-trained image features and the second pre-trained text features, the prompt information parameters are adjusted to obtain the first pre-trained model. The prompt information parameters can be a paradigm set for the image. By adjusting the prompt information parameters instead of adjusting the parameters of the first pre-trained model, the processing load during training can be reduced, thereby improving training efficiency.

[0090] In some optional embodiments of this application, during the process of adjusting the prompt information parameters based on the matching result between the first pre-trained image features and the second pre-trained text features to obtain the first pre-trained model, the content matching device performs dimension mapping on the first pre-trained image features according to the feature dimension information of the second pre-trained text features, through the mapping parameters of the first model to be trained, to obtain the feature mapping result. Then, based on the matching result between the feature mapping result and the second pre-trained text features, the prompt information parameters and the mapping parameters are adjusted to obtain the first pre-trained model. In this embodiment, the feature dimensions of the first pre-trained image features output by the student model do not match with the feature dimensions of the second pre-trained text features. Therefore, the mapper maps the dimensions of the first pre-trained image features to the feature dimensions of the second pre-trained text features, and then matches them with the second pre-trained text features, thereby adjusting the prompt information parameters and the mapping parameters. By using a mapper for feature dimension matching, the adaptability of the model to image features of different dimensions can be improved, increasing the flexibility of the scheme.

[0091] The following describes the implementation details of the technical solution of this application embodiment using specific examples. In actual processing, the operator or manager of the content matching model first constructs a training dataset for training the model. The model to be trained is then trained using this training dataset to obtain a trained content matching model. This model is then used for actual image-text matching and to execute downstream tasks. Specifically, the solution in this application generates corresponding descriptive text based on an existing image dataset to construct a training dataset. For example, ten thousand images with a resolution of 1024x1024 or higher can be randomly selected from an existing image data source. Corresponding training text is then generated for each image, and the images and text are merged to form the training dataset. Please refer to [link to relevant documentation]. Figure 4 , Figure 4 This is a schematic diagram illustrating the automatic text data annotation process in an embodiment of this application. For example... Figure 4 As shown, the input image I contains three training targets: target 1, target 2, and target 3. The input image is fed into an image segmentation model for automatic segmentation, resulting in mask information for several targets. After filtering out masks that are too small, too large, or too proportionally large based on area and aspect ratio, the final targets to be labeled and their corresponding mask representations M = {M1, M2, M3} remain. Subsequently, the input image I and the mask M are fed into a multimodal large language model for image-to-text conversion, yielding text descriptions for each target, which are then merged into a detailed long text description T = {T1, T2, T3}.

[0092] The training dataset, containing high resolution and long text, was used to train the content matching model. Specifically, the scheme of this application first trains the image encoder of the teacher model based on the training dataset. and text encoder φ T Fine-tuning is performed to achieve the ability to understand high-resolution images and long texts. During the training of the teacher model, the training dataset can be divided into multiple batches. For each batch of data, it is first processed by an image encoder. and text encoder φ T The image and text features are processed to obtain them. Then, the inner product of all text and image features is calculated separately, resulting in a matrix. From the image perspective, the row direction represents a classifier, and from the text perspective, the column direction also represents a classifier. The matching relationships between images and text in the training dataset are known; therefore, the objective function is to maximize the inner product of the same pair of image and text features (i.e., the elements on the matrix diagonal) while minimizing the inner product with irrelevant features. In the training of the teacher model, target-level region feature constraints are added to achieve finer-grained feature training. The input image resolution in this application is 512x512 and 1024x1024, but other reasonable image resolutions are also acceptable. The same method is used for pre-training of the student model to obtain higher-quality weights to improve the distillation effect of the student model. Specifically, ViT-L / 14CLIP can be used as the teacher visual model. ViT-B / 16 CLIP as a student visual model A 24-layer / 1024-dimensional Transformer as a teacher text encoder φ T A 12-layer / 512-dimensional Transformer as a student text encoder φ S Please see. Figure 5 , Figure 5 This is a schematic flowchart illustrating the model training process in an embodiment of this application. Figure 5 As shown, given an input high-resolution image I, which is processed by the teacher's visual model and student visual models After processing, the global image features f of the teacher model and the student model are obtained respectively. I and l I The region image features are obtained by pooling the target location information M = {M1, M2, M3} in the feature space within the region of interest. and For image description It is transmitted via the teacher text encoder φ T and student text encoder After processing, the global text features f of the teacher model and the student model are obtained respectively. T / l T and regional text features During the training phase, three multi-granularity distillation loss functions are used to constrain the learning of the student network. The first is the global loss, which is the loss based on global image features. I and global text features l T The first is the contrast loss between the two. The second is the target-level loss, i.e., the loss based on region image features. and regional text features The contrast loss between the two models is used to enhance the student model's perception of fine-grained targets. The third type is the distillation loss of the teacher model over the student model, which includes global and local components. The global relevance probability distribution is given by f. I and f T The calculations show that the distillation of the student model is achieved by using l. I and l T The resulting global correlation probability distribution. The local correlation probability distribution is determined by the teacher. The calculated correlation matrix serves as the truth value, constraining the students. The relevance matrix is ​​calculated. Specifically, the student image encoder and student text encoder can be trained using a paradigm and learnable parameters of the mapper, with the weights of the pre-trained student model frozen during training. The trained student model can then be used as a content matching model for image-text matching.

[0093] It should be noted that although the steps of the method in this application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0094] The following describes the implementation of the apparatus of this application, which can be used to execute the content matching method in the above embodiments of this application. Figure 6 A schematic block diagram illustrating the composition of the content matching device in an embodiment of this application is shown. For example... Figure 6As shown, the content matching device 600 mainly includes:

[0095] The acquisition module 610 is configured to acquire an image to be processed and text to be processed, wherein the image to be processed contains a target to be identified, and the text to be processed contains target description text of the target to be identified in the image to be processed;

[0096] The image encoding module 620 is configured to encode the image to be processed using a content matching model to obtain global image features for the image to be processed and regional image features for the target to be identified.

[0097] The text encoding module 630 is configured to perform text encoding on the text to be processed through the content matching model to obtain global text features for the text to be processed and regional text features for the target description text.

[0098] The result determination module 640 is configured to determine the matching result between the image to be processed and the text to be processed based on the contrast loss between the global image features and the global text features and the contrast loss between the regional image features and the regional text features.

[0099] In some embodiments of this application, based on the above technical solutions, the content matching device further includes:

[0100] The training data acquisition module is configured to acquire a training dataset, which includes training images and corresponding training text. The training images include training targets and target location information of the training targets in the training images. The training text includes training description text for each training target in the training images.

[0101] The training module is configured to perform image encoding on the training image and the target location information through a first pre-trained model to obtain first global training image features and first region training image features, and to perform text encoding on the training text and the training description text to obtain first global training text features and first region training text features.

[0102] The comparison module is configured to compare the first global training image features and the first global training text features to obtain the global training loss, and to compare the first region training image features and the first region training text features to obtain the target training loss.

[0103] The parameter adjustment module is configured to adjust the parameters of the first pre-trained model based on the global training loss and the target training loss to obtain the content matching model.

[0104] In some embodiments of this application, based on the above technical solutions, the parameter adjustment module is specifically configured to: perform image encoding on the training image and the target location information using a second pre-trained model to obtain second global training image features and second region training image features; and perform text encoding on the training text and the training description text to obtain second global training text features and second region training text features, wherein the number of parameters of the second pre-trained model is higher than the number of parameters of the first pre-trained model; perform correlation calculation based on the first global training image features and the first global training text features to determine the first global correlation distribution of the first pre-trained model; and perform correlation calculation based on the second global training image features and the second global training text features. The process involves: calculating and determining the second global relevance distribution of the second pre-trained model; performing relevance calculations based on the training image features and training text features of the first region to determine the first region relevance distribution of the first pre-trained model; performing relevance calculations based on the training image features and training text features of the second region to determine the second region relevance distribution of the second pre-trained model; obtaining a relevance loss based on the divergence between the first and second global relevance distributions and the divergence between the first and second region relevance distributions; and adjusting the parameters of the first pre-trained model based on the global training loss, the target training loss, and the relevance loss to obtain the content matching model.

[0105] In some embodiments of this application, based on the above technical solutions, the training data acquisition module is specifically configured to: acquire a set of resource images, each resource image in the set containing a training target; perform image segmentation on each resource image to obtain mask information of the training target, the mask information serving as target location information indicating the training target in the training image; perform target recognition based on the mask information of the training target and the corresponding resource image to obtain descriptive text of the training target; merge each resource image and the mask information of the training target in each resource image to obtain a training image, and merge the descriptive text of the training target in each resource image to obtain the training text corresponding to the training image, which serves as the training dataset.

[0106] In some embodiments of this application, based on the above technical solutions, the resource image set further includes labeling information of the training target, which is used to indicate the position of the training target in the resource image; the training data acquisition module is specifically configured to: perform image encoding on the resource image to obtain resource image encoding; generate resource hint encoding corresponding to the training target based on the preset embedding corresponding to the position encoding of the labeling information of the training target in the resource image and the category of the labeling information; and perform mask prediction based on the resource image encoding and the resource hint encoding to obtain the mask information of the training target.

[0107] In some embodiments of this application, based on the above technical solutions, the second training module is further configured to: encode the training images in the training dataset using the image encoder of the second model to be trained to obtain second pre-trained image features, wherein the pre-trained image features are determined based on regional feature constraints for the training target in the training images; encode the training text in the training dataset using the text encoder of the second pre-trained model to obtain second pre-trained text features; and adjust the parameters of the second model to be trained according to the matching result between the second pre-trained image features and the second pre-trained text features to obtain the second pre-trained model.

[0108] In some embodiments of this application, based on the above technical solutions, the first training module is further configured to: merge the prompt information parameters of the first model to be trained with the training images in the training dataset to obtain training input data; encode the training input data through the image encoder of the first model to be trained to obtain first pre-trained image features; and adjust the prompt information parameters according to the matching result between the first pre-trained image features and the second pre-trained text features to obtain the first pre-trained model.

[0109] In some embodiments of this application, based on the above technical solutions, the first training module is specifically configured to: perform dimensional mapping on the first pre-trained image features according to the feature dimension information of the second pre-trained text features, and obtain a feature mapping result by using the mapping parameters of the first model to be trained; adjust the prompt information parameters and the mapping parameters according to the matching result between the feature mapping result and the second pre-trained text features, and obtain the first pre-trained model.

[0110] In some embodiments of this application, based on the above technical solutions, the result determination module 640 is further configured to: determine the target description text corresponding to each target to be identified in the image to be processed according to the matching result; and classify each target to be identified in the image to be processed according to the target description text.

[0111] In some embodiments of this application, based on the above technical solutions, the result determination module 640 is further configured to: input the text to be processed into an image generation model for text encoding to obtain a generated image embedding; adjust the parameters of the image generation model based on the matching result between the generated image embedding and the image features of the image to be processed that match the text to be processed; perform embedding prediction based on the input text through the image generation model to obtain a predicted image embedding; and decode the predicted image embedding through the image generation model to obtain a generated image corresponding to the input text.

[0112] It should be noted that the apparatus provided in the above embodiments and the method provided in the above embodiments belong to the same concept, and the specific way in which each module performs the operation has been described in detail in the method embodiments, and will not be repeated here.

[0113] Figure 7 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0114] It should be noted that, Figure 7 The computer system 700 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0115] like Figure 7 As shown, the computer system 700 includes a Central Processing Unit (CPU) 701, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 702 or programs loaded from storage section 708 into Random Access Memory (RAM) 703. The RAM 703 also stores various programs and data required for system operation. The CPU 701, ROM 702, and RAM 703 are interconnected via a bus 704. An Input / Output (I / O) interface 705 is also connected to the bus 704.

[0116] The following components are connected to I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), and speakers, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN (Local Area Network) card and a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 710 as needed so that computer programs read from it can be installed into storage section 708 as needed.

[0117] Specifically, according to embodiments of this application, the processes described in the various method flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by central processing unit (CPU) 701, it performs various functions defined in the system of this application.

[0118] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0119] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0120] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0121] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0122] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0123] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A content matching method, characterized in that, include: Obtain an image to be processed and text to be processed, wherein the image to be processed contains a target to be identified, and the text to be processed contains target description text of the target to be identified in the image to be processed; The image to be processed is encoded using a content matching model to obtain global image features for the image to be processed and regional image features for the target to be identified. The text to be processed is encoded using the content matching model to obtain global text features for the text to be processed and regional text features for the target descriptive text. The matching result between the image to be processed and the text to be processed is determined based on the contrast loss between the global image features and the global text features and the contrast loss between the regional image features and the regional text features.

2. The method according to claim 1, characterized in that, The method further includes: Obtain a training dataset, which includes training images and corresponding training text. The training images include training targets and their target location information within the training images. The training text includes training description text for each training target in the training images. The training image and the target location information are image encoded by the first pre-trained model to obtain the first global training image features and the first region training image features, and the training text and the training description text are text encoded to obtain the first global training text features and the first region training text features. The global training loss is obtained by comparing the first global training image features and the first global training text features, and the target training loss is obtained by comparing the first region training image features and the first region training text features. Based on the global training loss and the target training loss, the parameters of the first pre-trained model are adjusted to obtain the content matching model.

3. The method according to claim 2, characterized in that, The step of adjusting the parameters of the first pre-trained model based on the global training loss and the target training loss to obtain the content matching model includes: The training image and the target location information are image encoded by the second pre-trained model to obtain the second global training image features and the second region training image features. The training text and the training description text are text encoded to obtain the second global training text features and the second region training text features. The number of parameters of the second pre-trained model is higher than that of the first pre-trained model. Based on the first global training image features and the first global training text features, a correlation calculation is performed to determine the first global correlation distribution of the first pre-trained model; and based on the second global training image features and the second global training text features, a correlation calculation is performed to determine the second global correlation distribution of the second pre-trained model. Based on the features of the first region training image and the features of the first region training text, a correlation calculation is performed to determine the first region correlation distribution of the first pre-trained model; and based on the features of the second region training image and the features of the second region training text, a correlation calculation is performed to determine the second region correlation distribution of the second pre-trained model. The correlation loss is obtained based on the divergence between the first global correlation distribution and the second global correlation distribution, and based on the divergence between the first regional correlation distribution and the second regional correlation distribution. Based on the global training loss, the target training loss, and the relevance loss, the parameters of the first pre-trained model are adjusted to obtain the content matching model.

4. The method according to claim 2, characterized in that, The acquisition of the training dataset includes: Obtain a set of resource images, wherein each resource image in the set contains the training target; Each resource image is segmented to obtain the mask information of the training target, and the mask information serves as the target location information of the training target in the training image; Based on the mask information of the training target and the corresponding resource image, target recognition is performed to obtain the descriptive text of the training target; The training image is obtained by merging the mask information of each resource image and the training target in each resource image, and the training text corresponding to the training image is obtained by merging the description text of each training target in each resource image, which serves as the training dataset.

5. The method according to claim 4, characterized in that, The resource image set also includes labeling information of the training target, which is used to indicate the location of the training target in the resource image; The step of segmenting each resource image to obtain the mask information of the training target includes: The resource image is image encoded to obtain the resource image code; Based on the position encoding of the label information of the training target in the resource image and the preset embedding corresponding to the category of the label information, a resource cue code corresponding to the training target is generated; Mask prediction is performed based on the resource image encoding and the resource cue encoding to obtain the mask information of the training target.

6. The method according to claim 3, characterized in that, After obtaining the training dataset, the method further includes: The training images in the training dataset are encoded by the image encoder of the second model to be trained to obtain the second pre-trained image features, wherein the pre-trained image features are determined based on the regional feature constraints of the training target in the training images; The training text in the training dataset is encoded by the text encoder of the second pre-trained model to obtain the second pre-trained text features; Based on the matching results between the second pre-trained image features and the second pre-trained text features, the parameters of the second model to be trained are adjusted to obtain the second pre-trained model.

7. The method according to claim 6, characterized in that, The method further includes: The training input data is obtained by merging the prompt information parameters of the first model to be trained with the training images in the training dataset; The training input data is encoded by the image encoder of the first model to be trained to obtain the first pre-trained image features; Based on the matching results between the first pre-trained image features and the second pre-trained text features, the prompt information parameters are adjusted to obtain the first pre-trained model.

8. The method according to claim 7, characterized in that, The step of adjusting the prompt information parameters based on the matching result between the first pre-trained image features and the second pre-trained text features to obtain the first pre-trained model includes: Based on the feature dimension information of the second pre-trained text features, the first pre-trained image features are dimension-mapped using the mapping parameters of the second model to be trained, and the feature mapping result is obtained. Based on the matching results between the feature mapping results and the second pre-trained text features, the prompt information parameters and the mapping parameters are adjusted to obtain the first pre-trained model.

9. The method according to claim 1, characterized in that, The method further includes: Based on the matching results, determine the target description text corresponding to each target to be identified in the image to be processed; The target objects in the image to be processed are categorized according to the target description text.

10. The method according to claim 1, characterized in that, The method further includes: The text to be processed is input into the image generation model for text encoding to obtain the generated image embedding; Based on the matching results between the generated image embedding and the image features of the image to be processed that match the text to be processed, the parameters of the image generation model are adjusted; The image generation model performs embedding prediction based on the input text to obtain the predicted image embedding; The predicted image embedding is decoded using the image generation model to obtain a generated image corresponding to the input text.

11. A content matching device, characterized in that, include: The acquisition module is configured to acquire an image to be processed and text to be processed, wherein the image to be processed contains a target to be identified, and the text to be processed contains target description text of the target to be identified in the image to be processed; The image encoding module is configured to encode the image to be processed using a content matching model to obtain global image features for the image to be processed and regional image features for the target to be identified. The text encoding module is configured to encode the text to be processed using the content matching model to obtain global text features for the text to be processed and regional text features for the target description text. The result determination module is configured to determine the matching result between the image to be processed and the text to be processed based on the contrast loss between the global image features and the global text features and the contrast loss between the regional image features and the regional text features.

12. An electronic device, characterized in that, include: processor; Memory for storing the executable instructions of the processor; The processor is configured to execute the content matching method of any one of claims 1 to 10 by executing the executable instructions.

13. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the content matching method as described in any one of claims 1 to 10.

14. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium, a processor of a computer device reading the computer instructions from the computer-readable storage medium, and the processor executing the computer instructions to cause the computer device to perform the content matching method as described in any one of claims 1 to 10.

Images