Text image matching model training method, picture labeling method, device and equipment

Abstract

The application relates to a text image matching model training method and a picture labeling method. The method comprises the following steps: obtaining sample pictures, sample texts and labeling information of the sample texts and the sample pictures; extracting global features, local features and region of interest features of the sample pictures through an image feature extractor of a text image matching model; extracting original text features and abstract text features of the sample texts through a text feature extractor of the text image matching model, wherein the abstract text features are generated based on the labeling information of the sample texts and the sample pictures; performing contrast learning based on the global features, the abstract text features, the local features and the original text features, the region of interest features and the original text features, the region of interest features and the abstract text features to generate loss terms; calculating a Hungarian loss based on the loss terms; and training the text image matching model according to the Hungarian loss. The method can automatically label pictures.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a text-image matching model training method, a picture labeling method, a device, a computer device, a storage medium and a computer program product. BACKGROUND

[0002] Looking at various basic tasks of computer vision, such as image classification, object detection, semantic segmentation, etc., the research and application route of the data level has long followed the learning paradigm of accurate labeling of picture data sets. Visual detection tasks based on this paradigm have achieved good results. However, this method has been limited to using only limited computer vision field label learning mechanism, and requires high labor labeling cost.

[0003] However, with the relatively mature development trend of deep learning in the field of deep learning, researchers pursue more extreme model learning and generalization ability, and application users gradually increase the requirements for human cost, task learning period and deployment efficiency. The current learning paradigm undoubtedly hinders the development of the task. SUMMARY

[0004] Therefore, it is necessary to provide a text-image matching model training method, a picture labeling method, a device, a computer device, a storage medium and a computer program product capable of automatically establishing the matching relationship between pictures and texts in view of the above technical problems.

[0005] A text-image matching model training method, the method comprising:

[0006] obtaining a sample picture, a sample text, and labeling information of the sample text and the sample picture;

[0007] extracting global features, local features and region of interest features of the sample picture by an image feature extractor of a text-image matching model;

[0008] extracting original text features and summary text features of the sample text by a text feature extractor of the text-image matching model, the summary text features being generated based on the labeling information of the sample text and the sample picture;

[0009] generating a first loss term based on a comparison learning of the global features and the summary text features, generating a second loss term based on a comparison learning of the local features and the original text features, generating a third loss term based on a comparison learning of the region of interest features and the original text features, and generating a fourth loss term based on a comparison learning of the region of interest features and the summary text features;

[0010] calculate a hungarian loss based on the first loss term, the second loss term, the third loss term, and the fourth loss term;

[0011] train the text-image matching model according to the hungarian loss.

[0012] In one of the embodiments, the process of extracting the global feature of the sample picture comprises:

[0013] crop the sample picture according to a first cropping ratio to obtain a global picture;

[0014] extract a feature from the global picture to obtain a global feature;

[0015] the process of extracting the local feature of the sample picture comprises:

[0016] crop the sample picture according to a second cropping ratio to obtain a local picture, the second cropping ratio being smaller than the first cropping ratio;

[0017] extract a feature from the local picture to obtain a local feature;

[0018] the process of extracting the region-of-interest feature of the sample picture comprises:

[0019] identify a region-of-interest of the sample picture, and obtain region position information of the region-of-interest;

[0020] encode the region-of-interest to obtain an image feature, and obtain a region-of-interest feature according to the image feature and the region position information.

[0021] In one of the embodiments, the process of extracting the original text feature of the sample text comprises:

[0022] extract a text feature of the sample text as an original text feature;

[0023] the process of extracting the summary text feature of the sample text comprises:

[0024] filter the sample text according to the annotation information of the sample text and the sample picture;

[0025] extract a text feature of the filtered sample text as a summary text feature.

[0026] In one of the embodiments, the process of generating the first loss term based on the global feature and the summary text feature comprises:

[0027] calculate a first similarity between the global feature and the summary text feature, and a second similarity between the summary text feature and the local feature;

[0028] generate a first loss term based on the first similarity and the second similarity;

[0029] generate a second loss term based on the local feature and the original text feature through contrast learning, comprising:

[0030] calculate a third similarity of the local feature and the original text feature, and a fourth similarity of the original text feature and the local feature;

[0031] generate a second loss term based on the third similarity and the fourth similarity;

[0032] generate a third loss term based on the region of interest feature and the original text feature through contrast learning, comprising:

[0033] calculate a fifth similarity of the region of interest feature and the original text feature, and a sixth similarity of the original text feature and the region of interest feature;

[0034] generate a third loss term based on the fifth similarity and the sixth similarity;

[0035] generate a fourth loss term based on the region of interest feature and the summary text feature through contrast learning, comprising:

[0036] calculate a seventh similarity of the region of interest feature and the summary text feature, and an eighth similarity of the summary text feature and the region of interest feature;

[0037] generate a fourth loss term based on the seventh similarity and the eighth similarity.

[0038] A picture labeling method, comprising:

[0039] receiving a picture to be processed and text to be processed;

[0040] inputting the picture to be processed and the text to be processed into a text-image matching model trained according to any one of the above embodiments to obtain labeled text corresponding to a position of the picture to be processed.

[0041] A text-image matching model training device, comprising:

[0042] a sample acquisition module configured to acquire a sample picture, a sample text, and labeled information of the sample text and the sample picture;

[0043] an image feature extraction module configured to extract global features, local features, and region of interest features of the sample picture by an image feature extractor of the text-image matching model;

[0044] a text feature extraction module configured to extract original text features and summary text features of the sample text by a text feature extractor of the text-image matching model, the summary text features being generated based on the sample text and the annotation information of the sample picture;

[0045] a loss term generation module configured to generate a first loss term based on the global features and the summary text features through contrastive learning, generate a second loss term based on the local features and the original text features through contrastive learning, generate a third loss term based on the region of interest features and the original text features through contrastive learning, and generate a fourth loss term based on the region of interest features and the summary text features through contrastive learning;

[0046] a hungarian loss calculation module configured to calculate a hungarian loss based on the first loss term, the second loss term, the third loss term, and the fourth loss term;

[0047] a training module configured to train the text-image matching model according to the hungarian loss.

[0048] A picture annotation apparatus, the picture annotation apparatus comprising:

[0049] a receiving module configured to receive a picture to be processed and text to be processed;

[0050] an annotation module configured to input the picture to be processed and the text to be processed into a text-image matching model trained according to any one of the above embodiments to obtain annotation text of a corresponding position of the picture to be processed.

[0051] A computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing steps of the method described in any one of the above embodiments when executing the computer program.

[0052] A computer readable storage medium storing a computer program, the computer program implementing steps of the method described in any one of the above embodiments when executed by a processor.

[0053] A computer program product comprising a computer program, the computer program implementing steps of the method described in any one of the above embodiments when executed by a processor.

[0054] The text image matching model training method, picture labeling method, device, computer equipment, storage medium and computer program product can more accurately align images and texts in a hierarchical structure. A double-flow network based on texts and images is constructed with an input pyramid having different semantic levels on both sides. According to global features, local features and region of interest features, original text features and abstract text features are subjected to visual modeling and language modeling. Then, a first loss term is generated through contrast learning based on the global features and the abstract text features, a second loss term is generated through contrast learning based on the local features and the original text features, a third loss term is generated through contrast learning based on the region of interest features and the original text features, and a fourth loss term is generated through contrast learning based on the region of interest features and the abstract text features. The Hungarian loss is calculated based on the first loss term, the second loss term, the third loss term and the fourth loss term, so that the matching is more accurate and the potential of text-picture cross-modal visual representation learning is stimulated. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 An application environment diagram of a text image matching model training method in an embodiment;

[0056] Figure 2 A flowchart of a text image matching model training method in an embodiment;

[0057] Figure 3 A schematic diagram of adversarial learning in an embodiment;

[0058] Figure 4 A flowchart of a picture labeling method in an embodiment;

[0059] Figure 5 A structural diagram of a model in an embodiment;

[0060] Figure 6 A structural block diagram of a text image matching model training device in an embodiment;

[0061] Figure 7 A structural block diagram of a picture labeling device in an embodiment;

[0062] Figure 8 An internal structural diagram of a computer equipment in an embodiment. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0064] The text-picture matching model training method and picture labeling method provided by the embodiments of the present application can be applied to an application environment as shown in Figure 1 The terminal 102 communicates with the server 104 through a network. The data storage system can store data required to be processed by the server 104. The data storage system can be integrated on the server 104, or placed on a cloud or other network server.

[0065] The server 104 obtains sample pictures, sample texts, and annotation information of the sample texts and the sample pictures; extracts global features, local features, and region of interest features of the sample pictures through a picture feature extractor of a text-picture matching model; extracts original text features and summary text features of the sample texts through a text feature extractor of the text-picture matching model, the summary text features being generated based on the annotation information of the sample texts and the sample pictures; generates a first loss term based on a comparison learning of the global features and the summary text features, generates a second loss term based on a comparison learning of the local features and the original text features, generates a third loss term based on a comparison learning of the region of interest features and the original text features, and generates a fourth loss term based on a comparison learning of the region of interest features and the summary text features; calculates a Hungarian loss based on the first loss term, the second loss term, the third loss term, and the fourth loss term; and trains the text-picture matching model according to the Hungarian loss. In this way, after the model is trained, it can be installed on a terminal to facilitate picture labeling.

[0066] The text-picture matching model training method described above aligns pictures and texts in a hierarchical structure more accurately, constructs an input pyramid with different semantic levels on both sides of a dual-flow network based on texts and pictures, and performs visual modeling and language modeling on global features, local features, and region of interest features, original text features, and summary text features, and then generates a first loss term based on a comparison learning of the global features and the summary text features, generates a second loss term based on a comparison learning of the local features and the original text features, generates a third loss term based on a comparison learning of the region of interest features and the original text features, and generates a fourth loss term based on a comparison learning of the region of interest features and the summary text features; calculates a Hungarian loss based on the first loss term, the second loss term, the third loss term, and the fourth loss term, so that the matching is more accurate and the potential of text-picture cross-modal visual representation learning is stimulated.

[0067] The terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle-mounted device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server 104 can be implemented by a stand-alone server or a server cluster composed of multiple servers.

[0068] In one embodiment, as shown in Figure 2 , a text-picture matching model training method is provided. Taking the server in Figure 1 as an example, the method includes the following steps:

[0069] S202: Obtain a sample picture, a sample text, and annotation information of the sample text and the sample picture.

[0070] Specifically, the sample picture and the sample text are a corresponding sample, wherein the user pre-annotates the sample picture and the sample text to obtain standard information, for example, annotates a region or the whole of the sample picture by the text in the sample text, so as to establish the association between the local text or the whole text in the sample text and the local or whole of the sample picture.

[0071] S204: Extract global features, local features, and region-of-interest features of the sample picture by a picture feature extractor of the text-picture matching model.

[0072] Specifically, the global features refer to the features of the whole of the sample picture, the local features refer to the features of the region obtained by cropping the sample picture, and the region-of-interest features can refer to the features of a region in the sample picture.

[0073] S206: Extract original text features and abstract text features of the sample text by a text feature extractor of the text-picture matching model, wherein the abstract text features are generated based on the annotation information of the sample text and the sample picture.

[0074] Specifically, the original text features are the text features extracted before the text is deleted, and the abstract text features are the text features extracted after the text is deleted. In the sample, the abstract text features are generated based on the annotation information of the sample text and the sample picture, for example, the annotated sample text is taken as the abstract text, so as to extract the abstract text features.

[0075] S208: generating a first loss term based on the global feature and the summary text feature, generating a second loss term based on the local feature and the original text feature, generating a third loss term based on the region of interest feature and the original text feature, and generating a fourth loss term based on the region of interest feature and the summary text feature.

[0076] Specifically, in combination with Figure 3 as shown in FIG. 2, where Figure 3 is a schematic diagram of the adversarial learning in an embodiment. Semantic mismatch between the visual modality and the language modality usually exists in the text-picture pair, for example, title redundancy, picture redundancy, missing relationship between target instances, mutual compatibility between data, and the like. For example, a certain text / picture can correspond to another text / picture or multiple texts / pictures, and the traditional method directly regards the other as a negative sample without considering the relevance, which can cause model overfitting. To this end, the embodiment more accurately aligns the picture and the text in the form of a hierarchical structure, and based on the dual-flow network of the text and the picture, an input pyramid with different semantic levels is constructed on both sides thereof, and the features of the global picture, the local picture region, and the salient instances in the picture are used to perform visual modeling and language modeling on the original text and the text summary.

[0077] Specifically, for the intra-hierarchical semantic alignment, since the global region of the picture and the text summary both contain global semantic information, and the local region and the original text contain more fine-grained semantic information, they are regarded as two pairs of positive samples, that is, a first loss term is generated based on the global feature and the summary text feature, and a second loss term is generated based on the local feature and the original text feature.

[0078] For the cross-hierarchical relationship alignment, in order to avoid the modeling of the target relationship by the visual encoder being overwhelmed by the scene semantic modeling, the embodiment aligns the relationship between the target instances with the language elements, that is, a third loss term is generated based on the region of interest feature and the original text feature, and a fourth loss term is generated based on the region of interest feature and the summary text feature.

[0079] S210: calculating a Hungarian loss based on the first loss term, the second loss term, the third loss term, and the fourth loss term.

[0080] S212: training the text-picture matching model according to the Hungarian loss.

[0081] Specifically, for the compatibility problem between the picture / text pair, the embodiment softens the loss term of the negative sample, that is, the unpaired sample, in the contrast learning process, so as to alleviate the strict loss constraint, and thus weaken the negative effect of certain local similarity.

[0082] For N image-text pairs in a batch Where i represents the i-th pair, and normalized embedding vectors of the same dimension are obtained through a two-stream encoder. Specifically, the image encoder generates features from the global cropped image G, the local cropped image L, and the ROI feature sequence, respectively, while the image encoder generates global features from the global cropped image G, the local cropped image L, and the ROI feature sequence, respectively. Local features Region of Interest Features The text encoder extracts text summaries T from the text. S Generate original text features from the original text T Text summarization features

[0083] Then, using the above vector set, four monitoring signals L are constructed. GS L LT L RS L RT It is used for in-batch contrastive learning, aiming to achieve alignment between visual representations and language representations from different semantic levels.

[0084] In one embodiment, a first loss term is generated by comparative learning based on global features and summary text features, including: calculating a first similarity between global features and summary text features, and a second similarity between summary text features and local features; generating a first loss term based on the first and second similarities; a second loss term is generated by comparative learning based on local features and original text features, including: calculating a third similarity between local features and original text features, and a fourth similarity between original text features and local features; generating a second loss term based on the third and fourth similarities; a third loss term is generated by comparative learning based on region of interest features and original text features, including: calculating a fifth similarity between region of interest features and original text features, and a sixth similarity between original text features and region of interest features; generating a third loss term based on the fifth and sixth similarities; and a fourth loss term is generated by comparative learning based on region of interest features and summary text features, including: calculating a seventh similarity between region of interest features and summary text features, and an eighth similarity between summary text features and region of interest features; generating a fourth loss term based on the seventh and eighth similarities.

[0085] For ease of understanding, The first loss term L GS For example, for the i-th pair, the normalized image-language similarity and the language-image similarity can be calculated using the following formula:

[0086]

[0087]

[0088] a first similarity of the global feature and the summary text feature, a second similarity of the summary text feature and the local feature, sim is a similarity calculation, τ is a constant term, with the similarity distinction, the matching between each patch, the loss function calculation here adopts the Hungarian loss, to get a more accurate matching result:

[0089]

[0090] where b i is a vector that defines the center coordinates of the true box and its height and width relative to the size of the image, c i is the target class label, which cannot be an empty set, identifies the probability of class c i , and the prediction box is L box is the score of the bounding box.

[0091] In this embodiment, for the semantically aligned picture patch, the Hungarian matching loss is added according to the similarity, so that the matching is more accurate, and the potential of text-picture cross-modal visual representation learning is stimulated.

[0092] The above text-picture matching model training method more accurately aligns the picture and the text in a hierarchical structure, constructs an input pyramid with different semantic levels on both sides of the text and picture double-flow network, and performs visual modeling and language modeling on the global feature, the local feature, and the region of interest feature, the original text feature, and the summary text feature. Then, based on the global feature and the summary text feature, a first loss term is generated by contrast learning, based on the local feature and the original text feature, a second loss term is generated by contrast learning, based on the region of interest feature and the original text feature, a third loss term is generated by contrast learning, and based on the region of interest feature and the summary text feature, a fourth loss term is generated by contrast learning. Based on the first loss term, the second loss term, the third loss term, and the fourth loss term, the Hungarian loss is calculated to make the matching more accurate and stimulate the potential of text-picture cross-modal visual representation learning.

[0093] In one embodiment, the process of extracting global features of the sample image includes: cropping the sample image according to a first cropping ratio to obtain a global image; and extracting features from the global image to obtain global features. The process of extracting local features of the sample image includes: cropping the sample image according to a second cropping ratio, where the second cropping ratio is smaller than the first cropping ratio; and extracting features from the local image to obtain local features. The process of extracting region of interest (ROI) features of the sample image includes: identifying the ROI of the sample image and obtaining the ROI's location information; encoding the ROI to obtain image features; and obtaining the ROI features based on the image and the ROI's location information.

[0094] In one embodiment, the process of extracting the original text features of the sample text includes: extracting the text features of the sample text as the original text features; the process of extracting the summary text features of the sample text includes: filtering the sample text based on the annotation information of the sample text and the sample image; and extracting the text features of the filtered sample text as the summary text features.

[0095] For semantic alignment within a hierarchy, since the global regions of the image and text summary contain global semantic information, while the local regions and the original text contain finer-grained semantic information, they are considered as two pairs of positive samples.

[0096] Specifically, the global view G is first randomly cropped, with the cropping ratio set to [0.9, 1.0]. This operation still retains all the information from the original image. In other embodiments, the cropping ratio ranges from 0.9 to 1.0. Assuming a random cropping ratio of 0.93 (93%) is chosen, a 93% random cropping of an image still preserves most of the effective area. Furthermore, this cropping maintains the variability of the data. This operation is primarily used for data augmentation, enriching the training data format to obtain a model with better generalization performance. (Text summary T) S The original text T was compressed, removing some redundant and overly detailed information. G and T S Both capture global information and can be used as paired positive samples. Through contrastive learning, G and T... S Projection embedding v g and l S It shortens the distance, where V (vision) refers to the image and l (language) refers to the text.

[0097] For fine-grained local information comparison learning, since the global view G and the aforementioned text summary T... SThe alignment is relatively coarse, thus discarding a large amount of fine-grained information. This embodiment aims to align image sub-regions with certain descriptions in the title. Therefore, this embodiment introduces fine-grained local contrast. The random cropping ratio used to generate the local view L is set to [0.6, 1], focusing on sub-regions of image I. The original text T contains many detailed descriptions, making it more suitable to consider it as a positive sample of L. Then, the projection embeddings of L and T are... l and l T They are also combined using contrast loss. Here, V (vision) refers to the image, and l (language) refers to the text.

[0098] For cross-level relationship alignment, to avoid the visual encoder's modeling of target relationships being overwhelmed by scene semantic modeling, this embodiment aligns the relationships between target instances with linguistic elements. To further improve alignment accuracy, ROI feature sequences of salient objects in the image are introduced to provide more supervision. Specifically, given an image I with M salient objects, a pre-trained object detector is used to extract the visual semantics of each object region, defined as [o′]. m , z m ], where m represents the Mth object, o′ m It is a 2048-dimensional feature vector, z m It is a 4-dimensional normalized position vector, representing the coordinates of the top left and bottom right corners.

[0099] Through cascading o′ m and z m This yields a 2048-dimensional location-sensitive ROI feature vector O. m This forms a ROI feature sequence. To enhance the text encoder's ability to model conceptual relationships while avoiding weakening the visual encoder's reasoning ability, (v r , l s ) and (v r , l t ) are used as the other two positive pairs, while minimizing v r and l s The distance between them and v r and l t The distance between them. Since the visual modality uses very fine-grained instance-level input, while the language modality uses complete sentences, i.e., text summaries and original text, this training process is referred to here as cross-level relation alignment.

[0100] In summary, a linear projection layer projects the patch tokens to a higher dimension and reshapes them. Next, a 3×3 depthwise convolution is used to capture local information. Then, the features are mapped onto the token sequence and reprojected back to the initial dimension. The cls token remains unchanged throughout the process and is concatenated with the locally enhanced patch tokens to generate the final output.

[0101] In the above embodiments, for the text-image data usage method, Hungarian matching based on patch similarity is introduced to maximize the use of image patch similarity and obtain better matching results. Secondly, it solves the problem of traditional technology directly using text-image pairs for task learning, which ignores the text information redundancy, image information redundancy, the relationship between image target instances, and the relationship between different target instances in this type of training data.

[0102] In one embodiment, such as Figure 4 As shown, an image annotation method is provided, which can be applied to... Figure 1 Taking a server or terminal as an example, the following steps are included:

[0103] S402: Receive the image to be processed and the text to be processed.

[0104] S404: Input the image to be processed and the text to be processed into the text-image matching model trained in any of the above input embodiments to obtain the labeled text at the corresponding position of the image to be processed.

[0105] Specifically, in combination Figure 5 As shown, for a given text-image pair, the goal is for the text-image matching model to learn a general visual representation. The text description contains rich semantic information about the target scene in the corresponding image, such as the target object category, color, space, and action state. This rich information representation is extremely valuable for downstream vision tasks such as image classification or object detection.

[0106] To this end, the input text is initially encoded to obtain a semantic representation corresponding to the image. An image decoder extracts visual features from the image. Then, the semantic representation output by the text extractor is fused with the extracted image features for learning. This step uses a linear mapping dependent on different modalities to map the image and text features obtained in the first two steps to a shared embedding space. The image and text feature vectors in the shared space are then concatenated to obtain a complete and independent sequence containing both image and text features. The goal is to learn a scene descriptor for the image content, which predicts and outputs a visual content representation of the input image, thus completing the image annotation.

[0107] In the above embodiments, the text-picture semantic alignment makes it possible and efficient to produce a large amount of high-quality training data, and no longer relies on manual annotation, thereby shortening the task development cycle of algorithm personnel and improving business application efficiency.

[0108] It should be understood that, although each step in the flowchart involved in each of the above embodiments is shown in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each of the above embodiments can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0109] Based on the same inventive concept, the embodiments of the present application also provide a text-picture matching model training device and a picture labeling device for implementing the above-mentioned text-picture matching model training method and picture labeling method. The implementation scheme for solving problems provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more text-picture matching model training device and picture labeling device embodiments provided below can refer to the limitations of the text-picture matching model training method and picture labeling method in the above text, which will not be repeated here.

[0110] In one embodiment, as shown in Figure 6 A text-picture matching model training device is provided, comprising: a sample acquisition module 601, a picture feature extraction module 602, a text feature extraction module 603, a loss term generation module 604, a Hungarian loss calculation module 605, and a training module 606, wherein:

[0111] The sample acquisition module 601 is configured to acquire a sample picture, a sample text, and annotation information of the sample text and the sample picture.

[0112] The picture feature extraction module 602 is configured to extract global features, local features, and region of interest features of the sample picture by a picture feature extractor of the text-picture matching model.

[0113] The text feature extraction module 603 is configured to extract original text features and summary text features of the sample text by a text feature extractor of the text-picture matching model, the summary text features being generated based on the annotation information of the sample text and the sample picture.

[0114] The loss term generation module 604 is configured to generate a first loss term based on the global feature and the summary text feature through contrast learning, generate a second loss term based on the local feature and the original text feature through contrast learning, generate a third loss term based on the region of interest feature and the original text feature through contrast learning, and generate a fourth loss term based on the region of interest feature and the summary text feature through contrast learning.

[0115] The Hungarian loss calculation module 605 is configured to calculate a Hungarian loss based on the first loss term, the second loss term, the third loss term, and the fourth loss term.

[0116] The training module 606 is configured to train the text-picture matching model according to the Hungarian loss.

[0117] In one of the embodiments, the picture feature extraction module 602 is further configured to crop the sample picture according to a first cropping ratio to obtain a global picture, extract a global feature from the global picture, crop the sample picture according to a second cropping ratio to obtain a local picture, the second cropping ratio being smaller than the first cropping ratio, extract a local feature from the local picture, identify a region of interest of the sample picture, obtain region position information of the region of interest, encode the region of interest to obtain a picture feature, and obtain a region of interest feature based on the picture feature and the region position information.

[0118] In one of the embodiments, the text feature extraction module 603 is further configured to extract a text feature of the sample text as an original text feature, filter the sample text according to the annotation information of the sample text and the sample picture, and extract a text feature of the filtered sample text as a summary text feature.

[0119] In one of the embodiments, the loss term generation module 604 is further configured to calculate a first similarity between the global feature and the summary text feature, and a second similarity between the summary text feature and the local feature, generate the first loss term based on the first similarity and the second similarity, calculate a third similarity between the local feature and the original text feature, and a fourth similarity between the original text feature and the local feature, generate the second loss term based on the third similarity and the fourth similarity, calculate a fifth similarity between the region of interest feature and the original text feature, and a sixth similarity between the original text feature and the region of interest feature, generate the third loss term based on the fifth similarity and the sixth similarity, calculate a seventh similarity between the region of interest feature and the summary text feature, and an eighth similarity between the summary text feature and the region of interest feature, and generate the fourth loss term based on the seventh similarity and the eighth similarity.

[0120] In one of the embodiments, as shown in FIG. 6, the text-picture matching model includes a picture feature extraction module 602, a text feature extraction module 603, a loss term generation module 604, a Hungarian loss calculation module 605, and a training module 606. Figure 7As shown in the figure, a picture labeling device is provided, comprising a receiving module 701 and a labeling module 702, wherein:

[0121] The receiving module 701 is configured to receive a picture to be processed and text to be processed.

[0122] The labeling module 702 is configured to input the picture to be processed and the text to be processed into the text-picture matching model trained in any one of the above embodiments, to obtain labeled text of a corresponding position of the picture to be processed.

[0123] The modules in the text-picture matching model training device and the picture labeling device can be all or partially implemented by software, hardware and combinations thereof. The modules can be embedded in or independent of a processor in a computer device in a hardware form, or can be stored in a memory in a computer device in a software form, so as to be called and executed by a processor to perform operations corresponding to the modules.

[0124] In one embodiment, a computer device is provided, which can be a server, and an internal structure diagram thereof can be as shown in the figure. Figure 8 The computer device comprises a processor, a memory and a network interface connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement a text-picture matching model training method and a picture labeling method.

[0125] Those skilled in the art can understand that Figure 8 the structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can comprise more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0126] In one embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program: obtaining a sample picture, a sample text, and annotation information of the sample text and the sample picture; extracting global features, local features, and region of interest features of the sample picture by a picture feature extractor of a text-picture matching model; extracting original text features and summary text features of the sample text by a text feature extractor of the text-picture matching model, the summary text features being generated based on the annotation information of the sample text and the sample picture; performing contrast learning based on the global features and the summary text features to generate a first loss term, performing contrast learning based on the local features and the original text features to generate a second loss term, performing contrast learning based on the region of interest features and the original text features to generate a third loss term, and performing contrast learning based on the region of interest features and the summary text features to generate a fourth loss term; calculating a Hungarian loss based on the first loss term, the second loss term, the third loss term, and the fourth loss term; and training the text-picture matching model according to the Hungarian loss.

[0127] In one embodiment, the extraction process of the global features of the sample picture involved when the processor executes the computer program comprises: cropping the sample picture according to a first cropping ratio to obtain a global picture; and performing feature extraction on the global picture to obtain the global features. The extraction process of the local features of the sample picture involved when the processor executes the computer program comprises: cropping the sample picture according to a second cropping ratio to obtain a local picture, the second cropping ratio being smaller than the first cropping ratio; and performing feature extraction on the local picture to obtain the local features. The extraction process of the region of interest features of the sample picture involved when the processor executes the computer program comprises: identifying a region of interest of the sample picture, and obtaining region position information of the region of interest; performing picture encoding on the region of interest to obtain picture features, and obtaining the region of interest features according to the picture features and the region position information.

[0128] In one embodiment, the extraction process of the original text features of the sample text involved when the processor executes the computer program comprises: extracting text features of the sample text as the original text features. The extraction process of the summary text features of the sample text involved when the processor executes the computer program comprises: filtering the sample text according to the annotation information of the sample text and the sample picture; and extracting text features of the filtered sample text as the summary text features.

[0129] In one embodiment, the processor, when executing the computer program, implements contrastive learning based on the global feature and the summary text feature to generate a first loss term, including: calculating a first similarity of the global feature and the summary text feature, and a second similarity of the summary text feature and the local feature; generating the first loss term based on the first similarity and the second similarity; the processor, when executing the computer program, implements contrastive learning based on the local feature and the original text feature to generate a second loss term, including: calculating a third similarity of the local feature and the original text feature, and a fourth similarity of the original text feature and the local feature; generating the second loss term based on the third similarity and the fourth similarity; the processor, when executing the computer program, implements contrastive learning based on the region of interest feature and the original text feature to generate a third loss term, including: calculating a fifth similarity of the region of interest feature and the original text feature, and a sixth similarity of the original text feature and the region of interest feature; generating the third loss term based on the fifth similarity and the sixth similarity; the processor, when executing the computer program, implements contrastive learning based on the region of interest feature and the summary text feature to generate a fourth loss term, including: calculating a seventh similarity of the region of interest feature and the summary text feature, and an eighth similarity of the summary text feature and the region of interest feature; generating the fourth loss term based on the seventh similarity and the eighth similarity.

[0130] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor, when executing the computer program, implements the following steps: receiving a picture to be processed and text to be processed; inputting the picture to be processed and the text to be processed into the text-picture matching model trained in any one of the above embodiments to obtain the labeled text of the corresponding position of the picture to be processed.

[0131] In one embodiment, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the following steps: obtaining a sample picture, a sample text, and annotation information of the sample text and the sample picture; extracting, by a picture feature extractor of a text-picture matching model, global features, local features, and region-of-interest features of the sample picture; extracting, by a text feature extractor of the text-picture matching model, original text features and summary text features of the sample text, the summary text features being generated based on the annotation information of the sample text and the sample picture; performing contrast learning based on the global features and the summary text features to generate a first loss term, performing contrast learning based on the local features and the original text features to generate a second loss term, performing contrast learning based on the region-of-interest features and the original text features to generate a third loss term, and performing contrast learning based on the region-of-interest features and the summary text features to generate a fourth loss term; calculating a Hungarian loss based on the first loss term, the second loss term, the third loss term, and the fourth loss term; and training the text-picture matching model according to the Hungarian loss.

[0132] In one embodiment, the extraction process of the global features of the sample picture involved when the computer program is executed by the processor includes: cropping the sample picture according to a first cropping ratio to obtain a global picture; and performing feature extraction on the global picture to obtain the global features. The extraction process of the local features of the sample picture involved when the computer program is executed by the processor includes: cropping the sample picture according to a second cropping ratio to obtain a local picture, the second cropping ratio being smaller than the first cropping ratio; and performing feature extraction on the local picture to obtain the local features. The extraction process of the region-of-interest features of the sample picture involved when the computer program is executed by the processor includes: identifying a region of interest of the sample picture, and obtaining region position information of the region of interest; performing picture encoding on the region of interest to obtain picture features, and obtaining the region-of-interest features based on the picture features and the region position information.

[0133] In one embodiment, the extraction process of the original text features of the sample text involved when the computer program is executed by the processor includes: extracting text features of the sample text as the original text features. The extraction process of the summary text features of the sample text involved when the computer program is executed by the processor includes: filtering the sample text according to the annotation information of the sample text and the sample picture; and extracting text features of the filtered sample text as the summary text features.

[0134] In one embodiment, the computer program, when executed by the processor, implements contrastive learning based on the global feature and the summary text feature to generate a first loss term, including: calculating a first similarity of the global feature and the summary text feature, and a second similarity of the summary text feature and the local feature; generating the first loss term based on the first similarity and the second similarity; the computer program, when executed by the processor, implements contrastive learning based on the local feature and the original text feature to generate a second loss term, including: calculating a third similarity of the local feature and the original text feature, and a fourth similarity of the original text feature and the local feature; generating the second loss term based on the third similarity and the fourth similarity; the computer program, when executed by the processor, implements contrastive learning based on the region of interest feature and the original text feature to generate a third loss term, including: calculating a fifth similarity of the region of interest feature and the original text feature, and a sixth similarity of the original text feature and the region of interest feature; generating the third loss term based on the fifth similarity and the sixth similarity; the computer program, when executed by the processor, implements contrastive learning based on the region of interest feature and the summary text feature to generate a fourth loss term, including: calculating a seventh similarity of the region of interest feature and the summary text feature, and an eighth similarity of the summary text feature and the region of interest feature; generating the fourth loss term based on the seventh similarity and the eighth similarity.

[0135] In one embodiment, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, and the computer program, when executed by a processor, implements the following steps: receiving a picture to be processed and text to be processed; inputting the picture to be processed and the text to be processed into the text-picture matching model trained in any one of the above embodiments to obtain the labeled text of the corresponding position of the picture to be processed.

[0136] In one embodiment, a computer program product is provided, including a computer program, and the computer program, when executed by a processor, implements the following steps: obtaining a sample picture, sample text, and labeled information of the sample text and the sample picture; extracting, by a picture feature extractor of a text-picture matching model, a global feature, a local feature, and a region of interest feature of the sample picture; extracting, by a text feature extractor of the text-picture matching model, an original text feature and a summary text feature of the sample text, the summary text feature being generated based on the labeled information of the sample text and the sample picture; generating a first loss term based on contrastive learning of the global feature and the summary text feature, generating a second loss term based on contrastive learning of the local feature and the original text feature, generating a third loss term based on contrastive learning of the region of interest feature and the original text feature, and generating a fourth loss term based on contrastive learning of the region of interest feature and the summary text feature; calculating a Hungarian loss based on the first loss term, the second loss term, the third loss term, and the fourth loss term; and training the text-picture matching model according to the Hungarian loss.

[0137] In an embodiment, the extraction process of the global feature of the sample picture involved when the computer program is executed by the processor comprises: cropping the sample picture according to a first cropping ratio to obtain a global picture; and performing feature extraction on the global picture to obtain the global feature; the extraction process of the local feature of the sample picture involved when the computer program is executed by the processor comprises: cropping the sample picture according to a second cropping ratio to obtain a local picture, the second cropping ratio being smaller than the first cropping ratio; and performing feature extraction on the local picture to obtain the local feature; and the extraction process of the region-of-interest feature of the sample picture involved when the computer program is executed by the processor comprises: identifying a region-of-interest of the sample picture, and obtaining region position information of the region-of-interest; performing picture encoding on the region-of-interest to obtain a picture feature, and obtaining the region-of-interest feature according to the picture feature and the region position information.

[0138] In an embodiment, the extraction process of the original text feature of the sample text involved when the computer program is executed by the processor comprises: extracting a text feature of the sample text as the original text feature; and the extraction process of the summary text feature of the sample text involved when the computer program is executed by the processor comprises: filtering the sample text according to the annotation information of the sample text and the sample picture; and extracting a text feature of the filtered sample text as the summary text feature.

[0139] In an embodiment, the first loss term generated by contrast learning based on the global feature and the summary text feature and implemented when the computer program is executed by the processor comprises: calculating a first similarity of the global feature and the summary text feature, and a second similarity of the summary text feature and the local feature; and generating the first loss term based on the first similarity and the second similarity; the second loss term generated by contrast learning based on the local feature and the original text feature and implemented when the computer program is executed by the processor comprises: calculating a third similarity of the local feature and the original text feature, and a fourth similarity of the original text feature and the local feature; and generating the second loss term based on the third similarity and the fourth similarity; the third loss term generated by contrast learning based on the region-of-interest feature and the original text feature and implemented when the computer program is executed by the processor comprises: calculating a fifth similarity of the region-of-interest feature and the original text feature, and a sixth similarity of the original text feature and the region-of-interest feature; and generating the third loss term based on the fifth similarity and the sixth similarity; and the fourth loss term generated by contrast learning based on the region-of-interest feature and the summary text feature and implemented when the computer program is executed by the processor comprises: calculating a seventh similarity of the region-of-interest feature and the summary text feature, and an eighth similarity of the summary text feature and the region-of-interest feature; and generating the fourth loss term based on the seventh similarity and the eighth similarity.

[0140] In one embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the following steps: receiving a picture to be processed and text to be processed; inputting the picture to be processed and the text to be processed into the text-picture matching model trained in any one of the above embodiments to obtain the labeled text of the corresponding position of the picture to be processed.

[0141] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties.

[0142] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to a memory, database or other medium used in the embodiments provided by the present application can include at least one of a non-volatile and volatile memory. The non-volatile memory can include a read-only memory (Read-Only Memory, ROM), a magnetic tape, a floppy disk, a flash memory, an optical storage, a high-density embedded non-volatile memory, a resistive memory (ReRAM), a magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), a ferroelectric memory (Ferroelectric Random Access Memory, FRAM), a phase change memory (Phase Change Memory, PCM), a graphene memory, etc. The volatile memory can include a random access memory (Random Access Memory, RAM) or an external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided by the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided by the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0143] Any technical features in the above embodiments can be combined, and for the sake of brevity, not all possible combinations are described above, however, as long as the combinations of technical features do not have contradictions, they shall be considered within the scope of the present disclosure.

[0144] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it shall not be understood as a limitation on the patent scope of the present application. It shall be pointed out that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, and these shall be within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A method for training a text-image matching model, characterized in that, The method comprises: obtaining a sample picture, a sample text, and annotation information of the sample text and the sample picture; extracting global features, local features, and region of interest features of the sample picture through an image feature extractor of a text-image matching model; extracting original text features and summary text features of the sample text through a text feature extractor of the text-image matching model, the summary text features being generated based on the annotation information of the sample text and the sample picture; generating a first loss term based on contrast learning of the global features and the summary text features, generating a second loss term based on contrast learning of the local features and the original text features, generating a third loss term based on contrast learning of the region of interest features and the original text features, and generating a fourth loss term based on contrast learning of the region of interest features and the summary text features; calculating a Hungarian loss based on the first loss term, the second loss term, the third loss term, and the fourth loss term; training the text-image matching model according to the Hungarian loss.

2. The method of claim 1, wherein, The extraction process of the global features of the sample picture comprises: cropping the sample picture according to a first cropping ratio to obtain a global picture; extracting features of the global picture to obtain global features; The extraction process of the local features of the sample picture comprises: cropping the sample picture according to a second cropping ratio to obtain a local picture, the second cropping ratio being smaller than the first cropping ratio; extracting features of the local picture to obtain local features; The extraction process of the region of interest features of the sample picture comprises: identifying a region of interest of the sample picture, and obtaining region position information of the region of interest; image encoding the region of interest to obtain image features, and obtaining region of interest features according to the image features and the region position information.

3. The method of claim 1, wherein, The extraction process of the original text features of the sample text comprises: extracting text features of the sample text as original text features; The extraction process of the summary text features of the sample text comprises: filtering the sample text according to the annotation information of the sample text and the sample picture; extracting text features of the filtered sample text as summary text features.

4. The method of claim 1, wherein, The process of generating the first loss term based on contrast learning of the global features and the summary text features comprises: calculating a first similarity of the global features and the summary text features, and a second similarity of the summary text features and the global features; generating a first loss term based on the first similarity and the second similarity; The process of generating the second loss term based on contrast learning of the local features and the original text features comprises: calculating a third similarity of the local features and the original text features, and a fourth similarity of the original text features and the local features; generating a second loss term based on the third similarity and the fourth similarity; The process of generating the third loss term based on contrast learning of the region of interest features and the original text features comprises: fifth similarity between the region-of-interest feature and the original text feature, and a sixth similarity between the original text feature and the region-of-interest feature; generating a third loss term based on the fifth similarity and the sixth similarity; generating a fourth loss term based on the contrastive learning of the region-of-interest feature and the summary text feature, comprising: calculating a seventh similarity between the region-of-interest feature and the summary text feature, and an eighth similarity between the summary text feature and the region-of-interest feature; generating a fourth loss term based on the seventh similarity and the eighth similarity.

5. A picture labeling method characterized by, The picture labeling method comprises: receiving a picture to be processed and text to be processed; inputting the picture to be processed and the text to be processed into the text-image matching model trained according to any one of claims 1 to 4, to obtain labeled text corresponding to a position of the picture to be processed. 6.A text image matching model training apparatus, characterized by comprising: The device comprises: a sample acquisition module configured to acquire a sample picture, sample text, and labeling information of the sample text and the sample picture; an image feature extraction module configured to extract, by an image feature extractor of a text-image matching model, global features, local features, and region-of-interest features of the sample picture; a text feature extraction module configured to extract, by a text feature extractor of the text-image matching model, original text features and summary text features of the sample text, the summary text features being generated based on the labeling information of the sample text and the sample picture; a loss term generation module configured to generate a first loss term based on the contrastive learning of the global features and the summary text features, generate a second loss term based on the contrastive learning of the local features and the original text features, generate a third loss term based on the contrastive learning of the region-of-interest features and the original text features, and generate a fourth loss term based on the contrastive learning of the region-of-interest features and the summary text features; a Hungarian loss calculation module configured to calculate a Hungarian loss based on the first loss term, the second loss term, the third loss term, and the fourth loss term; a training module configured to train the text-image matching model according to the Hungarian loss.

7. An image labeling apparatus characterized by comprising: The picture labeling device comprises: a receiving module configured to receive a picture to be processed and text to be processed; a labeling module configured to input the picture to be processed and the text to be processed into the text-image matching model trained according to claim 6, to obtain labeled text corresponding to a position of the picture to be processed.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor, when executing the computer program, implements the steps of the method of any one of claims 1 to 4 or 5.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 4 or 5.

10. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 4 or 5.

Images