Model training and image-text matching method and device, electronic equipment and storage medium

Abstract

The application discloses a model training and image-text matching method and device, electronic equipment and a storage medium. The method comprises the following steps: obtaining an initial training data set, wherein the initial training data set comprises a plurality of first image-text pairs, each first image-text pair comprises an image and a first description text corresponding to the content of the image; generating a target training data set according to the initial training data set, wherein the target training data set comprises a plurality of second image-text pairs, each second image-text pair comprises a positive image-text pair, a negative image-text pair and an approximate image-text pair; and fine-tuning the initial image-text alignment model by using the target training data set to obtain a target image-text alignment model. The application solves the technical problem that the model has poor effect in negative semantic understanding due to the fact that the related art only constructs negative image-text pairs to enhance the understanding ability of the model to negative semantics.

Description

Technical Field

[0001] This application relates to the field of multimodal large model technology, and more specifically, to a model training and image-text matching method, apparatus, electronic device and storage medium. Background Technology

[0002] CLIP (Contrastive Language-Image Pretraining) is a multimodal model capable of aligning information from both image and text modalities. It demonstrates powerful capabilities in image-text alignment and serves as a foundational model for multimodal tasks such as image-text multimodal retrieval and visual question answering. While CLIP performs well on public datasets, its understanding of negation semantics remains a significant weakness.

[0003] However, the related technologies only enhance the model's ability to understand negation semantics by constructing negation graph-text pairs, which creates a technical problem that the model is not performing well in understanding negation semantics.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a model training and image-text matching method, apparatus, electronic device, and storage medium to at least solve the technical problem that the model's performance in understanding negative semantics is poor because related technologies only enhance the model's ability to understand negative semantics by constructing negative image-text pairs.

[0006] According to one aspect of the embodiments of this application, a model training method is provided, comprising: acquiring an initial training dataset, wherein the initial training dataset includes multiple first image-text pairs, each first image-text pair including: an image and a first descriptive text corresponding to the content of the image; generating a target training dataset based on the initial training dataset, wherein the target training dataset includes multiple sets of second image-text pairs, each set of second image-text pairs including: positive image-text pairs, negative image-text pairs, and approximate image-text pairs, wherein the positive image-text pairs include: positive descriptive text with affirmative semantics and an image corresponding to the descriptive content of the positive descriptive text, the negative image-text pairs include: negative descriptive text with negative semantics and an image corresponding to the descriptive content of the negative descriptive text, and the approximate image-text pairs include approximate descriptive text based on the relationship between affirmative and negative semantics and an image corresponding to the descriptive content of the approximate descriptive text; and fine-tuning the initial image-text alignment model using the target training dataset to obtain a target image-text alignment model, wherein the target image-text alignment model is used to match images corresponding to descriptive text.

[0007] Optionally, generating the target training dataset based on the initial training dataset includes: randomly selecting a first image-text pair from the initial training dataset; using a text analysis agent to analyze the first descriptive text in the first image-text pair to generate candidate text elements, wherein the candidate text elements are text elements not mentioned in the first descriptive text but related to the scene described by the first descriptive text, and the text elements include at least one of the following: action, target object, attribute; using a multimodal agent to analyze the candidate text elements and the images in the first image-text pair to determine whether the images in the first image-text pair contain the content described by the candidate text elements; if the images in the first image-text pair do not contain the content described by the candidate text elements, the candidate text elements are determined as target text elements; and generating a second image-text pair in the target training dataset based on the target text elements and the first descriptive text.

[0008] Optionally, generating a second image-text pair in the target training dataset based on the target text element and the first descriptive text includes: using a text analysis agent to add a negative word to the target text element, and adding the target text element with the added negative word to the first descriptive text to obtain a negative descriptive text; identifying the negative descriptive text and the image in the first image-text pair as the negative image-text pair; and replacing the original text element in the first descriptive text that corresponds to the type of the target text element with the target text element to obtain a positive descriptive text; generating an image corresponding to the content described by the positive descriptive text, and identifying the image and the positive descriptive text as a positive image-text pair.

[0009] Optionally, the method further includes: determining approximate text elements corresponding to the target text element, wherein the semantic similarity between the approximate text element and the target text element exceeds a preset similarity threshold and the text elements are of the same type; replacing the original text elements in the first descriptive text that correspond to the type of the approximate text element with the approximate text element to obtain approximate descriptive text; generating an image corresponding to the content described by the approximate descriptive text, and determining the image and the approximate descriptive text as an approximate image-text pair.

[0010] Optionally, the initial image-text alignment model is fine-tuned using the target training dataset to obtain the target image-text alignment model. This includes: dividing the target training dataset into multiple training batches, where each training batch contains multiple sets of second image-text pairs; using the initial image-text alignment model, analyzing the data from the multiple training batches to obtain model analysis results, and determining negation semantic vectors based on the model analysis results, where negation semantic vectors are used to capture the feature representation of negation semantics; determining loss function values ​​based on the negation semantic vectors, and adjusting the model parameters of the text encoder in the initial image-text alignment model based on the loss function values ​​to obtain the target image-text alignment model.

[0011] Optionally, determining the negation semantic vector based on the model analysis results includes: identifying the correctly matched negation image-text pairs in the model analysis results, and obtaining the positive descriptive text from the positive image-text pairs belonging to the same group as the negation image-text pairs; determining feature vectors based on the positive descriptive text and the negation descriptive text in the negation image-text pairs, and constructing a feature matrix based on the feature vectors, wherein the feature vectors are used to characterize the feature difference between the positive descriptive text and the negation descriptive text, and the feature matrix is ​​used to record the feature vectors corresponding to the correctly matched negation image-text pairs in multiple training batches; performing dimensionality reduction processing on the feature matrix using principal component analysis, and performing nonlinear transformation on the dimensionality-reduced feature matrix to obtain the negation semantic vector.

[0012] Optionally, constructing the feature matrix based on the feature vectors includes: obtaining the feature vectors corresponding to the correctly matched negation text-image pairs in the model analysis results of the current training batch; adding the feature vectors to the feature matrix when the negation text-image pair is the first correct match; obtaining the historical feature vectors corresponding to the identifiers of the negation text-image pairs in the feature matrix when the negation text-image pair has been correctly matched in previous training batches; performing weighted fusion on the feature vectors obtained in the current training batch and the previously stored historical feature vectors in the feature matrix to obtain new feature vectors, and replacing the previously stored historical feature vectors in the feature matrix with the new feature vectors, wherein, during weighted fusion, the weight value corresponding to the feature vectors obtained in the current training batch is greater than the weight value corresponding to the previously stored historical feature vectors in the feature matrix.

[0013] Optionally, determining the loss function value based on the negation semantic vector includes: determining a first loss value based on the matching of image-text pairs represented by the model analysis results and the correspondence of image-text pairs in the target training dataset, wherein the first loss value is used to characterize the accuracy of the model in image-text alignment; determining the correctly matched negation image-text pairs in the model analysis results and determining the feature vectors corresponding to the negation image-text pairs, wherein the feature vectors are used to characterize the feature difference between the positive descriptive text and the negation descriptive text in the positive image-text pairs belonging to the same group as the negation image-text pairs; determining a second loss value based on the feature vectors and the negation semantic vector, wherein the second loss value is used to characterize the similarity between the feature vectors and the negation semantic vectors; and determining the loss function value based on the first loss value and the second loss value.

[0014] According to another aspect of the embodiments of this application, a text-image matching method is provided, comprising: acquiring descriptive text to be matched; and analyzing the descriptive text to be matched using a target text-image alignment model to determine a target image corresponding to the descriptive text to be matched. The target text-image alignment model is obtained by fine-tuning an initial text-image alignment model based on a target training dataset. The target training dataset contains multiple sets of second text-image pairs. Each set of second text-image pairs includes: a positive text-image pair, a negative text-image pair, and an approximate text-image pair. The positive text-image pair includes: a positive descriptive text with affirmative semantics and an image corresponding to the descriptive content of the positive descriptive text. The negative text-image pair includes: a negative descriptive text with negative semantics and an image corresponding to the descriptive content of the negative descriptive text. The approximate text-image pair includes: an approximate descriptive text based on the relationship between affirmative and negative semantics and an image corresponding to the descriptive content of the approximate descriptive text.

[0015] According to another aspect of the embodiments of this application, a model training apparatus is also provided, comprising: a training set acquisition module, configured to acquire an initial training dataset, wherein the initial training dataset includes multiple first image-text pairs, each first image-text pair including: an image and a first descriptive text corresponding to the content of the image; a training set optimization module, configured to generate a target training dataset based on the initial training dataset, wherein the target training dataset includes multiple sets of second image-text pairs, each set of second image-text pairs including: positive image-text pairs, negative image-text pairs, and approximate image-text pairs, wherein the positive image-text pairs include: positive descriptive text with affirmative semantics and an image corresponding to the descriptive content of the positive descriptive text, the negative image-text pairs include: negative descriptive text with negative semantics and an image corresponding to the descriptive content of the negative descriptive text, and the approximate image-text pairs include approximate descriptive text based on the relationship between affirmative and negative semantics and an image corresponding to the descriptive content of the approximate descriptive text; and a fine-tuning training module, configured to fine-tune the initial image-text alignment model using the target training dataset to obtain a target image-text alignment model, wherein the target image-text alignment model is used to match images corresponding to descriptive text.

[0016] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the program executes a model training or image-text matching method during runtime.

[0017] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored computer program, wherein the device where the non-volatile storage medium is located executes a model training or image-text matching method by running the computer program.

[0018] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program, wherein when the computer program is executed by a processor, it implements the steps of a model training or image-text matching method.

[0019] In this embodiment, an initial training dataset is obtained, which includes multiple first image-text pairs. Each first image-text pair contains an image and a first descriptive text corresponding to the content of the image. Based on the initial training dataset, a target training dataset is generated, which contains multiple sets of second image-text pairs. Each set of second image-text pairs includes positive image-text pairs, negative image-text pairs, and approximate image-text pairs. Positive image-text pairs contain positive descriptive text with affirmative semantics and an image corresponding to the descriptive content of the positive descriptive text. Negative image-text pairs contain negative descriptive text with negative semantics and an image corresponding to the descriptive content of the negative descriptive text. Approximate image-text pairs contain approximate descriptive text based on the relationship between affirmative and negative semantics and an image corresponding to the descriptive content of the approximate descriptive text. Using the target training dataset, the initial image-text alignment model is fine-tuned to obtain a target image-text alignment model. The target image-text alignment model is used to match images corresponding to descriptive text. By generating image-text pairs containing positive, negative, and approximate descriptions, the model is exposed to more diverse semantic scenarios during training. Simultaneously, PCA (principal components) is introduced. The unsupervised learning method of principal component analysis (PCA) effectively improves the model's ability to understand and extract negation semantics, thereby solving the technical problem that the model's performance in understanding negation semantics is poor because related technologies only enhance the model's ability to understand negation semantics by constructing negation graph-text pairs. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0021] Figure 1 This is a hardware structure block diagram of a computer terminal (or electronic device) for implementing a method for model training or image-text matching, according to an embodiment of this application.

[0022] Figure 2 This is a schematic diagram of a model training method according to an embodiment of this application;

[0023] Figure 3 This is a schematic diagram illustrating the composition of a training image-text pair according to an embodiment of this application;

[0024] Figure 4This is a schematic diagram of constructing a pipeline using graphic pairs according to an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of a model fine-tuning training process provided according to an embodiment of this application;

[0026] Figure 6 This is a schematic diagram of a method flow for image-text matching according to an embodiment of this application;

[0027] Figure 7 This is a schematic diagram of a model training device provided according to an embodiment of this application. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] To facilitate a better understanding of the embodiments of this application by those skilled in the art, some technical terms or nouns involved in the embodiments of this application are explained as follows:

[0031] PCA (principal components analysis): Principal component analysis, also known as principal component analysis, aims to use the idea of ​​dimensionality reduction to transform multiple indicators into a few comprehensive indicators.

[0032] CLIP (Contrastive Language-Image Pretraining): A large-scale, multimodal model based on contrastive learning for text and images.

[0033] AI Agent: refers to an intelligent agent capable of proactive thinking and action, which can work in a human-like manner, "understand" user needs through large models, proactively "plan" to achieve goals, use various "tools" to complete tasks, and ultimately "act" to execute these tasks.

[0034] Negation is a fundamental phenomenon in linguistics that can reverse the meaning of a sentence. Negative semantics refers to adding a negative word to a positive semantic description. Common negative words include, but are not limited to, "neither," "without," "no," and "not." For example, the positive text description is "A little girl wearing a hat," but the description after adding negative semantics is "A little girl not wearing a hat."

[0035] Because the publicly available training set used by CLIP does not have targeted supplementary graph-text pairs with negative semantic descriptions, its ability to understand negative semantics is insufficient. The optimization ideas for CLIP's negative semantic understanding ability in related technologies mainly rely on building a data production pipeline with agents and multimodal large models to construct negative graph-text pairs. There are two methods: offline generation and online generation. At the same time, contrastive learning methods are combined to fine-tune CLIP.

[0036] However, the improvements in data at the data level are limited to the construction of negative image-text pairs, which restricts the model's ability to generalize and the accuracy of understanding negative semantics. At the same time, apart from relying on contrastive learning methods, the innovation of related technologies in training methods is also relatively lacking.

[0037] To address the aforementioned issues, this application provides relevant solutions. Adhering to the approach of combining data and methods, a more refined approach to constructing negation semantic graph-text pairs is introduced. At the same time, an unsupervised learning method is introduced to assist in fine-tuning CLIP, thereby improving the extraction and understanding capabilities of negation semantics. These are described in detail below.

[0038] According to the embodiments of this application, a method embodiment for model training or image-text matching is provided. It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0039] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1A hardware block diagram of a computer terminal (or electronic device) for implementing model training or image-text matching methods is shown. Figure 1 As shown, the computer terminal 10 (or electronic device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0040] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or electronic device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0041] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the model training or image-text matching method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-mentioned model training or image-text matching method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0042] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0043] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or electronic device).

[0044] Under the above operating environment, this application provides a model training method. Figure 2 This is a schematic diagram of a model training method flow provided in an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0045] Step S202: Obtain the initial training dataset, wherein the initial training dataset includes multiple first image-text pairs, and each first image-text pair contains: an image and a first descriptive text corresponding to the content of the image;

[0046] Step S204: Based on the initial training dataset, generate the target training dataset. The target training dataset contains multiple sets of second image-text pairs. Each set of second image-text pairs contains: positive image-text pairs, negative image-text pairs, and approximate image-text pairs. Positive image-text pairs contain: positive descriptive text with affirmative semantics and an image corresponding to the description content of the positive descriptive text. Negative image-text pairs contain: negative descriptive text with negative semantics and an image corresponding to the description content of the negative descriptive text. Approximate image-text pairs contain approximate descriptive text based on the relationship between affirmative and negative semantics and an image corresponding to the description content of the approximate descriptive text.

[0047] Step S206: Using the target training dataset, the initial image-text alignment model is fine-tuned to obtain the target image-text alignment model, wherein the target image-text alignment model is used to match the image corresponding to the descriptive text.

[0048] Through the above steps, by generating graph-text pairs containing positive, negative, and approximate descriptions, the model is exposed to more diverse semantic scenarios during training. At the same time, the introduction of PCA-based unsupervised learning methods effectively improves the model's ability to understand and extract negative semantics. This solves the technical problem that related technologies only enhance the model's ability to understand negative semantics by constructing negative graph-text pairs, resulting in poor performance in understanding negative semantics.

[0049] The model training method in steps S202 to S206 of the embodiments of this application will be further described below.

[0050] In this embodiment of the application, the composition of image-text pairs in the target training dataset used to train the image-text alignment model is as follows: Figure 3 As shown, "+" indicates a positive title (i.e., positive descriptive text), "-" indicates a negative title (i.e., negative descriptive text), and "+ / -" indicates an approximate title (i.e., approximate descriptive text). Figure 3 In each row, the three images on the left and the three texts on the right correspond one-to-one, forming a second image-text pair.

[0051] In other words, a second set of text-image pairs includes not only the positive title and corresponding image (i.e., positive text-image pairs), but also the negative semantics and corresponding image (i.e., negative text-image pairs), as well as approximate titles and corresponding images that fall between positive and negative semantics (i.e., approximate text-image pairs). For example, "A giraffe is playing in the water" is neither a reverse description of the positive title nor does it express part of the meaning of the negative title. Another example is "The man is dribbling the ball," which is parallel to the positive title ("dribbling" and "shouting" are parallel semantics rather than opposite semantics), and also has a semantic inclusion relationship with "The man is not shooting a ball."

[0052] By generating image-text pairs containing positive, negative, and approximate descriptions as the target training dataset, interference samples related to negative image-text pairs, in addition to the regular negative image-text pairs, can be introduced during subsequent model fine-tuning. The introduction of competing samples forces the model to ignore the interference caused by related concepts, actions, or attributes in negative semantic extraction, allowing it to focus more on extracting the negative features inherent in the negative title. For example, if the negative title contains "ball" and... Figure 3The second image in the first row is unrelated, meaning that positive image-text pairs provide a clear baseline for affirmative semantics, while negative and approximate image-text pairs guide the model to identify and distinguish subtle differences in semantics, especially in the presence of negative words. This helps the model to accurately understand and represent semantic inversion or partial inversion information, thereby enabling the model to learn the diversity of semantic descriptions more comprehensively during fine-tuning, particularly for the understanding and expression of negative semantics.

[0053] The following describes the process steps for constructing the target training dataset in the embodiments of this application, such as... Figure 4 The diagram shows the flow of image-text pair data required for offline production fine-tuning CLIP in this embodiment of the application, as detailed below.

[0054] First, a first image-text pair can be randomly selected from the initial training dataset, i.e., an image and its corresponding text (i.e., the first descriptive text), denoted as I and T respectively. For example, in Figure 4 In the first image-text pair, the first descriptive text is: "The man is shouting".

[0055] Then, a second set of image-text pairs can be generated based on the first image-text pair. The specific steps are as follows.

[0056] In some embodiments of this application, generating a target training dataset based on an initial training dataset includes the following steps: randomly selecting a first image-text pair from the initial training dataset; using a text analysis agent to analyze the first descriptive text in the first image-text pair to generate candidate text elements, wherein the candidate text elements are text elements not mentioned in the first descriptive text but related to the scene described by the first descriptive text, and the text elements include at least one of the following: action, target object, attribute; using a multimodal agent to analyze the candidate text elements and the images in the first image-text pair to determine whether the images in the first image-text pair contain the content described by the candidate text elements; if the images in the first image-text pair do not contain the content described by the candidate text elements, the candidate text elements are determined as target text elements; and generating a second image-text pair in the target training dataset based on the target text elements and the first descriptive text.

[0057] Specifically, a text analysis agent can be used to analyze the first descriptive text in the first image-text pair to generate candidate text elements. It should be noted that the categories involved in the candidate text elements include, but are not limited to, actions, target objects, and attributes. In this embodiment, the action category is used as an example for specific explanation. The implementation process for the target object and attribute categories is similar.

[0058] For example, an instruction can be issued to the LLM Agent to query for actions (i.e., candidate text elements) that may exist in the first description text T but do not actually exist. Figure 4 The candidate text element identified is "shooting"; then, for at least one Action returned by the LLM Agent, at least one Action not found in image I is queried, denoted as LA (i.e., the target text element). That is, if the image in the first image-text pair does not contain the content described by the candidate text element, the candidate text element is identified as the target text element. For example, in... Figure 4 If it is determined that the image in the first image-text pair does not contain the action "shooting", then "shooting" is identified as the target text element.

[0059] Then, using the target text elements and the first descriptive text, the second image-text pair in the target training dataset can be generated. The specific steps are as follows.

[0060] In some embodiments of this application, generating a second image-text pair in the target training dataset based on the target text element and the first descriptive text includes the following steps: using a text analysis agent to add negative words to the target text element, and adding the target text element with the added negative words to the first descriptive text to obtain negative descriptive text; determining the negative image-text pair as the negative image-text pair by using the negative text element; and replacing the original text element in the first descriptive text that corresponds to the type of the target text element with the target text element to obtain positive descriptive text; generating an image corresponding to the content described by the positive descriptive text, and determining the image and the positive descriptive text as a positive image-text pair.

[0061] In some embodiments of this application, the method further includes: determining an approximate text element corresponding to the target text element, wherein the semantic similarity between the approximate text element and the target text element exceeds a preset similarity threshold and the text elements are of the same type; replacing the original text element in the first descriptive text that corresponds to the type of the approximate text element with the approximate text element to obtain an approximate descriptive text; generating an image corresponding to the content described by the approximate descriptive text, and determining the image and the approximate descriptive text as an approximate image-text pair.

[0062] Specifically, an instruction can be issued to the text analysis agent LLM Agent to add LA (i.e., the target text element) to T (i.e., the first descriptive text) to generate a new text containing a negative word, denoted as T- (i.e., the negative descriptive text). For example, Figure 4The generated negative description text is "The man is shouting and not shooting a ball"; then the generated negative description text and the original image in the first image-text pair can be identified as the negative image-text pair.

[0063] Simultaneously, instructions can be issued to the LLM using LA (target text element) and I (first descriptive text) as input to generate an Action that most closely approximates LA, denoted as LAS (approximate text element). For example, in... Figure 4 The approximate text element generated in the middle is "dribbling";

[0064] Then, the existing Actions in T (the first descriptive text) can be replaced with LA (the target text element) and LAS (the approximate text element) respectively, to obtain T+ (the positive descriptive text) and T+ / - (the approximate descriptive text), for example, Figure 4 The generated positive descriptive text is "The man is shooting a ball", and the generated approximate descriptive text is "The man is dribbling a ball". Then, the text-to-image (T2I) method can be used to generate two corresponding images from the generated positive and approximate descriptive texts, thus obtaining positive image-text pairs and approximate image-text pairs.

[0065] Finally, a second set of image-text pairs is generated using the selected first image-text pair. By repeatedly executing the above process to generate image-text pairs of a certain size, this can be used as the target training dataset for fine-tuning CLIP.

[0066] By refining the graph-text pair generation process, not only were basic positive and negative graph-text pairs constructed, but also approximate graph-text pairs in between were creatively introduced. With the assistance of an intelligent agent, a large number of training samples can be generated automatically and efficiently, thus forming a richer training dataset. This enables the model to demonstrate better recognition ability and creativity when dealing with complex semantic scenarios, especially when dealing with ambiguous situations between affirmative and negative semantics, significantly enhancing the model's semantic parsing power and adaptability.

[0067] On the other hand, regarding the model training process, the unsupervised assisted training method introduced in this application continuously collects correctly matched negation graph-text pairs during training and extracts stable expressions of negation semantics through PCA. Combined with the aforementioned graph-text pair construction method, it bidirectionally drives the model to understand negation semantics, which has stronger interpretability than learning negation semantics solely based on data-driven methods. The fine-tuning training method in this application embodiment is described below.

[0068] After generating the offline data (target training dataset), the overall fine-tuning architecture of the training method in this embodiment is as follows: Figure 5 As shown, the PCA-based assisted training method can gradually capture the stable negation semantics learned during model training, and promote the extraction of negation semantics in subsequent training processes by incorporating a loss function. The specific process is as follows.

[0069] In some embodiments of this application, the initial image-text alignment model is fine-tuned and trained using a target training dataset to obtain the target image-text alignment model, including the following steps: dividing the target training dataset into multiple training batches, wherein each training batch contains multiple sets of second image-text pairs; using the initial image-text alignment model, analyzing the data of multiple training batches respectively to obtain model analysis results, and determining the negation semantic vector based on the model analysis results, wherein the negation semantic vector is used to capture the feature representation of negation semantics; determining the loss function value based on the negation semantic vector, and adjusting the model parameters of the text encoder in the initial image-text alignment model based on the loss function value to obtain the target image-text alignment model.

[0070] In order to preserve the embedding space of the original model, this embodiment of the application only fine-tunes the text encoder (corresponding to the Text Encoder) and freezes the visual encoder (corresponding to the Image Encoder) when fine-tuning the model. The specific fine-tuning process is as follows.

[0071] Specifically, the target training dataset can be divided into multiple training batches. For example, some samples belonging to the same group (each group has 3 image-text pairs) (the second image-text pair) can be randomly selected to form a batch of length 1536; the batch is then input into the initial image-text alignment model (taking CLIP as an example) to obtain the output result, i.e., the model analysis result.

[0072] Then, based on the model analysis results, the negation semantic vector can be determined. The specific steps are as follows.

[0073] In some embodiments of this application, determining the negation semantic vector based on the model analysis results includes the following steps: determining the correctly matched negation image-text pairs in the model analysis results, and obtaining the positive descriptive text in the positive image-text pairs belonging to the same group as the negation image-text pairs; determining feature vectors based on the positive descriptive text and the negation descriptive text in the negation image-text pairs, and constructing a feature matrix based on the feature vectors, wherein the feature vectors are used to characterize the feature difference between the positive descriptive text and the negation descriptive text, and the feature matrix is ​​used to record the feature vectors corresponding to the correctly matched negation image-text pairs in multiple training batches; performing dimensionality reduction processing on the feature matrix using principal component analysis, and performing nonlinear transformation on the dimensionality-reduced feature matrix to obtain the negation semantic vector.

[0074] Specifically, first, select the correctly matching negative image-text pairs from the model analysis results, and obtain the positive descriptive text from the positive image-text pairs that belong to the same group as the negative image-text pairs; then, based on the positive descriptive text and the negative descriptive text, construct a dynamic feature matrix A, as follows.

[0075] In some embodiments of this application, constructing a feature matrix based on feature vectors includes the following steps: obtaining the feature vectors corresponding to the correctly matched negation text pairs in the model analysis results of the current training batch; adding the feature vectors to the feature matrix when the negation text pairs are correctly matched for the first time; obtaining the historical feature vectors corresponding to the identifiers of the negation text pairs in the feature matrix when the negation text pairs have been correctly matched in previous training batches; performing weighted fusion on the feature vectors obtained in the current training batch and the previously stored historical feature vectors in the feature matrix to obtain new feature vectors, and replacing the previously stored historical feature vectors in the feature matrix with the new feature vectors, wherein, during weighted fusion, the weight value corresponding to the feature vectors obtained in the current training batch is greater than the weight value corresponding to the previously stored historical feature vectors in the feature matrix.

[0076] Specifically, during training, a key-value cache can be maintained to indicate which negative image-text pairs were successfully matched earlier, where k is the key value. Text pairs, This indicates a negative title (negative descriptive text). Indicates and The corresponding positive title (positive description text), v is a unique ID (ID is assigned sequentially from 0); sequentially query whether the text pair formed by the negative title (negative description text) and the corresponding positive title (positive description text) of each correctly matched negative image-text pair in the current model analysis results exists in kv;

[0077] If the text pair does not exist, a new text pair is added to the key-value pair (kv). For the text pair that appears in the kv for the first time, the difference between the negative title (negative descriptive text) feature and the positive title (positive descriptive text) feature is calculated to obtain a feature vector of dimension 512 (representing the extracted negative semantics), and this vector is stored in the feature matrix. In the matrix, the row number is the ID of the text pair, and the feature of that row is denoted as . ;

[0078] For text pairs that are not appearing in the key-value pair for the first time, Perform an exponentially weighted average, i.e. The effect of this approach is that the feature vector of each row is mostly composed of feature differences generated by the most recent time window, with the proportion of feature differences decreasing as they come up. In other words, the weight of the current feature vector is higher than the weight of the historical feature vector. This mechanism ensures that the feature matrix can gradually accumulate and stably express the negation semantics learned by the model during training. Because the longer the training time, the more stable the negation semantics extracted by the model will be. Through continuous updating and optimization of the feature matrix, the model's ability to understand and distinguish negation semantics can be effectively promoted, thereby improving the model's accuracy and robustness in handling complex contexts containing negation words.

[0079] Furthermore, for the feature matrix PCA technology can be used to analyze and update the feature matrix A, effectively capturing the stable expression of negation semantics during model training and obtaining the minimum... The eigenvectors corresponding to each principal component; because a larger principal component indicates a lack of similarity in features between different samples, while a smaller principal component indicates similarity in features between different samples; considering the richness and flexibility of textual expression, and referencing the 95th percentile concept in statistics, this proposal only considers the eigenvectors corresponding to the lowest 5% of principal components. Therefore, this proposal will... =5, these eigenvectors form a shape of... matrix ;

[0080] Subsequently, the feature matrix after PCA dimensionality reduction can be further transformed using the MLP module to remove potential noise, focus on the core negation features, and ultimately form a negation semantic vector. For example, the text pairs used for training are mostly short texts, and the array... This may encode irrelevant information; adding a dimension would be helpful. The learnable MLP module (also known as a multilayer perceptron) is used to focus on core information, and finally outputs a shape. The matrix is ​​used to calculate the mean of each column, resulting in the final negation semantic vector. , shape .

[0081] The above process can be continued across multiple training batches. In each iteration, new feature vectors are determined and supplemented based on the correctly matched negative image-text pairs and their corresponding positive descriptive texts. In the process, after PCA dimensionality reduction and nonlinear transformation by the MLP module, It gradually became a highly generalized and pure expression of negation semantics (i.e., a negation semantic vector).

[0082] In the subsequent fine-tuning training of the model, negation semantic vectors can be used as a basis. This is used to determine the loss function value, thereby adjusting the model parameters, as detailed below.

[0083] In some embodiments of this application, determining the loss function value based on the negation semantic vector includes: determining a first loss value based on the matching of image-text pairs represented by the model analysis results and the correspondence of image-text pairs in the target training dataset, wherein the first loss value is used to characterize the accuracy of the model in image-text alignment; determining the correctly matched negation image-text pairs in the model analysis results and determining the feature vectors corresponding to the negation image-text pairs, wherein the feature vectors are used to characterize the feature difference between the positive descriptive text and the negation descriptive text in the positive image-text pairs belonging to the same group as the negation image-text pairs; determining a second loss value based on the feature vectors and the negation semantic vector, wherein the second loss value is used to characterize the similarity between the feature vectors and the negation semantic vector; and determining the loss function value based on the first loss value and the second loss value.

[0084] In this embodiment of the application, the loss function is determined by... (First loss value) and (Second loss value) constitutes, where, The InfoNCE loss used is the same as that used in the original CLIP model. This is the negative semantic alignment loss function proposed in the embodiments of this application, and the specific formula is as follows: The meaning is the sum of the feature differences between all negative and positive headlines. The two values ​​are kept as close as possible, and their sum is used as the total loss function. Finally, backpropagation is performed to update the training parameters of the text encoder.

[0085] Specifically, the first loss value L1 evaluates the model's performance on the image-text alignment task by comparing the model's output image-text matching results with the matching results in the actual target training dataset, ensuring the basic contrastive learning effect. The second loss value L2 focuses on the similarity between feature vectors (reflecting the feature differences between positive and negative descriptions within the same group) and the negative semantic vectors extracted through PCA, driving the learned negative features to evolve towards standardized and stable expressions, enhancing the model's robustness and generalization ability in handling negative semantics. Finally, L1 and L2 are combined to form the total loss function value, guiding the parameter updates of the text encoder and ensuring that the model significantly enhances its ability to capture and understand negative semantics while maintaining its original functionality.

[0086] This application optimizes the model from both data and methodology perspectives. It not only generates richer and more detailed text-image pairs for training and fine-tuning the CLIP model, exposing it to more diverse semantic scenarios during training and thus improving its generalization ability to understand and process negation semantics, but also introduces a PCA-based unsupervised learning method. This allows the model to gradually learn and capture stable expressions of negation semantics, helping it to more accurately understand and match images when processing text containing negation words. This effectively enhances the model's ability to understand and extract negation semantics, making the extracted negation semantics more reliable and reasonable. Furthermore, it leverages large-scale data mining to uncover the essential features of negation semantics, further improving the model's generalization ability to understand negation semantics. In practical applications, such as monitoring violations in urban governance scenarios (e.g., unlicensed vehicles on the road, workers not wearing safety helmets, chefs not wearing masks), it also brings significant performance improvements and user experience enhancements.

[0087] According to embodiments of this application, this application also provides a text-image matching method. Figure 6 This is a schematic diagram of a text-image matching method provided in an embodiment of this application, such as... Figure 6 As shown, the method includes the following steps:

[0088] Step S602: Obtain the description text to be matched;

[0089] Step S604: Using a target image-text alignment model, the descriptive text to be matched is analyzed to determine the target image corresponding to the descriptive text to be matched. The target image-text alignment model is obtained by fine-tuning the initial image-text alignment model based on the target training dataset. The target training dataset contains multiple sets of second image-text pairs. Each set of second image-text pairs contains: positive image-text pairs, negative image-text pairs, and approximate image-text pairs. Positive image-text pairs contain: positive descriptive text with affirmative semantics and an image corresponding to the descriptive content of the positive descriptive text. Negative image-text pairs contain: negative descriptive text with negative semantics and an image corresponding to the descriptive content of the negative descriptive text. Approximate image-text pairs contain approximate descriptive text based on the relationship between affirmative and negative semantics and an image corresponding to the descriptive content of the approximate descriptive text.

[0090] It should be noted that the image-text matching method provided in this embodiment is... Figure 2 The model training method shown corresponds to the method embodiment from the perspective of model application. Therefore, the relevant explanations and descriptions of the above model training method also apply to the embodiments of this application, and will not be repeated here.

[0091] According to an embodiment of this application, an embodiment of a model training apparatus is also provided. Figure 7 This is a schematic diagram of a model training device provided according to an embodiment of this application. Figure 7 As shown, the device includes:

[0092] The training set acquisition module 70 is used to acquire the initial training dataset, which includes multiple first image-text pairs. Each first image-text pair contains: an image and a first descriptive text corresponding to the content of the image.

[0093] The training set optimization module 72 is used to generate a target training dataset based on the initial training dataset. The target training dataset contains multiple sets of second image-text pairs. Each set of second image-text pairs contains: positive image-text pairs, negative image-text pairs, and approximate image-text pairs. Positive image-text pairs contain: positive descriptive text with affirmative semantics and an image corresponding to the description content of the positive descriptive text. Negative image-text pairs contain: negative descriptive text with negative semantics and an image corresponding to the description content of the negative descriptive text. Approximate image-text pairs contain approximate descriptive text based on the relationship between affirmative and negative semantics and an image corresponding to the description content of the approximate descriptive text.

[0094] The fine-tuning training module 74 is used to fine-tune the initial image-text alignment model using the target training dataset to obtain the target image-text alignment model, wherein the target image-text alignment model is used to match the image corresponding to the descriptive text.

[0095] Optionally, generating the target training dataset based on the initial training dataset includes: randomly selecting a first image-text pair from the initial training dataset; using a text analysis agent to analyze the first descriptive text in the first image-text pair to generate candidate text elements, wherein the candidate text elements are text elements not mentioned in the first descriptive text but related to the scene described by the first descriptive text, and the text elements include at least one of the following: action, target object, attribute; using a multimodal agent to analyze the candidate text elements and the images in the first image-text pair to determine whether the images in the first image-text pair contain the content described by the candidate text elements; if the images in the first image-text pair do not contain the content described by the candidate text elements, the candidate text elements are determined as target text elements; and generating a second image-text pair in the target training dataset based on the target text elements and the first descriptive text.

[0096] Optionally, generating a second image-text pair in the target training dataset based on the target text element and the first descriptive text includes: using a text analysis agent to add a negative word to the target text element, and adding the target text element with the added negative word to the first descriptive text to obtain a negative descriptive text; identifying the negative descriptive text and the image in the first image-text pair as the negative image-text pair; and replacing the original text element in the first descriptive text that corresponds to the type of the target text element with the target text element to obtain a positive descriptive text; generating an image corresponding to the content described by the positive descriptive text, and identifying the image and the positive descriptive text as a positive image-text pair.

[0097] Optionally, it further includes: determining approximate text elements corresponding to the target text element, wherein the semantic similarity between the approximate text element and the target text element exceeds a preset similarity threshold and the text elements are of the same type; replacing the original text elements in the first descriptive text that correspond to the type of the approximate text element with the approximate text element to obtain approximate descriptive text; generating an image corresponding to the content described by the approximate descriptive text, and determining the image and the approximate descriptive text as an approximate image-text pair.

[0098] Optionally, the initial image-text alignment model is fine-tuned using the target training dataset to obtain the target image-text alignment model. This includes: dividing the target training dataset into multiple training batches, where each training batch contains multiple sets of second image-text pairs; using the initial image-text alignment model, analyzing the data from the multiple training batches to obtain model analysis results, and determining negation semantic vectors based on the model analysis results, where negation semantic vectors are used to capture the feature representation of negation semantics; determining loss function values ​​based on the negation semantic vectors, and adjusting the model parameters of the text encoder in the initial image-text alignment model based on the loss function values ​​to obtain the target image-text alignment model.

[0099] Optionally, determining the negation semantic vector based on the model analysis results includes: identifying the correctly matched negation image-text pairs in the model analysis results, and obtaining the positive descriptive text from the positive image-text pairs belonging to the same group as the negation image-text pairs; determining feature vectors based on the positive descriptive text and the negation descriptive text in the negation image-text pairs, and constructing a feature matrix based on the feature vectors, wherein the feature vectors are used to characterize the feature difference between the positive descriptive text and the negation descriptive text, and the feature matrix is ​​used to record the feature vectors corresponding to the correctly matched negation image-text pairs in multiple training batches; performing dimensionality reduction processing on the feature matrix using principal component analysis, and performing nonlinear transformation on the dimensionality-reduced feature matrix to obtain the negation semantic vector.

[0100] Optionally, constructing the feature matrix based on the feature vectors includes: obtaining the feature vectors corresponding to the correctly matched negation text-image pairs in the model analysis results of the current training batch; adding the feature vectors to the feature matrix when the negation text-image pair is the first correct match; obtaining the historical feature vectors corresponding to the identifiers of the negation text-image pairs in the feature matrix when the negation text-image pair has been correctly matched in previous training batches; performing weighted fusion on the feature vectors obtained in the current training batch and the previously stored historical feature vectors in the feature matrix to obtain new feature vectors, and replacing the previously stored historical feature vectors in the feature matrix with the new feature vectors, wherein, during weighted fusion, the weight value corresponding to the feature vectors obtained in the current training batch is greater than the weight value corresponding to the previously stored historical feature vectors in the feature matrix.

[0101] Optionally, determining the loss function value based on the negation semantic vector includes: determining a first loss value based on the matching of image-text pairs represented by the model analysis results and the correspondence of image-text pairs in the target training dataset, wherein the first loss value is used to characterize the accuracy of the model in image-text alignment; determining the correctly matched negation image-text pairs in the model analysis results and determining the feature vectors corresponding to the negation image-text pairs, wherein the feature vectors are used to characterize the feature difference between the positive descriptive text and the negation descriptive text in the positive image-text pairs belonging to the same group as the negation image-text pairs; determining a second loss value based on the feature vectors and the negation semantic vector, wherein the second loss value is used to characterize the similarity between the feature vectors and the negation semantic vectors; and determining the loss function value based on the first loss value and the second loss value.

[0102] It should be noted that each module in the above-mentioned model training device can be a program module (e.g., a set of program instructions to implement a certain function) or a hardware module. For the latter, it can be manifested in the following forms, but is not limited to them: each of the above modules is manifested as a processor, or the functions of each of the above modules are implemented by a processor.

[0103] It should be noted that the model training device provided in this embodiment can be used to perform... Figure 2 The model training method shown above is also applicable to the embodiments of this application, and will not be repeated here.

[0104] This application embodiment also provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the following model training or image-text matching method by running the computer program:

[0105] Obtain an initial training dataset, which includes multiple first image-text pairs. Each first image-text pair contains an image and a first descriptive text corresponding to the image's content. Based on the initial training dataset, generate a target training dataset, which contains multiple sets of second image-text pairs. Each set of second image-text pairs includes: positive image-text pairs, negative image-text pairs, and approximate image-text pairs. Positive image-text pairs contain positive descriptive text with affirmative semantics and an image corresponding to the descriptive content of the positive descriptive text. Negative image-text pairs contain negative descriptive text with negative semantics and an image corresponding to the descriptive content of the negative descriptive text. Approximate image-text pairs contain approximate descriptive text based on the relationship between affirmative and negative semantics and an image corresponding to the descriptive content of the approximate descriptive text. Using the target training dataset, fine-tune the initial image-text alignment model to obtain a target image-text alignment model, which is used to match images corresponding to descriptive text.

[0106] Alternatively, obtain the descriptive text to be matched; use a target image-text alignment model to analyze the descriptive text to be matched and determine the target image corresponding to the descriptive text to be matched. The target image-text alignment model is obtained by fine-tuning the initial image-text alignment model based on the target training dataset. The target training dataset contains multiple sets of second image-text pairs. Each set of second image-text pairs contains: positive image-text pairs, negative image-text pairs, and approximate image-text pairs. Positive image-text pairs contain: positive descriptive text with affirmative semantics and an image corresponding to the descriptive content of the positive descriptive text. Negative image-text pairs contain: negative descriptive text with negative semantics and an image corresponding to the descriptive content of the negative descriptive text. Approximate image-text pairs contain approximate descriptive text based on the relationship between affirmative and negative semantics and an image corresponding to the descriptive content of the approximate descriptive text.

[0107] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the model training or image-text matching methods described in various embodiments of this application:

[0108] Obtain an initial training dataset, which includes multiple first image-text pairs. Each first image-text pair contains an image and a first descriptive text corresponding to the image's content. Based on the initial training dataset, generate a target training dataset, which contains multiple sets of second image-text pairs. Each set of second image-text pairs includes: positive image-text pairs, negative image-text pairs, and approximate image-text pairs. Positive image-text pairs contain positive descriptive text with affirmative semantics and an image corresponding to the descriptive content of the positive descriptive text. Negative image-text pairs contain negative descriptive text with negative semantics and an image corresponding to the descriptive content of the negative descriptive text. Approximate image-text pairs contain approximate descriptive text based on the relationship between affirmative and negative semantics and an image corresponding to the descriptive content of the approximate descriptive text. Using the target training dataset, fine-tune the initial image-text alignment model to obtain a target image-text alignment model, which is used to match images corresponding to descriptive text.

[0109] Alternatively, obtain the descriptive text to be matched; use a target image-text alignment model to analyze the descriptive text to be matched and determine the target image corresponding to the descriptive text to be matched. The target image-text alignment model is obtained by fine-tuning the initial image-text alignment model based on the target training dataset. The target training dataset contains multiple sets of second image-text pairs. Each set of second image-text pairs contains: positive image-text pairs, negative image-text pairs, and approximate image-text pairs. Positive image-text pairs contain: positive descriptive text with affirmative semantics and an image corresponding to the descriptive content of the positive descriptive text. Negative image-text pairs contain: negative descriptive text with negative semantics and an image corresponding to the descriptive content of the negative descriptive text. Approximate image-text pairs contain approximate descriptive text based on the relationship between affirmative and negative semantics and an image corresponding to the descriptive content of the approximate descriptive text.

[0110] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0111] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0112] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0113] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0114] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0115] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0116] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A model training method, characterized in that, include: Obtain an initial training dataset, wherein the initial training dataset includes multiple first image-text pairs, and each first image-text pair contains: an image and a first descriptive text corresponding to the content of the image; Based on the initial training dataset, a target training dataset is generated, wherein the target training dataset contains multiple sets of second image-text pairs. Each set of second image-text pairs contains: positive image-text pairs, negative image-text pairs, and approximate image-text pairs. The positive image-text pairs contain: positive descriptive text with affirmative semantics and an image corresponding to the descriptive content of the positive descriptive text. The negative image-text pairs contain: negative descriptive text with negative semantics and an image corresponding to the descriptive content of the negative descriptive text. The approximate image-text pairs contain approximate descriptive text based on the positive semantics and the negative semantics and an image corresponding to the descriptive content of the approximate descriptive text. Using the target training dataset, the initial image-text alignment model is fine-tuned to obtain the target image-text alignment model, wherein the target image-text alignment model is used to match the image corresponding to the descriptive text; The process of fine-tuning the initial image-text alignment model using the target training dataset to obtain the target image-text alignment model includes: dividing the target training dataset into multiple training batches, wherein each training batch contains multiple sets of the second image-text pairs; using the initial image-text alignment model, analyzing the data of each of the multiple training batches to obtain model analysis results, and determining the negative semantic vector based on the model analysis results, including: determining the correctly matched negative image-text pairs in the model analysis results, and obtaining the positive descriptive text in the positive image-text pairs belonging to the same group as the negative image-text pairs; determining the feature vector based on the positive descriptive text and the negative descriptive text in the negative image-text pairs, and based on the... A feature matrix is ​​constructed from the described feature vectors, wherein the feature vectors are used to characterize the feature difference between the positive and negative descriptive texts, and the feature matrix is ​​used to record the feature vectors corresponding to the correctly matched negative image-text pairs in the multiple training batches; the feature matrix is ​​dimensionality-reduced using principal component analysis, and a nonlinear transformation is performed on the dimensionality-reduced feature matrix to obtain the negative semantic vector, wherein the negative semantic vector is used to capture the feature representation of negative semantics; based on the negative semantic vector, a loss function value is determined, and based on the loss function value, the model parameters of the text encoder in the initial image-text alignment model are adjusted to obtain the target image-text alignment model.

2. The model training method according to claim 1, characterized in that, Based on the initial training dataset, the target training dataset is generated as follows: The first image-text pair is randomly selected from the initial training dataset. The first descriptive text in the first image-text pair is analyzed by a text analysis agent to generate candidate text elements. The candidate text elements are text elements that are not mentioned in the first descriptive text but are related to the scene described by the first descriptive text. The text elements include at least one of the following: action, target object, and attribute. A multimodal intelligent agent is used to analyze the candidate text element and the image in the first image-text pair to determine whether the image in the first image-text pair contains the content described by the candidate text element. If the image in the first image-text pair does not contain the content described by the candidate text element, the candidate text element is determined as the target text element. Based on the target text element and the first descriptive text, the second image-text pair in the target training dataset is generated.

3. The model training method according to claim 2, characterized in that, Generating the second image-text pair in the target training dataset based on the target text element and the first descriptive text includes: Using the text analysis agent, a negative word is added to the target text element, and the target text element with the added negative word is added to the first descriptive text to obtain the negative descriptive text. The negative description text and the image in the first image-text pair are identified as the negative image-text pair; Furthermore, the target text element is used to replace the original text element in the first descriptive text that corresponds to the type of the target text element, in order to obtain the positive descriptive text; Generate an image corresponding to the content described by the positive descriptive text, and determine the image and the positive descriptive text as the positive image-text pair.

4. The model training method according to claim 3, characterized in that, The method further includes: Identify approximate text elements corresponding to the target text element, wherein the semantic similarity between the approximate text element and the target text element exceeds a preset similarity threshold, and the text elements are of the same type; The approximate descriptive text is obtained by replacing the original text element in the first descriptive text with the approximate text element whose type corresponds to the approximate text element. Generate an image corresponding to the content described by the approximate descriptive text, and determine the image and the approximate descriptive text as the approximate image-text pair.

5. The model training method according to claim 1, characterized in that, Based on the eigenvectors, constructing the feature matrix includes: Obtain the feature vector corresponding to the correctly matched negative image-text pairs in the model analysis results of the current training batch; If the negative image-text pair is a correct match for the first time, the feature vector is added to the feature matrix; If the negative image-text pair has been correctly matched in a training batch prior to the current training batch, obtain the historical feature vector in the feature matrix corresponding to the identifier of the negative image-text pair; The feature vector obtained from the current training batch and the historical feature vector previously stored in the feature matrix are weighted and fused to obtain a new feature vector. The new feature vector replaces the historical feature vector previously stored in the feature matrix. When performing the weighted fusion, the weight value corresponding to the feature vector obtained from the current training batch is greater than the weight value corresponding to the historical feature vector previously stored in the feature matrix.

6. The model training method according to claim 1, characterized in that, Based on the negation semantic vector, the loss function value is determined as follows: Based on the matching of the image-text pairs represented by the model analysis results and the correspondence of the image-text pairs in the target training dataset, a first loss value is determined, wherein the first loss value is used to characterize the accuracy of the model in image-text alignment; Determine the correctly matched negative image-text pairs in the model analysis results, and determine the feature vectors corresponding to the negative image-text pairs, wherein the feature vectors are used to characterize the feature difference between the positive description text and the negative description text in the positive image-text pairs that belong to the same group as the negative image-text pairs; Based on the feature vector and the negation semantic vector, a second loss value is determined, wherein the second loss value is used to characterize the degree of similarity between the feature vector and the negation semantic vector; The loss function value is determined based on the first loss value and the second loss value.

7. A method for matching images and text, characterized in that, include: Get the description text to be matched; A target image-text alignment model is used to analyze the descriptive text to be matched and determine the target image corresponding to the descriptive text. The target image-text alignment model is obtained by fine-tuning an initial image-text alignment model based on a target training dataset. The target training dataset contains multiple sets of second image-text pairs. Each set of second image-text pairs includes: positive image-text pairs, negative image-text pairs, and approximate image-text pairs. The positive image-text pairs include: positive descriptive text with affirmative semantics and an image corresponding to the descriptive content of the positive descriptive text. The negative image-text pairs include: negative descriptive text with negative semantics and an image corresponding to the descriptive content of the negative descriptive text. The approximate image-text pairs include: approximate descriptive text based on the positive semantics and the negative semantics and an image corresponding to the descriptive content of the approximate descriptive text. The process of fine-tuning the initial image-text alignment model using the target training dataset to obtain the target image-text alignment model includes: dividing the target training dataset into multiple training batches, wherein each training batch contains multiple sets of the second image-text pairs; using the initial image-text alignment model, analyzing the data of each of the multiple training batches to obtain model analysis results, and determining the negative semantic vector based on the model analysis results, including: determining the correctly matched negative image-text pairs in the model analysis results, and obtaining the positive descriptive text in the positive image-text pairs belonging to the same group as the negative image-text pairs; determining the feature vector based on the positive descriptive text and the negative descriptive text in the negative image-text pairs, and based on the... A feature matrix is ​​constructed from the described feature vectors, wherein the feature vectors are used to characterize the feature difference between the positive and negative descriptive texts, and the feature matrix is ​​used to record the feature vectors corresponding to the correctly matched negative image-text pairs in the multiple training batches; the feature matrix is ​​dimensionality-reduced using principal component analysis, and a nonlinear transformation is performed on the dimensionality-reduced feature matrix to obtain the negative semantic vector, wherein the negative semantic vector is used to capture the feature representation of negative semantics; based on the negative semantic vector, a loss function value is determined, and based on the loss function value, the model parameters of the text encoder in the initial image-text alignment model are adjusted to obtain the target image-text alignment model.

8. A model training device, characterized in that, include: The training set acquisition module is used to acquire an initial training dataset, wherein the initial training dataset includes multiple first image-text pairs, and each first image-text pair contains: an image and a first descriptive text corresponding to the content of the image; The training set optimization module is used to generate a target training dataset based on the initial training dataset. The target training dataset contains multiple sets of second image-text pairs. Each set of second image-text pairs contains: positive image-text pairs, negative image-text pairs, and approximate image-text pairs. The positive image-text pairs contain: positive descriptive text with affirmative semantics and an image corresponding to the descriptive content of the positive descriptive text. The negative image-text pairs contain: negative descriptive text with negative semantics and an image corresponding to the descriptive content of the negative descriptive text. The approximate image-text pairs contain approximate descriptive text based on the positive semantics and the negative semantics and an image corresponding to the descriptive content of the approximate descriptive text. The fine-tuning training module is used to fine-tune the initial image-text alignment model using the target training dataset to obtain the target image-text alignment model, wherein the target image-text alignment model is used to match the image corresponding to the descriptive text. The process of fine-tuning the initial image-text alignment model using the target training dataset to obtain the target image-text alignment model includes: dividing the target training dataset into multiple training batches, wherein each training batch contains multiple sets of the second image-text pairs; using the initial image-text alignment model, analyzing the data of each of the multiple training batches to obtain model analysis results, and determining the negative semantic vector based on the model analysis results, including: determining the correctly matched negative image-text pairs in the model analysis results, and obtaining the positive descriptive text in the positive image-text pairs belonging to the same group as the negative image-text pairs; determining the feature vector based on the positive descriptive text and the negative descriptive text in the negative image-text pairs, and based on the... A feature matrix is ​​constructed from the described feature vectors, wherein the feature vectors are used to characterize the feature difference between the positive and negative descriptive texts, and the feature matrix is ​​used to record the feature vectors corresponding to the correctly matched negative image-text pairs in the multiple training batches; the feature matrix is ​​dimensionality-reduced using principal component analysis, and a nonlinear transformation is performed on the dimensionality-reduced feature matrix to obtain the negative semantic vector, wherein the negative semantic vector is used to capture the feature representation of negative semantics; based on the negative semantic vector, a loss function value is determined, and based on the loss function value, the model parameters of the text encoder in the initial image-text alignment model are adjusted to obtain the target image-text alignment model.

9. An electronic device, characterized in that, include: A memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when running, executes the model training method of any one of claims 1 to 6 or the image-text matching method of claim 7.

10. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored computer program, wherein the device containing the non-volatile storage medium executes the model training method according to any one of claims 1 to 6 or the image-text matching method according to claim 7 by running the computer program.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the model training method according to any one of claims 1 to 6 or the image-text matching method according to claim 7.

Images