Training method of cross-modal model, retrieval method based on cross-modal model, device and electronic equipment

Abstract

The application discloses a training method of a cross-modal model, a retrieval method based on the cross-modal model, a device and an electronic device. The method comprises the following steps: obtaining a first training data set, wherein the data set comprises a plurality of first sample data, and the first sample data comprises a first image and a first description text; determining a text similarity between each first sample data and each preset concept term, wherein the text similarity is used to represent the similarity between the first description text in the first sample data and the corresponding preset concept term; screening sample data in the first training data set based on the text similarity between the plurality of first sample data and the plurality of preset concept terms to obtain a second training data set; and training an initial cross-modal model by using the second training data set to obtain a cross-modal model, wherein the cross-modal model is used to retrieve the description text corresponding to the image according to the image. The cross-modal model is trained by using the above method, cross-modal retrieval is performed, and the accuracy of retrieval can be improved.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to a training method for a cross-modal model, a retrieval method based on a cross-modal model, an apparatus, and an electronic device. Background Technology

[0002] With the development of the Internet, Internet data is growing rapidly, including multimedia information such as text and images. This information is rich in content and diverse in form, making it urgent to solve the problem of how to extract the information that we really need from the massive amount of data.

[0003] Currently, cross-modal coding models are used for image-text cross-modal retrieval. Cross-modal coding models are a type of model that can process and fuse multiple different modal data, aiming to map information from different modalities into a unified feature space for retrieval.

[0004] In the process of converting data from different modalities into a unified feature space, some information unique to the original modality may be lost, which may result in the inability to accurately match relevant data during retrieval, leading to low retrieval accuracy. Summary of the Invention

[0005] This application provides a method for training a cross-modal model, a retrieval method, apparatus, and electronic device based on the cross-modal model. Using the trained cross-modal model for cross-modal retrieval can improve the accuracy of retrieval.

[0006] In a first aspect, embodiments of this application provide a method for training a cross-modal model, the method comprising:

[0007] Obtain a first training dataset, which includes multiple first sample data, the first sample data including: a first image, and a first descriptive text for describing the first image;

[0008] Determine the text similarity between each of the first sample data and each preset concept term, wherein the text similarity is used to characterize the similarity between the first descriptive text in the first sample data and the corresponding preset concept term;

[0009] Based on the text similarity between the plurality of first sample data and the plurality of preset concept terms, sample data in the first training dataset is filtered to obtain the second training dataset;

[0010] The initial cross-modal model is trained using the second training dataset to obtain a cross-modal model, which is used to retrieve the image description text corresponding to the image.

[0011] In one embodiment of this application, determining the text similarity between each of the first sample data and each preset concept term includes:

[0012] The preset concept terms are encoded to obtain a first vector corresponding to the preset concept terms; and the first descriptive text in the first sample data is encoded to obtain a second vector corresponding to the first descriptive text.

[0013] Based on the second vector and the first vector, the text similarity between the first sample data and the preset concept term is determined.

[0014] In one embodiment of this application, the step of filtering sample data in the first training dataset based on the text similarity between the plurality of first sample data and the plurality of preset concept terms to obtain a second training dataset includes:

[0015] Obtain the loss function value corresponding to each of the first sample data, and the loss function value is associated with the initial cross-modal model;

[0016] Based on the loss function value corresponding to each first sample data in the plurality of first sample datasets, and the text similarity between each first sample data and each preset concept term in the plurality of preset concept terms, the mastery level value of each preset concept term is determined, and the mastery level value is used to characterize the mastery level of the concept term by the initial cross-modal model;

[0017] Based on the mastery level values ​​of each of the preset concept terms, at least one first concept term and at least one second concept term are determined among the plurality of preset concept terms, wherein the mastery level value of each first concept term is greater than the mastery level value of each second concept term.

[0018] The first training dataset is filtered according to the at least one first conceptual term and the at least one second conceptual term to obtain the second training dataset. The second training dataset includes multiple second sample data, which include: a second image and a second descriptive text for describing the second image.

[0019] In one embodiment of this application, obtaining the loss function value corresponding to each of the first sample data includes:

[0020] The sample data in the first training dataset is divided into multiple data subsets;

[0021] For each data subset, the following processing is performed: each of the first sample data in the data subset is input into the initial cross-modal model to obtain a first loss function value. The first loss function value is determined based on a second loss function value and a third loss function value. The second loss function value is the loss value of the first image relative to the first descriptive text, and the third loss function value is the loss function value of the first descriptive text relative to the first image.

[0022] The first loss function value is determined as the loss function value corresponding to each of the first sample data in the data subset.

[0023] In one embodiment of this application, the step of filtering the first training dataset according to the at least one first conceptual term and the at least one second conceptual term to obtain the second training dataset includes:

[0024] The following processing is performed on each of the first sample data in the first training dataset:

[0025] Based on the text similarity between the first sample data and each of the first concept terms, the average text similarity of the first sample data relative to the multiple first concept terms is calculated to obtain a first value;

[0026] Based on the text similarity between the first sample data and each of the second concept terms, the average text similarity of the first sample data relative to the multiple second concept terms is calculated to obtain the second value;

[0027] If the second value is greater than the first value, then the first sample data is used as the second sample data in the second training dataset.

[0028] In one embodiment of this application, determining the mastery level value of each preset concept term based on the loss function value corresponding to each first sample data in the plurality of first sample datasets and the text similarity between each first sample data and each preset concept term in the plurality of preset concept terms includes:

[0029] One of the multiple preset concept terms is taken as the target concept term;

[0030] For each first sample data in the first sample dataset, the following processing is performed: the loss function value corresponding to the first sample data is multiplied by the first text similarity to obtain the third value corresponding to the first sample data, wherein the first text similarity is the text similarity between the first sample data and the target concept term;

[0031] The mastery level value of the target concept term is obtained by weighted summing of the third values ​​corresponding to each first sample data in the first sample dataset;

[0032] If a third concept term exists among the plurality of preset concept terms, then the third concept term is used as the target concept term, and the process jumps to the step of multiplying the loss function value corresponding to the first sample data with the first text similarity to obtain the third value corresponding to the first sample data, wherein the third concept term is a concept term that has not participated in the calculation.

[0033] Secondly, embodiments of this application provide a retrieval method based on a cross-modal model, the method comprising:

[0034] Retrieve the image to be searched and multiple reference descriptive texts;

[0035] The image and multiple reference description texts are input into a cross-modal model for retrieval to obtain the target reference description text corresponding to the image. The cross-modal model is obtained based on the training method of the cross-modal model described in the first aspect.

[0036] Thirdly, embodiments of this application provide a training apparatus for a cross-modal model, the apparatus comprising:

[0037] The first acquisition module is used to acquire a first training dataset, which includes multiple first sample data, and the first sample data includes: a first image and a first descriptive text for describing the first image;

[0038] The determination module is used to determine the text similarity between each of the first sample data and each preset concept term, wherein the text similarity is used to characterize the similarity between the first descriptive text in the first sample data and the corresponding preset concept term;

[0039] The filtering module is used to filter sample data in the first training dataset based on the text similarity between the plurality of first sample data and the plurality of preset concept terms, respectively, to obtain a second training dataset;

[0040] The training module is used to train the initial cross-modal model using the second training dataset to obtain a cross-modal model, which is used to retrieve the image based on the image to obtain the corresponding descriptive text.

[0041] In one embodiment of this application, the first acquisition module includes: an encoding submodule and a determination submodule;

[0042] The encoding submodule is used to encode the preset concept term to obtain a first vector corresponding to the preset concept term; and to encode the first descriptive text in the first sample data to obtain a second vector corresponding to the first descriptive text.

[0043] The determination submodule is used to determine the text similarity between the first sample data and the preset concept term based on the second vector and the first vector.

[0044] In one embodiment of this application, the determining sub-module includes: obtaining sub-units, calculating sub-units, and filtering sub-units;

[0045] A sub-unit is used to obtain the loss function value corresponding to each of the first sample data, and the loss function value is associated with the initial cross-modal model;

[0046] The calculation subunit is used to determine the mastery level value of each preset concept term based on the loss function value corresponding to each first sample data in the plurality of first sample datasets and the text similarity between each first sample data and each preset concept term in the plurality of preset concept terms. The mastery level value is used to characterize the mastery level of the concept term by the initial cross-modal model. Based on the mastery level value of each preset concept term, at least one first concept term and at least one second concept term are determined among the plurality of preset concept terms, and the mastery level value of each first concept term is greater than the mastery level value of each second concept term.

[0047] A filtering subunit is configured to filter the first training dataset according to the at least one first concept term and the at least one second concept term to obtain a second training dataset. The second training dataset includes multiple second sample data, which include: a second image and a second descriptive text for describing the second image.

[0048] In one embodiment of this application, the sub-unit is specifically used to divide the sample data in the first training dataset to obtain multiple data subsets;

[0049] For each subset of data, perform the following processing:

[0050] The acquisition subunit is specifically used to input each of the first sample data in the data subset into the initial cross-modal model to obtain a first loss function value. The first loss function value is determined based on a second loss function value and a third loss function value. The second loss function value is the loss value of the first image relative to the first descriptive text, and the third loss function value is the loss function value of the first descriptive text relative to the first image.

[0051] The sub-unit is specifically used to determine the first loss function value as the loss function value corresponding to each of the first sample data in the data subset.

[0052] In one embodiment of this application, the first sample data in the first training dataset is processed as follows: a filtering subunit is specifically used to calculate the average text similarity of the first sample data with respect to multiple first concept terms based on the text similarity between the first sample data and each first concept term, to obtain a first value; based on the text similarity between the first sample data and each second concept term, calculate the average text similarity of the first sample data with respect to multiple second concept terms, to obtain a second value; if the second value is greater than the first value, then the first sample data is used as the second sample data in the second training dataset.

[0053] In one embodiment of this application, the calculation subunit is specifically used to take one of the plurality of preset concept terms as the target concept term; and to perform the following processing on each of the first sample data in the first sample dataset: multiply the loss function value corresponding to the first sample data by the first text similarity to obtain the third value corresponding to the first sample data, wherein the first text similarity is the text similarity between the first sample data and the target concept term;

[0054] The calculation subunit is specifically used to perform a weighted summation of the third value corresponding to each of the first sample data in the first sample dataset to obtain the mastery level value of the target concept term.

[0055] The calculation subunit is specifically used to, if there is a third concept term among the plurality of preset concept terms, take the third concept term as the target concept term and jump to the step of multiplying the loss function value corresponding to the first sample data with the first text similarity to obtain the third value corresponding to the first sample data, wherein the third concept term is a concept term that has not participated in the calculation.

[0056] Fourthly, embodiments of this application provide a retrieval device based on a cross-modal model, the device comprising:

[0057] The second acquisition module is used to acquire the image to be retrieved and multiple reference description texts;

[0058] The retrieval module is used to input the image and multiple reference description texts into the cross-modal model for retrieval, and obtain the target reference description text corresponding to the image. The cross-modal model is obtained based on the training method of the cross-modal model described in the first aspect.

[0059] Fifthly, embodiments of this application provide an electronic device, including: a processor and a memory storing computer program instructions;

[0060] When the processor executes the computer program instructions, it implements the cross-modal model training method as described in the first aspect, or the cross-modal model-based retrieval method as described in the second aspect.

[0061] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the cross-modal model training method as described in the first aspect, or the cross-modal model-based retrieval method as described in the second aspect.

[0062] In a seventh aspect, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform a cross-modal model training method as described in the first aspect, or a cross-modal model-based retrieval method as described in the second aspect.

[0063] This application discloses a cross-modal model training method, a cross-modal model-based retrieval method, apparatus, and electronic device. The method involves acquiring a first training dataset, which includes multiple first sample data sets. Each first sample data set includes a first image and first descriptive text describing the first image. The method further involves determining the text similarity between each first sample data set and each preset concept term, where the text similarity characterizes the similarity between the first descriptive text in the first sample data set and the corresponding preset concept term. Based on the text similarity between the multiple first sample data sets and the multiple preset concept terms, the method filters sample data from the first training dataset to obtain a second training dataset. The second training dataset is then used to train an initial cross-modal model to obtain a cross-modal model. This cross-modal model is used for image retrieval to obtain the corresponding descriptive text. The cross-modal model trained using the above method enables cross-modal retrieval, achieving interaction between different modalities. The cross-modal model, used for image retrieval to obtain corresponding text descriptions, can improve retrieval accuracy. Attached Figure Description

[0064] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 This is a flowchart illustrating a cross-modal model training method provided in an embodiment of this application;

[0066] Figure 2 This is a schematic diagram of the similarity matrix provided in the embodiments of this application;

[0067] Figure 3 This is a data filtering diagram provided in an embodiment of this application;

[0068] Figure 4 This is a schematic diagram of the level of mastery values ​​provided in the embodiments of this application;

[0069] Figure 5 This is another flowchart illustrating the cross-modal model training method provided in the embodiments of this application;

[0070] Figure 6 This is a flowchart illustrating a retrieval method based on a cross-modal model provided in an embodiment of this application;

[0071] Figure 7 This is a schematic diagram of the structure of the training device for the cross-modal model provided in the embodiments of this application;

[0072] Figure 8 This is a schematic diagram of the structure of the retrieval device based on a cross-modal model provided in an embodiment of this application;

[0073] Figure 9 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0074] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0075] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0076] In all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. Additionally, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments obtained.

[0077] To address the problems of existing technologies, embodiments of this application provide a method for training a cross-modal model, a retrieval method based on a cross-modal model, an apparatus, and an electronic device. The method for training a cross-modal model provided in this application embodiment will be described first below.

[0078] Figure 1 A flowchart illustrating a training method for a cross-modal model provided in an embodiment of this application is shown. Figure 1 As shown in the embodiments of this application, the cross-modal model training method is applied to electronic devices and includes the following steps 101-104, wherein:

[0079] Step 101: Obtain a first training dataset. The first training dataset includes multiple first sample data, which include: a first image and a first descriptive text for describing the first image.

[0080] In this embodiment, a first training dataset is obtained, which includes multiple first sample data. Each first sample data includes a first image and a first descriptive text for describing the first image. The first descriptive text is used to describe the content in the first image, which may be a pathological image of the user.

[0081] Taking the first image as the first pathological image and the first descriptive text as the first pathological text as an example, as shown in Table 1, the first training dataset may include multiple first sample data from different parts, or multiple first sample data from the same part; for example, the amount of data for the pleura is 39, that is, there are 39 first sample data related to the pleura. Each first sample data includes a first pathological image and a first pathological text used to describe the first pathological image.

[0082] Table 1

[0083]

[0084]

[0085] It should be noted that when acquiring pathological images and texts, the user's individual permission or consent will be obtained through pop-up windows or redirection to a confirmation page. Only after obtaining the user's individual permission or consent will the necessary user-related data be acquired to enable the embodiments of this disclosure to operate normally.

[0086] Step 102: Determine the text similarity between each of the first sample data and each preset concept term. The text similarity is used to characterize the similarity between the first descriptive text in the first sample data and the corresponding preset concept term.

[0087] In this embodiment, for multiple preset concept terms, the text similarity between each first sample data and each preset concept term is determined, that is, the similarity between the first descriptive text in the first sample data and the corresponding preset concept term. By encoding the first descriptive text to obtain an encoding vector, and encoding the preset concept term to obtain an encoding vector, the text similarity between the first descriptive text and the preset concept term is calculated based on the two vectors, and the obtained similarity is used as the similarity between the first descriptive text and the corresponding preset concept term.

[0088] Taking the first image as the first pathological image and the first descriptive text as the first pathological text as an example, see [link to relevant documentation]. Figure 2 The first sample data can also be called an image-text pair. An image-text pair includes multiple pairs, each containing both text and an image, i.e., the first pathological text and the first pathological image. The terminology includes multiple predefined conceptual terms, such as... Figure 2 As shown, there are 8 texts and 8 terms. The text similarity between each text-image pair and each term is determined. The text similarity is used to characterize the similarity between the text and image in the text-image pair and the corresponding term (that is, the text similarity between each first sample data and each preset concept term is determined above). The text similarity between each text-image pair and each term can form a similarity matrix.

[0089] Step 103: Based on the text similarity between the plurality of first sample data and the plurality of preset concept terms, filter the sample data in the first training dataset to obtain the second training dataset.

[0090] In this embodiment, the sample data in the first training dataset is filtered according to the similarity between each first sample data and each preset concept term. For example, the mastery level value of each preset concept term is further determined according to the similarity, and the sample data in the first training dataset is filtered according to the mastery level value to obtain the second training dataset.

[0091] Step 104: Train the initial cross-modal model using the second training dataset to obtain a cross-modal model. The cross-modal model is used to retrieve the image based on the image and obtain the descriptive text corresponding to the image.

[0092] In this embodiment, the initial cross-modal model is trained using a second training dataset. During the training process, the model loss value is obtained. If the loss value is greater than or equal to a preset loss value, the initial cross-modal model is used as the cross-modal model. The cross-modal model is used to perform cross-modal retrieval based on the image to obtain the descriptive text corresponding to the image. Cross-modal retrieval breaks away from the limitations of traditional single-modal retrieval and enables cross-modal retrieval across different modalities.

[0093] In this embodiment, a first training dataset is obtained, which includes multiple first sample data, each comprising a first image and a first descriptive text describing the first image. The text similarity between each first sample data and each preset concept term is determined, whereby the text similarity characterizes the similarity between the first descriptive text in the first sample data and the corresponding preset concept term. Based on the text similarity between the multiple first sample data and the multiple preset concept terms, sample data in the first training dataset is filtered to obtain a second training dataset. The second training dataset is used to train an initial cross-modal model to obtain a cross-modal model. This cross-modal model is used to retrieve images and obtain the corresponding descriptive text. By training the cross-modal model using the above method, cross-modal retrieval is performed, enabling interaction between different modalities. The cross-modal model is used to retrieve corresponding text descriptions based on images, which can improve the accuracy of retrieval.

[0094] In one embodiment of this application, step 102, determining the text similarity between each of the first sample data and each preset concept term, includes:

[0095] The preset concept terms are encoded to obtain a first vector corresponding to the preset concept terms; and the first descriptive text in the first sample data is encoded to obtain a second vector corresponding to the first descriptive text.

[0096] Based on the second vector and the first vector, the text similarity between the first sample data and the preset concept term is determined.

[0097] In this embodiment, for each preset concept term, the preset concept term is encoded to obtain a first vector corresponding to the preset concept term; for each first sample data, the first descriptive text is encoded to obtain a second vector corresponding to the first descriptive text. Based on the first vector and the second vector, the text similarity between the first sample data and the preset concept term is determined. Specifically, the cosine similarity between the first vector and the second vector is calculated to obtain a first result. The first result is used as the text similarity between the sample data and the preset concept term. The similarity between the two vectors is measured by calculating the cosine value of the angle between them. The smaller the angle, the closer the cosine value is to 1, and the more similar the vectors are.

[0098] Alternatively, calculate the Euclidean distance (also known as Euclidean distance) between the first and second vectors. Euclidean distance measures the degree of difference between two vectors. Generally, the smaller the Euclidean distance, the higher the similarity between the vectors; the larger the Euclidean distance, the lower the similarity between the vectors. The two are negatively correlated.

[0099] Alternatively, calculate the Manhattan distance between the first and second vectors. Manhattan distance, also known as city block distance, measures the degree of difference between two vectors, thus reflecting their similarity. Generally, the smaller the Manhattan distance, the higher the similarity between the two vectors; the larger the Manhattan distance, the lower the similarity between the two vectors.

[0100] Alternatively, calculate the Pearson correlation coefficient between the first and second vectors. The Pearson correlation coefficient is used to measure the linear correlation between two vectors. The closer the value is to 1 or -1, the stronger the correlation, which means that the vectors have similar trends to some extent.

[0101] The text similarity between each first sample data and each preset concept term is calculated using one of the methods described above.

[0102] Similarity can represent the relationship between descriptive text and conceptual terms. The higher the similarity, the stronger the relationship. By filtering data through similarity, representative data can be selected, which can more effectively train the model.

[0103] In one embodiment of this application, the step of filtering sample data in the first training dataset based on the text similarity between the plurality of first sample data and the plurality of preset concept terms to obtain a second training dataset includes:

[0104] Obtain the loss function value corresponding to each of the first sample data, and the loss function value is associated with the initial cross-modal model;

[0105] Based on the loss function value corresponding to each first sample data in the plurality of first sample datasets, and the text similarity between each first sample data and each preset concept term in the plurality of preset concept terms, the mastery level value of each preset concept term is determined, and the mastery level value is used to characterize the mastery level of the concept term by the initial cross-modal model;

[0106] Based on the mastery level values ​​of each of the preset concept terms, at least one first concept term and at least one second concept term are determined among the plurality of preset concept terms, wherein the mastery level value of each first concept term is greater than the mastery level value of each second concept term.

[0107] The first training dataset is filtered according to the at least one first conceptual term and the at least one second conceptual term to obtain the second training dataset. The second training dataset includes multiple second sample data, which include: a second image and a second descriptive text for describing the second image.

[0108] In this embodiment, the loss function value corresponding to each first sample data is obtained. The loss function value is associated with the cross-modal model. For example, multiple first sample data are input into the cross-modal model to obtain the loss function value, and the obtained loss function value is used as the loss function value of each first sample data. Alternatively, the first training dataset is grouped to obtain multiple data subsets. Each data subset includes multiple first sample data. The multiple first sample data in each data subset are used in the cross-modal model to obtain the loss function value, and the obtained loss function value is used as the loss function value of each first sample data in the data subset.

[0109] In the above process, for each first sample data in the plurality of first sample datasets, the mastery level value of each preset concept term is calculated based on the loss function value corresponding to the first sample data and the text similarity between the first sample data and each preset concept term in the plurality of preset concept terms. The mastery level value is used to characterize the mastery level of the cross-modal model on the term. The higher the mastery level of the model on the term, the larger the mastery level value; the lower the mastery level of the model on the term, the smaller the mastery level value.

[0110] By balancing the model's learning of the correspondence between text and images under each concept and term, the model's effectiveness in linking text and images within the relevant domain is optimized. "Mastery" describes the extent to which the model can associate text and images related to a given concept or term. Mastery refers to the model's understanding of the correspondence between images and text related to that concept or term.

[0111] For example, to describe the model's mastery of the conceptual term "lung cancer," it is necessary to calculate the average distance between multiple different lung cancer-related pathological images and their corresponding pathological texts in the model's encoding space. The larger the average distance, the less well the model can match the pathological images and texts related to the conceptual term "lung cancer," indicating a lower level of mastery. Further learning of the image-text pairing relationships related to the conceptual term is needed. If the model's mastery of different conceptual terms, such as "lung cancer," "stomach cancer," and "breast cancer," is roughly the same, then the model has a similar level of learning for "cancer"-related conceptual terms. In this case, the model will not be affected by a learning bias towards a particular conceptual term, thus making it easier to produce optimal image-text pairing results in fields such as "cancer."

[0112] Based on the mastery level values ​​of each preset conceptual term, at least one first conceptual term and at least one second conceptual term are selected from multiple preset conceptual terms. Optionally, see [link to relevant documentation]. Figure 3 The mastery levels of each preset concept term are sorted to obtain the sorted mastery levels. From the sorted results, the concept terms with the highest mastery levels are selected as the first concept terms, such as... Figure 3 C in High The corresponding first-concept terms; and from the ranking results, select several concept terms with lower mastery values, that is, second-concept terms, such as... Figure 3 C in Low The corresponding second-concept term represents the level of mastery for each first-concept term. First-concept terms indicate relatively high mastery, while second-concept terms indicate relatively low mastery.

[0113] Furthermore, the first training dataset is filtered according to at least one first concept term and at least one second concept term to obtain a second training dataset. The second training dataset includes second sample data, which includes a second image and a second descriptive text for describing the second image.

[0114] By calculating the degree of mastery through text similarity, first-concept terms provide less relevance, while second-concept terms provide more relevance. The data is filtered based on the degree of mastery of the concepts and terms, and the filtered data can be used to train the model more effectively.

[0115] In one embodiment of this application, obtaining the loss function value corresponding to each of the first sample data includes:

[0116] The sample data in the first training dataset is divided into multiple data subsets;

[0117] For each data subset, the following processing is performed: each of the first sample data in the data subset is input into the initial cross-modal model to obtain a first loss function value. The first loss function value is determined based on a second loss function value and a third loss function value. The second loss function value is the loss value of the first image relative to the first descriptive text, and the third loss function value is the loss function value of the first descriptive text relative to the first image.

[0118] The first loss function value is determined as the loss function value corresponding to each of the first sample data in the data subset.

[0119] In this embodiment, the sample data in the first training dataset is divided into multiple data subsets, i.e., batch processing is performed. For each data subset, the following processing is performed: each first sample data in the data subset is input into the initial cross-modal model to obtain the first loss function value.

[0120] The initial cross-modal model can be a Contrastive Language-Image Pre-training (CLIP) model. CLIP is a model that includes a two-stream encoder, namely an image encoder and a text encoder. It encodes a batch of first images and first descriptive text. The image encoder encodes each first image to obtain the feature vector f corresponding to the first image. Ii Based on each feature vector, an image feature sequence F is constructed. I =[f I1 ,f I2 ,…,f IN Each first description text is encoded using a text encoder to obtain the feature vector f corresponding to the first description text. Ti Based on each feature vector, a text feature sequence F is constructed. T =[f T1 ,f T2 ,…,f TNThen, the first and second feature vectors are normalized so that their magnitudes are 1, resulting in the normalized first and second feature sequences.

[0121] For image feature sequences and text feature sequences, calculate the similarity matrix S between them, where the elements of matrix S are... ij Image I i eigenvector f Ii With text T j eigenvector f Tj The dot product between the features is equivalent to the cosine similarity, S, since the feature vectors have been normalized. ij =f I ·f Tj .

[0122] The second loss function value is calculated as follows: For each first image corresponding to the normalized image feature sequence, its positive sample is the corresponding first descriptive text, and the remaining N-1 first descriptive texts are negative samples. The loss value of the first image relative to the first descriptive text is calculated using the following formula:

[0123]

[0124] Where τ is a hyperparameter used to adjust the learning difficulty for each first image I. i Calculate its relationship with the positive sample (corresponding to the first descriptive text T). i The logarithmic probability of the similarity score of the image is the sum of the similarity scores of the image with the descriptive text of all images, including the first, second and other descriptive texts present in the dataset.

[0125] The third loss function is calculated as follows: For each first descriptive text corresponding to the normalized text feature sequence, its positive sample is the corresponding first image, and the remaining N-1 first images are negative samples. The loss function value of the first descriptive text relative to the first image is calculated using the following formula:

[0126]

[0127] Where τ is a hyperparameter, for each first descriptive text T i Calculate its relationship with the positive sample (corresponding to the first image I). i The logarithmic probability of the similarity score of the text is the sum of the similarity scores of the text with all images, including the first, second and other images present in the dataset.

[0128] The optimization objective of cross-modal models is to minimize the total loss when training them. This is achieved by continuously adjusting the parameters of the image encoder and text encoder to maximize the similarity of matched image-text pairs in the feature space and minimize the similarity of unmatched pairs.

[0129] The first loss function value is calculated based on the second and third loss function values, as follows:

[0130] L=Lit+Lti (3)

[0131] Where L is the first loss function value, Lit is the second loss function value, and Lti is the third loss function value.

[0132] For each subset of data, the first loss function value is determined as the loss function value corresponding to each first sample data.

[0133] In one embodiment of this application, the step of filtering the first training dataset according to the at least one first conceptual term and the at least one second conceptual term to obtain the second training dataset includes:

[0134] The following processing is performed on each of the first sample data in the first training dataset:

[0135] Based on the text similarity between the first sample data and each of the first concept terms, the average text similarity of the first sample data relative to the multiple first concept terms is calculated to obtain a first value;

[0136] Based on the text similarity between the first sample data and each of the second concept terms, the average text similarity of the first sample data relative to the multiple second concept terms is calculated to obtain the second value;

[0137] If the second value is greater than the first value, then the first sample data is used as the second sample data in the second training dataset.

[0138] In this embodiment, each first sample data in the first training dataset is processed as follows: based on the text similarity between the first sample data and each first concept term, the average text similarity of the first sample data relative to multiple first concept terms is calculated. For example, if there are three first concept terms among multiple preset concept terms, namely concept term a, concept term b, and concept term c, the text similarity between the first sample data and concept term a is x1, the text similarity between the first sample data and concept term b is x2, and the text similarity between the first sample data and concept term c is x3. Calculate (x1+x2+x3) / 3 to obtain the average text similarity of the first sample data relative to multiple first concept terms, i.e., the first value.

[0139] Based on the text similarity between the first sample data and each second concept term, the average text similarity of the first sample data relative to multiple second concept terms is calculated. The average value is then calculated based on the multiple text similarities to obtain the second value. The first value and the second value are compared, and the data is filtered based on the comparison results.

[0140] Conceptual terminology is explained using pathological terminology as an example; see [link / reference]. Figure 3 As shown, the first pathological term is related to C. High The corresponding pathological terminology, the second pathological term is related to C Low The corresponding pathological terms are as follows: the first pathological term corresponds to each first sample data in the first training dataset, and the second pathological term corresponds to each first sample data in the first training dataset. For each first sample data: the first value and the second value are compared, and it is determined whether to retain the first sample data based on the comparison result. If the second value is greater than the first value, the first sample data is retained and used as the second sample data in the second training dataset. The first pathological image is used as the second pathological image, and the first pathological text is used as the second pathological text. The second training dataset includes multiple second sample data, which include: the second pathological image and the second pathological text used to describe the second pathological image.

[0141] By comparing the average text similarity, the data is filtered out, removing redundant data. This allows the model to converge faster, significantly saving training time and improving training efficiency. Furthermore, selecting representative data enables more effective model training.

[0142] In one embodiment of this application, determining the mastery level value of each preset concept term based on the loss function value corresponding to each first sample data in the plurality of first sample datasets and the text similarity between each first sample data and each preset concept term in the plurality of preset concept terms includes:

[0143] One of the multiple preset concept terms is taken as the target concept term;

[0144] For each first sample data in the first sample dataset, the following processing is performed: the loss function value corresponding to the first sample data is multiplied by the first text similarity to obtain the third value corresponding to the first sample data, wherein the first text similarity is the text similarity between the first sample data and the target concept term;

[0145] The mastery level value of the target concept term is obtained by weighted summing of the third values ​​corresponding to each first sample data in the first sample dataset;

[0146] If a third concept term exists among the plurality of preset concept terms, then the third concept term is used as the target concept term, and the process jumps to the step of multiplying the loss function value corresponding to the first sample data with the first text similarity to obtain the third value corresponding to the first sample data, wherein the third concept term is a concept term that has not participated in the calculation.

[0147] In this embodiment, one of the multiple preset concept terms is taken as the target concept term. For each first sample data in the first sample dataset, the following processing is performed: the loss function value corresponding to the first sample data is multiplied by the first text similarity, where the first text similarity is the text similarity between the first sample data and the target concept term. The multiplication yields the third value corresponding to the first sample data.

[0148] For the first sample dataset, the third value corresponding to each first sample data point is weighted and summed to obtain the mastery level value of the target concept term. The mastery level value is expressed as:

[0149] h(M,C)=-∑ X∈D R(X,C)l(X,M) (4)

[0150] Where h(M,C) represents the mastery level value, that is, the degree of mastery of the target concept term by the cross-modal model, R(X,C) represents the text similarity between the first sample data and the target concept term, D represents the first sample dataset, l(X,M) represents the loss function value corresponding to the first sample data, the loss function value is obtained through the cross-modal model, and M represents the cross-modal model.

[0151] See Figure 4 As shown, h represents the mastery level value, and the first sample dataset also becomes the image-text pair dataset. The mastery level value of each preset concept term is obtained according to the above method.

[0152] See Figure 5 , Figure 5 This is another flowchart illustrating the training method for cross-modal models, where the first training dataset (i.e.) is obtained. Figure 5 The first training dataset includes multiple first sample data, each first sample data including a first image and a first descriptive text describing the first image. Further, the correlation between the data (i.e., the first sample data) and terms (i.e., multiple preset concept terms) is evaluated (i.e., determining the text similarity between each first sample data and each preset concept term as described above). Further, the model-term mastery is evaluated (i.e., determining the mastery value of each preset concept term based on the loss function value corresponding to each first sample data in the multiple first sample datasets and the text similarity between each first sample data and each preset concept term in the multiple preset concept terms as described above). Further, based on... The process involves sampling stage data based on the level of mastery (i.e., determining at least one first concept term and at least one second concept term from among the multiple preset concept terms based on the mastery level values ​​of each preset concept term, wherein the mastery level value of each first concept term is greater than the mastery level value of each second concept term; and filtering the first training dataset based on the at least one first concept term and the at least one second concept term to obtain the second training dataset). Further, the stage data is used to train the parameters of the cross-modal model. The training process is divided into several stages, with each stage involving a certain number of parameter updates, resulting in a trained cross-modal model (i.e., training the cross-modal model using the second training dataset as described above to obtain the cross-modal model).

[0153] A model is used for cross-modal retrieval to enable interaction between different modalities. The model is used to retrieve corresponding descriptive text based on images, which can improve the accuracy of retrieval and the efficiency of obtaining descriptive text.

[0154] Figure 6 This illustration shows a flowchart of a retrieval method based on a cross-modal model provided in an embodiment of this application. Figure 6 As shown, the retrieval method based on a cross-modal model provided in this application embodiment is applied to an electronic device and includes the following steps 601-602, wherein:

[0155] Step 601: Obtain the image to be retrieved and multiple reference descriptive texts.

[0156] In this embodiment, the image to be retrieved and multiple reference description texts are obtained. Multiple reference description texts are stored in the database, and multiple reference description texts are retrieved from the database.

[0157] The image to be retrieved can be a pathological image, and the reference descriptive text can be a reference pathological text.

[0158] Step 602: Input the image and multiple reference description texts into the cross-modal model for retrieval to obtain the target reference description text corresponding to the image. The cross-modal model is obtained based on the cross-modal model training method.

[0159] In this embodiment, an image and multiple reference descriptive texts are input into a cross-modal model. Specifically, the cross-modal model includes an image encoder and a text encoder. The image encoder encodes the image to be retrieved, obtaining its feature vector. The text encoder encodes each reference descriptive text, obtaining its feature vector. The similarity, i.e., cosine similarity, between the feature vector of the image to be retrieved and the feature vector of each reference descriptive text is calculated. During training, the cross-modal model uses contrastive learning, ensuring that images and their corresponding texts are close in semantic space, while non-corresponding images and texts are far apart. Therefore, the semantic relevance between images and texts can be measured by calculating the cosine similarity between feature vectors. Based on the calculated similarity scores, the multiple reference descriptive texts are ranked. Reference descriptive texts with higher similarity scores have a higher semantic match with the input image. The reference descriptive text with the highest similarity score is selected as the target reference descriptive text. The target reference descriptive text is the text that can describe the image and is provided for user reference.

[0160] In this embodiment, the image to be retrieved and multiple reference description texts are obtained. The image and multiple reference description texts are input into a cross-modal model to obtain the target reference description text output by the model. This realizes cross-modal retrieval of images and texts, and can obtain the target reference description text corresponding to the image, thereby improving the accuracy of the retrieved text.

[0161] Figure 7 A structural diagram of a training apparatus for a cross-modal model provided in an embodiment of this application is shown. Figure 7 As shown, the cross-modal model training device 700 includes:

[0162] The first acquisition module 701 is used to acquire a first training dataset, the first training dataset including multiple first sample data, the first sample data including: a first image, and a first descriptive text for describing the first image;

[0163] The determining module 702 is used to determine the text similarity between each of the first sample data and each preset concept term, wherein the text similarity is used to characterize the similarity between the first descriptive text in the first sample data and the corresponding preset concept term;

[0164] The filtering module 703 is used to filter sample data in the first training dataset based on the text similarity between the plurality of first sample data and the plurality of preset concept terms, respectively, to obtain a second training dataset.

[0165] The training module 704 is used to train the initial cross-modal model using the second training dataset to obtain a cross-modal model, which is used to retrieve the image based on the image to obtain the corresponding descriptive text.

[0166] In one embodiment of this application, the first acquisition module includes: an encoding submodule and a determination submodule;

[0167] The encoding submodule is used to encode the preset concept term to obtain a first vector corresponding to the preset concept term; and to encode the first descriptive text in the first sample data to obtain a second vector corresponding to the first descriptive text.

[0168] The determination submodule is used to determine the text similarity between the first sample data and the preset concept term based on the second vector and the first vector.

[0169] In one embodiment of this application, the determining sub-module includes: obtaining sub-units, calculating sub-units, and filtering sub-units;

[0170] A sub-unit is used to obtain the loss function value corresponding to each of the first sample data, and the loss function value is associated with the initial cross-modal model;

[0171] The calculation subunit is used to determine the mastery level value of each preset concept term based on the loss function value corresponding to each first sample data in the plurality of first sample datasets and the text similarity between each first sample data and each preset concept term in the plurality of preset concept terms. The mastery level value is used to characterize the mastery level of the concept term by the initial cross-modal model. Based on the mastery level value of each preset concept term, at least one first concept term and at least one second concept term are determined among the plurality of preset concept terms, and the mastery level value of each first concept term is greater than the mastery level value of each second concept term.

[0172] A filtering subunit is configured to filter the first training dataset according to the at least one first concept term and the at least one second concept term to obtain a second training dataset. The second training dataset includes multiple second sample data, which include: a second image and a second descriptive text for describing the second image.

[0173] In one embodiment of this application, the sub-unit is specifically used to divide the sample data in the first training dataset to obtain multiple data subsets;

[0174] For each subset of data, perform the following processing:

[0175] The acquisition subunit is specifically used to input each of the first sample data in the data subset into the initial cross-modal model to obtain a first loss function value. The first loss function value is determined based on a second loss function value and a third loss function value. The second loss function value is the loss value of the first image relative to the first descriptive text, and the third loss function value is the loss function value of the first descriptive text relative to the first image.

[0176] The sub-unit is specifically used to determine the first loss function value as the loss function value corresponding to each of the first sample data in the data subset.

[0177] In one embodiment of this application, the first sample data in the first training dataset is processed as follows: a filtering subunit is specifically used to calculate the average text similarity of the first sample data with respect to multiple first concept terms based on the text similarity between the first sample data and each first concept term, to obtain a first value; based on the text similarity between the first sample data and each second concept term, calculate the average text similarity of the first sample data with respect to multiple second concept terms, to obtain a second value; if the second value is greater than the first value, then the first sample data is used as the second sample data in the second training dataset.

[0178] In one embodiment of this application, the calculation subunit is specifically used to take one of the plurality of preset concept terms as the target concept term; and to perform the following processing on each of the first sample data in the first sample dataset: multiply the loss function value corresponding to the first sample data by the first text similarity to obtain the third value corresponding to the first sample data, wherein the first text similarity is the text similarity between the first sample data and the target concept term;

[0179] The calculation subunit is specifically used to perform a weighted summation of the third value corresponding to each of the first sample data in the first sample dataset to obtain the mastery level value of the target concept term.

[0180] The calculation subunit is specifically used to, if there is a third concept term among the plurality of preset concept terms, take the third concept term as the target concept term and jump to the step of multiplying the loss function value corresponding to the first sample data with the first text similarity to obtain the third value corresponding to the first sample data, wherein the third concept term is a concept term that has not participated in the calculation.

[0181] The cross-modal model training apparatus provided in this application embodiment can implement all the processes implemented in the aforementioned cross-modal model training method embodiment and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0182] Figure 8 A structural diagram of a cross-modal model-based retrieval device provided in an embodiment of this application is shown. Figure 8 As shown, the retrieval device 800 based on a cross-modal model includes:

[0183] The second acquisition module 801 is used to acquire the image to be retrieved and multiple reference description texts;

[0184] The retrieval module 802 is used to input the image and multiple reference description texts into the cross-modal model for retrieval, and obtain the target reference description text corresponding to the image. The cross-modal model is obtained based on the training method of the cross-modal model described above.

[0185] The retrieval device based on cross-modal models provided in this application can realize all the processes implemented in the aforementioned retrieval method embodiments based on cross-modal models and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0186] Figure 9 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.

[0187] The electronic device may include a processor 901 and a memory 902 storing computer program instructions.

[0188] Specifically, the processor 901 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0189] Memory 902 may include mass storage for data or instructions. For example, and not limitingly, memory 902 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 902 may include removable or non-removable (or fixed) media. Where appropriate, memory 902 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 902 is non-volatile solid-state memory.

[0190] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the methods according to the first or second aspect of this disclosure.

[0191] The processor 901 reads and executes computer program instructions stored in the memory 902 to implement any of the information auditing methods in the above embodiments.

[0192] In one example, the electronic device may also include a communication interface 903 and a bus 910. Wherein, as... Figure 9 As shown, the processor 901, memory 902, and communication interface 903 are connected through bus 910 and complete communication with each other.

[0193] The communication interface 903 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0194] Bus 910 includes hardware, software, or both, that couples components of an information auditing method or verification device together. For example, and not as a limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 910 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0195] Furthermore, in conjunction with the cross-modal model training methods in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the cross-modal model training methods in the above embodiments, or a cross-modal model-based retrieval method.

[0196] Alternatively, this application embodiment can provide a computer program product for implementation. When the instructions in the computer program product are executed by the processor of an electronic device, the electronic device implements any of the cross-modal model training methods or cross-modal model-based retrieval methods in the above embodiments.

[0197] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described as examples. However, the method process of this application is not limited to the specific steps described. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0198] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0199] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0200] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0201] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A training method for a cross-modal model, characterized in that, The method includes: Obtain a first training dataset, which includes multiple first sample data, the first sample data including: a first image, and a first descriptive text for describing the first image; Determine the text similarity between each of the first sample data and each preset concept term, wherein the text similarity is used to characterize the similarity between the first descriptive text in the first sample data and the corresponding preset concept term; Based on the text similarity between the plurality of first sample data and the plurality of preset concept terms, sample data in the first training dataset is filtered to obtain a second training dataset. The second training dataset is obtained as follows: the mastery level value of each preset concept term is determined according to the text similarity; the sample data in the first training dataset is filtered according to the mastery level value of each preset concept term to obtain the second training dataset; the mastery level value of each preset concept term is determined as follows: the loss function value corresponding to each first sample data is multiplied by the first text similarity to obtain a third value corresponding to each first sample data; the third values ​​corresponding to each first sample data in the first sample dataset are weighted and summed to obtain the mastery level value of the target concept term, wherein the first text similarity is the text similarity between each first sample data and the target concept term, and the target concept term is one of the plurality of preset concept terms; The initial cross-modal model is trained using the second training dataset to obtain a cross-modal model, which is used to retrieve the image description text corresponding to the image.

2. The training method for the cross-modal model according to claim 1, characterized in that, Determining the text similarity between each of the first sample data and each preset concept term includes: The preset concept terms are encoded to obtain a first vector corresponding to the preset concept terms; and the first descriptive text in the first sample data is encoded to obtain a second vector corresponding to the first descriptive text. Based on the second vector and the first vector, the text similarity between the first sample data and the preset concept term is determined.

3. The training method for the cross-modal model according to claim 2, characterized in that, The step of filtering sample data from the first training dataset based on the text similarity between the plurality of first sample data and a plurality of preset concept terms to obtain a second training dataset includes: Obtain the loss function value corresponding to each of the first sample data, and the loss function value is associated with the initial cross-modal model; Based on the loss function value corresponding to each first sample data in multiple first sample datasets and the text similarity between each first sample data and each preset concept term in multiple preset concept terms, the mastery level value of each preset concept term is determined. The mastery level value is used to characterize the mastery level of the concept term by the initial cross-modal model. Based on the mastery level values ​​of each of the preset concept terms, at least one first concept term and at least one second concept term are determined among the plurality of preset concept terms, wherein the mastery level value of each first concept term is greater than the mastery level value of each second concept term. The first training dataset is filtered according to the at least one first conceptual term and the at least one second conceptual term to obtain the second training dataset. The second training dataset includes multiple second sample data, which include: a second image and a second descriptive text for describing the second image.

4. The training method for the cross-modal model according to claim 3, characterized in that, The step of obtaining the loss function value corresponding to each of the first sample data includes: The sample data in the first training dataset is divided into multiple data subsets; For each data subset, the following processing is performed: each of the first sample data in the data subset is input into the initial cross-modal model to obtain a first loss function value. The first loss function value is determined based on a second loss function value and a third loss function value. The second loss function value is the loss value of the first image relative to the first descriptive text, and the third loss function value is the loss function value of the first descriptive text relative to the first image. The first loss function value is determined as the loss function value corresponding to each of the first sample data in the data subset.

5. The training method for the cross-modal model according to claim 3, characterized in that, The step of filtering the first training dataset according to the at least one first conceptual term and the at least one second conceptual term to obtain the second training dataset includes: The following processing is performed on each of the first sample data in the first training dataset: Based on the text similarity between the first sample data and each of the first concept terms, the average text similarity of the first sample data relative to the multiple first concept terms is calculated to obtain a first value; Based on the text similarity between the first sample data and each of the second concept terms, the average text similarity of the first sample data relative to the multiple second concept terms is calculated to obtain the second value; If the second value is greater than the first value, then the first sample data is used as the second sample data in the second training dataset.

6. The training method for the cross-modal model according to claim 3, characterized in that, The step of determining the mastery level value of each preset concept term based on the loss function value corresponding to each first sample data in multiple first sample datasets and the text similarity between each first sample data and each preset concept term in multiple preset concept terms includes: One of the multiple preset concept terms is taken as the target concept term; For each first sample data in the first sample dataset, the following processing is performed: the loss function value corresponding to the first sample data is multiplied by the first text similarity to obtain the third value corresponding to the first sample data, wherein the first text similarity is the text similarity between the first sample data and the target concept term; The mastery level value of the target concept term is obtained by weighted summing of the third values ​​corresponding to each first sample data in the first sample dataset; If a third concept term exists among the plurality of preset concept terms, then the third concept term is used as the target concept term, and the process jumps to the step of multiplying the loss function value corresponding to the first sample data with the first text similarity to obtain the third value corresponding to the first sample data, wherein the third concept term is a concept term that has not participated in the calculation.

7. A retrieval method based on a cross-modal model, characterized in that, The method includes: Retrieve the image to be searched and multiple reference descriptive texts; The image and multiple reference description texts are input into a cross-modal model for retrieval to obtain the target reference description text corresponding to the image. The cross-modal model is obtained based on the training method of any one of the cross-modal models in claims 1 to 6.

8. A training device for a cross-modal model, characterized in that, The device includes: The first acquisition module is used to acquire a first training dataset, which includes multiple first sample data, and the first sample data includes: a first image and a first descriptive text for describing the first image; The determination module is used to determine the text similarity between each of the first sample data and each preset concept term, wherein the text similarity is used to characterize the similarity between the first descriptive text in the first sample data and the corresponding preset concept term; A filtering module is used to filter sample data in the first training dataset based on the text similarity between the plurality of first sample data and the plurality of preset concept terms, to obtain a second training dataset. The second training dataset is obtained as follows: the mastery level value of each preset concept term is determined based on the text similarity; the sample data in the first training dataset is filtered based on the mastery level value of each preset concept term to obtain the second training dataset; the mastery level value of each preset concept term is determined as follows: the loss function value corresponding to each first sample data is multiplied by the first text similarity to obtain a third value corresponding to each first sample data; the third values ​​corresponding to each first sample data in the first sample dataset are weighted and summed to obtain the mastery level value of the target concept term, wherein the first text similarity is the text similarity between each first sample data and the target concept term, and the target concept term is one of the plurality of preset concept terms; The training module is used to train the initial cross-modal model using the second training dataset to obtain a cross-modal model, which is used to retrieve the image based on the image to obtain the corresponding descriptive text.

9. A retrieval device based on a cross-modal model, characterized in that, The device includes: The second acquisition module is used to acquire the image to be retrieved and multiple reference description texts; The retrieval module is used to input the image and multiple reference description texts into a cross-modal model for retrieval, and obtain the target reference description text corresponding to the image. The cross-modal model is obtained based on the training method of any one of the cross-modal models in claims 1 to 6.

10. An electronic device, characterized in that, include: Processor and memory storing computer program instructions; When the processor executes the computer program instructions, it implements the cross-modal model training method as described in any one of claims 1-6 or the cross-modal model-based retrieval method as described in claim 7.

Images