Image-text cross-retrieval methods, devices, equipment, and storage media

Abstract

This invention provides a method, apparatus, device, and storage medium for image-text cross-referencing. The method includes: acquiring a target image and target text; acquiring a target deep learning model, the target deep learning model including an image modality and a text modality, wherein the image vector corresponding to the image modality and the text vector corresponding to the text modality have the same dimension; performing feature extraction operations on the target image and target text to obtain a first feature vector and a second feature vector; and analyzing the first and second feature vectors through the target deep learning model to obtain retrieval results. This application retrieves target images and target text using a target deep learning model. Since the target deep learning model includes both image and text modalities, and their corresponding vector dimensions are the same, it can ensure the consistency of image and text semantics, thereby improving the accuracy of retrieval results. For the retrieval of medical images and medical text in the medical field, it can effectively improve the accuracy of retrieval.

Description

Technical Field

[0001] This application relates to the field of intelligent medical technology, and in particular to a method, device, computer equipment, and computer-readable storage medium for mutual image and text retrieval. Background Technology

[0002] With the rapid development of medical imaging technology, hospitals generate a massive amount of medical image data and medical texts every day. How to effectively and quickly retrieve this vast amount of medical image data and medical texts has become an urgent problem to be solved. Searching within medical images and medical texts can assist diagnosing doctors and improve their work efficiency to a certain extent.

[0003] In existing technologies, retrieval of medical image data or medical text is typically achieved through image-text interoperability retrieval, i.e., retrieving medical images from medical text or vice versa. However, due to the rich semantic information in medical images, medical text usually cannot provide a complete description of the images, resulting in poor accuracy of retrieval results. Furthermore, retrieval of medical text often requires manual annotation of medical images, leading to low retrieval efficiency. Summary of the Invention

[0004] This application provides a method, apparatus, computer device, and computer-readable storage medium for image-text cross-referencing, aiming to retrieve target images and text using a target deep learning model, thereby improving the accuracy and efficiency of retrieval results. It is particularly effective in improving the accuracy and efficiency of retrieval for medical images and text in the medical field.

[0005] To achieve the above objectives, this application provides a method for image-text cross-retrieval, the method comprising:

[0006] Acquire a target image and target text, and acquire a target deep learning model, wherein the target deep learning model includes an image modality and a text modality, and the image vector corresponding to the image modality and the text vector corresponding to the text modality have the same dimension;

[0007] Perform feature extraction operations on the target image and the target text to obtain the corresponding first feature vector and second feature vector;

[0008] The target deep learning model is used to analyze the first feature vector and the second feature vector respectively to obtain the corresponding retrieval results.

[0009] To achieve the above objectives, this application also provides an image-text interoperability retrieval device, the image-text interoperability retrieval device comprising:

[0010] The acquisition module is used to acquire a target image and a target text, and to acquire a target deep learning model, wherein the target deep learning model includes an image modality and a text modality, and the image vector corresponding to the image modality and the text vector corresponding to the text modality have the same dimension;

[0011] A feature extraction module is used to perform feature extraction operations on the target image and the target text to obtain corresponding first feature vectors and second feature vectors.

[0012] The retrieval module is used to analyze the first feature vector and the second feature vector respectively through the target deep learning model to obtain the corresponding retrieval results.

[0013] In addition, to achieve the above objectives, this application also provides a computer device, which includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, implement the steps of the image-text mutual retrieval method provided in any of the embodiments of this application.

[0014] In addition, to achieve the above objectives, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to implement the steps of the image-text mutual retrieval method provided in any of the embodiments of this application.

[0015] The image-text interoperability retrieval method, device, computer equipment, and computer-readable storage medium disclosed in this application can analyze target images and text using a target deep learning model to obtain corresponding retrieval results. Since the target deep learning model includes both image and text modalities, and their corresponding image and text vectors have the same dimension, the consistency of image and text semantics can be guaranteed, thereby improving the accuracy of retrieval results. Especially for the retrieval of medical images and medical text in the medical field, it can effectively improve the accuracy and efficiency of retrieval results. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of a scenario based on a text-image mutual retrieval method provided in an embodiment of this application;

[0018] Figure 2 This is a schematic diagram illustrating the steps of a text-image mutual retrieval method provided in an embodiment of this application;

[0019] Figure 3 This is a schematic diagram of the process for obtaining a target deep learning model according to an embodiment of this application;

[0020] Figure 4 This is a schematic diagram of the process for determining the target loss value provided in an embodiment of this application;

[0021] Figure 5 This is a schematic diagram of the process for determining the target loss value provided in another embodiment of this application;

[0022] Figure 6 This is a schematic block diagram of a text-image retrieval device provided in an embodiment of this application;

[0023] Figure 7 This is a schematic block diagram of a computer device provided in one embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily need to be performed in the described order. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation. Furthermore, although functional modules are divided in the device diagram, in some cases, a different module division may be used.

[0026] The term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items, as well as all possible combinations, and includes such combinations.

[0027] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0028] like Figure 1 As shown, the image-text mutual retrieval method provided in this application embodiment can be applied to, for example... Figure 1The application environment shown includes a terminal device 110 and a server 120. The terminal device 110 can communicate with the server 120 via a network. Specifically, the server 120 acquires a target image and target text, and acquires a target deep learning model. The target deep learning model includes an image modality and a text modality, where the image vector corresponding to the image modality and the text vector corresponding to the text modality have the same dimension. Further, the server 120 performs feature extraction operations on the target image and the target text to obtain corresponding first feature vectors and second feature vectors. The server then analyzes the first feature vector and the second feature vector using the target deep learning model to obtain corresponding search results, which are then sent to the terminal device 110. The server 120 can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal device 110 may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. Terminal device 110 and server 120 can be directly or indirectly connected via wired or wireless communication, and this application does not impose any restrictions.

[0029] Please see Figure 2 , Figure 2 This is a schematic diagram illustrating the steps of a text-image cross-retrieval method provided in an embodiment of this application. This text-image cross-retrieval method can be applied to computer devices, thereby enabling the acquisition of relatively accurate retrieval results through a target deep learning model.

[0030] like Figure 2 As shown, the image-text mutual retrieval method includes steps S11 to S13.

[0031] Step S11: Obtain the target image and target text, and obtain the target deep learning model.

[0032] In this context, the target image and target text refer to the image corresponding to the text to be retrieved and the text corresponding to the image to be retrieved. Furthermore, the target image can be a medical image from the medical field, and the target text can be medical text from the medical field; alternatively, the target image and target text can be images and text from other fields, etc., and this application does not impose any limitations on these aspects.

[0033] Furthermore, the target deep learning model includes an image modality and a text modality, with the image vector corresponding to the image modality having the same dimension as the text vector corresponding to the text modality.

[0034] Understandably, since the image vectors corresponding to the image modality and the text vectors corresponding to the text modality in the target deep learning model have the same dimension, the representation of images and text can be kept consistent, thereby improving the accuracy of retrieval results.

[0035] In this embodiment, medical text in the medical field can be acquired for retrieving corresponding medical images in the medical field, and medical images in the medical field can be acquired for retrieving corresponding medical text in the medical field. Furthermore, a target deep learning model can be acquired to ensure consistency between the representation of images and text, thereby improving the accuracy of the retrieval results for medical text or medical images in the medical field.

[0036] Step S12: Perform feature extraction on the target image and target text to obtain the corresponding first feature vector and second feature vector.

[0037] Wherein, the first feature vector is the feature vector obtained after feature extraction of the target image; the second feature vector is the feature vector obtained after feature extraction of the target text. Furthermore, the target text can be medical text in the medical field; the target image can be medical image in the medical field, and this application will use this as an example for illustration.

[0038] It should be noted that feature extraction refers to extracting feature vectors from medical text or medical images in the medical field to represent textual information. This application does not limit the methods for feature extraction from medical text or medical images in the medical field. For example, feature extraction operations on target text can be performed using methods such as TF-IDF, word frequency methods, text frequency methods, and mutual information; feature extraction operations on medical images in the medical field can be performed using algorithms such as LBP (Local Binary Patterns), HOG (Histogram of Oriented Gradient) feature extraction algorithm, and SIFT (Scale-invariant feature transform).

[0039] This application uses the method of mutual information for target text feature extraction and the SIFT operator for target image feature extraction as an example for illustration.

[0040] Mutual information, a concept in information theory, represents the relationship between information items and is a numerical representation of the statistical correlation between two random variables. Feature extraction using mutual information theory is based on the assumption that terms that appear frequently in a specific category but less frequently in other categories have a high mutual information with that category. The mutual information between a feature and its category reflects the degree of correlation between the feature and the category, and is a widely used standard for building statistical models of word association. Therefore, for each topic, the greater the mutual information of a feature, the greater its co-occurrence probability with that topic. Thus, when using mutual information as an evaluation criterion for feature extraction, the features with the highest mutual information should be selected.

[0041] The SIFT operator can obtain features from an image by finding feature points and their related size and orientation descriptors, and then perform image feature point matching. It has the characteristics of scale invariance and rotation invariance. Its main steps are as follows: scale space generation, detection of scale space extrema, removal of bad feature points, assignment of orientation parameters for each key point, generation of key point descriptors, and image matching based on SIFT.

[0042] In this embodiment, feature extraction can be performed on the target text using mutual information and on the target image using the SIFT operator to obtain the corresponding feature vector.

[0043] Step S13: Analyze the first feature vector and the second feature vector using the target deep learning model to obtain the corresponding retrieval results.

[0044] Specifically, the first feature vector and the second feature vector can be input into the target deep learning model to obtain the retrieval results corresponding to medical texts in the medical field, as well as the retrieval results corresponding to medical images in the medical field.

[0045] Alternatively, a similarity calculation method can be used to calculate the similarity between medical text and medical images in the medical field and each corresponding search result, and the search result with the highest similarity can be used as the final search result.

[0046] In this embodiment, the first feature vector and the second feature vector can be analyzed using a target deep learning model to obtain the corresponding retrieval results. Furthermore, the similarity between the target text and the target image and each corresponding retrieval result can be calculated, allowing the retrieval result with the highest similarity to be selected as the final retrieval result.

[0047] The image-text cross-referencing method disclosed in this application can analyze target images and target text using a target deep learning model to obtain corresponding retrieval results. Since the target deep learning model includes both image and text modalities, and their corresponding image and text vectors have the same dimension, the consistency of image and text semantics can be guaranteed, thereby improving the accuracy of the retrieval results. Especially for the retrieval of medical images and medical text in the medical field, it can effectively improve the accuracy and efficiency of the retrieval results.

[0048] Please continue reading. Figure 3 , Figure 3 This is a schematic diagram illustrating the process of obtaining a target deep learning model according to an embodiment of this application. Figure 3 As shown, the target deep learning model can be obtained through steps S111 to S114.

[0049] Step S111: Obtain the target sample and the initial deep learning model.

[0050] The target samples include image samples and text samples corresponding to the image samples.

[0051] It should be noted that this application does not limit the image sample and its corresponding text sample. For example, they can be medical images and the corresponding medical text. This application will use this as an example for illustration.

[0052] Step S112: Construct the corresponding negative sample based on the target sample.

[0053] Negative samples include negative samples corresponding to medical images and negative samples corresponding to medical texts.

[0054] Alternatively, negative samples can be constructed by using a BERT model to identify the text sample with the highest similarity but belonging to a different category, and using that text sample as the negative sample corresponding to the text sample. Similarly, by using a YOLO model to identify the image sample with the highest similarity but belonging to a different category, and using that image as the negative sample corresponding to the image sample.

[0055] In this embodiment, target samples and an initial deep learning model can be obtained, and corresponding negative samples can be constructed based on the target samples. Thus, the initial deep learning model can be trained using the target samples and negative samples.

[0056] Step S113: Determine the target loss value of the initial deep learning model based on the target sample and the negative sample.

[0057] Specifically, the loss function of the initial deep learning model can be determined first, and feature extraction operations can be performed on the target sample and its corresponding negative sample to obtain the corresponding feature vector. The feature vector is then substituted into the loss function to obtain the target loss value in the initial deep learning model.

[0058] It should be noted that this application does not limit the type of the loss function mentioned above. For example, the loss function may include at least one of regression loss function and mean square loss function.

[0059] Please continue reading. Figure 4 , Figure 4 This is a schematic diagram of the process for determining the target loss value provided in an embodiment of this application. For example... Figure 4 As shown, the target loss value includes the first loss value, which can be determined through steps S1131 to S1134.

[0060] Step S1131: Perform feature extraction operations on the image samples and text samples respectively to obtain the corresponding third feature vector and fourth feature vector.

[0061] Understandably, the third feature vector is the feature vector obtained after performing feature extraction on the image sample; the fourth feature vector is the feature vector obtained after performing feature extraction on the text sample.

[0062] Optionally, feature extraction operations can be performed on text samples using mutual information, and feature extraction operations can be performed on image samples using the SIFT operator, thereby obtaining the corresponding feature vectors. This application does not limit this to any particular method.

[0063] Optionally, data augmentation processing can be performed on the image samples and text samples to obtain first augmented data and second augmented data, respectively. Then, feature extraction operations can be performed on the first augmented data and second augmented data to obtain the corresponding third feature vector and fourth feature vector.

[0064] The first enhanced data is the data obtained after enhancing image sample data; the second enhanced data is the data obtained after enhancing text sample data.

[0065] It's important to note that data augmentation, also known as data enlargement, essentially involves making limited data generate value equivalent to more data without substantially increasing the amount of data. For example, image samples can be smoothed, sharpened, eroded, and dilated, while text samples can undergo stop word removal and non-Chinese character removal. This reduces irrelevant features in image and text samples.

[0066] Optionally, based on the above embodiments, feature extraction operations are performed on image samples and text samples to obtain corresponding third and fourth feature vectors, including: performing feature extraction operations on first enhanced data and second enhanced data to obtain corresponding third and fourth feature vectors.

[0067] Understandably, after obtaining the first and second augmented data, feature extraction operations can be performed based on the first and second augmented data to obtain the third and fourth feature vectors. In this way, feature extraction operations can be performed on image and text samples to extract relevant features, thereby fundamentally improving the overall performance of the model.

[0068] In this embodiment, feature extraction operations can be performed on image samples and text samples to obtain corresponding third and fourth feature vectors, thereby enabling the training of the initial deep learning model based on the third and fourth feature vectors. Furthermore, data augmentation operations can be performed on the image and text samples to reduce irrelevant features, thus preventing the extraction of irrelevant features after feature extraction and fundamentally improving the overall performance of the model.

[0069] Step S1132: Perform semantic alignment on the third and fourth feature vectors to obtain the fifth and sixth feature vectors.

[0070] Among them, the fifth feature vector is the feature vector obtained by semantic alignment of the third feature vector, and the sixth feature vector is the feature vector obtained by semantic alignment of the fourth feature vector.

[0071] Understandably, semantic alignment ensures the consistency of the semantics of the fifth and sixth feature vectors, reducing interference from other irrelevant elements and thus improving the accuracy of model retrieval.

[0072] Optionally, the semantic alignment operation includes vector segmentation. The above-mentioned feature extraction operation on the image samples to obtain the corresponding third feature vector includes: determining several target combinations in the image samples; performing feature extraction operation on each target combination to obtain each corresponding third feature vector.

[0073] Furthermore, each third and fourth eigenvector can be vector-segmented to obtain several fifth and sixth eigenvectors.

[0074] Specifically, each target in the sample can be identified and then combined to obtain several target combinations. Further, feature extraction can be performed on each target combination to obtain a corresponding third feature vector. Each third and fourth feature vector is then vector-segmented to obtain several fifth and sixth feature vectors, where the dimensions of the fifth and sixth feature vectors are the same.

[0075] In this embodiment, the third and fourth feature vectors can be semantically aligned to obtain the fifth and sixth feature vectors. Specifically, each third and fourth feature vector can be segmented using vector segmentation to obtain the fifth and sixth feature vectors with the same dimensions, thus ensuring semantic consistency.

[0076] Step S1133: Determine the first similarity value based on the fifth and sixth feature vectors, determine the second similarity value based on the negative samples corresponding to the image samples and text samples, and determine the third similarity value based on the negative samples corresponding to the text samples and image samples.

[0077] The first similarity value is the similarity between the image sample and the text sample; the second similarity value is the similarity between the negative sample corresponding to the image sample and the text sample; and the third similarity value is the similarity between the negative sample corresponding to the text sample and the text sample.

[0078] Specifically, the first similarity value can be determined using the following formula:

[0079]

[0080] Where S(I,T) is the first similarity value; I is the image sample; T is the text sample; q is the correlation matrix between image samples and text samples; q is the number of fifth eigenvectors; p is the number of sixth eigenvectors.

[0081] The correlation matrix A can be obtained using the following formula:

[0082]

[0083] in, The correlation matrix between image samples and text samples; v bi The third eigenvector; t bj This is the fourth eigenvector.

[0084] Similarly, the second similarity value can be determined based on the above formula. and the third similarity value

[0085] Step S1134: Determine the first loss value based on the first similarity value, the second similarity value, and the third similarity value.

[0086] Specifically, the first loss value can be determined using the following formula:

[0087]

[0088] Where α is the boundary parameter, manually set to 0.3; D is the number of samples; S(I,T) is the first similarity value; This is the second similarity value; This is the third similarity value.

[0089] In the embodiments of this application, a first similarity value, a second similarity value, and a third similarity value can be determined respectively, and then a first loss value can be determined based on the first similarity value, the second similarity value, and the third similarity value.

[0090] Alternatively, please continue reading Figure 5 , Figure 5 This is a schematic diagram of the process for determining the target loss value provided in another embodiment of this application. For example... Figure 5 As shown, the target loss value also includes a second loss value. Therefore, steps S1135 to S1138 can also be used to determine the target loss value of the initial deep learning model based on the target sample and the negative sample.

[0091] Step S1135: Based on the third feature vector corresponding to the first target combination and the third feature vector corresponding to the second target combination, determine the cross-correlation matrix of the first target combination and the second target combination.

[0092] The cross-correlation matrix includes the diagonal lines and all other positions except the diagonal lines.

[0093] Since the image sample contains several target combinations, any two target combinations in the image sample can be determined, that is, the first target combination and the second target combination can be determined. Furthermore, after determining the first target combination and the second target combination, their corresponding feature vectors can be obtained, and the cross-correlation matrix of the first target combination and the second target combination can be determined based on their corresponding feature vectors.

[0094] It should be noted that the cross-correlation matrix can be calculated by calculating the cross-correlation coefficient and cross-covariance between the i-th and j-th eigenvectors (x,y), and this application does not limit this calculation.

[0095] Step S1136: Determine the first dimension of the third feature vector corresponding to the first target combination and the second dimension of the third feature vector corresponding to the second target combination.

[0096] In this matrix, the rows of the cross-correlation matrix represent the first dimension of the third eigenvector, and the columns of the cross-correlation matrix represent the second dimension of the third eigenvector.

[0097] Step S1137: Determine the position values ​​other than the diagonal line based on the first dimension and the second dimension, and determine the diagonal line value based on the first dimension.

[0098] Specifically, the position values ​​other than the diagonal can be determined using the following formula:

[0099]

[0100] Among them, C iJ , where is the position value excluding the diagonal; b is a sample in a batch during training; i is the first dimension; j is the second dimension; A and B represent the first target combination and the second target combination, respectively.

[0101] Furthermore, the diagonal value can be determined using the following formula:

[0102]

[0103] Among them, C ii is the diagonal value; b is a sample in a batch during training; i is the first dimension; A and B represent the first target combination and the second target combination, respectively.

[0104] Step S1138: Determine the second loss value based on the position values ​​other than the diagonal and the diagonal value.

[0105] Specifically, after obtaining the position values ​​other than the diagonal line and the diagonal line value, the second loss value can be determined using the following formula:

[0106]

[0107] Among them, L reg The second loss value; C iJ For positions other than the diagonal; C ii λ is the diagonal value; λ is a hyperparameter, set to 0.005.

[0108] Therefore, the second loss value can be determined using the formula described above.

[0109] Optionally, based on the above embodiments, determining the target loss value of the initial deep learning model based on the target sample and the negative sample includes: summing the first loss value and the second loss value to obtain the target loss value of the initial deep learning model.

[0110] Specifically, the target loss value of the initial deep learning model can be obtained using the following formula:

[0111] L = m +L reg

[0112] Where L is the target loss value; L m This is the first loss value; L reg This is the second loss value.

[0113] In this embodiment, a target loss value can be determined based on the target sample and negative samples for training the initial deep learning model. The target loss value includes at least one of a first loss value and a second loss value. Furthermore, after obtaining the first and second loss values, they can be summed to obtain the target loss value. This aligns the image modality and the text modality, making the target loss value more accurate, thereby improving the accuracy of the target deep learning model and reducing the cost of later model maintenance.

[0114] Step S114: Train the initial deep learning model using the target loss value to obtain the target deep learning model.

[0115] After obtaining the target loss value, the initial deep learning model can be trained using the target loss value to obtain the target deep learning model.

[0116] Specifically, training can be performed by introducing the target loss value into each intermediate layer of the initial deep learning model, so that each intermediate layer has progressively complex feature representation capabilities after training, and the feature representation capabilities between each layer are consistent. This accelerates training convergence, improves the feature transfer capability of each intermediate layer, achieves an overall improvement in model performance, and thus obtains the target deep learning model.

[0117] In this embodiment, the target deep learning model is derived based on a target loss function, which utilizes the overlapping effects of image and text modalities. Therefore, the target deep learning model exhibits good training performance and model capabilities. Especially for processing electronic medical records and clinical trial texts in the medical field, it can improve the model's efficiency and accuracy in text processing.

[0118] Please see Figure 6 , Figure 6 This is a schematic block diagram of a text-image retrieval device provided in an embodiment of this application. The text-image retrieval device can be configured in a server to perform the aforementioned text-image retrieval method.

[0119] like Figure 6 As shown, the image-text mutual retrieval device 200 includes: an acquisition module 201, a feature extraction module 202, and a retrieval module 203.

[0120] The acquisition module 201 is used to acquire a target image and target text, and acquire a target deep learning model, wherein the target deep learning model includes an image modality and a text modality, and the image vector corresponding to the image modality and the text vector corresponding to the text modality have the same dimension.

[0121] The feature extraction module 202 is used to perform feature extraction operations on the target image and the target text to obtain the corresponding first feature vector and second feature vector.

[0122] The retrieval module 203 is used to analyze the first feature vector and the second feature vector respectively through the target deep learning model to obtain the corresponding retrieval results.

[0123] The acquisition module 201 is further configured to acquire target samples and an initial deep learning model, wherein the target samples include image samples and text samples corresponding to the image samples; construct corresponding negative samples based on the target samples, wherein the negative samples include negative samples corresponding to the image samples and negative samples corresponding to the text samples; determine the target loss value of the initial deep learning model based on the target samples and the negative samples; and train the initial deep learning model using the target loss value to obtain the target deep learning model.

[0124] The acquisition module 201 is further configured to perform feature extraction operations on the image sample and the text sample respectively to obtain corresponding third feature vectors and fourth feature vectors; perform semantic alignment operations on the third feature vector and the fourth feature vector to obtain fifth feature vector and sixth feature vector; determine a first similarity value based on the fifth feature vector and the sixth feature vector, and determine a second similarity value based on the negative samples corresponding to the image sample and the text sample, and determine a third similarity value based on the negative samples corresponding to the text sample and the image sample; and determine a first loss value based on the first similarity value, the second similarity value, and the third similarity value.

[0125] The acquisition module 201 is further configured to determine several target combinations in the image sample; perform feature extraction operation on each target combination to obtain each corresponding third feature vector; perform vector segmentation on each third feature vector and the fourth feature vector to obtain several fifth feature vectors and the sixth feature vector, wherein the dimension of the fifth feature vector is the same as the dimension of the sixth feature vector.

[0126] The acquisition module 201 is further configured to: determine the cross-correlation matrix of the first target combination and the second target combination based on the third feature vector corresponding to the first target combination and the third feature vector corresponding to the second target combination, wherein the cross-correlation matrix includes the diagonal line and other positions excluding the diagonal line; determine the first dimension of the third feature vector corresponding to the first target combination and the second dimension of the third feature vector corresponding to the second target combination; determine the values ​​of the other positions excluding the diagonal line based on the first dimension and the second dimension, and determine the diagonal line value based on the first dimension; and determine the second loss value based on the other positions excluding the diagonal line and the diagonal line value.

[0127] The acquisition module 201 is further configured to sum the first loss value and the second loss value to obtain the target loss value of the initial deep learning model.

[0128] The acquisition module 201 is further configured to perform data augmentation processing on the image samples and the text samples to obtain first augmented data and second augmented data respectively; and to perform feature extraction operation on the first augmented data and the second augmented data to obtain the corresponding third feature vector and the fourth feature vector.

[0129] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described apparatus and its modules and units can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0130] The methods and apparatus of this application can be used in a wide variety of general-purpose or special-purpose computing system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer terminal devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0131] For example, the above-described method and apparatus can be implemented as a computer program, which can be used in, for example... Figure 7 It runs on the computer device shown.

[0132] Please see Figure 7 , Figure 7 This is a schematic diagram of a computer device provided in an embodiment of this application. The computer device may be a server.

[0133] like Figure 7As shown, the computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include volatile storage media, non-volatile storage media, and internal memory.

[0134] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform steps of any image-text retrieval method.

[0135] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0136] Internal memory provides an environment for the execution of computer programs stored in non-volatile storage media. When executed by a processor, the computer program enables the processor to perform the steps of any image-text retrieval method.

[0137] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that the structure of this computer device is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0138] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0139] In some embodiments, the processor is used to run a computer program stored in a memory to perform the following steps: acquiring a target image and target text; acquiring a target deep learning model, wherein the target deep learning model includes an image modality and a text modality, and the image vector corresponding to the image modality and the text vector corresponding to the text modality have the same dimension; performing feature extraction operations on the target image and the target text to obtain corresponding first feature vectors and second feature vectors; and analyzing the first feature vector and the second feature vector through the target deep learning model to obtain corresponding retrieval results.

[0140] In some embodiments, the processor is further configured to acquire target samples and an initial deep learning model, wherein the target samples include image samples and text samples corresponding to the image samples; construct corresponding negative samples based on the target samples, wherein the negative samples include negative samples corresponding to the image samples and negative samples corresponding to the text samples; determine a target loss value for the initial deep learning model based on the target samples and the negative samples; and train the initial deep learning model using the target loss value to obtain the target deep learning model.

[0141] In some embodiments, the processor is further configured to perform feature extraction operations on the image sample and the text sample respectively to obtain corresponding third feature vectors and fourth feature vectors; perform semantic alignment operations on the third feature vector and the fourth feature vector to obtain fifth feature vector and sixth feature vector; determine a first similarity value based on the fifth feature vector and the sixth feature vector, and determine a second similarity value based on the negative samples corresponding to the image sample and the text sample, and determine a third similarity value based on the negative samples corresponding to the text sample and the image sample; and determine a first loss value based on the first similarity value, the second similarity value, and the third similarity value.

[0142] In some embodiments, the processor is further configured to determine a plurality of target combinations in the image samples; perform feature extraction operation on each target combination to obtain a corresponding third feature vector; and perform vector segmentation on each third feature vector and the fourth feature vector to obtain a plurality of fifth feature vectors and the sixth feature vector, wherein the dimension of the fifth feature vector is the same as the dimension of the sixth feature vector.

[0143] In some embodiments, the processor is further configured to: determine a cross-correlation matrix of the first target combination and the second target combination based on a third feature vector corresponding to a first target combination and a third feature vector corresponding to a second target combination, wherein the cross-correlation matrix includes a diagonal line and other positions excluding the diagonal line; determine a first dimension of the third feature vector corresponding to the first target combination and a second dimension of the third feature vector corresponding to the second target combination; determine the values ​​of the other positions excluding the diagonal line based on the first dimension and the second dimension, and determine the diagonal line value based on the first dimension; and determine a second loss value based on the other position values ​​excluding the diagonal line and the diagonal line value.

[0144] In some implementations, the processor is further configured to sum the first loss value and the second loss value to obtain the target loss value of the initial deep learning model.

[0145] In some embodiments, the processor is further configured to perform data augmentation processing on the image samples and the text samples to obtain first augmented data and second augmented data, respectively; and to perform feature extraction operations on the first augmented data and the second augmented data to obtain the corresponding third feature vector and the fourth feature vector.

[0146] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed, implement the steps of any of the image-text mutual retrieval methods provided in this application.

[0147] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.

[0148] Furthermore, the computer-readable storage medium may primarily include a program storage area and a data storage area, wherein the program storage area may store the operating system, at least one application program required for a function, etc.

[0149] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for mutual retrieval of text and images, characterized in that, The method includes: Acquire a target image and target text, and acquire a target deep learning model, wherein the target deep learning model includes an image modality and a text modality, and the image vector corresponding to the image modality and the text vector corresponding to the text modality have the same dimension; Perform feature extraction operations on the target image and the target text to obtain the corresponding first feature vector and second feature vector; The target deep learning model is used to analyze the first feature vector and the second feature vector respectively to obtain the corresponding retrieval results; The acquisition of the target deep learning model includes: Obtain target samples and an initial deep learning model, wherein the target samples include image samples and text samples corresponding to the image samples; Based on the target sample, a corresponding negative sample is constructed, wherein the negative sample includes the negative sample corresponding to the image sample and the negative sample corresponding to the text sample; The target loss value of the initial deep learning model is determined based on the target sample and the negative sample. The initial deep learning model is trained using the target loss value to obtain the target deep learning model; Wherein, the target loss value includes a first loss value, and determining the target loss value of the initial deep learning model based on the target sample and the negative sample includes: Feature extraction operations are performed on the image samples and the text samples respectively to obtain the corresponding third feature vector and fourth feature vector; The third and fourth feature vectors are semantically aligned to obtain the fifth and sixth feature vectors. A first similarity value is determined based on the fifth feature vector and the sixth feature vector, a second similarity value is determined based on the negative samples corresponding to the image sample and the text sample, and a third similarity value is determined based on the negative samples corresponding to the text sample and the image sample. The first loss value is determined based on the first similarity value, the second similarity value, and the third similarity value; The semantic alignment operation includes vector segmentation, performing feature extraction on the image samples to obtain the corresponding third feature vector, including: Determine several target combinations in the image samples; Perform feature extraction on each target combination to obtain the corresponding third feature vector; The target loss value further includes a second loss value, and the step of determining the target loss value of the initial deep learning model based on the target sample and the negative sample includes: Based on the third feature vector corresponding to the first target combination and the third feature vector corresponding to the second target combination, the cross-correlation matrix of the first target combination and the second target combination is determined, wherein the cross-correlation matrix includes the diagonal line and other positions except the diagonal line; Determine the first dimension of the third feature vector corresponding to the first target combination, and the second dimension of the third feature vector corresponding to the second target combination; The position values ​​other than the diagonal are determined based on the first dimension and the second dimension, and the diagonal value is determined based on the first dimension; The second loss value is determined based on the other position values ​​excluding the diagonal line and the diagonal line value.

2. The method according to claim 1, characterized in that, The step of performing semantic alignment on the third and fourth feature vectors to obtain the fifth and sixth feature vectors includes: Each of the third and fourth feature vectors is vector-segmented to obtain several fifth and sixth feature vectors, wherein the dimensions of the fifth feature vectors are the same as the dimensions of the sixth feature vectors.

3. The method according to claim 1, characterized in that, Determining the target loss value of the initial deep learning model based on the target sample and the negative sample includes: The first loss value and the second loss value are summed to obtain the target loss value of the initial deep learning model.

4. The method according to claim 1, characterized in that, Before performing feature extraction operations on the image samples and the text samples respectively to obtain the corresponding third feature vector and fourth feature vector, the following steps are included: The image samples and the text samples are subjected to data augmentation processing to obtain first augmented data and second augmented data, respectively. The step of performing feature extraction operations on the image samples and the text samples respectively to obtain the corresponding third feature vector and fourth feature vector includes: Feature extraction is performed on the first enhanced data and the second enhanced data to obtain the corresponding third feature vector and the fourth feature vector.

5. A text-image retrieval device, characterized in that, For implementing the image-text mutual retrieval method as described in any one of claims 1-4, the image-text mutual retrieval device comprises: The acquisition module is used to acquire a target image and a target text, and to acquire a target deep learning model, wherein the target deep learning model includes an image modality and a text modality, and the image vector corresponding to the image modality and the text vector corresponding to the text modality have the same dimension; A feature extraction module is used to perform feature extraction operations on the target image and the target text to obtain corresponding first feature vectors and second feature vectors. The retrieval module is used to analyze the first feature vector and the second feature vector respectively through the target deep learning model to obtain the corresponding retrieval results.

6. A computer device, characterized in that, include: A memory and a processor; wherein the memory is connected to the processor and is used to store a program, and the processor is used to implement the steps of the image-text mutual retrieval method as described in any one of claims 1-4 by running the program stored in the memory.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to implement the steps of the image-text mutual retrieval method as described in any one of claims 1-4.

Images