Cross-modal target retrieval method and device, equipment, storage medium and program product

Abstract

The invention relates to a cross-modal target retrieval method and device, equipment, a storage medium and a program product. The method comprises the steps of obtaining a to-be-detected target object; the target object is input into a cross-modal retrieval model, a retrieval result output by the cross-modal retrieval model is obtained, and the retrieval result comprises similar objects matched with the target object retrieved in a database of multiple modal types; wherein the cross-modal retrieval model is used for projecting to-be-matched objects and / or target objects in databases of multiple modal types to a shared feature space, and determining a matching result of the to-be-matched objects and the target objects according to shared features of the to-be-matched objects and / or the target objects projected in the shared feature space. By adopting the method, cross-modal object matching can be realized.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a cross-modal target retrieval method, device, equipment, storage medium and program product. Background Art

[0002] In recent years, target re-identification technology has played a huge practical role in security systems and has attracted more and more attention. The purpose of target re-identification is to match target images captured by different cameras with non-overlapping fields of view.

[0003] In the related art, facing many complex scenes, the target image captured only by a visible light camera cannot effectively provide accurate appearance information of the target, and usually different types of cameras or devices such as infrared light are required to effectively capture the target.

[0004] However, current target re-identification mainly focuses on matching target images captured from visible light cameras. Cross-modal matching cannot be achieved for different types of target objects, such as visible light images and infrared images. Summary of the invention

[0005] Based on this, it is necessary to provide a cross-modal target retrieval method, device, equipment, storage medium and program product that can achieve cross-modal object matching in response to the above technical problems.

[0006] In a first aspect, the present application provides a cross-modal target retrieval method, comprising:

[0007] Obtain the target object to be detected;

[0008] Inputting the target object into a cross-modal retrieval model, and obtaining a retrieval result output by the cross-modal retrieval model, wherein the retrieval result includes similar objects matching the target object retrieved from databases of multiple modal types;

[0009] Among them, the cross-modal retrieval model is used to project the objects to be matched and / or the target objects in the database of the multiple modal types into a shared feature space, and determine the matching results of the objects to be matched and the target objects based on the shared features projected by the objects to be matched and / or the target objects in the shared feature space.

[0010] In one of the embodiments, the target object and the object to be matched both include image data, and the cross-modal retrieval model includes an image feature processing unit and a matching unit;

[0011] The image feature processing unit is used to divide the image data into a plurality of image block data of preset sizes; project the plurality of image block data of preset sizes to the shared feature space through a linear projection layer to obtain a shared feature of each image block data; and determine a feature sequence of the image data according to the shared feature of each image block data;

[0012] The matching unit is used to determine the matching result between the object to be matched and the target object according to the feature sequences of the image data of the target object and the object to be matched.

[0013] In one of the embodiments, projecting the plurality of image block data of preset sizes to the shared feature space through a linear projection layer to obtain the shared features of each image block data includes:

[0014] Determine the mapping matrices corresponding to the linear projection layers of the target object and the object to be matched, respectively, and weight parameters of the mapping matrices;

[0015] According to the mapping matrix and the weight parameter, the plurality of image block data of preset sizes are projected into the shared feature space to obtain the shared features of each image block data.

[0016] In one of the embodiments, the target object also includes text data; the cross-modal retrieval model includes a text feature processing unit;

[0017] The text feature processing unit is used to segment and encode the text data to obtain the encoding of multiple language units in the text data; and determine the feature sequence of the text data according to the encoding of the multiple language units;

[0018] The matching unit is further used to determine a matching result between the object to be matched and the target object according to a feature sequence of the text data and a feature sequence of the image data of the object to be matched.

[0019] In one embodiment, the method further comprises:

[0020] Acquire a sample set, wherein the sample set includes a plurality of image-text pairs and a matching relationship label of each image-text pair;

[0021] The cross-modal retrieval model is retrievaled using the sample set to adjust model parameters in the cross-modal retrieval model.

[0022] In one of the embodiments, the cross-modal retrieval model includes an image-to-text matching loss function and a text-to-image matching loss function;

[0023] Among them, the image-to-text matching loss function is constructed by the relative entropy divergence of the probability of image-to-text projection and the true pairing probability of image-to-text, and the text-to-image matching loss function is determined by the projection of text onto image and the image-to-text matching loss function.

[0024] In a second aspect, the present application also provides a cross-modal target retrieval device, comprising:

[0025] An acquisition module, used for acquiring a target object to be detected;

[0026] A matching module, used to input the target object into a cross-modal retrieval model and obtain a retrieval result output by the cross-modal retrieval model, wherein the retrieval result includes similar objects matching the target object retrieved from databases of multiple modal types;

[0027] Among them, the cross-modal retrieval model is used to project the objects to be matched and / or the target objects in the database of the multiple modal types into a shared feature space, and determine the matching results of the objects to be matched and the target objects based on the shared features projected by the objects to be matched and / or the target objects in the shared feature space.

[0028] In one of the embodiments, the target object and the object to be matched both include image data, and the cross-modal retrieval model includes an image feature processing unit and a matching unit;

[0029] The image feature processing unit is used to divide the image data into a plurality of image block data of preset sizes; project the plurality of image block data of preset sizes to the shared feature space through a linear projection layer to obtain a shared feature of each image block data; and determine a feature sequence of the image data according to the shared feature of each image block data;

[0030] The matching unit is used to determine the matching result between the object to be matched and the target object according to the feature sequences of the image data of the target object and the object to be matched.

[0031] In one of the embodiments, the image feature processing unit is also used to determine the mapping matrices corresponding to the linear projection layers of the target object and the object to be matched, and the weight parameters of the mapping matrices; according to the mapping matrices and the weight parameters, the multiple image block data of preset sizes are projected into the shared feature space to obtain the shared features of each image block data.

[0032] In one of the embodiments, the target object also includes text data; the cross-modal retrieval model includes a text feature processing unit;

[0033] The text feature processing unit is used to segment and encode the text data to obtain the encoding of multiple language units in the text data; and determine the feature sequence of the text data according to the encoding of the multiple language units;

[0034] The matching unit is further used to determine a matching result between the object to be matched and the target object according to a feature sequence of the text data and a feature sequence of the image data of the object to be matched.

[0035] In one of the embodiments, the cross-modal target retrieval device further includes:

[0036] The training module is used to obtain a sample set, wherein the sample set includes multiple image-text pairs and matching relationship labels of each image-text pair; and use the sample set to the cross-modal retrieval model to adjust the model parameters in the cross-modal retrieval model.

[0037] In one of the embodiments, the cross-modal retrieval model includes an image-to-text matching loss function and a text-to-image matching loss function;

[0038] Among them, the image-to-text matching loss function is constructed by the relative entropy divergence of the probability of image-to-text projection and the true pairing probability of image-to-text, and the text-to-image matching loss function is determined by the projection of text onto image and the image-to-text matching loss function.

[0039] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the cross-modal target retrieval method of the first aspect when executing the computer program.

[0040] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the cross-modal target retrieval method of the first aspect described above is implemented.

[0041] In a fifth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the cross-modal target retrieval method of the first aspect mentioned above.

[0042] The above-mentioned cross-modal target retrieval method, device, equipment, storage medium and program product first obtain the target object to be detected. Subsequently, the target object is input into the cross-modal retrieval model, and the retrieval result output by the cross-modal retrieval model is obtained, and the retrieval result includes similar objects that match the target object retrieved from databases of multiple modal types. Among them, the cross-modal retrieval model is used to project the object to be matched and / or the target object in the database of the multiple modal types into a shared feature space, and determine the matching result of the object to be matched and the target object based on the shared features projected by the object to be matched and / or the target object in the shared feature space. Since the cross-modal retrieval model can project objects of multiple modalities into a shared feature space, and determine the match through the shared features projected in the shared feature space, it is possible to achieve matching of cross-modal objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0044] FIG1 is an application environment diagram of a cross-modal target retrieval method provided by an embodiment of the present application;

[0045] Figure 2 A schematic diagram of a flow chart of a cross-modal target retrieval method provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of a cross-modal search provided in an embodiment of the present application;

[0047] Figure 4 A schematic diagram of another cross-modal retrieval provided in an embodiment of the present application;

[0048] Figure 5 A schematic diagram of another cross-modal search provided in an embodiment of the present application;

[0049] Figure 6 A schematic diagram of the structure of a cross-modal retrieval model provided in an embodiment of the present application;

[0050] Figure 7 A schematic diagram of a flow chart of another cross-modal target retrieval method provided in an embodiment of the present application;

[0051] Figure 8 A structural block diagram of a cross-modal target retrieval device provided in an embodiment of the present application;

[0052] FIG. 9 is an internal structure diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] First, the related technology will be described below.

[0055] In recent years, target re-identification technology has played a huge practical role in security systems and has received increasing attention. The purpose of target re-identification is to match target images captured by different cameras with non-overlapping fields of view. In related technologies, target re-identification technology still has the following shortcomings.

[0056] First, in many complex scenes, the target image captured only by a visible light camera cannot effectively provide accurate appearance information of the target. Usually, different types of cameras or devices such as infrared light are needed to effectively capture the target. However, current target re-identification mainly focuses on matching target images captured from visible light cameras, and cross-modal matching cannot be achieved for different types of target objects such as visible light images and infrared images.

[0057] For example, when the light is weak at night, an infrared camera is used to effectively collect pedestrians, vehicles and other targets, and when the light is sufficient during the day, a visible light camera is used to effectively collect pedestrians, vehicles and other targets. Since both infrared light images and visible light images are collected, when matching targets, it is necessary to complete the matching and recognition of targets across visible light and infrared light.

[0058] Secondly, target re-identification in related technologies often tends to ignore appearance changes caused by subtle changes in the target's attributes, which can easily lead to mismatching problems, further reducing the success rate of cross-modal matching. For example, because the target disguises the face and facial information cannot be collected, in order to match the target, the witness can provide different text descriptions of the target. For example, a witness may provide "blue scarf and tattoo on the left arm", while another witness may provide "wearing sunglasses and a green jacket". Current target re-identification cannot complete the matching of targets between different modalities based on the text description information provided by the witness. Accordingly, current target re-identification cannot align the text description information with images of different modalities based on the introduction of text description information, and achieve alignment between images of different modalities.

[0059] To solve the above problems, the embodiments of the present application provide a cross-modal target retrieval method, apparatus, device, storage medium and program product, which project objects of multiple modalities into a shared feature space through a cross-modal retrieval model, and determine matches through shared features projected in the shared feature space, thereby enabling matching of cross-modal objects.

[0060] The following describes the application environment of the cross-modal target retrieval method provided in the embodiment of the present application.

[0061] The cross-modal target retrieval method provided in the embodiment of the present application can be applied in the application environment shown in FIG1. ​​In which, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers.

[0062] The terminal 102 sends a retrieval request to the server 104, and the retrieval request includes the target object to be detected. Subsequently, the server 104 obtains the target object to be detected. Subsequently, the server 104 inputs the target object into the cross-modal retrieval model, and obtains the retrieval results output by the cross-modal retrieval model, and the retrieval results include similar objects that match the target object retrieved from databases of multiple modal types. Among them, the cross-modal retrieval model is used to project the objects to be matched and / or the target objects in the databases of multiple modal types into a shared feature space, and determine the matching results of the objects to be matched and the target objects based on the shared features projected by the objects to be matched and / or the target objects in the shared feature space.

[0063] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0064] In an exemplary embodiment, Figure 2 As shown, a cross-modal target retrieval method is provided, and the method is applied to the server in FIG. 1 as an example for explanation. The cross-modal target retrieval method includes S201-S202:

[0065] S201: Acquire a target object to be detected.

[0066] In the present application, when cross-modal target retrieval is required, the server can obtain the target object to be detected.

[0067] It should be understood that the embodiments of the present application do not limit the above-mentioned target objects, and may include text information or image data of multiple modalities. The image data of multiple modalities may include infrared image data, near-infrared image data, visible light image data, depth image data, etc.

[0068] Exemplarily, the target of the cross-modal target retrieval may be a specific pedestrian, and the above-mentioned text information may be description information of the characteristics of the specific pedestrian, and the above-mentioned image data may be image data containing the specific pedestrian.

[0069] In some embodiments, the terminal may send a retrieval request to the server, where the retrieval request includes the target object to be detected. After receiving the retrieval request, the server may obtain the target object to be detected from the retrieval request.

[0070] S202: Input the target object into the cross-modal retrieval model, and obtain the retrieval result output by the cross-modal retrieval model.

[0071] In this step, after the server obtains the target object to be detected, the target object can be input into the cross-modal retrieval model, and the retrieval result output by the cross-modal retrieval model can be obtained.

[0072] The search results include similar objects that match the target object and are retrieved from databases of multiple modality types.

[0073] It should be understood that the above-mentioned databases of multiple modality types may include a database of the same modality as the target object, or a database of a different modality from the target object, and the embodiments of the present application do not limit this. Exemplarily, the databases of multiple modality types may include a database of visible light images, a database of near-infrared images, a database of depth images, etc.

[0074] For example, Figure 3 A schematic diagram of a cross-modal search provided in an embodiment of the present application, such as Figure 3 As shown, if the target object obtained by the server is text information, the cross-modal retrieval model can search in the visible light image database, thereby returning a visible light image similar to the object described by the text information; and, the cross-modal retrieval model can search in the near-infrared image database, thereby returning a near-infrared image similar to the object described by the text information.

[0075] For example, Figure 4 A schematic diagram of another cross-modal search provided in an embodiment of the present application, such as Figure 4 As shown, if the target object acquired by the server is an infrared image, the cross-modal retrieval model can search in the visible light image database to return a visible light image similar to the target object; and, the cross-modal retrieval model can search in the near-infrared image database to return a near-infrared image similar to the target object.

[0076] For example, Figure 5 A schematic diagram of another cross-modal search provided in an embodiment of the present application is shown as follows: Figure 5 As shown, if the target object acquired by the server is a visible light image, the cross-modal retrieval model can search in the visible light image database to return a visible light image similar to the target object; and, the cross-modal retrieval model can search in the near-infrared light image database to return a near-infrared light image similar to the target object.

[0077] The above cross-modal retrieval model is explained below.

[0078] In an embodiment of the present application, a cross-modal retrieval model is used to project the objects to be matched and / or target objects in a database of multiple modal types into a shared feature space, and determine the matching results of the objects to be matched and the target objects based on the shared features projected by the objects to be matched and / or target objects in the shared feature space.

[0079] The above-mentioned object to be matched may be image data in a database of any modality type, and the shared feature space may be a feature space shared by data of different modality types after projection.

[0080] Figure 6 A schematic diagram of the structure of a cross-modal retrieval model provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the target object and the object to be matched both include image data, and the cross-modal retrieval model includes an image feature processing unit and a matching unit.

[0081] The image feature processing unit is used to divide the image data into a plurality of image block data of preset sizes. The image feature processing unit then projects the plurality of image block data of preset sizes into a shared feature space through a linear projection layer to obtain the shared features of each image block data, and determines the feature sequence of the image data according to the shared features of each image block data.

[0082] The preset size may be set according to actual conditions, for example, it may be a PxP pixel size.

[0083] For example, the target object may be a visible light pedestrian image. , the object to be matched can be a near-infrared pedestrian image ,in, Used to characterize visible light pedestrian images, Used to characterize near-infrared pedestrian images, H, W, and C represent the height, width, and number of channels of the image, respectively. Visible light pedestrian images can be evenly divided into N non-overlapping image blocks according to the size of PxP pixels. , N is an integer greater than or equal to 1. The near-infrared pedestrian image can also be evenly divided into N non-overlapping image blocks according to the size of PxP pixels. , N is an integer greater than or equal to 1. The visible light pedestrian image and the near infrared pedestrian image are converted into N D-dimensional image block vectors through their corresponding linear projection layers. For example, the image block corresponding to the visible light pedestrian image is transformed into N D-dimensional image block vectors through the visible light linear projection layer. The corresponding mapping matrix is ​​converted into an image block vector, and the near-infrared pedestrian image is passed through the near-infrared linear projection layer The corresponding mapping matrix is ​​converted into an image block vector. Images of different modalities are projected to obtain a shared feature space M.

[0084] The embodiments of the present application do not limit how to determine the shared features of each image block data. In some embodiments, the image feature processing unit can first determine the mapping matrices corresponding to the linear projection layers of the target object and the object to be matched, and the weight parameters of the mapping matrices. Subsequently, the image feature processing unit projects multiple image block data of preset sizes into the shared feature space according to the mapping matrix and the weight parameters to obtain the shared features of each image block data.

[0085] It should be understood that since the image block data of different modalities are quite different, it is very difficult to directly convert them into a shared feature space. Therefore, when projecting multiple image block data of preset sizes into a shared feature space, one modality can take into account the information of other modalities to obtain a more suitable matrix to obtain a common shared feature space.

[0086] For example, the process of sharing features of image blocks corresponding to the visible light pedestrian image and the near infrared light pedestrian image through the mapping matrix and weight parameters can be shown as formula (1):

[0087] (1)

[0088] in, , N is an integer greater than or equal to 1; is the shared feature obtained by weighting the k-th image block corresponding to the visible light pedestrian image and the near-infrared pedestrian image through their respective linear mappings; is a learnable weight parameter used to adjust the weighting of different mapping matrices. is the kth image block corresponding to the infrared pedestrian image, is the kth image block corresponding to the visible light pedestrian image, is the mapping matrix corresponding to the linear projection layer of visible light, It is the mapping matrix corresponding to the linear projection layer of near-infrared light.

[0089] For example, if there are other modal data that need to be matched, more other modal data can be introduced in the mapping matrix learning. For example, if cross-modal feature learning of visible light pedestrian images, infrared light pedestrian images, and deep pedestrian images is required, the process of sharing features of image blocks corresponding to visible light pedestrian images, infrared light pedestrian images, and deep pedestrian images through mapping matrices and weight parameters can be shown as formula (2):

[0090] (2)

[0091] in, , N is an integer greater than or equal to 1; is the shared feature obtained by weighting the k-th image block corresponding to the visible light pedestrian image, near infrared pedestrian image and deep behavior image through their respective linear mappings; and are all learnable weight parameters, which are used to adjust the weights of different mapping matrices. is the kth image block corresponding to the infrared pedestrian image, is the kth image block corresponding to the visible light pedestrian image, is the kth image block corresponding to the depth pedestrian image, is the mapping matrix corresponding to the linear projection layer of visible light, is the mapping matrix corresponding to the linear projection layer of near-infrared light, It is the mapping matrix corresponding to the linear projection layer of the depth image.

[0092] In some embodiments, after the shared features of each image block data are determined, the shared features of each image block data may be concatenated to determine a feature sequence of the image data.

[0093] For example, a learnable Classification Token Embedding (CLS) embedding vector can be inserted at the beginning of the feature sequence of the image data to capture sequence-level information. In order to enable the model to understand the relative position relationship between image blocks, a position embedding vector can also be added to the feature sequence of the image data. , thus obtaining the feature sequence of image data as shown in formula (3).

[0094] (3)

[0095] in, is the feature sequence of image data, is a linear projection layer that maps image patches to D-dimensional vectors. is the CLS embedding vector, and P is the position embedding. Exemplarily, the image feature processing unit may include a Vision Transformer (ViT) network, and the feature sequence of the image data output by the last layer of the ViT network after learning can be expressed as , the output CLS embedding vector , which can be used as a global feature of pedestrian images .

[0096] Continue to refer Figure 6 , the target object also includes text data; the cross-modal retrieval model includes a text feature processing unit.

[0097] In some embodiments, the text feature processing unit is used to segment and encode the text data to obtain the encoding of multiple language units in the text data; and determine the feature sequence of the text data according to the encoding of the multiple language units. Correspondingly, the matching unit is also used to determine the matching result between the object to be matched and the target object according to the feature sequence of the text data and the feature sequence of the image data of the object to be matched.

[0098] The above-mentioned word segmentation and encoding can be achieved through Byte Pair Encoding (BPE). BPE can decompose text data into smaller language units based on a dictionary containing 49,152 words, and assign a unique code to each language unit.

[0099] For example, in the process of determining the feature sequence of text data, in order to ensure the standardization of processing, special symbols can be implanted at the beginning and end of the feature sequence of text data, including the Start Of Sequence (SOS) symbol and the End Of Sequence (EOS) symbol. The SOS symbol is used to mark the beginning of a sentence, and the EOS symbol is used to mark the end of a sentence. The SOS symbol and the EOS symbol strengthen the boundary clarity of the processing flow.

[0100] For example, considering the effective use of computing resources, the length of text data can be limited. For example, the maximum sequence of feature sequences of text data can be set to 77 elements to optimize processing speed and efficiency. For example, in order to enhance the model's ability to understand syntactic structure and word order, the positional embedding (P) mechanism can be integrated into the feature sequence of text data, so that the model can capture and understand the relative position information of language units in the text, improve the grasp of contextual dependence, and thus achieve accurate expression and distinction of semantics at a higher dimension. Among them, the dimension of positional embedding P can be , the position embedding P is added to the feature sequence of the text data, and the feature sequence of the obtained text data is shown in formula (4).

[0101] (4)

[0102] in, is the feature sequence of text data, is the feature corresponding to the SOS symbol, The features corresponding to the EOS symbol, is the code of the first language unit, is the code for the second language unit, and so on.

[0103] The following is an explanation of the training process of the cross-modal retrieval model.

[0104] In some embodiments, the server may first obtain a sample set, which includes a plurality of image-text pairs and matching relationship labels of each image-text pair, and then use the sample set to adjust the model parameters in the cross-modal retrieval model.

[0105] For example, in order to minimize the distance between the text modality and the image modality, the loss function of the commonly used image-text matching task (Cross-Modal Projection Matching, CMPM) can be referred to. Accordingly, a small batch set consisting of multiple image-text pairs can be given, and the small batch set and the matching relationship label of each image-text pair can form a sample set ,in, express Describing the goal The targets shown in the above image are similar objects. express Describe the goals and The target shown in the above image is not a similar object. The matching relationship can be shown as formula (5):

[0106] (5)

[0107] in, is the normalized text representation vector. It is expressed as the ratio of the similarity between the image and all texts. is a learnable parameter that can be used to control the range of similarity.

[0108] In some embodiments, the cross-modal retrieval model includes an image-to-text matching loss function and a text-to-image matching loss function.

[0109] Among them, the image-to-text matching loss function is constructed by the relative entropy divergence of the probability of image-to-text projection and the true pairing probability of image-to-text, and the text-to-image matching loss function is determined by the projection of text to image and the image-to-text matching loss function.

[0110] For example, in a small batch of samples, the number of matched positive samples may be more than one. This paper uses a nonlinear operation for the true matching probability and normalizes it using the loss (softmax) function. The true matching probability is expressed by formula (6):

[0111] (6)

[0112] in, Expressed as the probability distribution of the true label, is a learnable parameter that can be used to control the range of similarity.

[0113] It should be understood that by quantifying the relative entropy divergence (Kullback-Leibler, KL) between the probability of image-to-text projection and the true pairing probability of image-to-text, an image-to-text matching loss function can be constructed, which essentially measures the difference in the probability distribution of the two. The image-to-text matching loss function can be shown in formula (7).

[0114] (7)

[0115] in, is a very small number to prevent numerical overflow. Formula (7) can make the shape of the pi distribution curve close to the shape of the qi distribution curve, so that the projection value of the matching image-text pair is maximized and the projection value of the unmatched image-text pair is minimized.

[0116] It should be understood that due to the symmetry of the loss function, by calculating the projection of the text representation onto the image representation, the text-to-image matching loss function can be obtained: Correspondingly, the total loss of the cross-modal retrieval model is .in, It can be used to adjust the image-to-text matching loss function and the text-to-image matching loss function.

[0117] In the embodiments of the present application, a robust feature expression for highly accurate cross-modal text description alignment can be achieved, matching between text descriptions and various cross-modal images can be achieved, and matching between various cross-modal images can be supported.

[0118] The cross-modal target retrieval method provided in an embodiment of the present application first obtains the target object to be detected. Subsequently, the target object is input into a cross-modal retrieval model, and the retrieval result output by the cross-modal retrieval model is obtained, and the retrieval result includes similar objects that match the target object retrieved from databases of multiple modal types. Among them, the cross-modal retrieval model is used to project the objects to be matched and / or the target objects in the databases of multiple modal types into a shared feature space, and determine the matching results of the objects to be matched and the target object based on the shared features projected by the objects to be matched and / or the target objects in the shared feature space. Since the cross-modal retrieval model can project objects of multiple modalities into a shared feature space, and determine the match through the shared features projected in the shared feature space, it is possible to achieve matching of cross-modal objects.

[0119] The following explains how to train a cross-modal retrieval model. Figure 7 A flow chart of another cross-modal target retrieval method provided in an embodiment of the present application is shown as follows: Figure 7 As shown, the cross-modal target retrieval method includes S301-S304:

[0120] S301 : Obtain a sample set, where the sample set includes multiple image-text pairs and a matching relationship label for each image-text pair.

[0121] S302: Use the sample set to adjust the model parameters in the cross-modal retrieval model.

[0122] In some embodiments, the cross-modal retrieval model includes an image-to-text matching loss function and a text-to-image matching loss function.

[0123] Among them, the image-to-text matching loss function is constructed by the relative entropy divergence of the probability of image-to-text projection and the true pairing probability of image-to-text, and the text-to-image matching loss function is determined by the projection of text to image and the image-to-text matching loss function.

[0124] Among them, the cross-modal retrieval model is used to project the objects to be matched and / or target objects in the database of multiple modal types into a shared feature space, and determine the matching results of the objects to be matched and the target objects based on the shared features projected by the objects to be matched and / or target objects in the shared feature space.

[0125] S303: Acquire a target object to be detected.

[0126] S304: Input the target object into the cross-modal retrieval model, and obtain retrieval results output by the cross-modal retrieval model, where the retrieval results include similar objects matching the target object retrieved from databases of multiple modal types.

[0127] The cross-modal target retrieval method provided in an embodiment of the present application first obtains the target object to be detected. Subsequently, the target object is input into a cross-modal retrieval model, and the retrieval result output by the cross-modal retrieval model is obtained, and the retrieval result includes similar objects that match the target object retrieved from databases of multiple modal types. Among them, the cross-modal retrieval model is used to project the objects to be matched and / or the target objects in the databases of multiple modal types into a shared feature space, and determine the matching results of the objects to be matched and the target object based on the shared features projected by the objects to be matched and / or the target objects in the shared feature space. Since the cross-modal retrieval model can project objects of multiple modalities into a shared feature space, and determine the match through the shared features projected in the shared feature space, it is possible to achieve matching of cross-modal objects.

[0128] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0129] Based on the same inventive concept, the embodiment of the present application also provides a cross-modal target retrieval device for implementing the cross-modal target retrieval method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more cross-modal target retrieval device embodiments provided below can refer to the limitations on the cross-modal target retrieval method above, and will not be repeated here.

[0130] In an exemplary embodiment, Figure 8 As shown, a cross-modal target retrieval device 400 is provided, comprising: an acquisition module 401 and a matching module 402, wherein:

[0131] The acquisition module 401 is used to acquire the target object to be detected.

[0132] The matching module 402 is used to input the target object into the cross-modal retrieval model and obtain the retrieval results output by the cross-modal retrieval model, wherein the retrieval results include similar objects matching the target object retrieved from databases of multiple modal types.

[0133] Among them, the cross-modal retrieval model is used to project the objects to be matched and / or target objects in the database of multiple modal types into a shared feature space, and determine the matching results of the objects to be matched and the target objects based on the shared features projected by the objects to be matched and / or target objects in the shared feature space.

[0134] In one of the embodiments, both the target object and the object to be matched include image data, and the cross-modal retrieval model includes an image feature processing unit and a matching unit.

[0135] The image feature processing unit is used to divide the image data into a plurality of image block data of preset sizes; project the plurality of image block data of preset sizes into a shared feature space through a linear projection layer to obtain the shared features of each image block data; and determine the feature sequence of the image data according to the shared features of each image block data.

[0136] The matching unit is used to determine the matching result between the object to be matched and the target object according to the feature sequence of the image data of the target object and the object to be matched.

[0137] In one of the embodiments, the image feature processing unit is also used to determine the mapping matrices corresponding to the linear projection layers of the target object and the object to be matched, and the weight parameters of the mapping matrices; according to the mapping matrices and the weight parameters, multiple image block data of preset sizes are projected into a shared feature space to obtain shared features of each image block data.

[0138] In one of the embodiments, the target object also includes text data; and the cross-modal retrieval model includes a text feature processing unit.

[0139] The text feature processing unit is used to segment and encode the text data to obtain the encoding of multiple language units in the text data; and determine the feature sequence of the text data according to the encoding of the multiple language units.

[0140] The matching unit is further used to determine the matching result between the object to be matched and the target object according to the feature sequence of the text data and the feature sequence of the image data of the object to be matched.

[0141] In one embodiment, the cross-modal object retrieval apparatus 400 further includes:

[0142] The training module 403 is used to obtain a sample set, which includes multiple image-text pairs and matching relationship labels of each image-text pair; and use the sample set to the cross-modal retrieval model to adjust the model parameters in the cross-modal retrieval model.

[0143] In one of the embodiments, the cross-modal retrieval model includes an image-to-text matching loss function and a text-to-image matching loss function.

[0144] Among them, the image-to-text matching loss function is constructed by the relative entropy divergence of the probability of image-to-text projection and the true pairing probability of image-to-text, and the text-to-image matching loss function is determined by the projection of text to image and the image-to-text matching loss function.

[0145] Each module in the above cross-modal target retrieval device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0146] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG9. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a cross-modal target retrieval method is implemented.

[0147] Those skilled in the art will understand that the structure shown in FIG. 9 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0148] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above-mentioned cross-modal target retrieval method when executing the computer program.

[0149] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned cross-modal target retrieval method is implemented.

[0150] In one embodiment, a computer program product is provided, including a computer program, which implements the above-mentioned cross-modal target retrieval method when executed by a processor.

[0151] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0152] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0153] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A cross-modal target retrieval method, characterized in that: The method comprises: Obtain the target object to be detected; Inputting the target object into a cross-modal retrieval model, and obtaining a retrieval result output by the cross-modal retrieval model, wherein the retrieval result includes similar objects matching the target object retrieved from databases of multiple modal types; Among them, the cross-modal retrieval model is used to project the objects to be matched and / or the target objects in the database of the multiple modal types into a shared feature space, and determine the matching results of the objects to be matched and the target objects based on the shared features projected by the objects to be matched and / or the target objects in the shared feature space.

2. The method according to claim 1, characterized in that The target object and the object to be matched both include image data, and the cross-modal retrieval model includes an image feature processing unit and a matching unit; The image feature processing unit is used to divide the image data into a plurality of image block data of preset sizes; project the plurality of image block data of preset sizes into the shared feature space through a linear projection layer to obtain a shared feature of each image block data; Determining a feature sequence of the image data according to the shared features of each image block data; The matching unit is used to determine the matching result between the object to be matched and the target object according to the feature sequences of the image data of the target object and the object to be matched.

3. The method according to claim 2, characterized in that The step of projecting the plurality of image block data of preset sizes onto the shared feature space through a linear projection layer to obtain a shared feature of each image block data comprises: Determine mapping matrices corresponding to the linear projection layers of the target object and the object to be matched, respectively, and weight parameters of the mapping matrices; According to the mapping matrix and the weight parameter, the plurality of image block data of preset sizes are projected into the shared feature space to obtain the shared features of each image block data.

4. The method according to claim 2, characterized in that: The target object also includes text data; the cross-modal retrieval model includes a text feature processing unit; The text feature processing unit is used to segment and encode the text data to obtain the encoding of multiple language units in the text data; and determine the feature sequence of the text data according to the encoding of the multiple language units; The matching unit is further used to determine a matching result between the object to be matched and the target object according to a feature sequence of the text data and a feature sequence of the image data of the object to be matched.

5. The method according to claim 1, characterized in that The method further comprises: Acquire a sample set, wherein the sample set includes a plurality of image-text pairs and a matching relationship label of each image-text pair; The cross-modal retrieval model is retrievaled using the sample set to adjust model parameters in the cross-modal retrieval model.

6. The method according to any one of claims 1 to 5, characterized in that: The cross-modal retrieval model includes an image-to-text matching loss function and a text-to-image matching loss function; Among them, the image-to-text matching loss function is constructed by the relative entropy divergence of the probability of image-to-text projection and the true pairing probability of image-to-text, and the text-to-image matching loss function is determined by the projection of text onto image and the image-to-text matching loss function.

7. A cross-modal target retrieval device, characterized in that: The device comprises: An acquisition module, used for acquiring a target object to be detected; A matching module, used to input the target object into a cross-modal retrieval model and obtain a retrieval result output by the cross-modal retrieval model, wherein the retrieval result includes similar objects matching the target object retrieved from databases of multiple modal types; Among them, the cross-modal retrieval model is used to project the objects to be matched and / or the target objects in the database of the multiple modal types into a shared feature space, and determine the matching results of the objects to be matched and the target objects based on the shared features projected by the objects to be matched and / or the target objects in the shared feature space.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images