A training method and device of a cross-modal retrieval model, equipment and a storage medium

Abstract

The application provides a training method and device of a cross-modal retrieval model, equipment and a storage medium. The method comprises: obtaining image training samples corresponding to each visual feature type from video image data corresponding to a retrieval object; inputting the image training samples corresponding to each visual feature type into an image encoder in an initial model to obtain visual features of the retrieval object under each visual feature type, and taking a splicing processing result of each visual feature as a visual description feature of the retrieval object; inputting text description information of the retrieval object into a text encoder in the initial model to obtain a text description feature of the retrieval object; and adjusting model parameters of the initial model according to a feature distance between the text description feature and the visual description feature to obtain the cross-modal retrieval model. In this way, the application can obtain a visual description feature that more comprehensively describes the retrieval object visually, and improves the accuracy of cross-modal retrieval of the retrieval object.

Description

Technical Field

[0001] This application relates to the field of model training technology, and more specifically, to a training method, apparatus, device, and storage medium for a cross-modal retrieval model. Background Technology

[0002] Cross-modal retrieval is a retrieval scheme that leverages the associations between different modalities of the same retrieval object to search for samples of the same object in a different modality, based on samples of the same object in one modality. For example, when searching for a vehicle, one can use the textual description of the target vehicle to perform cross-modal retrieval of its image.

[0003] Currently, taking cross-modal vehicle retrieval as an example, existing cross-modal vehicle retrieval mainly uses a visual model pre-trained on the ReID (Re-identification) task and a general pre-trained language model as the feature extraction network for vehicle images and text, respectively. Then, a matching loss is used to optimize the features of images and text based on the one-to-one correspondence between text and images, thereby performing visual-language cross-modal retrieval of target vehicles.

[0004] However, the aforementioned cross-modal vehicle retrieval method requires the introduction of an additional ReID task during the model training phase to pre-train the visual model on the training dataset, resulting in inefficient overall model training and a waste of training resources. Furthermore, when the amount of training data is low, the image and text feature extraction networks within this method have limited fitting capabilities to the training dataset, which can easily lead to low model training efficiency. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a training method, apparatus, device and storage medium for a cross-modal retrieval model, so as to learn visual features under different visual feature types by targeted training of the model without introducing redundant ReID tasks, so that the model can obtain visual descriptive features that more comprehensively describe the retrieval object from a visual perspective, thereby helping to improve the retrieval accuracy of the model for cross-modal retrieval of the retrieval object.

[0006] In a first aspect, embodiments of this application provide a training method for a cross-modal retrieval model, the training method comprising:

[0007] Based on a variety of preset visual feature types, image training samples corresponding to each visual feature type are obtained from the video image data corresponding to the search object.

[0008] For each visual feature type, the target image training sample corresponding to that visual feature type is input into the target image encoder associated with that visual feature type in the initial model to obtain the target visual features of the retrieval object under that visual feature type output by the target image encoder. The target visual features of the retrieval object under each visual feature type are then spliced ​​to obtain the visual description features of the retrieval object.

[0009] The text description information of the search object is input into the text encoder in the initial model to obtain the text description features of the search object output by the text encoder;

[0010] Based on the feature distance between the textual description features and the visual description features of the retrieval object, the model parameters of the initial model are adjusted to obtain a cross-modal retrieval model including the adjusted model parameters.

[0011] In one optional implementation, the step of obtaining image training samples corresponding to each visual feature type from the video image data corresponding to the retrieval object according to a preset set of multiple visual feature types includes:

[0012] Based on the first visual feature type included in the multiple visual feature types, the video image frame in which the search object first appears is obtained from the video image data corresponding to the search object as the image frame to be processed; wherein, the first visual feature type is used to characterize the visual feature type to which the appearance feature of the search object belongs;

[0013] From the image frame to be processed, the bounding box image of the retrieval object is extracted as the first image training sample corresponding to the first visual feature type.

[0014] In one optional implementation, the step of obtaining image training samples corresponding to each visual feature type from the video image data corresponding to the retrieval object according to a preset set of multiple visual feature types includes:

[0015] Based on the second visual feature type included in the multiple visual feature types, video image frames containing the search object are extracted from the video image data corresponding to the search object as second image training samples corresponding to the second visual feature type; wherein, the second visual feature type is used to characterize the visual feature type to which the surrounding environment features of the search object belong.

[0016] In one optional implementation, the step of obtaining image training samples corresponding to each visual feature type from the video image data corresponding to the retrieval object according to a preset set of multiple visual feature types includes:

[0017] Based on the third visual feature type included in the multiple visual feature types, all video image frames including the search object are obtained from the video image data corresponding to the search object as a group of image frames to be sampled; wherein, the third visual feature type is used to characterize the visual feature type to which the motion trajectory feature of the search object belongs;

[0018] According to a preset image sampling number threshold, the image frame group to be sampled is uniformly sampled to obtain a target image sequence with the same sequence length as the image sampling number threshold;

[0019] The target image sequence is used as the third image training sample corresponding to the third visual feature type.

[0020] In one optional implementation, the step of inputting the target image training samples corresponding to the visual feature type into the target image encoder associated with the visual feature type within the initial model to obtain the target visual features of the retrieval object under the visual feature type output by the target image encoder includes:

[0021] The first image training sample corresponding to the first visual feature type is input into the first image encoder associated with the first visual feature type in the initial model to obtain the first visual feature vector of the search object output by the first image encoder; wherein, the first visual feature vector is used to characterize the appearance features of the search object.

[0022] In one optional implementation, the step of inputting the target image training samples corresponding to the visual feature type into the target image encoder associated with the visual feature type within the initial model to obtain the target visual features of the retrieval object under the visual feature type output by the target image encoder includes:

[0023] The second image training sample corresponding to the second visual feature type is input into the second image encoder associated with the second visual feature type in the initial model to obtain the second visual feature vector of the search object output by the second image encoder; wherein, the second visual feature vector is used to characterize the surrounding environment features of the search object.

[0024] In one optional implementation, the step of inputting the target image training samples corresponding to the visual feature type into the target image encoder associated with the visual feature type within the initial model to obtain the target visual features of the retrieval object under the visual feature type output by the target image encoder includes:

[0025] The training samples of the third image corresponding to the third visual feature type are input into the third image encoder associated with the third visual feature type in the initial model to obtain the third visual feature vector of the search object output by the third image encoder; wherein, the third visual feature vector is used to characterize the motion trajectory features of the search object.

[0026] In one optional implementation, the step of inputting the text description information of the search object into the text encoder within the initial model to obtain the text description features of the search object output by the text encoder includes:

[0027] The text description information of the search object is input into the text encoder in the initial model to obtain the text description features of the search object output by the text encoder; wherein, the text description features include at least: a first text feature for describing the appearance of the search object, a second text feature for describing the surrounding environment of the search object, and a third text feature for describing the movement trajectory of the search object.

[0028] In one optional implementation, adjusting the model parameters of the initial model based on the feature distance between the textual description features and the visual description features of the search object includes:

[0029] When the feature distance between the text description feature and the visual description feature is less than or equal to the feature matching distance threshold, the model parameters of the initial model are adjusted according to the first adjustment strategy; wherein, the first adjustment strategy is used to characterize the model parameter adjustment strategy for reducing the feature matching loss between the text description feature and the visual description feature;

[0030] When the feature distance between the text description feature and the visual description feature is greater than the feature matching distance threshold, the model parameters of the initial model are adjusted according to the second adjustment strategy; wherein, the second adjustment strategy is used to characterize the model parameter adjustment strategy that increases the feature matching loss between the text description feature and the visual description feature.

[0031] In one optional implementation, the initial model includes multiple image encoders; wherein each image encoder corresponds to a preset visual feature type; the training method further includes:

[0032] For any two image encoders among the plurality of image encoders, if the visual feature types corresponding to each of the two image encoders belong to the first feature type, then it is determined that the two image encoders are trained in a parameter-sharing manner during the model training process of the initial model; wherein, the first feature type is used to characterize the visual feature type that is independent of the motion state of the retrieval object;

[0033] When any two image encoders include a second feature type in their respective visual feature types, it is determined that the two image encoders are trained in a non-shared parameter manner during the model training process of the initial model; wherein, the second feature type is used to characterize the visual feature type related to the motion state of the retrieved object.

[0034] Secondly, embodiments of this application also provide a cross-modal retrieval method, the cross-modal retrieval method comprising:

[0035] The text description information required by the target object during the retrieval process is input into a pre-trained cross-modal retrieval model. The cross-modal retrieval model is used to perform cross-modal retrieval between vision and language for the target object, and the image retrieval result for the target object is obtained by the cross-modal retrieval model output by the cross-modal retrieval model. The cross-modal retrieval model is obtained by training according to the training method in any of the optional embodiments of the first aspect above.

[0036] Thirdly, embodiments of this application provide a training apparatus for a cross-modal retrieval model, the training apparatus comprising:

[0037] The sample acquisition module is used to acquire image training samples corresponding to each visual feature type from the video image data corresponding to the search object, based on a variety of preset visual feature types.

[0038] The visual feature extraction module is used to input the target image training sample corresponding to each visual feature type into the target image encoder associated with that visual feature type in the initial model, for each visual feature type, to obtain the target visual features of the search object under that visual feature type output by the target image encoder, and to perform splicing processing on the target visual features of the search object under each visual feature type to obtain the visual description features of the search object.

[0039] The text feature extraction module is used to input the text description information of the search object into the text encoder in the initial model to obtain the text description features of the search object output by the text encoder;

[0040] The model training module is used to adjust the model parameters of the initial model based on the feature distance between the textual description features and the visual description features of the retrieval object, so as to obtain a cross-modal retrieval model including the adjusted model parameters.

[0041] Fourthly, embodiments of this application provide a cross-modal retrieval device, characterized in that the cross-modal retrieval device comprises:

[0042] The retrieval module is used to input the text description information required by the target object during the retrieval process into a pre-trained cross-modal retrieval model. The cross-modal retrieval model performs cross-modal retrieval between vision and language on the target object, and obtains the image retrieval results for the target object output by the cross-modal retrieval model.

[0043] Fifthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the training method for any of the cross-modal retrieval models described above.

[0044] Sixthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the training method for any of the cross-modal retrieval models described above.

[0045] In a seventh aspect, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the cross-modal retrieval methods described above.

[0046] Eighthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of any of the cross-modal retrieval methods described above.

[0047] The technical solutions provided by the embodiments of this application may include the following beneficial effects:

[0048] This application provides a training method, apparatus, device, and storage medium for a cross-modal retrieval model. Based on a preset set of multiple visual feature types, image training samples corresponding to each visual feature type are obtained from video image data corresponding to the retrieval object. For each visual feature type, the target image training sample corresponding to that visual feature type is input into a target image encoder associated with that visual feature type within an initial model. This yields the target visual features of the retrieval object under that visual feature type, output by the target image encoder. The target visual features of the retrieval object under each visual feature type are then concatenated to obtain the visual description features of the retrieval object. The textual description information of the retrieval object is input into a text encoder within the initial model to obtain the textual description features of the retrieval object output by the text encoder. Based on the feature distance between the textual description features and the visual description features of the retrieval object, the model parameters of the initial model are adjusted to obtain a cross-modal retrieval model including the adjusted model parameters. In this way, without introducing redundant ReID tasks, this application can learn visual features under different visual feature types by specifically training the model, so that the model can obtain visual descriptive features that more comprehensively describe the retrieved object, thereby helping to improve the retrieval accuracy of the model for cross-modal retrieval of the retrieved object.

[0049] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0050] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 The diagram illustrates a flowchart of a training method for a cross-modal retrieval model provided in an embodiment of this application.

[0052] Figure 2 A flowchart illustrating the method for obtaining a first image training sample provided in an embodiment of this application is shown.

[0053] Figure 3 A flowchart illustrating the method for obtaining second image training samples provided in an embodiment of this application is shown.

[0054] Figure 4 A flowchart illustrating the method for obtaining third image training samples provided in an embodiment of this application is shown.

[0055] Figure 5 This illustration shows a flowchart of a method for encoding input first image training samples using a first image encoder, as provided in an embodiment of this application.

[0056] Figure 6 This illustration shows a flowchart of a method for encoding input second image training samples using a second image encoder, as provided in an embodiment of this application.

[0057] Figure 7 This illustration shows a flowchart of a method for encoding input third image training samples using a third image encoder, as provided in an embodiment of this application.

[0058] Figure 8 This illustration shows a flowchart of a method for determining whether parameter sharing is required between different image encoders within an initial model, as provided in an embodiment of this application.

[0059] Figure 9 A flowchart illustrating a cross-modal retrieval method provided in an embodiment of this application is shown;

[0060] Figure 10 This paper shows a schematic diagram of the structure of a training device for a cross-modal retrieval model provided in an embodiment of this application;

[0061] Figure 11 This illustration shows a schematic diagram of the structure of a cross-modal retrieval device provided in an embodiment of this application;

[0062] Figure 12 This is a schematic diagram of the structure of an electronic device 1200 provided in an embodiment of this application;

[0063] Figure 13 This is a schematic diagram of the structure of another electronic device 1300 provided in an embodiment of this application. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0065] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0066] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0067] Currently, taking cross-modal vehicle retrieval as an example, existing cross-modal vehicle retrieval methods require the additional introduction of a ReID task to pre-train the visual model on the training dataset during the model training phase. This results in inefficient overall model training and a waste of model training resources. Furthermore, when the amount of training data is low, the image and text feature extraction networks within this method have limited fitting ability to the training dataset, which can easily lead to low model training efficiency.

[0068] Based on this, embodiments of this application provide a training method, apparatus, device, and storage medium for a cross-modal retrieval model. According to a preset set of multiple visual feature types, image training samples corresponding to each visual feature type are obtained from the video image data corresponding to the retrieval object. For each visual feature type, the target image training sample corresponding to that visual feature type is input into the target image encoder associated with that visual feature type within the initial model, obtaining the target visual features of the retrieval object under that visual feature type output by the target image encoder. The target visual features of the retrieval object under each visual feature type are then concatenated to obtain the visual description features of the retrieval object. The textual description information of the retrieval object is input into the text encoder within the initial model, obtaining the textual description features of the retrieval object output by the text encoder. Based on the feature distance between the textual description features and the visual description features of the retrieval object, the model parameters of the initial model are adjusted to obtain a cross-modal retrieval model including the adjusted model parameters. In this way, without introducing redundant ReID tasks, this application can learn visual features under different visual feature types by specifically training the model, so that the model can obtain visual descriptive features that more comprehensively describe the retrieved object, thereby helping to improve the retrieval accuracy of the model for cross-modal retrieval of the retrieved object.

[0069] The following provides a detailed description of a training method, apparatus, device, and storage medium for a cross-modal retrieval model provided in the embodiments of this application.

[0070] Reference Figure 1 As shown, Figure 1 The diagram illustrates a flowchart of a training method for a cross-modal retrieval model provided in an embodiment of this application. The training method includes steps S101-S104; specifically:

[0071] S101: Based on multiple preset visual feature types, obtain image training samples corresponding to each visual feature type from the video image data corresponding to the retrieval object.

[0072] S102, for each visual feature type, the target image training sample corresponding to the visual feature type is input into the target image encoder associated with the visual feature type in the initial model to obtain the target visual features of the search object under the visual feature type output by the target image encoder, and the target visual features of the search object under each visual feature type are spliced ​​to obtain the visual description features of the search object.

[0073] S103, input the text description information of the search object into the text encoder in the initial model to obtain the text description features of the search object output by the text encoder.

[0074] S104, Based on the feature distance between the textual description features and the visual description features of the retrieval object, the model parameters of the initial model are adjusted to obtain a cross-modal retrieval model including the adjusted model parameters.

[0075] This application provides a training method for a cross-modal retrieval model. Based on a set of preset visual feature types, image training samples corresponding to each visual feature type are obtained from the video image data corresponding to the retrieval object. For each visual feature type, the target image training sample is input into the target image encoder associated with that visual feature type within the initial model to obtain the target visual features of the retrieval object under that visual feature type, output by the target image encoder. The target visual features of the retrieval object under each visual feature type are then concatenated to obtain the visual description features of the retrieval object. The textual description information of the retrieval object is input into the text encoder within the initial model to obtain the textual description features of the retrieval object output by the text encoder. Based on the feature distance between the textual description features and the visual description features of the retrieval object, the model parameters of the initial model are adjusted to obtain a cross-modal retrieval model including the adjusted model parameters. In this way, without introducing redundant ReID tasks, this application can learn visual features under different visual feature types by specifically training the model, so that the model can obtain visual descriptive features that more comprehensively describe the retrieved object, thereby helping to improve the retrieval accuracy of the model for cross-modal retrieval of the retrieved object.

[0076] The following is an exemplary description of each step in the training method of the cross-modal retrieval model provided in the embodiments of this application:

[0077] S101: Based on multiple preset visual feature types, obtain image training samples corresponding to each visual feature type from the video image data corresponding to the retrieval object.

[0078] Here, the specific entity type to which the search object belongs can be determined based on the application scenario of the cross-modal retrieval model. For example, when the trained cross-modal retrieval model is used in an intelligent transportation system, the specific entity type to which the search object belongs can be a vehicle; when the trained cross-modal retrieval model is used in a zoo management system, the specific entity type to which the search object belongs can be an animal. Based on this, the embodiments of this application do not impose any limitations on the specific entity type to which the search object belongs.

[0079] Here, the video image data corresponding to the search object is used to represent multiple consecutive video image frames containing the search object. However, this application embodiment does not limit the specific number of images contained in the video image data corresponding to the search object.

[0080] Specifically, the video image data corresponding to the aforementioned search object can also be determined based on the application scenario of the cross-modal retrieval model; for example, when the trained cross-modal retrieval model is used in an intelligent transportation system, the search object can be a vehicle, and the video image data corresponding to the search object can be the monitoring video data of the vehicle in the intelligent transportation system.

[0081] It should be noted that the specific types of visual features mentioned above can be determined based on the actual retrieval needs of the retrieval target; however, this application does not impose any limitations on the specific types of visual features.

[0082] For example, when the search object is a vehicle, considering that the text description information used when searching for a vehicle is generally related to the vehicle's appearance (such as vehicle shape, vehicle color, vehicle model, etc.), the vehicle's driving environment (such as driving on a highway, driving on an overpass, etc.), and the vehicle's driving route (or driving trajectory), based on this, when the search object is a vehicle, the preset multiple visual feature types can be: vehicle appearance feature type, vehicle surrounding environment feature type, vehicle driving trajectory feature type, etc.

[0083] S102, for each visual feature type, the target image training sample corresponding to the visual feature type is input into the target image encoder associated with the visual feature type in the initial model to obtain the target visual features of the search object under the visual feature type output by the target image encoder, and the target visual features of the search object under each visual feature type are spliced ​​to obtain the visual description features of the search object.

[0084] Specifically, considering that the search objects are usually objects that can move, such as vehicles and animals as shown in the above example; based on this, when the search object in the embodiment of this application is an object that can move, the above-mentioned preset visual feature type includes at least: a first visual feature type for characterizing the appearance features of the search object, a second visual feature type for characterizing the surrounding environment features of the search object, and a third visual feature type for characterizing the movement trajectory features of the search object.

[0085] Here, taking the preset multiple visual feature types, including the aforementioned first visual feature type, second visual feature type, and third visual feature type, as an example, based on the different visual feature types used to characterize the different visual features of the search object, this application embodiment needs to obtain image training samples corresponding to each visual feature type from the video image data corresponding to the search object, so as to train the initial model to learn the visual features under different visual feature types in a more targeted manner.

[0086] In this embodiment of the application, the initial model is used to characterize the cross-modal retrieval model that has not yet completed training. At this time, since the cross-modal retrieval model provided in this embodiment of the application is used to realize cross-modal retrieval of the retrieval object between text (i.e., language) and image (i.e., vision), the initial model can be a visual language model, such as CLIP (Contrastive Language Image Pretraining) model, etc.

[0087] Here, the initial model includes multiple image encoders and at least one text encoder. The number of image encoders can be the same as the number of visual feature types; that is, in the initial model, each visual feature type is associated with a corresponding target image encoder. This application does not limit the specific encoder model of the aforementioned image encoders.

[0088] For example, if the preset multiple visual feature types include the first visual feature type, the second visual feature type, and the third visual feature type, then the initial model may include three image encoders, wherein the target image encoder associated with the first visual feature type is the first image encoder, the target image encoder associated with the second visual feature type is the second image encoder, and the target image encoder associated with the third visual feature type is the third image encoder.

[0089] Specifically, the aforementioned first visual feature type and the aforementioned second visual feature type are used to characterize the visual features exhibited by the retrieval object in a static image (i.e., the aforementioned appearance features and the aforementioned surrounding environment features). Therefore, as an optional embodiment, a single video image frame containing the retrieval object can be obtained from the video image data corresponding to the retrieval object as a first image training sample corresponding to the aforementioned first visual feature type. The first image training sample is then input to the first image encoder associated with the first visual feature type to obtain the appearance features of the retrieval object under the first visual feature type (i.e., the target visual features of the retrieval object under the first visual feature type) output by the first image encoder. Alternatively, a single video image frame containing the retrieval object can be obtained from the video image data corresponding to the retrieval object as a second image training sample corresponding to the aforementioned second visual feature type. The second image training sample is then input to the second image encoder associated with the second visual feature type to obtain the surrounding environment features of the retrieval object under the second visual feature type (i.e., the target visual features of the retrieval object under the second visual feature type) output by the second image encoder.

[0090] Specifically, the aforementioned third visual feature type is used to characterize the visual features (i.e., the aforementioned motion trajectory features) exhibited by the retrieval object in dynamic images (equivalent to multiple consecutive video image frames with consecutive timestamps). Therefore, as an optional embodiment, multiple consecutive video image frames with consecutive timestamps can be obtained from the video image data corresponding to the retrieval object as third image training samples corresponding to the aforementioned third visual feature type. The third image training samples are then input into the third image encoder associated with the third visual feature type to obtain the motion trajectory features of the retrieval object under the third visual feature type (i.e., the target visual features of the retrieval object under the third visual feature type) output by the third image encoder.

[0091] Here, in the image encoder, the target visual features of the retrieved object under each visual feature type are usually output in the form of vectors. Therefore, the above-mentioned splicing process can be used to characterize the first and last splicing of the target visual features of the retrieved object under each visual feature type.

[0092] For example, taking the first visual feature type and the third visual feature type as an example, after obtaining the appearance feature vector of the search object under the first visual feature type output by the first image encoder, the surrounding environment feature vector of the search object under the second visual feature type output by the second image encoder, and the motion trajectory feature vector of the search object under the third visual feature type output by the third image encoder, a concat operation (i.e., the above-mentioned splicing process) is performed on the appearance feature vector, the surrounding environment feature vector, and the motion trajectory feature vector to obtain the visual description features that can comprehensively describe the search object from the three visual branches of appearance, surrounding environment, and motion trajectory.

[0093] S103, input the text description information of the search object into the text encoder in the initial model to obtain the text description features of the search object output by the text encoder.

[0094] Here, text description information is used to represent the descriptive information that describes the search object in text form. In other words, text description information is equivalent to the text-based information that a user enters when searching for a search object. For example, it can be the natural language description information of the search object.

[0095] Specifically, when the preset visual feature types in the embodiments of this application are the first visual feature type - the third visual feature type mentioned above, as an optional embodiment, the text description information of the search object may include: the appearance description information of the search object (e.g., the appearance description information such as the shape characteristics and appearance color of the search object), the surrounding environment description information of the search object (e.g., the search object is located on an overpass, and the surrounding environment description information includes bushes, etc.), and the movement trajectory description information of the search object (e.g., the movement trajectory description information such as the search object is traveling on highway x towards exit x1).

[0096] Based on the above optional embodiments, in this application embodiment, step S103 can be performed according to the following step a, specifically:

[0097] Step a: Input the text description information of the search object into the text encoder in the initial model to obtain the text description features of the search object output by the text encoder.

[0098] At this point, the aforementioned text description features include at least: a first text feature for describing the appearance of the retrieved object, a second text feature for describing the surrounding environment of the retrieved object, and a third text feature for describing the movement trajectory of the retrieved object.

[0099] Specifically, when the initial model is the CLIP model, the CLIP model has already published its pre-trained network parameters obtained after training on a very large and diverse visual text dataset. Therefore, the image encoders (i.e., the first, second, and third image encoders mentioned above) and text encoders provided in the CLIP model used in this application embodiment have already been pre-loaded with the aforementioned pre-trained network parameters before training. That is, before the initial model starts training, this application embodiment has already pre-initialized any one of the target image encoders in step S102 and the text encoder in step S103 using the aforementioned pre-trained network parameters. In this way, on the one hand, by initializing the aforementioned image encoders and text encoders, the learning speed of the initial model can be accelerated, which is conducive to obtaining a better initial model performance; on the other hand, based on the large training set of the CLIP model, the aforementioned pre-trained network parameters have great potential to solve various problems, including the cross-modal vehicle retrieval problem that this application embodiment needs to solve.

[0100] It should be noted that, taking the CLIP model as an example, the target image encoder (i.e., the first image encoder, the second image encoder, and the third image encoder) and the text encoder can be image encoders and text encoders provided within the CLIP model; for example, the target image encoder can be a ResNet or Transformer network provided within the CLIP model; the specific encoder models of the text encoder and the target image encoder are not limited in this application embodiment.

[0101] S104, Based on the feature distance between the textual description features and the visual description features of the retrieval object, the model parameters of the initial model are adjusted to obtain a cross-modal retrieval model including the adjusted model parameters.

[0102] In this embodiment of the application, when adjusting the model parameters of the initial model, a feature matching threshold can be preset. When the feature distance between the text description feature and the visual description feature is less than or equal to the preset feature matching threshold, it is determined that the matching degree between the text description feature and the visual description feature meets the cross-modal retrieval requirements for the retrieval object. When the feature distance between the text description feature and the visual description feature is greater than the preset feature matching threshold, it is determined that the matching degree between the text description feature and the visual description feature does not meet the cross-modal retrieval requirements for the retrieval object.

[0103] Specifically, when the feature distance between the aforementioned textual description features and the aforementioned visual description features is less than or equal to the feature matching distance threshold, the model parameters of the initial model can be adjusted according to the first adjustment strategy.

[0104] Here, the first adjustment strategy is used to characterize the model parameter adjustment strategy for reducing the feature matching loss between the text description features and the visual description features; that is, when the feature distance between the text description features and the visual description features is less than or equal to the feature matching distance threshold, based on the fact that the matching degree between the text description features and the visual description features meets the cross-modal retrieval requirements for the retrieval object, the model parameters of the initial model can be adjusted by narrowing the feature distance between the text description features and the visual description features (i.e., reducing the feature matching loss between the text description features and the visual description features).

[0105] Specifically, when the feature distance between the aforementioned textual description features and the aforementioned visual description features is greater than the feature matching distance threshold, the model parameters of the initial model can be adjusted according to the second adjustment strategy.

[0106] Here, the second adjustment strategy described above is used to characterize the model parameter adjustment strategy for increasing the feature matching loss between the text description features and the visual description features; that is, when the feature distance between the text description features and the visual description features is greater than the feature matching distance threshold, the model parameters of the initial model can be adjusted by increasing the feature distance between the text description features and the visual description features (i.e., increasing the feature matching loss between the text description features and the visual description features).

[0107] It should be noted that the reduction / increase of the feature matching loss between the text description features and the visual description features can be performed by using a preset fixed coefficient; or by using a non-fixed coefficient (e.g., dynamically reducing / increasing the loss based on the difference between the feature distance and the feature matching distance threshold). This application does not limit the specific reduction or increase methods used when reducing / increasing the feature matching loss between the text description features and the visual description features.

[0108] The following section uses the example of a vehicle as the search target to illustrate the specific implementation process of each of the above steps in this embodiment of the application:

[0109] Regarding the specific implementation process of step S101 above, when the search object belongs to a vehicle, the preset multiple visual feature types in step S101 include at least the first visual feature type, the second visual feature type, and the third visual feature type. The following details the different implementation methods of step S101 under these three different visual feature types:

[0110] Regarding the aforementioned first visual feature type, in one optional implementation scheme, such as Figure 2 As shown, Figure 2 The diagram illustrates a flowchart of a method for obtaining first image training samples according to an embodiment of this application. Specifically, when performing step S101, the method includes steps S201-S202.

[0111] S201, based on the first visual feature type included in the multiple visual feature types, obtain the first video image frame of the search object from the video image data corresponding to the search object as the image frame to be processed.

[0112] Here, the aforementioned first visual feature type is still used to characterize the visual feature type to which the appearance feature of the search object belongs; wherein, the appearance feature may include the shape feature of the search object (such as vehicle size, vehicle shape, chassis height, etc.), color feature (such as vehicle color), etc.; the specific feature type of the above appearance feature is not limited in this application embodiment.

[0113] S202, extract the bounding box image of the search object from the image frame to be processed as the first image training sample corresponding to the first visual feature type.

[0114] Here, the bounding box image of the search object can be obtained by cropping the image region of the search object in the image frame to be processed according to a preset image bounding box; the specific image format of the bounding box image of the search object is not limited in this embodiment.

[0115] It should be noted that, unlike the first method of obtaining the first image training sample given in step S102 above (i.e., obtaining a single video image frame containing the search object from the video image data corresponding to the search object as the first image training sample corresponding to the first visual feature type), the specific reason for limiting the extraction of the bounding box image of the search object from the video image frame where the search object first appears as the first image training sample in steps S201-S202 is explained as follows:

[0116] Reason 1: Compared with the existing image sample acquisition method (i.e., first extract the bounding box image of the retrieval object in each video image frame, and then average all the extracted bounding box images to obtain the first image training sample), it saves more image processing resources and reduces the burden of image data processing.

[0117] Reason 2: Experimental results obtained through multiple model training experiments show that, compared to the method of randomly selecting a single video image frame in the first acquisition method given in step S102 above, when the bounding box image of the retrieval object is extracted from the video image frame where the retrieval object first appears as the first image training sample in steps S201-S202 above, and the first image encoder in the initial model is trained, the model performance of the cross-modal retrieval model obtained after the final training is better.

[0118] Regarding the aforementioned second visual feature type, in one optional implementation, such as Figure 3 As shown, Figure 3 The diagram illustrates a flowchart of a method for obtaining second image training samples according to an embodiment of this application. Specifically, when performing step S101, the method includes step S301.

[0119] S301, based on the second visual feature type included in the multiple visual feature types, extract video image frames containing the search object from the video image data corresponding to the search object as second image training samples corresponding to the second visual feature type.

[0120] Here, the aforementioned second visual feature type is still used to characterize the visual feature type to which the surrounding environment features of the retrieved object belong.

[0121] Specifically, when the search object is a vehicle, the surrounding environmental features of the search object can be the vehicle's driving environment, such as the vehicle driving on a highway or an overpass. This application does not limit the specific feature type of the above-mentioned surrounding environmental features.

[0122] In this embodiment, when the retrieval object is a vehicle, considering that vehicles are usually in a driving state (equivalent to a moving state) in video image data, the surrounding environmental features of the vehicle may change dynamically in different video image frames (e.g., video image frame A shows the vehicle driving on a highway, while video image frame E shows the vehicle driving on an overpass). Therefore, in order to save image processing resources, in a preferred embodiment, a video image frame containing the retrieval object can be randomly selected from the video image data corresponding to the retrieval object as a second image training sample, in accordance with the above step S301.

[0123] It should be noted that the method of obtaining the second image training samples shown in the above preferred embodiments is only for the purpose of saving image processing resources. In addition to the implementation method given in the above preferred embodiments, the second image training samples can also be obtained in other ways according to actual usage needs (e.g., using a specified number of video image frames containing the search object that meet the clarity requirements as the second image training samples, or using each video image frame containing the search object as the second image training sample, etc.). As long as the obtained image frames can reflect the environmental characteristics around the search object, they can be used as the second image training samples. The specific method and quantity of obtaining the second image training samples are not limited in the embodiments of this application.

[0124] Regarding the aforementioned third visual feature type, in one optional implementation scheme, such as Figure 4 As shown, Figure 4 The diagram illustrates a flowchart of a method for obtaining third image training samples according to an embodiment of this application. Specifically, when performing step S101, the method includes steps S401-S403.

[0125] S401, based on the third visual feature type included in the multiple visual feature types, obtain all video image frames including the search object from the video image data corresponding to the search object as a group of image frames to be sampled.

[0126] Here, the aforementioned third visual feature type is still used to characterize the visual feature type to which the motion trajectory feature of the search object belongs; that is, when the search object belongs to a vehicle, the aforementioned third visual feature type can be used to characterize the visual feature type to which the vehicle's driving trajectory feature belongs.

[0127] S402, according to the preset image sampling quantity threshold, the image frame group to be sampled is uniformly sampled to obtain a target image sequence with the same sequence length as the image sampling quantity threshold.

[0128] Here, the specific value of the preset image sampling number threshold is not limited in this embodiment; among them, the experimental results obtained after multiple model training experiments show that the preset image sampling number threshold can be 10, so as to obtain a cross-modal retrieval model with better training effect.

[0129] In this embodiment of the application, considering that the number of video image frames contained in the actual obtained image frame group to be sampled is generally difficult to be exactly equal to the preset image sampling number threshold, when the number of video image frames contained in the actual obtained image frame group to be sampled is not equal to the preset image sampling number threshold, the target image sequence with the same sequence length as the above-mentioned image sampling number threshold can be obtained by following the steps b1-b2, specifically:

[0130] Step b1: When the number of video image frames contained in the image frame group to be sampled is greater than the preset image sampling number threshold, the number of video image frames in the image frame group to be sampled can be extracted as the target image sequence by means of equal interval sampling.

[0131] For example, if the preset image sampling threshold is 10, and the image frame group to be sampled contains 20 video image frames P1-P20 with consecutive shooting time, then by sampling at equal intervals, 10 video image frames can be extracted from the image frame group to be sampled, and the target image sequence is: video image frames P1, P3, P5...P19.

[0132] Step b2: When the number of video image frames contained in the image frame group to be sampled is less than the preset image sampling number threshold, zero-padding can be performed to pad the insufficient number of video image frames in the image frame group to be sampled, so as to obtain a target image sequence with the same sequence length as the image sampling number threshold.

[0133] For example, taking a preset image sampling threshold of 10 as an example, if the image frame group to be sampled contains 9 video image frames P1-P9 with consecutive shooting time, then by zero-padding, a blank image P0 is added to the end of the image frame group to be sampled, and the target image sequence is: video image frames P1, P2, P3...P9, P0.

[0134] S403, the target image sequence is used as the third image training sample corresponding to the third visual feature type.

[0135] Here, regarding the implementation of steps S401-S403 above, it should be noted that: in order to balance the conflict between the richness of image sequence information and the reduction of GPU (graphics processing unit) memory usage (i.e., to reduce GPU memory usage while obtaining as much image sequence information as possible), in this embodiment, the above-mentioned image frame group to be sampled can be uniformly sampled according to a preset image sampling number threshold. This ensures that the sequence length of the target image sequence is not affected by the number of video image frames included in the image frame group to be sampled, so that regardless of whether the actual number of video image frames in the image frame group to be sampled is sufficient, a target image sequence with a sequence length equal to the preset image sampling number threshold can be obtained. That is, in addition to the method of obtaining the third image training sample shown in steps S401-S403 above, the third image training sample can also be obtained in other ways (e.g., using each video image frame included in the image frame group to be sampled as the third image training sample, etc.); this embodiment does not limit the specific method of obtaining the third image training sample.

[0136] Regarding the specific implementation process of step S102 above, when the search object belongs to a vehicle, the above-mentioned multiple visual feature types include at least the first visual feature type, the second visual feature type, and the third visual feature type. The following details the different implementation methods of step S102 under these three different visual feature types:

[0137] Regarding the aforementioned first visual feature type, in one optional implementation scheme, such as Figure 5 As shown, Figure 5 The diagram illustrates a flowchart of a method for encoding input first image training samples using a first image encoder, as provided in an embodiment of this application. Specifically, when executing step S102, the method includes step S501.

[0138] S501, the first image training sample corresponding to the first visual feature type is input into the first image encoder associated with the first visual feature type in the initial model to obtain the first visual feature vector of the search object output by the first image encoder.

[0139] Here, the first visual feature vector is used to characterize the appearance features of the retrieved object.

[0140] Specifically, the first image training sample can be obtained by referring to the method shown in steps S201-S202, and the repeated content will not be repeated here.

[0141] It should be noted that the first image encoder can be an image encoder within the CLIP model or other types of image encoders. This application does not limit the specific encoder type of the first image encoder.

[0142] Regarding the aforementioned second visual feature type, in one optional implementation, such as Figure 6 As shown, Figure 6 The diagram illustrates a flowchart of a method for encoding input second image training samples using a second image encoder, as provided in an embodiment of this application. Specifically, when executing step S102, the method includes step S601.

[0143] S601, the second image training sample corresponding to the second visual feature type is input into the second image encoder associated with the second visual feature type in the initial model to obtain the second visual feature vector of the search object output by the second image encoder.

[0144] Here, the second visual feature vector is used to characterize the environmental features surrounding the retrieved object.

[0145] Specifically, the second image training sample can be obtained by referring to the method shown in step S301, and the repeated content will not be repeated here.

[0146] It should be noted that the second image encoder can be an image encoder within the CLIP model or other types of image encoders. This application does not limit the specific encoder type of the second image encoder.

[0147] Regarding the aforementioned third visual feature type, in one optional implementation scheme, such as Figure 7 As shown, Figure 7 The diagram illustrates a flowchart of a method for encoding input third image training samples using a third image encoder, as provided in an embodiment of this application. Specifically, when executing step S102, the method includes step S701.

[0148] S701, input the training sample of the third image corresponding to the third visual feature type into the third image encoder associated with the third visual feature type in the initial model to obtain the third visual feature vector of the search object output by the third image encoder.

[0149] Here, the third visual feature vector is used to characterize the motion trajectory features of the retrieved object.

[0150] Specifically, the third image training sample can be obtained by referring to the method shown in steps S401-S403, and the repeated content will not be repeated here.

[0151] It should be noted that the third image encoder can be an image encoder within the CLIP model or other types of image encoders. This application does not limit the specific encoder type of the third image encoder.

[0152] In this embodiment of the application, in addition to directly using the output of the third image encoder as the third visual feature vector of the retrieval object, as another preferred embodiment, after executing step S701, the target visual feature vector that can more comprehensively reflect the motion trajectory features of the retrieval object can be obtained by following the methods shown in steps c1-c3. Specifically:

[0153] Step c1: Perform vector recombination twice on the third visual feature vector output by the third image encoder to obtain the first recombined vector and the second recombined vector corresponding to the third visual feature vector.

[0154] Here, taking the sequence length of the target image sequence as N, the image dimension of each video image frame in the target image sequence as C (equivalent to the number of feature channels as C), and the number of samples in each batch of training samples during the initial model training process as B (equivalent to the number of retrieval objects in each batch during training) as B, the vector feature dimension of the above-mentioned third visual feature vector output by the third image encoder after image encoding of the input third image training samples (i.e., the target image sequence) can be expressed as: B×N×C.

[0155] Specifically, as an optional embodiment, the third visual feature vector can be reshaped twice to obtain a first recombined vector (wherein, the vector feature dimension of the first recombined vector can be adaptively represented as B×N×1×C) and a second recombined vector (wherein, the vector feature dimension of the second recombined vector can be adaptively represented as B×1×N×C).

[0156] Step c2: Use the vector difference between the first recombined vector and the second recombined vector as the target difference vector.

[0157] Here, the target difference vector can be obtained by subtracting the first recombined vector and the second recombined vector obtained in step c1 above.

[0158] Step c3: Input the target difference vector into the difference network in the initial model to obtain the target visual feature vector output by the difference network; wherein, the target visual feature vector is used to characterize the motion trajectory features of the retrieved object.

[0159] Here, the above-mentioned difference network includes at least one convolutional layer and one dimensionality reduction pooling layer of AdaptiveAvgPooling2d (binary adaptive mean pooling layer); and in step c3 above, it is in the difference network after the target difference vector input value has been initialized.

[0160] Regarding the specific implementation process of step S102 above, based on the analysis at step S102, it can be seen that in this embodiment, the initial model includes multiple image encoders; wherein each image encoder corresponds to a preset visual feature type; in an optional implementation, such as Figure 8 As shown, Figure 8 This illustration shows a flowchart of a method for determining whether parameter sharing is required between different image encoders within an initial model, as provided in an embodiment of this application. The determination method includes steps S801-S802; specifically:

[0161] S801, for any two image encoders among the plurality of image encoders, when the visual feature types corresponding to each of the two image encoders both belong to the first feature type, it is determined that the two image encoders are trained in the manner of parameter sharing during the model training process of the initial model.

[0162] Here, the first feature type is used to characterize visual feature types that are independent of the motion state of the retrieved object.

[0163] In this embodiment, for the three image encoders (i.e., the first image encoder, the second image encoder, and the third image encoder), it can be determined that both the first visual feature type and the second visual feature type belong to the first feature type. Therefore, in the initial model, the first image encoder and the second image encoder can be adaptively set to be trained in a parameter-sharing manner during the model training process of the initial model. This allows the first image encoder and the second image encoder, which have the same image feature focus (i.e., both are used to focus on static image features in the input image), to learn to extract image features with as few parameters as possible under limited training data, which is beneficial to accelerate the convergence speed of the initial model.

[0164] S802, when the visual feature types corresponding to any two image encoders each include the second feature type, it is determined that the two image encoders are trained in a non-shared parameter manner during the model training process of the initial model.

[0165] Here, the second feature type is used to characterize the visual feature type related to the motion state of the retrieved object.

[0166] Specifically, for the three types of image encoders mentioned above, it can be determined that the third visual feature type belongs to the second feature type. Therefore, in the initial model, the third image encoder can be adaptively set to be trained with the first and second image encoders in a non-shared parameter manner during the model training process of the initial model. This makes the third image encoder different from the first and second image encoders in terms of the image features they focus on, which is beneficial for the third image encoder to effectively capture the motion trajectory of the retrieved object in the target image sequence.

[0167] The training method for the cross-modal retrieval model provided in this application involves obtaining image training samples corresponding to each visual feature type from the video image data corresponding to the retrieval object, based on a preset set of multiple visual feature types. For each visual feature type, the target image training sample is input into the target image encoder associated with that visual feature type within the initial model to obtain the target visual features of the retrieval object under that visual feature type, output by the target image encoder. The target visual features of the retrieval object under each visual feature type are then concatenated to obtain the visual description features of the retrieval object. The textual description information of the retrieval object is input into the text encoder within the initial model to obtain the textual description features of the retrieval object output by the text encoder. Based on the feature distance between the textual description features and the visual description features of the retrieval object, the model parameters of the initial model are adjusted to obtain a cross-modal retrieval model including the adjusted model parameters. In this way, without introducing redundant ReID tasks, this application can learn visual features under different visual feature types by specifically training the model, so that the model can obtain visual descriptive features that more comprehensively describe the retrieved object, thereby helping to improve the retrieval accuracy of the model for cross-modal retrieval of the retrieved object.

[0168] In this embodiment of the application, after the cross-modal retrieval model has been trained, the application phase of the cross-modal retrieval model proceeds as follows:

[0169] like Figure 9 As shown, Figure 9The diagram shows a flowchart of a cross-modal retrieval method provided in an embodiment of this application. The cross-modal retrieval method includes step S901; specifically:

[0170] S901, input the text description information required by the target object during the retrieval process into the pre-trained cross-modal retrieval model, and perform cross-modal retrieval between vision and language on the target object through the cross-modal retrieval model to obtain the image retrieval result for the target object output by the cross-modal retrieval model.

[0171] Here, taking vehicle retrieval as an example, the cross-modal retrieval model can perform cross-modal retrieval of the target vehicle from the vehicle image database used for vehicle retrieval based on the input text description information of the target vehicle. Based on the degree of matching between the visual description features of each candidate vehicle reflected in the vehicle image and the text description information of the target vehicle, the model outputs vehicle images of multiple candidate vehicles with a matching degree higher than a preset threshold, or outputs vehicle images of each candidate vehicle in descending order of matching degree as the vehicle retrieval result for the target vehicle.

[0172] It should be noted that the cross-modal retrieval model is obtained after training according to the above steps S101-S104; the specific training process of the cross-modal retrieval model can be referred to the specific implementation process of the above steps, and the repeated parts will not be repeated here.

[0173] The cross-modal retrieval method provided in this application allows for cross-modal retrieval of target objects after the cross-modal retrieval model has been trained. This improves the retrieval efficiency and accuracy of target objects.

[0174] Based on the same inventive concept, this application also provides a training device for a cross-modal retrieval model corresponding to the training method of the cross-modal retrieval model in the above embodiments. Since the principle of the training device in this application is similar to that of the training method in the above embodiments of this application, the implementation of the training device can refer to the implementation of the aforementioned training method, and the repeated parts will not be described again.

[0175] Reference Figure 10 As shown, Figure 10 This illustration shows a schematic diagram of a training device for a cross-modal retrieval model provided in an embodiment of this application; the training device includes:

[0176] The sample acquisition module 1001 is used to acquire image training samples corresponding to each visual feature type from the video image data corresponding to the search object according to a variety of preset visual feature types.

[0177] The visual feature extraction module 1002 is used to input the target image training sample corresponding to each visual feature type into the target image encoder associated with the visual feature type in the initial model for each visual feature type, to obtain the target visual features of the search object under the visual feature type output by the target image encoder, and to perform splicing processing on the target visual features of the search object under each visual feature type to obtain the visual description features of the search object.

[0178] The text feature extraction module 1003 is used to input the text description information of the search object into the text encoder in the initial model to obtain the text description features of the search object output by the text encoder;

[0179] The model training module 1004 is used to adjust the model parameters of the initial model based on the feature distance between the textual description features and the visual description features of the retrieval object, so as to obtain a cross-modal retrieval model including the adjusted model parameters.

[0180] In an optional implementation, when obtaining image training samples corresponding to each visual feature type from the video image data corresponding to the retrieval object according to a preset set of multiple visual feature types, the sample acquisition module 1001 is used to:

[0181] Based on the first visual feature type included in the multiple visual feature types, the video image frame in which the search object first appears is obtained from the video image data corresponding to the search object as the image frame to be processed; wherein, the first visual feature type is used to characterize the visual feature type to which the appearance feature of the search object belongs;

[0182] From the image frame to be processed, the bounding box image of the retrieval object is extracted as the first image training sample corresponding to the first visual feature type.

[0183] In an optional implementation, when obtaining image training samples corresponding to each visual feature type from the video image data corresponding to the retrieval object according to a preset set of multiple visual feature types, the sample acquisition module 1001 is used to:

[0184] Based on the second visual feature type included in the multiple visual feature types, video image frames containing the search object are extracted from the video image data corresponding to the search object as second image training samples corresponding to the second visual feature type; wherein, the second visual feature type is used to characterize the visual feature type to which the surrounding environment features of the search object belong.

[0185] In an optional implementation, when obtaining image training samples corresponding to each visual feature type from the video image data corresponding to the retrieval object according to a preset set of multiple visual feature types, the sample acquisition module 1001 is used to:

[0186] Based on the third visual feature type included in the multiple visual feature types, all video image frames including the search object are obtained from the video image data corresponding to the search object as a group of image frames to be sampled; wherein, the third visual feature type is used to characterize the visual feature type to which the motion trajectory feature of the search object belongs;

[0187] According to a preset image sampling number threshold, the image frame group to be sampled is uniformly sampled to obtain a target image sequence with the same sequence length as the image sampling number threshold;

[0188] The target image sequence is used as the third image training sample corresponding to the third visual feature type.

[0189] In an optional implementation, when the training samples of the target image corresponding to the visual feature type are input into the target image encoder associated with the visual feature type in the initial model to obtain the target visual features of the retrieval object under the visual feature type output by the target image encoder, the visual feature extraction module 1002 is used to:

[0190] The first image training sample corresponding to the first visual feature type is input into the first image encoder associated with the first visual feature type in the initial model to obtain the first visual feature vector of the search object output by the first image encoder; wherein, the first visual feature vector is used to characterize the appearance features of the search object.

[0191] In an optional implementation, when the training samples of the target image corresponding to the visual feature type are input into the target image encoder associated with the visual feature type in the initial model to obtain the target visual features of the retrieval object under the visual feature type output by the target image encoder, the visual feature extraction module 1002 is used to:

[0192] The second image training sample corresponding to the second visual feature type is input into the second image encoder associated with the second visual feature type in the initial model to obtain the second visual feature vector of the search object output by the second image encoder; wherein, the second visual feature vector is used to characterize the surrounding environment features of the search object.

[0193] In an optional implementation, when the training samples of the target image corresponding to the visual feature type are input into the target image encoder associated with the visual feature type in the initial model to obtain the target visual features of the retrieval object under the visual feature type output by the target image encoder, the visual feature extraction module 1002 is used to:

[0194] The training samples of the third image corresponding to the third visual feature type are input into the third image encoder associated with the third visual feature type in the initial model to obtain the third visual feature vector of the search object output by the third image encoder; wherein, the third visual feature vector is used to characterize the motion trajectory features of the search object.

[0195] In an optional implementation, when the text description information of the search object is input into the text encoder within the initial model to obtain the text description features of the search object output by the text encoder, the text feature extraction module 1003 is used to:

[0196] The text description information of the search object is input into the text encoder in the initial model to obtain the text description features of the search object output by the text encoder; wherein, the text description features include at least: a first text feature for describing the appearance of the search object, a second text feature for describing the surrounding environment of the search object, and a third text feature for describing the movement trajectory of the search object.

[0197] In an optional implementation, when adjusting the model parameters of the initial model based on the feature distance between the textual description features and the visual description features of the search object, the model training module 1004 is configured to:

[0198] When the feature distance between the text description feature and the visual description feature is less than or equal to the feature matching distance threshold, the model parameters of the initial model are adjusted according to the first adjustment strategy; wherein, the first adjustment strategy is used to characterize the model parameter adjustment strategy for reducing the feature matching loss between the text description feature and the visual description feature;

[0199] When the feature distance between the text description feature and the visual description feature is greater than the feature matching distance threshold, the model parameters of the initial model are adjusted according to the second adjustment strategy; wherein, the second adjustment strategy is used to characterize the model parameter adjustment strategy that increases the feature matching loss between the text description feature and the visual description feature.

[0200] In one optional implementation, the initial model includes multiple image encoders; wherein each image encoder corresponds to a preset visual feature type; the training device further includes:

[0201] The first determining module is used to determine, for any two image encoders among the plurality of image encoders, that when the visual feature types corresponding to each of the two image encoders belong to the first feature type, the two image encoders are trained in a parameter-sharing manner during the model training process of the initial model; wherein, the first feature type is used to characterize the visual feature type that is unrelated to the motion state of the retrieval object.

[0202] The second determining module is used to determine that when the visual feature types corresponding to any two image encoders each include the second feature type, the two image encoders are trained in a non-shared parameter manner during the model training process of the initial model; wherein the second feature type is used to characterize the visual feature type related to the motion state of the retrieved object.

[0203] The training apparatus for the cross-modal retrieval model provided in this application embodiment obtains image training samples corresponding to each visual feature type from the video image data corresponding to the retrieval object based on a preset set of multiple visual feature types. For each visual feature type, the target image training sample corresponding to that visual feature type is input into the target image encoder associated with that visual feature type within the initial model to obtain the target visual features of the retrieval object under that visual feature type, output by the target image encoder. The target visual features of the retrieval object under each visual feature type are then concatenated to obtain the visual description features of the retrieval object. The text description information of the retrieval object is input into the text encoder within the initial model to obtain the text description features of the retrieval object output by the text encoder. Based on the feature distance between the text description features and the visual description features of the retrieval object, the model parameters of the initial model are adjusted to obtain a cross-modal retrieval model including the adjusted model parameters. In this way, without introducing redundant ReID tasks, this application can learn visual features under different visual feature types by specifically training the model, so that the model can obtain visual descriptive features that more comprehensively describe the retrieved object, thereby helping to improve the retrieval accuracy of the model for cross-modal retrieval of the retrieved object.

[0204] Based on the same inventive concept, this application also provides a cross-modal retrieval device corresponding to the cross-modal retrieval method in the above embodiments. Since the principle of the cross-modal retrieval device in this application is similar to that of the cross-modal retrieval method in the above embodiments of this application, the implementation of the cross-modal retrieval device can refer to the implementation of the aforementioned cross-modal retrieval method, and the repeated parts will not be described again.

[0205] Reference Figure 11 As shown, Figure 11 This illustration shows a structural schematic diagram of a cross-modal retrieval device provided in an embodiment of this application; the cross-modal retrieval device includes:

[0206] The retrieval module 1101 is used to input the text description information required by the target object during the retrieval process into a pre-trained cross-modal retrieval model, and to perform cross-modal retrieval between vision and language on the target object through the cross-modal retrieval model to obtain the image retrieval results for the target object output by the cross-modal retrieval model.

[0207] In one optional implementation, the cross-modal retrieval model is obtained by training according to the training method in any of the optional implementations described in the above embodiments; repeated details will not be repeated here.

[0208] With the cross-modal retrieval device provided in the embodiments of this application, after the cross-modal retrieval model is trained, the cross-modal retrieval model can be used to perform cross-modal retrieval of target objects, which is beneficial to improving the retrieval efficiency and accuracy of target objects.

[0209] Based on the same inventive concept, this application also provides an electronic device corresponding to the training method of the cross-modal retrieval model described above. Since the principle of solving the problem by the electronic device in the embodiments of this application is similar to the training method of the cross-modal retrieval model described above in the embodiments of this application, the implementation of the electronic device can refer to the implementation of the training method described above, and the repeated parts will not be described again.

[0210] Figure 12 A schematic diagram of an electronic device 1200 provided in this application embodiment includes: a processor 1201, a memory 1202, and a bus 1203. The memory 1202 stores machine-readable instructions executable by the processor 1201. When the electronic device runs a training method for a cross-modal retrieval model as described in the embodiment, the processor 1201 communicates with the memory 1202 via the bus 1203. The processor 1201 executes the machine-readable instructions, wherein the processor 1201 executes the machine-readable instructions to implement the following steps:

[0211] Based on a variety of preset visual feature types, image training samples corresponding to each visual feature type are obtained from the video image data corresponding to the search object.

[0212] For each visual feature type, the target image training sample corresponding to that visual feature type is input into the target image encoder associated with that visual feature type in the initial model to obtain the target visual features of the retrieval object under that visual feature type output by the target image encoder. The target visual features of the retrieval object under each visual feature type are then spliced ​​to obtain the visual description features of the retrieval object.

[0213] The text description information of the search object is input into the text encoder in the initial model to obtain the text description features of the search object output by the text encoder;

[0214] Based on the feature distance between the textual description features and the visual description features of the retrieval object, the model parameters of the initial model are adjusted to obtain a cross-modal retrieval model including the adjusted model parameters.

[0215] In an optional implementation, when obtaining image training samples corresponding to each visual feature type from the video image data corresponding to the retrieval object according to a preset set of multiple visual feature types, the processor 1201 is specifically used for:

[0216] Based on the first visual feature type included in the multiple visual feature types, the video image frame in which the search object first appears is obtained from the video image data corresponding to the search object as the image frame to be processed; wherein, the first visual feature type is used to characterize the visual feature type to which the appearance feature of the search object belongs;

[0217] From the image frame to be processed, the bounding box image of the retrieval object is extracted as the first image training sample corresponding to the first visual feature type.

[0218] In an optional implementation, when obtaining image training samples corresponding to each visual feature type from the video image data corresponding to the retrieval object according to a preset set of multiple visual feature types, the processor 1201 is specifically used for:

[0219] Based on the second visual feature type included in the multiple visual feature types, video image frames containing the search object are extracted from the video image data corresponding to the search object as second image training samples corresponding to the second visual feature type; wherein, the second visual feature type is used to characterize the visual feature type to which the surrounding environment features of the search object belong.

[0220] In an optional implementation, when obtaining image training samples corresponding to each visual feature type from the video image data corresponding to the retrieval object according to a preset set of multiple visual feature types, the processor 1201 is specifically used for:

[0221] Based on the third visual feature type included in the multiple visual feature types, all video image frames including the search object are obtained from the video image data corresponding to the search object as a group of image frames to be sampled; wherein, the third visual feature type is used to characterize the visual feature type to which the motion trajectory feature of the search object belongs;

[0222] According to a preset image sampling number threshold, the image frame group to be sampled is uniformly sampled to obtain a target image sequence with the same sequence length as the image sampling number threshold;

[0223] The target image sequence is used as the third image training sample corresponding to the third visual feature type.

[0224] In an optional implementation, when the target image training sample corresponding to the visual feature type is input into the target image encoder associated with the visual feature type in the initial model to obtain the target visual feature of the retrieval object under the visual feature type output by the target image encoder, the processor 1201 is specifically used for:

[0225] The first image training sample corresponding to the first visual feature type is input into the first image encoder associated with the first visual feature type in the initial model to obtain the first visual feature vector of the search object output by the first image encoder; wherein, the first visual feature vector is used to characterize the appearance features of the search object.

[0226] In an optional implementation, when the target image training sample corresponding to the visual feature type is input into the target image encoder associated with the visual feature type in the initial model to obtain the target visual feature of the retrieval object under the visual feature type output by the target image encoder, the processor 1201 is specifically used for:

[0227] The second image training sample corresponding to the second visual feature type is input into the second image encoder associated with the second visual feature type in the initial model to obtain the second visual feature vector of the search object output by the second image encoder; wherein, the second visual feature vector is used to characterize the surrounding environment features of the search object.

[0228] In an optional implementation, when the target image training sample corresponding to the visual feature type is input into the target image encoder associated with the visual feature type in the initial model to obtain the target visual feature of the retrieval object under the visual feature type output by the target image encoder, the processor 1201 is specifically used for:

[0229] The training samples of the third image corresponding to the third visual feature type are input into the third image encoder associated with the third visual feature type in the initial model to obtain the third visual feature vector of the search object output by the third image encoder; wherein, the third visual feature vector is used to characterize the motion trajectory features of the search object.

[0230] In an optional implementation, when the text description information of the retrieved object is input into the text encoder within the initial model to obtain the text description features of the retrieved object output by the text encoder, the processor 1201 is specifically used for:

[0231] The text description information of the search object is input into the text encoder in the initial model to obtain the text description features of the search object output by the text encoder; wherein, the text description features include at least: a first text feature for describing the appearance of the search object, a second text feature for describing the surrounding environment of the search object, and a third text feature for describing the movement trajectory of the search object.

[0232] In an optional implementation, when adjusting the model parameters of the initial model based on the feature distance between the textual description features and the visual description features of the search object, the processor 1201 is specifically configured to:

[0233] When the feature distance between the text description feature and the visual description feature is less than or equal to the feature matching distance threshold, the model parameters of the initial model are adjusted according to the first adjustment strategy; wherein, the first adjustment strategy is used to characterize the model parameter adjustment strategy for reducing the feature matching loss between the text description feature and the visual description feature;

[0234] When the feature distance between the text description feature and the visual description feature is greater than the feature matching distance threshold, the model parameters of the initial model are adjusted according to the second adjustment strategy; wherein, the second adjustment strategy is used to characterize the model parameter adjustment strategy that increases the feature matching loss between the text description feature and the visual description feature.

[0235] In one optional implementation, the initial model includes a plurality of image encoders; wherein each image encoder corresponds to a preset visual feature type; the processor 1201 is further configured to:

[0236] For any two image encoders among the plurality of image encoders, if the visual feature types corresponding to each of the two image encoders belong to the first feature type, then it is determined that the two image encoders are trained in a parameter-sharing manner during the model training process of the initial model; wherein, the first feature type is used to characterize the visual feature type that is independent of the motion state of the retrieval object;

[0237] When any two image encoders include a second feature type in their respective visual feature types, it is determined that the two image encoders are trained in a non-shared parameter manner during the model training process of the initial model; wherein, the second feature type is used to characterize the visual feature type related to the motion state of the retrieved object.

[0238] The electronic device provided in this application, based on multiple preset visual feature types, obtains image training samples corresponding to each visual feature type from the video image data corresponding to the retrieval object. For each visual feature type, the target image training sample is input into the target image encoder associated with that visual feature type within the initial model to obtain the target visual features of the retrieval object under that visual feature type, output by the target image encoder. The target visual features of the retrieval object under each visual feature type are then concatenated to obtain the visual description features of the retrieval object. The text description information of the retrieval object is input into the text encoder within the initial model to obtain the text description features of the retrieval object output by the text encoder. Based on the feature distance between the text description features and the visual description features of the retrieval object, the model parameters of the initial model are adjusted to obtain a cross-modal retrieval model including the adjusted model parameters. In this way, without introducing redundant ReID tasks, this application can learn visual features under different visual feature types by specifically training the model, enabling the model to obtain a more comprehensive visual description of the retrieval object, thereby helping to improve the retrieval accuracy of the model in cross-modal retrieval of the retrieval object.

[0239] Based on the same inventive concept, embodiments of this application also provide a computer-readable storage medium storing a computer program, which is executed by a processor, wherein the processor performs the following steps:

[0240] Based on a variety of preset visual feature types, image training samples corresponding to each visual feature type are obtained from the video image data corresponding to the search object.

[0241] For each visual feature type, the target image training sample corresponding to that visual feature type is input into the target image encoder associated with that visual feature type in the initial model to obtain the target visual features of the retrieval object under that visual feature type output by the target image encoder. The target visual features of the retrieval object under each visual feature type are then spliced ​​to obtain the visual description features of the retrieval object.

[0242] The text description information of the search object is input into the text encoder in the initial model to obtain the text description features of the search object output by the text encoder;

[0243] Based on the feature distance between the textual description features and the visual description features of the retrieval object, the model parameters of the initial model are adjusted to obtain a cross-modal retrieval model including the adjusted model parameters.

[0244] In an optional implementation, when obtaining image training samples corresponding to each visual feature type from the video image data corresponding to the retrieval object according to a preset plurality of visual feature types, the processor is specifically used for:

[0245] Based on the first visual feature type included in the multiple visual feature types, the video image frame in which the search object first appears is obtained from the video image data corresponding to the search object as the image frame to be processed; wherein, the first visual feature type is used to characterize the visual feature type to which the appearance feature of the search object belongs;

[0246] From the image frame to be processed, the bounding box image of the retrieval object is extracted as the first image training sample corresponding to the first visual feature type.

[0247] In an optional implementation, when obtaining image training samples corresponding to each visual feature type from the video image data corresponding to the retrieval object according to a preset plurality of visual feature types, the processor is specifically used for:

[0248] Based on the second visual feature type included in the multiple visual feature types, video image frames containing the search object are extracted from the video image data corresponding to the search object as second image training samples corresponding to the second visual feature type; wherein, the second visual feature type is used to characterize the visual feature type to which the surrounding environment features of the search object belong.

[0249] In an optional implementation, when obtaining image training samples corresponding to each visual feature type from the video image data corresponding to the retrieval object according to a preset plurality of visual feature types, the processor is specifically used for:

[0250] Based on the third visual feature type included in the multiple visual feature types, all video image frames including the search object are obtained from the video image data corresponding to the search object as a group of image frames to be sampled; wherein, the third visual feature type is used to characterize the visual feature type to which the motion trajectory feature of the search object belongs;

[0251] According to a preset image sampling number threshold, the image frame group to be sampled is uniformly sampled to obtain a target image sequence with the same sequence length as the image sampling number threshold;

[0252] The target image sequence is used as the third image training sample corresponding to the third visual feature type.

[0253] In an optional implementation, when the training samples of the target image corresponding to the visual feature type are input into the target image encoder associated with the visual feature type in the initial model to obtain the target visual features of the retrieval object under the visual feature type output by the target image encoder, the processor is specifically used to:

[0254] The first image training sample corresponding to the first visual feature type is input into the first image encoder associated with the first visual feature type in the initial model to obtain the first visual feature vector of the search object output by the first image encoder; wherein, the first visual feature vector is used to characterize the appearance features of the search object.

[0255] In an optional implementation, when the training samples of the target image corresponding to the visual feature type are input into the target image encoder associated with the visual feature type in the initial model to obtain the target visual features of the retrieval object under the visual feature type output by the target image encoder, the processor is specifically used to:

[0256] The second image training sample corresponding to the second visual feature type is input into the second image encoder associated with the second visual feature type in the initial model to obtain the second visual feature vector of the search object output by the second image encoder; wherein, the second visual feature vector is used to characterize the surrounding environment features of the search object.

[0257] In an optional implementation, when the training samples of the target image corresponding to the visual feature type are input into the target image encoder associated with the visual feature type in the initial model to obtain the target visual features of the retrieval object under the visual feature type output by the target image encoder, the processor is specifically used to:

[0258] The training samples of the third image corresponding to the third visual feature type are input into the third image encoder associated with the third visual feature type in the initial model to obtain the third visual feature vector of the search object output by the third image encoder; wherein, the third visual feature vector is used to characterize the motion trajectory features of the search object.

[0259] In one optional implementation, when the text description information of the search object is input into the text encoder within the initial model to obtain the text description features of the search object output by the text encoder, the processor is specifically used to:

[0260] The text description information of the search object is input into the text encoder in the initial model to obtain the text description features of the search object output by the text encoder; wherein, the text description features include at least: a first text feature for describing the appearance of the search object, a second text feature for describing the surrounding environment of the search object, and a third text feature for describing the movement trajectory of the search object.

[0261] In an optional implementation, when adjusting the model parameters of the initial model based on the feature distance between the textual description features and the visual description features of the search object, the processor is specifically configured to:

[0262] When the feature distance between the text description feature and the visual description feature is less than or equal to the feature matching distance threshold, the model parameters of the initial model are adjusted according to the first adjustment strategy; wherein, the first adjustment strategy is used to characterize the model parameter adjustment strategy for reducing the feature matching loss between the text description feature and the visual description feature;

[0263] When the feature distance between the text description feature and the visual description feature is greater than the feature matching distance threshold, the model parameters of the initial model are adjusted according to the second adjustment strategy; wherein, the second adjustment strategy is used to characterize the model parameter adjustment strategy that increases the feature matching loss between the text description feature and the visual description feature.

[0264] In one optional implementation, the initial model includes multiple image encoders; wherein each image encoder corresponds to a preset visual feature type; the processor is further configured to:

[0265] For any two image encoders among the plurality of image encoders, if the visual feature types corresponding to each of the two image encoders belong to the first feature type, then it is determined that the two image encoders are trained in a parameter-sharing manner during the model training process of the initial model; wherein, the first feature type is used to characterize the visual feature type that is independent of the motion state of the retrieval object;

[0266] When any two image encoders include a second feature type in their respective visual feature types, it is determined that the two image encoders are trained in a non-shared parameter manner during the model training process of the initial model; wherein, the second feature type is used to characterize the visual feature type related to the motion state of the retrieved object.

[0267] In this embodiment, the computer-readable storage medium can also execute other machine-readable instructions when the processor runs, to perform the training method of the cross-modal retrieval model as described in other embodiments. For the specific training method steps and principles, please refer to the description of the method-side embodiment, which will not be repeated here.

[0268] Using the computer-readable storage medium provided in this application embodiment, image training samples corresponding to each visual feature type are obtained from the video image data corresponding to the retrieval object according to a preset set of multiple visual feature types. For each visual feature type, the target image training sample corresponding to that visual feature type is input into the target image encoder associated with that visual feature type in the initial model to obtain the target visual features of the retrieval object under that visual feature type, output by the target image encoder. The target visual features of the retrieval object under each visual feature type are then concatenated to obtain the visual description features of the retrieval object. The text description information of the retrieval object is input into the text encoder in the initial model to obtain the text description features of the retrieval object output by the text encoder. Based on the feature distance between the text description features and the visual description features of the retrieval object, the model parameters of the initial model are adjusted to obtain a cross-modal retrieval model including the adjusted model parameters. In this way, without introducing redundant ReID tasks, this application can learn visual features under different visual feature types by specifically training the model, enabling the model to obtain a more comprehensive visual description of the retrieval object, thereby helping to improve the retrieval accuracy of the model in cross-modal retrieval of the retrieval object.

[0269] Based on the same inventive concept, this application also provides another electronic device corresponding to the above-mentioned cross-modal retrieval method. Since the principle of solving the problem by the other electronic device in the embodiments of this application is similar to that of the above-mentioned cross-modal retrieval method in the embodiments of this application, the implementation of the other electronic device can refer to the implementation of the above-mentioned cross-modal retrieval method, and the repeated parts will not be described again.

[0270] Figure 13 A schematic diagram of another electronic device 1300 provided in this application embodiment includes: a processor 1301, a memory 1302, and a bus 1303. The memory 1302 stores machine-readable instructions executable by the processor 1301. When the electronic device runs a cross-modal retrieval method as described in the embodiment, the processor 1301 and the memory 1302 communicate via the bus 1303. The processor 1301 executes the machine-readable instructions, wherein the processor 1301 performs the following steps when executing the machine-readable instructions:

[0271] The textual description information required for the target object during the retrieval process is input into a pre-trained cross-modal retrieval model. The cross-modal retrieval model performs cross-modal retrieval of the target object between vision and language, and obtains the image retrieval results for the target object output by the cross-modal retrieval model.

[0272] In one optional implementation, the cross-modal retrieval model is obtained by training according to the training method in any of the optional implementations described in the above embodiments; repeated details will not be repeated here.

[0273] With the electronic device provided in the embodiments of this application, after the cross-modal retrieval model is trained, the cross-modal retrieval model can be used to perform cross-modal retrieval of target objects, which is beneficial to improving the retrieval efficiency and accuracy of target objects.

[0274] Based on the same inventive concept, this application also provides another computer-readable storage medium storing a computer program, which is executed by a processor, wherein the processor performs the following steps:

[0275] The textual description information required for the target object during the retrieval process is input into a pre-trained cross-modal retrieval model. The cross-modal retrieval model performs cross-modal retrieval of the target object between vision and language, and obtains the image retrieval results for the target object output by the cross-modal retrieval model.

[0276] In one optional implementation, the cross-modal retrieval model is obtained by training according to the training method in any of the optional implementations described in the above embodiments; repeated details will not be repeated here.

[0277] With the computer-readable storage medium provided in the embodiments of this application, after the cross-modal retrieval model is trained, the cross-modal retrieval model can be used to perform cross-modal retrieval of target objects, which is beneficial to improving the retrieval efficiency and accuracy of target objects.

[0278] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0279] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0280] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0281] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0282] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0283] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A training method for a cross-modal retrieval model, characterized in that, The training method includes: Based on a set of preset visual feature types, image training samples corresponding to each visual feature type are obtained from the video image data corresponding to the retrieval object; wherein, the set of visual feature types includes: a first visual feature type for characterizing the appearance features of the retrieval object, a second visual feature type for characterizing the surrounding environment features of the retrieval object, and a third visual feature type for characterizing the motion trajectory features of the retrieval object; For each visual feature type, the corresponding image training sample is input into the target image encoder associated with that visual feature type within the initial model. This yields the target visual features of the retrieval object under that visual feature type, output by the target image encoder. The target visual features of the retrieval object under each visual feature type are then concatenated to obtain the visual description features of the retrieval object. The initial model includes multiple image encoders and at least one text encoder, where the multiple image encoders represent the target image encoders corresponding to the various visual feature types. The text description information of the search object is input into the text encoder in the initial model to obtain the text description features of the search object output by the text encoder; Based on the feature distance between the textual description features and the visual description features of the retrieval object, the model parameters of the initial model are adjusted to obtain a cross-modal retrieval model including the adjusted model parameters.

2. The training method according to claim 1, characterized in that, The step of obtaining image training samples corresponding to each visual feature type from the video image data corresponding to the retrieval object based on a preset set of multiple visual feature types includes: Based on the first visual feature type included in the multiple visual feature types, the video image frame in which the search object first appears is obtained from the video image data corresponding to the search object as the image frame to be processed. From the image frame to be processed, the bounding box image of the retrieval object is extracted as the first image training sample corresponding to the first visual feature type.

3. The training method according to claim 1, characterized in that, The step of obtaining image training samples corresponding to each visual feature type from the video image data corresponding to the retrieval object based on a preset set of multiple visual feature types includes: Based on the second visual feature type included in the multiple visual feature types, video image frames containing the search object are extracted from the video image data corresponding to the search object as second image training samples corresponding to the second visual feature type.

4. The training method according to claim 1, characterized in that, The step of obtaining image training samples corresponding to each visual feature type from the video image data corresponding to the retrieval object based on a preset set of multiple visual feature types includes: Based on the third visual feature type included in the multiple visual feature types, all video image frames including the search object are obtained from the video image data corresponding to the search object as a group of image frames to be sampled. According to a preset image sampling number threshold, the image frame group to be sampled is uniformly sampled to obtain a target image sequence with the same sequence length as the image sampling number threshold; The target image sequence is used as the third image training sample corresponding to the third visual feature type.

5. The training method according to claim 2, characterized in that, The step of inputting the target image training samples corresponding to this visual feature type into the target image encoder associated with this visual feature type in the initial model to obtain the target visual features of the search object under this visual feature type output by the target image encoder includes: The first image training sample corresponding to the first visual feature type is input into the first image encoder associated with the first visual feature type in the initial model to obtain the first visual feature vector of the search object output by the first image encoder; wherein, the first visual feature vector is used to characterize the appearance features of the search object.

6. The training method according to claim 3, characterized in that, The step of inputting the target image training samples corresponding to this visual feature type into the target image encoder associated with this visual feature type in the initial model to obtain the target visual features of the search object under this visual feature type output by the target image encoder includes: The second image training sample corresponding to the second visual feature type is input into the second image encoder associated with the second visual feature type in the initial model to obtain the second visual feature vector of the search object output by the second image encoder; wherein, the second visual feature vector is used to characterize the surrounding environment features of the search object.

7. The training method according to claim 4, characterized in that, The step of inputting the target image training samples corresponding to this visual feature type into the target image encoder associated with this visual feature type in the initial model to obtain the target visual features of the search object under this visual feature type output by the target image encoder includes: The training samples of the third image corresponding to the third visual feature type are input into the third image encoder associated with the third visual feature type in the initial model to obtain the third visual feature vector of the search object output by the third image encoder; wherein, the third visual feature vector is used to characterize the motion trajectory features of the search object.

8. The training method according to claim 1, characterized in that, The step of inputting the text description information of the search object into the text encoder within the initial model to obtain the text description features of the search object output by the text encoder includes: The text description information of the search object is input into the text encoder in the initial model to obtain the text description features of the search object output by the text encoder; wherein, the text description features include at least: a first text feature for describing the appearance of the search object, a second text feature for describing the surrounding environment of the search object, and a third text feature for describing the movement trajectory of the search object.

9. The training method according to claim 1, characterized in that, The step of adjusting the model parameters of the initial model based on the feature distance between the textual description features and the visual description features of the search object includes: When the feature distance between the text description feature and the visual description feature is less than or equal to the feature matching distance threshold, the model parameters of the initial model are adjusted according to the first adjustment strategy; wherein, the first adjustment strategy is used to characterize the model parameter adjustment strategy for reducing the feature matching loss between the text description feature and the visual description feature; When the feature distance between the text description feature and the visual description feature is greater than the feature matching distance threshold, the model parameters of the initial model are adjusted according to the second adjustment strategy; wherein, the second adjustment strategy is used to characterize the model parameter adjustment strategy that increases the feature matching loss between the text description feature and the visual description feature.

10. The training method according to claim 1, characterized in that, The initial model includes multiple image encoders; each image encoder corresponds to a preset visual feature type; the training method further includes: For any two image encoders among the plurality of image encoders, if the visual feature types corresponding to each of the two image encoders belong to the first feature type, then it is determined that the two image encoders are trained in a parameter-sharing manner during the model training process of the initial model; wherein, the first feature type is used to characterize the visual feature type that is independent of the motion state of the retrieval object; When any two image encoders include a second feature type in their respective visual feature types, it is determined that the two image encoders are trained in a non-shared parameter manner during the model training process of the initial model; wherein, the second feature type is used to characterize the visual feature type related to the motion state of the retrieved object.

11. A cross-modal retrieval method, characterized in that, The cross-modal retrieval method includes: The textual description information required for the target object during the retrieval process is input into a pre-trained cross-modal retrieval model. The cross-modal retrieval model is used to perform cross-modal retrieval between vision and language for the target object, and the image retrieval result for the target object is obtained by the cross-modal retrieval model output by the cross-modal retrieval model. The cross-modal retrieval model is obtained by training according to any one of the training methods described in claims 1 to 10.

12. A training device for a cross-modal retrieval model, characterized in that, The training device includes: The sample acquisition module is used to acquire image training samples corresponding to each visual feature type from the video image data corresponding to the retrieval object according to a preset variety of visual feature types; wherein, the variety of visual feature types include: a first visual feature type for characterizing the appearance features of the retrieval object, a second visual feature type for characterizing the surrounding environment features of the retrieval object, and a third visual feature type for characterizing the motion trajectory features of the retrieval object; A visual feature extraction module is used to input the target image training samples corresponding to each visual feature type into the target image encoder associated with that visual feature type within the initial model, for each visual feature type, to obtain the target visual features of the retrieval object under that visual feature type, output by the target image encoder. The module then concatenates the target visual features of the retrieval object under each visual feature type to obtain the visual description features of the retrieval object. The initial model includes multiple image encoders and at least one text encoder, where the multiple image encoders represent the target image encoders corresponding to the various visual feature types. The text feature extraction module is used to input the text description information of the search object into the text encoder in the initial model to obtain the text description features of the search object output by the text encoder; The model training module is used to adjust the model parameters of the initial model based on the feature distance between the textual description features and the visual description features of the retrieval object, so as to obtain a cross-modal retrieval model including the adjusted model parameters.

13. A cross-modal retrieval device, characterized in that, The cross-modal retrieval device includes: The retrieval module is used to input the text description information required by the target object during the retrieval process into a pre-trained cross-modal retrieval model, and to perform cross-modal retrieval between vision and language on the target object through the cross-modal retrieval model to obtain the image retrieval results for the target object output by the cross-modal retrieval model; wherein, the cross-modal retrieval model is obtained after training according to any one of the training methods according to claims 1 to 10.

14. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the training method as described in any one of claims 1 to 10, or the steps of the cross-modal retrieval method as described in claim 11.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the training method as described in any one of claims 1 to 10, or the steps of the cross-modal retrieval method as described in claim 11.

Images