Model training method and device, equipment and storage medium

Abstract

The application relates to a model training method and device, equipment and a storage medium. The method maps coarse-grained features (global video features and global text features), so that the information amount of fine-grained features (fine-grained visual features and fine-grained text features) is greatly increased, thereby improving the performance of the model when fine-grained features are used to optimize the network parameters of a graph neural network and a multilayer perceptron.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computers, and in particular to a model training method and device, equipment and a storage medium. BACKGROUND

[0002] Vehicle retrieval is an important topic in the academic and industrial circles, and has been an active research field in the past decade. An effective vehicle retrieval mechanism based on video content can retrieve similar vehicles by calculating the affinity of vehicle pictures and video sequences, helping the police locate lost vehicles and catch suspects, even if the suspects change license plates. Such image-based vehicle retrieval can still have good retrieval effect.

[0003] In many cases, there are no available photos of suspect vehicles, only some related text descriptions, so self-supervised contrast learning is introduced. Self-supervised contrast learning is a kind of contrast learning without label participation in contrast learning. It can learn potential representation using unlabeled data, mainly solving the problem of human cost caused by manual labeling.

[0004] Most current self-supervised contrast learning methods are applied to image classification and image segmentation, and few are applied to vehicle retrieval methods. Moreover, current self-supervised learning methods based on vehicle retrieval often learn vehicle features from a coarse granularity, which affects the final performance. SUMMARY

[0005] The present application provides a model training method, device, equipment and storage medium to solve the problem of learning vehicle features from a coarse granularity, which affects the performance of the model.

[0006] In a first aspect, a model training method is provided, comprising:

[0007] obtaining a video global feature of a sample video and a text global feature of a text description matched with the sample video based on a graph neural network, the sample video and the text description;

[0008] mapping the video global feature and the text global feature into a contrast learning space respectively by a multilayer perceptron to obtain a fine-grained visual feature corresponding to the video global feature and a fine-grained text feature corresponding to the text global feature;

[0009] optimizing network parameters of the graph neural network and the multilayer perceptron using the fine-grained visual feature and the fine-grained text feature.

[0010] Optionally, obtaining a video global feature of a sample video and a text global feature of a text description matched with the sample video based on a graph neural network, the sample video and the text description, comprises:

[0011] Obtaining video coding features of the sample video and text coding features of the text description;

[0012] Based on the graph neural network, the video encoding features and the text encoding features, the video global features and the text global features are obtained.

[0013] Optionally, obtaining the video global feature and the text global feature based on the graph neural network, the video encoding feature, and the text encoding feature includes:

[0014] Processing the video encoding features using the graph neural network to obtain video hidden state features of the sample video; and processing the text encoding features using the graph neural network to obtain text hidden state features of the text description;

[0015] Average pooling is performed on the video hidden state features and the text hidden state features respectively to obtain the video global features and the text global features.

[0016] Optionally, obtaining the video coding features of the sample video includes:

[0017] Based on the description time of the text description, sampling the sample video to obtain an aligned sample video;

[0018] Extracting video coding features of the aligned sample videos;

[0019] The video coding features of the aligned sample videos are determined as the video coding features of the sample videos.

[0020] Optionally, sampling the sample video based on the description time of the text description to obtain an aligned sample video includes:

[0021] Determining a sampling reference timestamp for sampling the sample video based on the description time;

[0022] Taking the video frame corresponding to the sampling reference timestamp in the sample video as a sampling starting point, sampling the sample video to obtain a sample video with a preset number of frames;

[0023] The sampling sample video is determined to be the aligned sample video.

[0024] Optionally, the text description includes at least two natural language descriptions, and different descriptions in the at least two natural language descriptions correspond to different description angles.

[0025] Optionally, the graph neural network and the multilayer perceptron belong to network layers in a weakly supervised learning model.

[0026] In a second aspect, a model training apparatus is provided, comprising:

[0027] a processing module configured to obtain a video global feature of a sample video and a text global feature of a text description matched with the sample video based on a graph neural network, the sample video and the text description;

[0028] a mapping module configured to map the video global feature and the text global feature into a contrast learning space respectively by a multi-layer perceptron to obtain a fine-grained visual feature corresponding to the video global feature and a fine-grained text feature corresponding to the text global feature;

[0029] an optimization module configured to optimize network parameters of the graph neural network and the multi-layer perceptron by using the fine-grained visual feature and the fine-grained text feature.

[0030] In a third aspect, an electronic device is provided, comprising a processor, a memory and a communication bus, wherein the processor and the memory complete mutual communication through the communication bus;

[0031] the memory is configured to store a computer program;

[0032] the processor is configured to execute the program stored in the memory to implement the model training method of the first aspect.

[0033] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the model training method of the first aspect.

[0034] The above technical solution provided by the embodiments of the present application has the following advantages compared with the prior art: after obtaining the video global feature of the sample video and the text global feature of the text description, the method provided by the embodiments of the present application performs feature mapping on the video global feature and the text global feature by the multi-layer perceptron to obtain the fine-grained visual feature and the fine-grained text feature, and finally optimizes the network parameters of the graph neural network and the multi-layer perceptron by using the fine-grained visual feature and the fine-grained text feature. That is, the present application increases the information amount of the fine-grained feature (the fine-grained visual feature and the fine-grained text feature) by mapping the coarse-grained feature (the video global feature and the text global feature), thereby improving the performance of the model when the fine-grained feature is used to optimize the network parameters of the graph neural network and the multi-layer perceptron. BRIEF DESCRIPTION OF DRAWINGS

[0035] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

[0037] Figure 1 A flowchart of a model training method in an embodiment of the present application;

[0038] Figure 2 Another flowchart of a model training method in an embodiment of the present application;

[0039] Figure 3 A structural diagram of a model training device in an embodiment of the present application;

[0040] Figure 4 A structural diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the following will combine the drawings in the embodiments of the present application to make a clear and complete description of the technical solutions in the embodiments of the present application. Obviously, the described embodiments are a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without any creative effort fall within the scope of protection of the present application.

[0042] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to the process, method, product or device.

[0043] Self-supervised contrastive learning is a kind of unsupervised contrastive learning method in contrastive learning. It can learn potential representation using unlabeled data, mainly solving the problem of human cost caused by manual annotation. Since this type of contrastive learning method is unsupervised, one or more proxy tasks need to be constructed for model training. One of the mainstream methods is to evaluate the contrastive loss based on the feature representation of two images extracted from the encoder network. For example, a sample is extracted from the training dataset, and a transformed version of the sample is retrieved by applying appropriate data augmentation techniques. During training, the data augmentation version of the original sample is considered as a positive sample, while the remaining samples in the same batch / dataset are considered as negative samples. Next, the model is trained in a way that learns to distinguish between positive and negative samples. Through this process, the model can learn the representation information of the sample and transfer the representation information to the downstream task.

[0044] Most of the current self-supervised contrastive learning methods are applied to image classification and image segmentation, and few are applied to vehicle retrieval methods. The current self-supervised learning method based on vehicle retrieval often learns vehicle features from a coarse granularity, which affects the final performance.

[0045] In order to solve the problems in the related art, the embodiment of the present application provides a model training method, which can be applied to an electronic device. The electronic device can include a terminal device or a server, and the embodiment of the present application does not limit it. The terminal device can be a mobile terminal such as a mobile phone, a tablet computer, a notebook computer, a palm computer, a PAD (Personal Digital Assistant), etc., and a fixed terminal such as a digital TV, a desktop computer, etc.

[0046] As shown in Figure 1 the method can include the following steps:

[0047] Step S101, based on a graph neural network, a sample video and a text description matched with the sample video, obtaining a video global feature of the sample video and a text global feature of the text description.

[0048] In this embodiment, a vehicle text retrieval dataset (CityFlow Benchmark) can be collected and organized in advance. The vehicle text retrieval dataset includes a plurality of sample videos, and each sample video is used to express a part of the vehicle trajectory of at least one vehicle.

[0049] Current self-supervised contrast learning can effectively learn potential expression information from unlabeled data, but it only uses two-dimensional visual modalities, which are low semantic and high redundant information modalities. If such modalities are used alone, the model will have certain difficulties in the self-supervised learning process, and therefore, from experiments, it can often be seen that self-supervised learning converges slowly or unstably.

[0050] Among them, the two-dimensional visual modality is generally composed of multiple information points in units of pixels, and each information point represents a pixel value, which is only a value of 0-255 and does not have any high-level semantics. In detail, a single or sparse information point does not have any semantics. For example, if a picture has a main content (i.e., high-level semantic information) of a dog, a single pixel or a small part of pixels cannot express the theme content. Moreover, if a small part of pixels is removed from the picture, it will not affect the main content of the whole picture, which is a high redundancy.

[0051] In order to reduce the training difficulty of the visual model and obtain a better visual model, the text description in the embodiment is described by natural language. Natural language is an information modality formed by human experience accumulation, which has high-level semantics to some extent and has low redundant information properties. Therefore, natural language description is not like visual information. Sometimes a word is missing in a sentence, the overall expression will change, and language has various high-level semantics rather than a simple numerical value. Therefore, the text description realized by natural language description can effectively reduce the training difficulty of the visual model.

[0052] In the embodiment, in order to eliminate ambiguity as much as possible, multiple natural language descriptions can be used to describe the same sample video from different description angles. For example, three natural language descriptions of a sample video can be: a large green freight truck drives through an intersection, and many smaller cars drive in different directions; a large green 18-wheel flatbed truck is driving straight; a green semi-truck drives through an intersection.

[0053] In the embodiment, the video encoding features of the sample video and the text encoding features of the text description can be obtained first, and the video global features and the text global features can be obtained based on the graph neural network, the video encoding features and the text encoding features.

[0054] Among them, the video encoding features and the text encoding features can be features directly extracted by the neural network in the related art. For example, the video encoding features are features extracted from the sample video by a pre-trained image encoder, and the text encoding features are features extracted from the text description by a pre-trained text encoder.

[0055] In this embodiment, the processing result of the graph neural network on the video coding features (i.e., the video hidden state feature) can be directly used as the video global feature, and the processing result on the text coding features (the text hidden state feature) can be directly used as the text global feature.

[0056] In this embodiment, in order to encourage the graph neural network to learn the features at the label level, which is beneficial to action localization and action segmentation and better for retrieval of the target object, after obtaining the video hidden state feature and the text hidden state feature, the video hidden state feature and the text hidden state feature can be subjected to average pooling, and the average pooling result of the video hidden state feature can be used as the video global feature, and the average pooling result of the text hidden state feature can be used as the text global feature.

[0057] In a specific implementation process, in one optional embodiment, the video global feature and the text global feature are obtained based on the graph neural network, the video coding features and the text coding features, which includes: processing the video coding features by using the graph neural network to obtain the video hidden state feature of the sample video; and processing the text coding features by using the graph neural network to obtain the text hidden state feature of the text description; and performing average pooling on the video hidden state feature and the text hidden state feature respectively to obtain the video global feature and the text global feature.

[0058] In this embodiment, in order to accurately retrieve the target object from the sample video, the sample alignment and the time alignment of the text description can be performed in the process of obtaining the video coding features of the sample video. Since the time of the text description is usually shorter than the time of the sample video, if the alignment is performed based on the sample video, there may be no corresponding text of the text description for matching. Therefore, in this embodiment, the sample video and the text description are compared based on the text of the text description.

[0059] In a specific implementation process, in one optional embodiment, the video coding features of the sample video are obtained, which includes: sampling the sample video based on the description time of the text description to obtain an aligned sample video; extracting the video coding features of the aligned sample video; and determining the video coding features of the aligned sample video as the video coding features of the sample video.

[0060] In this embodiment, when the sample video is sampled based on the description time of the text description, a start timestamp (i.e., a sampling reference time) is sampled from the boundary of the description time of the text description as the center of the aligned sample video, and then the sample video is sampled based on a preset frame number, so that the aligned sample video that is aligned with the text description in time is obtained.

[0061] In an optional embodiment, the sample video is sampled based on the description time of the text description to obtain an aligned sample video, including: determining a sampling reference timestamp for sampling the sample video based on the description time; sampling the sample video from a video frame corresponding to the sampling reference timestamp in the sample video as a sampling starting point to obtain a sampling sample video with a preset number of frames; and determining the sampling sample video as the aligned sample video.

[0062] The preset number of frames can be determined artificially based on experience or determined artificially according to requirements. For example, the preset number of frames can be set to 32 frames.

[0063] In this embodiment, the sampling reference timestamp includes but is not limited to any time point in the description time. In an application, in order to improve the alignment degree of the aligned sample video and the text description as much as possible, the sampling reference timestamp can be set to the center time point of the description time.

[0064] In this embodiment, when the number of frames of the sample video is relatively small, so that it is difficult to obtain a sampling sample video with a preset number of frames by sampling the sample video, the number of missing frames is filled with zeros. Taking the preset number of frames as 32 frames as an example, when the sample video is sampled, the first and last frames of the sample video are retained, and 30 frames are randomly sampled to form a 32-frame sampling sample video. If it is difficult to obtain 30 video frames from the sample video, the number of missing frames can be filled with zeros to obtain a 32-frame sampling sample video.

[0065] In step S102, the video global feature and the text global feature are respectively mapped into a contrast learning space by a multi-layer perceptron to obtain a fine-grained visual feature corresponding to the video global feature and a fine-grained text feature corresponding to the text global feature.

[0066] In step S103, the fine-grained visual feature and the fine-grained text feature are used to optimize the network parameters of the graph neural network and the multi-layer perceptron.

[0067] In order to achieve the precision that can even exceed the precision achieved by a supervised learning method by relying on less manually labeled data, in this embodiment, the graph neural network and the multi-layer perceptron are used in the weakly supervised learning model, that is, the graph neural network and the multi-layer perceptron are network parameters in the weakly supervised learning model.

[0068] In the technical scheme provided by the embodiment, after obtaining the video global feature of the sample video and the text global feature of the text description, the video global feature and the text global feature are subjected to feature mapping through a multilayer perceptron, so as to obtain fine-grained visual features and fine-grained text features, and finally the network parameters of the graph neural network and the multilayer perceptron are optimized by using the fine-grained visual features and the fine-grained text features. That is, by mapping the coarse-grained features (the video global feature and the text global feature), the information amount of the fine-grained features (the fine-grained visual features and the fine-grained text features) is greatly increased, so that the performance of the model is improved when the network parameters of the graph neural network and the multilayer perceptron are optimized by using the fine-grained features.

[0069] The model training method is described below with reference to a specific embodiment. As shown in the following Figure 2 , the method can include the following steps:

[0070] Step 201: sample the sample video to obtain an aligned sample video of 32 frames.

[0071] Step 202: extract fine-grained visual features of the aligned sample video and fine-grained text features of the natural language description matched with the aligned sample video.

[0072] Step 202 can be specifically divided into the following parts when implemented:

[0073] The first part is to convert the aligned sample video and the text of the natural language description into video encoding features and text encoding features. The video encoding features x v are extracted by using an image encoder SE-ResNeXt 101 model pre-trained on an ImageNet dataset. The model has a small number of parameters and a high accuracy, and can well encode features. Similarly, the text encoding features x t are extracted by using a debertav3 as a sentence description, color and type description, and action description text encoder.

[0074] The second part is to input the video encoding features x v and the text encoding features x t extracted by the video encoder and the text encoder into a transformer network (graph neural network) and respectively to obtain video hidden state features h v and text hidden state features h t , and the formulas are as follows:

[0075]

[0076] The third part is to perform average pooling on the video hidden state features and the text hidden state features respectively, and the formulas are as follows:

[0077] z v = AvgPool(h v ), z t = AvgPool(h t )

[0078] where z v and z t are the video global feature and text global feature by average pooling, using average pooling to encourage and to learn the mark-level representation which is beneficial to action localization and action segmentation, and better for vehicle retrieval.

[0079] The fourth part, using MLP (Multi-Layer Perceptron) as the projection head, maps the video hidden state feature and the text hidden state feature into the space of contrastive representation learning. Using MLP to map the average-pooled video hidden state feature to get the global fine-grained visual feature e v which can be expressed as:

[0080] e v = W i σ(BN(W i z v )),

[0081] where σ is the ReLU layer with the output dimension of 512, z v is the video hidden state feature, W i is the parameter in MLP, and BN is the normalization operator.

[0082] Using the projection head MLP to map the average-pooled text hidden state feature to get the global fine-grained text feature e t which can be expressed as:

[0083] e t = W i σ(LN(W i z t )),

[0084] where σ is the ReLU layer with the output dimension of 512, z v is the text hidden state feature, W i is the parameter in MLP, and LN is the normalization operator.

[0085] Step 203, performing symmetric weighted InfoNCE loss to achieve good alignment of fine-grained visual features and fine-grained text features.

[0086]

[0087]

[0088] wherein, L is a total loss function, B is a small batch of data sets for each round of training, also known as batch; τ is a temperature variable hyperparameter.

[0089] Based on the same idea, the embodiment of the present application provides a model training device. The specific implementation of the device can refer to the description of the method embodiment part, and the repeated parts will not be repeated. As shown in the Figure 3 The device mainly includes:

[0090] The processing module 301 is configured to obtain video global features of the sample video and text global features of the text description based on the graph neural network, the sample video and the text description matched with the sample video.

[0091] The mapping module 302 is configured to map the video global features and the text global features into a contrast learning space by a multilayer perceptron respectively, to obtain fine-grained visual features corresponding to the video global features and fine-grained text features corresponding to the text global features.

[0092] The optimization module 303 is configured to optimize network parameters of the graph neural network and the multilayer perceptron by using the fine-grained visual features and the fine-grained text features.

[0093] Optionally, the processing module 301 is configured to:

[0094] Obtain video coding features of the sample video and text coding features of the text description.

[0095] Obtain the video global features and the text global features based on the graph neural network, the video coding features and the text coding features.

[0096] Optionally, the processing module 301 is configured to:

[0097] Process the video coding features by using the graph neural network to obtain video hidden state features of the sample video, and process the text coding features by using the graph neural network to obtain text hidden state features of the text description.

[0098] Perform average pooling on the video hidden state features and the text hidden state features respectively to obtain the video global features and the text global features.

[0099] Optionally, the processing module 301 is configured to:

[0100] Sample the sample video based on description time of the text description to obtain an aligned sample video.

[0101] Extract video coding features of the aligned sample video.

[0102] Determine the video coding features of the aligned sample videos, which are the video coding features of the sample videos.

[0103] Optionally, the processing module 301 is configured to:

[0104] Determining a sampling base timestamp for sampling the sample video based on the description time;

[0105] Taking the video frame corresponding to the sampling reference timestamp in the sample video as the sampling starting point, sampling the sample video to obtain a sample video with a preset number of frames;

[0106] Determine the sampling sample video as the alignment sample video.

[0107] Optionally, the text description includes at least two natural language descriptions, and different descriptions in the at least two natural language descriptions correspond to different description angles.

[0108] Optionally, graph neural networks and multilayer perceptrons are network layers in the weakly supervised learning model.

[0109] Based on the same concept, an electronic device is also provided in the embodiment of the present application, such as Figure 4 As shown, the electronic device mainly includes: a processor 401, a memory 402 and a communication bus 403, wherein the processor 401 and the memory 402 communicate with each other via the communication bus 403. The memory 402 stores a program that can be executed by the processor 401, and the processor 401 executes the program stored in the memory 402 to implement the following steps:

[0110] Based on the graph neural network, sample videos and text descriptions matching the sample videos, the video global features of the sample videos and the text global features of the text descriptions are obtained;

[0111] The global features of the video and the global features of the text are mapped to the contrastive learning space through a multi-layer perceptron, and the fine-grained visual features corresponding to the global features of the video and the fine-grained text features corresponding to the global features of the text are obtained;

[0112] The network parameters of graph neural networks and multilayer perceptrons are optimized using fine-grained visual features and fine-grained text features.

[0113] The communication bus 403 mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus 403 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation,Figure 4 Although only one bus is shown, it should be understood that the computer system 400 can contain more buses. Any buses included in the computer system 400 are represented by a bus 406, which is used to transmit data to system components.

[0114] The memory 402 can include a random access memory (RAM) and can also include a non-volatile memory, such as at least one disk storage. Alternatively, the memory can further be at least one storage device located remotely from the aforementioned processor 401.

[0115] The processor 401 described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc., and can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic component, a discrete hardware component.

[0116] In still another embodiment of the present application, a computer readable storage medium is also provided, and the computer readable storage medium stores a computer program. When the computer program runs on a computer, the computer is caused to execute the model training method described in the above embodiments.

[0117] In the embodiments described above, all or some of the steps can be implemented by software, hardware or firmware, or any combination thereof. When implemented by software, all or some of the steps can be implemented in the form of one or more computer programs. The computer program can be stored in any computer readable medium, and loaded into the computer for execution. The computer readable medium includes: a computer storage medium and a computer communication medium. The computer storage medium includes: volatile memory and non-volatile memory. The computer storage medium includes: an electric medium, a magnetic medium, an optical medium, and a solid medium. The computer communication medium includes: computer communication networks and computer networks. The computer program can be loaded into many different computers and different computers can be caused to perform according to the computer program.

[0118] It should be noted that the terms "first" and "second" and the like in this text are used only to distinguish one entity or action from another, and do not necessarily require or imply these entities or actions have any such actual relationship or order. Also, the terms "comprises", "comprising", or any other variations thereof are intended to cover a non-exclusive inclusion, so that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by the statement "comprising a" does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0119] The above description is merely one specific implementation of the application. Many modifications and variations of the described embodiments can be implemented by those skilled in the art, and all such modifications and variations are believed to be within the spirit and scope of the application as defined by the appended claims. The embodiments are therefore to be considered in all respects as illustrative and not restrictive, and all changes coming within the meaning and equivalency range of the appended claims are intended to be embraced therein.

Claims

1. A model training method, characterized in that, include: Based on a graph neural network, a sample video, and a text description matching the sample video, obtaining a video global feature of the sample video and a text global feature of the text description; the sample video is used to represent a vehicle trajectory of at least one vehicle; Mapping the video global features and the text global features into a contrastive learning space through a multi-layer perceptron, respectively, to obtain fine-grained visual features corresponding to the video global features and fine-grained text features corresponding to the text global features; The fine-grained visual features and the fine-grained text features are used to optimize the network parameters of the graph neural network and the multi-layer perceptron; the graph neural network is used for vehicle retrieval.

2. The method of claim 1, wherein, Based on the graph neural network, the sample video, and the text description matching the sample video, obtaining the video global features of the sample video and the text global features of the text description, including: Obtaining video coding features of the sample video and text coding features of the text description; Based on the graph neural network, the video encoding features and the text encoding features, the video global features and the text global features are obtained.

3. The method of claim 2, wherein, Based on the graph neural network, the video encoding features, and the text encoding features, obtaining the video global features and the text global features includes: Processing the video encoding features using the graph neural network to obtain video hidden state features of the sample video; and processing the text encoding features using the graph neural network to obtain text hidden state features of the text description; Average pooling is performed on the video hidden state features and the text hidden state features respectively to obtain the video global features and the text global features.

4. The method according to claim 2 or 3, characterized in that, Obtaining video coding features of the sample video, including: Based on the description time of the text description, sampling the sample video to obtain an aligned sample video; Extracting video coding features of the aligned sample videos; The video coding features of the aligned sample videos are determined as the video coding features of the sample videos.

5. The method of claim 4, wherein, Sampling the sample video based on the description time of the text description to obtain an aligned sample video includes: Determining a sampling reference timestamp for sampling the sample video based on the description time; Taking the video frame corresponding to the sampling reference timestamp in the sample video as a sampling starting point, sampling the sample video to obtain a sample video with a preset number of frames; The sampling sample video is determined to be the aligned sample video.

6. The method according to any one of claims 1 to 5, characterized in that, The text description includes at least two natural language descriptions, and different descriptions in the at least two natural language descriptions correspond to different description angles.

7. The method according to any one of claims 1 to 5, characterized in that, The graph neural network and the multilayer perceptron belong to the network layers in the weakly supervised learning model.

8. A model training apparatus characterized by comprising: include: a processing module configured to obtain, based on a graph neural network, a sample video, and a text description matching the sample video, a video global feature of the sample video and a text global feature of the text description; the sample video is used to represent a vehicle trajectory of at least one vehicle; The mapping module is configured to map the video global feature and the text global feature into a contrast learning space respectively by using a multi-layer perceptron, so as to obtain a fine-grained visual feature corresponding to the video global feature and a fine-grained text feature corresponding to the text global feature. The optimization module is configured to optimize network parameters of the graph neural network and the multi-layer perceptron by using the fine-grained visual feature and the fine-grained text feature, and the graph neural network is used for vehicle retrieval.

9. An electronic device, comprising: Comprise: A processor, a memory and a communication bus, wherein the processor and the memory complete mutual communication through the communication bus; The memory is configured to store a computer program; The processor is configured to execute the program stored in the memory, and realize the model training method in any one of claims 1-7.

10. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to realize the model training method in any one of claims 1-7.

Images