Model training method and image-text comparison method

Abstract

The embodiments of this specification provide a model training method and a text-image comparison method, wherein the model training method includes: obtaining sample text and sample video, and inputting them into a text-image comparison model; extracting word unit features and long sentence features of the sample text through a text encoder in the text-image comparison model, and extracting block features and video frame features of the sample video through a video encoder in the text-image comparison model; constructing short sentence features based on word unit features, constructing trajectory features based on block features, and constructing video features based on video frame features; aligning word unit features and block features to obtain a first alignment score, aligning short sentence features and trajectory features to obtain a second alignment score, and aligning long sentence features and video features to obtain a third alignment score; adjusting parameters of the text-image comparison model based on the first alignment score, the second alignment score, and the third alignment score until a target text-image comparison model that meets the training stop condition is obtained.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of machine learning technology, and in particular to a model training method and a picture-text comparison method. Background Art

[0002] With the development of Internet technology, machine learning has been applied in more and more fields. Video text retrieval needs to be combined with machine learning models to improve retrieval efficiency. In the video text retrieval scenario, video text retrieval is an important research task in the field of cross-modal content understanding, which requires finding the potential correlation between video and text data to achieve relatively accurate cross-matching. In the existing technology, the video text retrieval model is obtained by fine-tuning the comparative image and text pre-training model and the additional modules on the downstream video-text dataset, so as to utilize the knowledge learned from large-scale image and text pre-training in the video field. However, most of the existing methods directly align the overall features of the video and text, ignoring the fine-grained cross-modal relationship. As a result, the overall performance after the text and video are aligned is not good. In the video text retrieval scenario, it is difficult to achieve higher accuracy, which affects the use of downstream services; therefore, an effective solution is urgently needed to solve the above problems. Summary of the Invention

[0003] In view of this, embodiments of this specification provide a model training method. One or more embodiments of this specification also involve a model training device, an image-text comparison method, an image-text comparison device, a computing device, a computer-readable storage medium, and a computer program to address technical deficiencies in the prior art.

[0004] According to a first aspect of an embodiment of this specification, a model training method is provided, comprising:

[0005] Obtain sample text and sample video, and input them into the image-text comparison model;

[0006] Extracting word unit features and long sentence features of the sample text through the text encoder in the image-text comparison model, and extracting image block features and video frame features of the sample video through the video encoder in the image-text comparison model;

[0007] Constructing a short sentence feature based on the word unit feature, constructing a trajectory feature based on the image block feature, and constructing a video feature based on the video frame feature;

[0008] aligning the word unit feature and the image block feature to obtain a first alignment score, aligning the short sentence feature and the trajectory feature to obtain a second alignment score, and aligning the long sentence feature and the video feature to obtain a third alignment score;

[0009] Based on the first alignment score, the second alignment score, and the third alignment score, the image-text comparison model is adjusted until a target image-text comparison model that meets a training stop condition is obtained.

[0010] According to a second aspect of the embodiments of this specification, a model training device is provided, comprising:

[0011] An acquisition module is configured to acquire sample text and sample video and input them into the image-text comparison model;

[0012] an extraction module configured to extract word unit features and long sentence features of the sample text through a text encoder in the image-text comparison model, and to extract image block features and video frame features of the sample video through a video encoder in the image-text comparison model;

[0013] a construction module configured to construct a short sentence feature based on the word unit feature, construct a trajectory feature based on the image block feature, and construct a video feature based on the video frame feature;

[0014] an alignment module configured to align the word unit feature with the image block feature to obtain a first alignment score, align the short sentence feature with the trajectory feature to obtain a second alignment score, and align the long sentence feature with the video feature to obtain a third alignment score;

[0015] The training module is configured to adjust the parameters of the image-text comparison model based on the first alignment score, the second alignment score and the third alignment score until a target image-text comparison model that meets the training stop condition is obtained.

[0016] According to a third aspect of the embodiments of this specification, a method for comparing images and texts is provided, including:

[0017] Obtaining a video query request carrying user text, and determining a video set corresponding to the user text;

[0018] Calculating candidate scores between the user text and candidate videos in the video set using the target image-text comparison model in the above method;

[0019] At least one target video is selected from the video collection based on the candidate scores as a response to the video query request.

[0020] According to a fourth aspect of the embodiments of this specification, there is provided a device for comparing images and texts, comprising:

[0021] A video request acquisition module is configured to acquire a video query request carrying user text and determine a video set corresponding to the user text;

[0022] a candidate score calculation module configured to calculate the candidate scores between the user text and the candidate videos in the video set using the target image-text comparison model in the above method;

[0023] The target video screening module is configured to screen out at least one target video from the video set based on the candidate scores as a response to the video query request.

[0024] According to a fifth aspect of the embodiments of this specification, another image-text comparison method is provided, including:

[0025] Obtaining a text query request carrying a user video, and determining a text set corresponding to the user video;

[0026] Calculating the candidate scores between the user video and the candidate texts in the text set using the target image-text comparison model in the above method;

[0027] At least one target text is screened out from the text collection based on the candidate scores as a response to the text query request.

[0028] According to a sixth aspect of the embodiments of this specification, another image-text comparison method is provided, including:

[0029] A text request acquisition module is configured to acquire a text query request carrying a user video and determine a text set corresponding to the user video;

[0030] A candidate score calculation module is configured to calculate the candidate scores between the user video and the candidate texts in the text set using the target image-text comparison model in the above method;

[0031] The target text screening module is configured to screen out at least one target text from the text collection based on the candidate scores as a response to the text query request.

[0032] According to a seventh aspect of the embodiments of this specification, another model training method is provided, which is applied to a cloud-side device, including:

[0033] Obtain sample text and sample video submitted by the client device and input them into the image-text comparison model;

[0034] Extracting word unit features and long sentence features of the sample text through the text encoder in the image-text comparison model, and extracting image block features and video frame features of the sample video through the video encoder in the image-text comparison model;

[0035] Constructing a short sentence feature based on the word unit feature, constructing a trajectory feature based on the image block feature, and constructing a video feature based on the video frame feature;

[0036] aligning the word unit feature and the image block feature to obtain a first alignment score, aligning the short sentence feature and the trajectory feature to obtain a second alignment score, and aligning the long sentence feature and the video feature to obtain a third alignment score;

[0037] Adjusting parameters of the image-text comparison model based on the first alignment score, the second alignment score, and the third alignment score until a target image-text comparison model that meets a training stop condition is obtained;

[0038] The target model parameters corresponding to the target image-text comparison model are sent to the terminal side device.

[0039] According to a seventh aspect of the embodiments of this specification, another model training device is provided, which is applied to a cloud-side device, including:

[0040] A sample acquisition module is configured to acquire sample text and sample video submitted by a client device and input them into the image-text comparison model;

[0041] a feature extraction module configured to extract word unit features and long sentence features of the sample text through the text encoder in the image-text comparison model, and to extract image block features and video frame features of the sample video through the video encoder in the image-text comparison model;

[0042] a feature construction module configured to construct a short sentence feature based on the word unit feature, construct a trajectory feature based on the image block feature, and construct a video feature based on the video frame feature;

[0043] an alignment feature module configured to align the word unit feature with the image block feature to obtain a first alignment score, align the short sentence feature with the trajectory feature to obtain a second alignment score, and align the long sentence feature with the video feature to obtain a third alignment score;

[0044] a model training module configured to adjust parameters of the image-text comparison model based on the first alignment score, the second alignment score, and the third alignment score until a target image-text comparison model that meets a training stop condition is obtained;

[0045] The parameter sending module is configured to send the target model parameters corresponding to the target image-text comparison model to the terminal side device.

[0046] According to an eighth aspect of the embodiments of this specification, there is provided a computing device, including:

[0047] memory and processor;

[0048] The memory is used to store computer-executable instructions, and the processor is used to implement any step of the above-mentioned model training method or image-text comparison method when executing the computer-executable instructions.

[0049] According to the ninth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned model training method or image-text comparison method.

[0050] According to the tenth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned model training method or image-text comparison method.

[0051] The model training method provided in this embodiment can achieve higher retrieval accuracy in video text retrieval scenarios. After obtaining sample text and sample video, they can be input into the image-text comparison model to be trained. First, the text encoder in the image-text comparison model extracts the word unit features and long sentence features corresponding to the sample text, and the video encoder extracts the block features and video frame features of the sample video. Based on this, short sentence features, trajectory features, and video features are constructed. This allows the extraction of multimodal features corresponding to the sample text and sample video, facilitating the subsequent establishment of cross-modal relationships. Secondly, the word unit features and block features are aligned to obtain a first alignment score, the short sentence features and trajectory features are aligned to obtain a second alignment score, and the long sentence features and video features are aligned to obtain a third alignment score. Starting from the smallest granularity, the dependency relationship between features at different granularities is captured step by step from the bottom up. Finally, the image-text comparison model is trained based on the above alignment scores to obtain the target image-text comparison model that meets actual usage requirements. This achieves higher retrieval performance and ensures retrieval accuracy in video text retrieval scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a schematic diagram of a model training method provided by an embodiment of this specification;

[0053] Figure 2 This is a flowchart of a model training method provided by one embodiment of this specification;

[0054] Figure 3 This is a flowchart of a processing process of a model training method provided by one embodiment of this specification;

[0055] Figure 4 This is a schematic diagram of the structure of a model training device provided by one embodiment of this specification;

[0056] Figure 5This is a flow chart of a method for comparing images and texts provided in one embodiment of this specification;

[0057] Figure 6 This is a structural diagram of an image-text comparison device provided by an embodiment of this specification;

[0058] Figure 7 This is a flowchart of another image-text comparison method provided by an embodiment of this specification;

[0059] Figure 8 This is a structural diagram of another image-text comparison device provided by an embodiment of this specification;

[0060] Figure 9 is a flowchart of another model training method provided by one embodiment of this specification;

[0061] Figure 10 This is a schematic diagram of the structure of another model training device provided by one embodiment of this specification;

[0062] Figure 11 This is a structural block diagram of a computing device provided by one embodiment of this specification. DETAILED DESCRIPTION

[0063] The following description sets forth many specific details to facilitate a thorough understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0064] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0065] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0066] First, the terms involved in one or more embodiments of this specification are explained.

[0067] Cross-modal search: Cross-modal search involves searching data from at least two modalities, typically using one modality as a query to retrieve relevant data from another modality. By identifying potential connections between data from different modalities, relatively accurate cross-matching is achieved.

[0068] Video-text retrieval: Video-text retrieval is one of the subtasks of cross-modal retrieval, which requires retrieving videos based on query text, or retrieving text based on query videos.

[0069] Contrastive image-text pre-training: The contrastive image-text pre-training method uses collected image-text pairs as training data, making the paired image and text features as similar as possible and as far away from other features as possible, thereby learning cross-modal features with good generalization performance.

[0070] Transfer learning: Transfer learning is a machine learning method in which a pre-trained model is reused for another task.

[0071] Model fine-tuning: Model fine-tuning is a specific solution for transfer learning. Given a pre-trained model, some of the model's parameters are trained on a downstream dataset to make the model more suitable for specific downstream tasks.

[0072] Cross-modal alignment: Cross-modal alignment refers to bringing features extracted from inputs of different modalities with similar semantics closer together, and moving semantically dissimilar cross-modal features further apart.

[0073] Multi-granularity alignment: Multi-granularity alignment aims to extract features of different granularities from the objects to be aligned and construct multiple alignment relationships to achieve better alignment results.

[0074] This specification provides a model training method. One or more embodiments of this specification also involve a model training device, an image-text comparison method, an image-text comparison device, a computing device, a computer-readable storage medium, and a computer program, each of which is described in detail in the following embodiments.

[0075] See also Figure 1As shown in the schematic diagram, the model training method provided by this embodiment can achieve higher retrieval accuracy in video text retrieval scenarios. After obtaining sample text and sample video, they can be input into the image-text comparison model to be trained. First, the text encoder in the image-text comparison model extracts the word unit features and long sentence features corresponding to the sample text, and the video encoder extracts the block features and video frame features of the sample video. Based on this, short sentence features, trajectory features, and video features are constructed. This allows the extraction of multimodal features corresponding to the sample text and sample video, facilitating the subsequent establishment of cross-modal relationships. Secondly, the word unit features and block features are aligned to obtain a first alignment score, the short sentence features and trajectory features are aligned to obtain a second alignment score, and the long sentence features and video features are aligned to obtain a third alignment score. Starting from the smallest granularity, the dependency relationship between features at different granularities is captured step by step from the bottom up. Finally, the image-text comparison model is trained based on the above alignment scores to obtain the target image-text comparison model that meets actual usage requirements. This achieves higher retrieval performance and ensures retrieval accuracy in video text retrieval scenarios.

[0076] It should be noted that the embodiments of the present application may involve the use of user data. In actual applications, user-specific personal data can be used in the scheme described herein within the scope permitted by applicable laws and regulations, subject to the requirements of applicable laws and regulations of the country where the user is located (for example, with the user's explicit consent, effective notification to the user, etc.).

[0077] Figure 2 A flowchart of a model training method provided according to an embodiment of this specification is shown, which specifically includes the following steps.

[0078] Step S202: Obtain sample text and sample video, and input them into the image-text comparison model.

[0079] The model training method provided in this embodiment is applied to the image-text comparison model used in the video text retrieval scenario. It is used to realize that after the image-text comparison model is trained, it can establish a multi-granularity video and text alignment relationship, so that in the video text retrieval scenario, the trained image-text comparison model can be combined to accurately identify the video associated with the target text, or the target text of the associated video, to facilitate the use of downstream services.

[0080] Specifically, the sample text and sample video specifically refer to the text and video used to train the image-text comparison model. When the text and video have a corresponding relationship, they are a positive sample pair; when the text and video do not have a corresponding relationship, they are a negative sample pair. Among them, the sample text specifically refers to the text content that describes the sample video, such as describing the video playback content; correspondingly, the sample video can be an instructional video, a movie, an entertainment video, an animation video, etc., and this embodiment does not impose any restrictions on this. Correspondingly, the image-text comparison model specifically refers to a model that needs to be trained. The model can predict the matching score between the text and the video. The higher the score, the higher the degree of matching between the text and the video. Conversely, the lower the score, the lower the degree of matching between the text and the video. It should be noted that the image-text comparison model is a pre-trained model, which is fully trained in the model training method provided in this embodiment to achieve a target text comparison model with high prediction accuracy.

[0081] Based on this, after obtaining sample text and video, in order to train a model that meets the application requirements, they can be input into the image-text comparison model to be trained. First, the text encoder in the image-text comparison model extracts word unit features and long sentence features corresponding to the sample text, and the video encoder extracts image block features and video frame features from the sample video. Based on these features, short sentence features, trajectory features, and video features are constructed. This allows the extraction of multimodal features corresponding to the sample text and sample video, facilitating the subsequent establishment of cross-modal relationships. Next, the word unit features and image block features are aligned to obtain a first alignment score, the short sentence features and trajectory features are aligned to obtain a second alignment score, and the long sentence features and video features are aligned to obtain a third alignment score. Starting from the smallest granularity, the dependencies between features at different granularities are captured step by step from the bottom up. Finally, the image-text comparison model is trained based on these alignment scores to obtain the target image-text comparison model that meets the actual application requirements. This achieves higher retrieval performance and ensures retrieval accuracy in video text retrieval scenarios.

[0082] It should be noted that the sample videos and sample text can be extracted from a pre-built sample set or uploaded by a party with model training needs. The image-text comparison model training service provided by the service provider is required. Specific applications can be tailored to actual needs and are not limited in this embodiment.

[0083] Step S204 , extracting word unit features and long sentence features of the sample text through the text encoder in the image-text comparison model, and extracting image block features and video frame features of the sample video through the video encoder in the image-text comparison model.

[0084] Specifically, after obtaining the sample video and sample text as described above, in order to further achieve cross-modal alignment of text and video from different granularity perspectives and implement application in video-text indexing scenarios, the text encoder in the image-text comparison model can first extract the word unit features and long sentence features corresponding to the sample text, and the video encoder can extract the image block features and video frame features corresponding to the sample video to facilitate subsequent use. In other words, the encoder in the image-text comparison model needs to extract the basic features corresponding to the sample for subsequent modeling and alignment.

[0085] Among them, the text encoder specifically refers to an encoder that can extract the basic features corresponding to the sample text. Correspondingly, the video encoder specifically refers to an encoder that can extract the basic features corresponding to the sample video. Correspondingly, the word unit feature specifically refers to the vector expression corresponding to each word in the sample text; the long sentence feature specifically refers to the vector expression corresponding to the sample text itself; the block feature specifically refers to the vector expression corresponding to the block contained in the video frame; the video frame feature specifically refers to the vector expression corresponding to each video frame after the sample video is framed.

[0086] Furthermore, in the process of extracting the block features and video frames corresponding to the sample video through the video encoder, considering that the sample video corresponds to a large number of video frames and adjacent video frames are similar, when extracting the block features and video frames, sampling can be performed first, and the features can be extracted after encoding. In this embodiment, the specific implementation method is as follows:

[0087] The sample video is sampled by the video encoder in the image comparison model to obtain a video frame sequence composed of video frames; and the block features and the video frame features are extracted from the video frames included in the video frame sequence by the video encoder.

[0088] Specifically, the video frame sequence refers to a set of video frames obtained after sampling the sample video, and the number of video frames included in the video frame sequence is less than or equal to the total number of video frames obtained after frame processing of the sample video.

[0089] Based on this, when encoding the sample video through the text encoder in the image-text comparison model, in order to extract the block features and video frame features that represent the sample text for subsequent use, the sample video can be sampled first through the video encoder in the image comparison model to achieve uniform sampling of the sample video to obtain a video frame sequence composed of video frames; then, the video encoder is used to extract the block features and video frame features from the video frames contained in the video frame sequence.

[0090] In actual applications, when using the text encoder and video encoder provided by the image-text comparison pre-training model to extract basic features from the input data, for the text side, the text encoder will extract word unit features from the input sample text. and long sentence features, t s ∈R d ; Where T represents the matrix of sample text features, its superscript w represents word, that is, the sample text word unit; correspondingly, t represents the matrix vector of word unit features, N w Indicates the number of word units, d t The dimension representing the word unit features. s The vector representing the long sentence feature, s represents sentence, and d is the feature dimension shared by the long sentence feature and the video frame feature.

[0091] For the video side, the video encoder will uniformly sample F frames of the input sample video and extract the block features and and video frame features Among them, V represents the matrix of video features, its superscript p represents patch, and j represents the jth frame in the video; N p Indicates the number of tiles in each frame, d v Represents the dimension of the tile features, A vector representing the features of the j-th video frame.

[0092] For example, the sample text T and the sample video S are input into the image-text comparison model. The sample text T is first encoded by the text encoder in the image-text comparison model to extract the features of each word unit in the sample text and the long sentence features corresponding to the entire sample text T; at the same time, the sample video S is encoded by the video encoder to uniformly sample F frames of the sample video, and then extract the video frame features of each frame and the block features corresponding to the blocks in each video frame for subsequent use.

[0093] In summary, in order to realize the subsequent multi-granularity cross-modal text-video matching calculation and improve the training accuracy of the model based on this, the video frames in the sample video can be uniformly sampled and then decoded when extracting block features and video frame features, so as to obtain block features and video frame features that meet the usage requirements.

[0094] Step S206 , constructing a short sentence feature based on the word unit feature, constructing a trajectory feature based on the image block feature, and constructing a video feature based on the video frame feature.

[0095] Specifically, after obtaining word unit features and long sentence features, as well as image block features and video frame features through the text encoder and video encoder in the above-mentioned image-text comparison model, in order to further improve the prediction accuracy after model training and to achieve alignment of text and video from multiple dimensions, features corresponding to different granularities can be constructed before the model is formally adjusted to facilitate subsequent use. Therefore, short sentence features can be constructed based on word unit features, trajectory features can be constructed based on image block features, and video features can be constructed based on video frame features. Feature expressions for text and video can be constructed separately at different granularities to facilitate subsequent use. Alignment processing can be completed at multiple granularities so that the model can learn this capability and use it in the application stage to improve the accuracy of the model's prediction of the matching degree between video and text.

[0096] Among them, short sentence features specifically refer to the vector expression obtained after reconstructing and splicing word unit features, and it should be noted that the short sentence content corresponding to the short sentence feature is smaller than the long sentence content corresponding to the long sentence feature; correspondingly, trajectory features specifically refer to the vector expression obtained after reconstructing and splicing block features, and it should be noted that trajectory features reflect the vector expression of the trajectory corresponding to the same object in the video, and when constructing trajectory features based on block features, it is actually obtained by combining blocks containing the same object in a set manner. Correspondingly, video features specifically refer to the vector expression obtained after reconstructing and splicing video frame features, and it should be noted that video features represent the features of sample videos.

[0097] Furthermore, when constructing short sentence features, we actually start from the most basic word unit features and perform feature splicing, thereby completing feature construction at a medium-granularity dimension to support subsequent multi-granularity cross-modal alignment processing; in this embodiment, the specific implementation method is as follows:

[0098] The word unit features are input into the temporal convolutional network unit in the image-text comparison model to construct sentences; the short sentence features are generated and output according to the sentence construction results; wherein the temporal convolutional network unit constructs sentences according to the set dimension and the set step size.

[0099] Specifically, the temporal convolutional network unit specifically refers to a temporal convolutional network CNN (Convolutional Neural Network) that can reconstruct word unit features; correspondingly, the set dimension specifically refers to the dimension of the temporal convolutional network when performing convolution processing, and the set step size specifically refers to the moving step size during convolution processing.

[0100] Based on this, when constructing short sentence features based on word unit features, the sub-word unit features contained in the word unit features are actually reconstructed and spliced. That is, the word unit features are input into the temporal convolutional network unit in the comparison model to construct sentences; so as to generate short sentence features according to the sentence construction results and output them; and during processing, the temporal convolutional network unit constructs sentences according to the set dimension and set step size. Among them, the set dimension and the set step size can be set according to actual needs, such as the dimension is set to one dimension and the step size is set to 2h+1. This embodiment does not impose any restrictions on this.

[0101] In summary, by using a temporal convolutional network to process word unit features according to the set step size and dimension, we can obtain a feature expression that can fully characterize the state of objects described in the text. Based on this, we can combine the features expressing the state of objects in subsequent alignment to improve the prediction accuracy of the image-text comparison model.

[0102] Furthermore, when constructing trajectory features, we actually start from the most basic tile features and perform feature splicing, thereby completing feature construction at a medium-granularity dimension to support subsequent multi-granularity cross-modal alignment processing. In this embodiment, the specific implementation method is as follows:

[0103] Clustering the sub-block features contained in the block features to obtain a block feature cluster based on the clustering result; inputting the sub-block features in the block feature cluster into the first feature construction unit in the image-text comparison model to perform trajectory construction, and generating the trajectory feature based on the trajectory construction result.

[0104] Specifically, the sub-tile feature refers to the vector expression corresponding to each tile. Correspondingly, clustering refers to aggregating tile features that may be of the same object across time and space dimensions to form a tile feature cluster. Correspondingly, the first feature construction unit refers to a unit that can reconstruct and splice sub-tile features, which can be implemented using Transformer.

[0105] Based on this, when constructing trajectory features based on block features, the sub-block features contained in the block features can be clustered to obtain block feature clusters based on the clustering results; on this basis, the sub-block features in the block feature cluster can be input into the first feature construction unit in the image-text comparison model for trajectory construction, and trajectory features can be generated based on the trajectory construction results, which is convenient for subsequent use.

[0106] In practical applications, considering that short sentences can express the state of an object in text and trajectories can express the position change of an object in a video, aligning the short sentence features and trajectory features can ensure that the model can more accurately learn to align cross-modal information. For the text side, a one-dimensional temporal convolutional network (CNN) with a step size of 2h+1 can be embedded in the image-text comparison model to construct short sentence features based on word unit features. The short sentence features are i represents the i-th word unit extracted from N word units.

[0107] For the video side, in order to accurately express the position change of the same object through the image blocks, all the image block features corresponding to the sample video can be clustered first to find the image blocks that may contain the same object across time and space. For the i-th cluster obtained by clustering, the Transformer in the image-text comparison model is used to perform spatiotemporal modeling on the image blocks in the cluster and take the average as the trajectory feature, where the trajectory feature is

[0108] Continuing with the previous example, after obtaining word unit features, we can use the temporal convolutional network (CNN) in the image-text comparison model to construct sentences from all word unit features, creating short sentence features of a set length for later use. At the same time, we cluster the image block features, and then generate clusters based on the clustering. By using the Transformer to perform spatiotemporal modeling on the image block features within the clusters and taking the average, we can obtain trajectory features for later use.

[0109] In summary, by using the first feature construction unit to process the block features, it is possible to obtain features that can fully characterize the position changes of objects in the video. Based on this, in subsequent alignment, the features that express the position changes of objects can be combined to improve the prediction accuracy of the image-text comparison model.

[0110] Furthermore, when constructing video features, the purpose is to complete alignment processing in a higher dimension. In this embodiment, the specific implementation is as follows:

[0111] The video frame features are input into the second feature construction unit in the image-text comparison model to perform video construction; and the video features are generated according to the video construction results.

[0112] Specifically, the second feature construction unit is a unit that reconstructs and splices video frame features, and can be implemented using a Transformer. Based on this, when constructing video features, the video frame features can be input into the second feature construction unit in the image-text comparison model for video construction. This allows for the generation of video features based on the video construction results, facilitating subsequent use.

[0113] In practical applications, in order to achieve the alignment of videos and long sentences in a coarse-grained dimension, the Transformer embedded in the image-text comparison model can be used to perform temporal modeling on the frame features of the entire video, and finally the average is taken to integrate the video features; among them, the video features are

[0114] Continuing with the previous example, in order to enable the image-text comparison model to learn multi-granularity alignment capabilities, we can construct video features of the sample video based on the video frame features. That is, through the Transformer embedded in the image-text comparison model, the frame features of the entire video are temporally modeled and finally the average is taken to integrate into the video features for subsequent use.

[0115] In summary, by using the second feature construction unit to process the video frame features, it is possible to obtain features that can fully characterize the characteristics of the video. Based on this, it is possible to combine the expression of video features in subsequent alignment to improve the prediction accuracy of the image-text comparison model.

[0116] Step S208 , aligning the word unit feature and the image block feature to obtain a first alignment score, aligning the short sentence feature and the trajectory feature to obtain a second alignment score, and aligning the long sentence feature and the video feature to obtain a third alignment score.

[0117] Specifically, after obtaining the word unit features, short sentence features and long sentence features corresponding to the sample text, and the block features, trajectory features and video features corresponding to the sample video, in order to achieve cross-modal alignment of text and video from multiple granularity angles, so that the model can learn the ability of cross-modal alignment, it is necessary to first perform cross-modal alignment processing at different granularities, and obtain the alignment score based on the alignment processing results, which is used to subsequently complete the model parameter adjustment, thereby obtaining the target image-text comparison model that meets the usage requirements.

[0118] Among them, the first alignment score specifically refers to the score obtained after aligning the basic features at the finest granularity, that is, the total matching score obtained after many-to-many alignment of word unit features and tile features; correspondingly, the second alignment score specifically refers to the score obtained after aligning the intermediate features at the medium granularity, that is, the total matching score obtained after many-to-many alignment of trajectory features and short sentence features; correspondingly, the third alignment score specifically refers to the score obtained after aligning the overall features at the coarsest granularity, that is, the total matching score obtained after one-to-one alignment of long sentence features and video features.

[0119] It should be noted that when aligning text and video at different granularities, it is a cross-modal alignment process, so it needs to be implemented by calculating the matching score between features.

[0120] Furthermore, when calculating the first alignment score, the alignment between the image block features and the word unit features is the finest-grained alignment process. In this embodiment, the specific implementation is as follows:

[0121] Calculate the tile importance corresponding to the sub-tile feature in the tile feature; filter the target sub-tile feature in the tile feature according to the tile importance to form a target tile feature; and calculate the fine-grained feature score between the word unit feature and the target tile feature according to a preset many-to-many calculation strategy as the first alignment score.

[0122] Specifically, the tile importance refers to the importance corresponding to each sub-tile feature, which can reflect the importance of each sub-tile feature, so that important tile features can be screened out for use according to the tile importance, that is, the screened out important tile features are the target sub-tile features; correspondingly, the many-to-many calculation strategy specifically refers to a strategy for many-to-many alignment of sub-tile features and sub-word unit features, which is used to calculate multiple fine-grained feature scores and then obtain the first alignment score by taking the total score.

[0123] Based on this, when aligning the tile features and word unit features, taking into account the large number of sub-tile features contained in the tile features, and the different importance of each tile compared to the video, the role they play during alignment is also different. Therefore, it is necessary to first calculate the tile importance corresponding to each sub-tile feature, and then filter according to the tile importance corresponding to each sub-tile feature to obtain the target sub-tile feature to form the target tile feature; on this basis, a many-to-many alignment process can be performed, that is, according to the preset many-to-many calculation strategy, the sub-fine-grained feature scores between the sub-word unit features in the word unit features and the target sub-tile features in the target tile features are calculated, so as to summarize all the sub-fine-grained feature scores according to the calculation results to obtain the first alignment score.

[0124] Before the alignment process, in order to ensure alignment accuracy, the tile importance can be calculated in the following way; in this embodiment, the specific implementation is as follows:

[0125] The sub-block features contained in the block feature are spliced to obtain a block feature sequence; the block feature sequence is input into the multi-layer perceptron network unit in the image-text comparison model to calculate the importance, and the importance calculation result is processed by the normalization unit in the image-text comparison model; and the block importance corresponding to the sub-block feature in the block feature is determined according to the processing result.

[0126] Specifically, the tile feature sequence specifically refers to the result obtained by splicing the tile features of the current frame in the spatial dimension. Correspondingly, the multi-layer perceptron network unit specifically refers to the multi-layer perceptron network MLP (Multilayer Perceptron) embedded in the image-text comparison model; correspondingly, the normalization unit specifically refers to the Softmax layer embedded in the image-text comparison model.

[0127] Based on this, when calculating the importance of sub-tile features, the sub-tile features contained in the tile feature can be concatenated to obtain a tile feature sequence. This tile feature sequence is then input into the multi-layer perceptron network unit in the image-text comparison model for importance calculation. This yields an importance score for each sub-tile feature. The importance calculation result is then processed by the normalization unit in the image-text comparison model. Based on the processing result, the tile importance corresponding to the sub-tile feature in the tile feature can be determined. This facilitates the subsequent calculation of the first alignment score in combination with the tile importance corresponding to the sub-tile feature.

[0128] In practical applications, considering that the sample video contains a large number of tiles, the information described by a large number of tiles may be irrelevant to the text semantics, and there may also be noise. Therefore, in order to align the basic feature dimensions and the word unit-tiles before aligning them, a tile selection module can be added to the image-text comparison model to predict the importance of each tile, so as to retain important tiles with rich information according to the screening results, while discarding unimportant tiles.

[0129] Specifically, for all the tiles on a given video frame, the features of the video frame are broadcasted in the spatial dimension and the tile features are spliced. The splicing results are then input into the multi-layer perceptron network (MLP) in the image-text comparison model. After being processed by the multi-layer perceptron network, the Softmax layer in the image-text comparison model is used to predict the importance of the tile, and the tile is obtained. Importance in, represents the importance matrix expression corresponding to the j-th tile, and [] refers to splicing on the channel dimension.

[0130] On this basis, after obtaining the importance corresponding to all tiles, the importance of all tiles compared to the sample video can be represented. In order to be able to screen out tiles with higher importance than the sample video and align them with word unit features, the top-k tiles can be selected according to the importance sorting results as the k tiles retained in each video frame.

[0131] Finally, when aligning the tile features and word unit features obtained above, the matching score of this stage is actually obtained through a many-to-many calculation method. That is, the matching score between each tile and all word unit features is calculated in turn as the sub-fine-grained feature score, and summarized after all calculations are completed to obtain the first alignment score between the tile features and the word unit features, so as to achieve the smallest granularity semantic information alignment that can be considered in retrieval.

[0132] The first alignment score can be calculated using the following formula:

[0133]

[0134] Among them, s w-p The first alignment score is obtained by aligning word unit features and image patch features, as can be determined from the superscripts; wp refers to word (word unit) and patch (image patch) alignment. It should be noted that the meanings of letters not explained here can be referred to the same descriptions in this embodiment and will not be repeated here.

[0135] In summary, when aligning tile features and word unit features at the minimum granularity, the target tile features can be first screened out from the tile features according to their importance, so as to discard tile features that have low relevance to the video and contain noise, thereby ensuring that accurate alignment scores can be obtained after comparison at the minimum granularity, so as to promote the improvement of the model's prediction accuracy at the minimum granularity dimension during the model parameter adjustment stage.

[0136] Furthermore, when calculating the second alignment score, the alignment between the trajectory block features and the short sentence features is a medium-granularity alignment process. In this embodiment, the specific implementation is as follows:

[0137] Determine a sub-sentence feature contained in the sentence feature and a sub-trajectory feature contained in the trajectory feature; calculate a sub-medium-granularity feature score between the sub-sentence feature in the sentence feature and the sub-trajectory feature in the trajectory feature according to a preset many-to-many calculation strategy; generate a medium-granularity feature score based on the sub-medium-granularity feature score and use it as the second alignment score.

[0138] Specifically, the sub-sentence feature refers to the vector expression corresponding to each sentence in the sample text; correspondingly, the sub-trajectory feature refers to the vector expression corresponding to the motion trajectory of each object in the video; correspondingly, the many-to-many calculation strategy refers to a strategy for performing many-to-many alignment processing on the sub-sentence features and the sub-trajectory features, which is used to calculate multiple medium-granularity feature scores and then obtain the second alignment score by taking the total score.

[0139] Based on this, when aligning trajectory features and short sentence features, a many-to-many alignment process can be performed, that is, first determine the sub-sentence features contained in the short sentence features, and the sub-trajectory features contained in the trajectory features; then, according to the preset many-to-many calculation strategy, calculate the sub-medium-granularity feature scores between the sub-sentence features in the short sentence features and the sub-trajectory features in the trajectory features; finally, summarize the sub-medium-granularity feature scores to obtain the medium-granularity feature score, and use it as the second alignment score.

[0140] In practical applications, when aligning short sentence features and trajectory features, a many-to-many calculation approach can be used to obtain the matching score at this stage. That is, the matching score between each sub-trajectory feature and sub-sentence feature is calculated in turn as the sub-medium-grained feature score. After all calculations are completed, the scores are summarized to obtain the second alignment score between the trajectory feature and the short sentence feature, thereby achieving medium-grained semantic information alignment in retrieval.

[0141] The second alignment score can be calculated using the following formula:

[0142]

[0143] Among them, s p-t Indicates the second alignment score. The superscript indicates that it is obtained by aligning the trajectory features and the short sentence features. pt refers to the alignment of phrases and trajectory. p and N t It should be noted that the meanings of the letters not explained here can refer to the same descriptions in this embodiment, and will not be repeated here.

[0144] In summary, when aligning trajectory features and short sentence features at a medium granularity, it is possible to achieve cross-modal alignment of the object's motion trajectory and object state description, thereby promoting the improvement of the model's prediction accuracy at a medium granularity during the model parameter adjustment stage.

[0145] Furthermore, when calculating the third alignment score, the alignment between the long sentence block features and the video features is the coarsest-grained alignment process. In this embodiment, the specific implementation is as follows:

[0146] According to a preset one-to-one alignment calculation strategy, a coarse-grained feature score between the long sentence feature and the short sentence feature is calculated; and the coarse-grained feature score is used as the third alignment score.

[0147] Specifically, the one-to-one alignment calculation strategy refers to a strategy that performs a one-to-one alignment process on long sentence features and video features, used to calculate a coarse-grained feature score as the third alignment score. Based on this, when aligning long sentence features with video features, a coarse-grained feature score can be calculated between the long sentence features and the short sentence features according to the preset one-to-one alignment calculation strategy; this coarse-grained feature score is then used as the third alignment score.

[0148] In practical applications, when aligning long sentence features and video features, a one-to-one calculation method can be used to obtain the matching score of this stage. That is, the matching score of the two can be directly calculated, and the obtained coarse-grained feature score can be used as the third alignment score to achieve coarse-grained semantic alignment in retrieval, that is, overall semantic alignment.

[0149] The third alignment score can be calculated using the following formula:

[0150] s s-v =(t s ) T v v

[0151] Among them, s s-v The third alignment score is obtained by aligning video features with sentence features, as can be determined from the superscripts. "sv" refers to sentence and video alignment. It should be noted that the meanings of letters not explained here refer to the same descriptions in this embodiment and are not further elaborated here.

[0152] Continuing with the above example, after all the tiles corresponding to the sample video are acquired, all the tiles corresponding to the video frame can be broadcasted in the spatial dimension and the tile features can be spliced. The spliced result is then input into a multi-layer perceptron network (MLP). After processing by the MLP network, the importance of the tiles is predicted by the Softmax layer in the image-text comparison model to obtain the importance corresponding to the tile features. After obtaining the importance corresponding to all tile features, the important tiles can be selected for use by sorting. After selecting k tile features, the tile features can be aligned with the word unit features, and the alignment score can be calculated. The trajectory features and short sentence features are then aligned, and the alignment score is calculated. At the same time, the video features and long sentence features are aligned, and the alignment score is calculated. This facilitates subsequent model optimization based on the combination of alignment scores of different granularities.

[0153] In summary, when aligning video features and long sentence features at the overall granularity, it is possible to achieve cross-modal alignment of video and text as a whole, so as to promote the improvement of the model's prediction accuracy at the overall granularity dimension during the model parameter adjustment stage.

[0154] Step S210: Adjust the parameters of the image-text comparison model based on the first alignment score, the second alignment score, and the third alignment score until a target image-text comparison model that meets the training stop condition is obtained.

[0155] Specifically, after completing the alignment processing between the sample text and the sample video at three granularities, further, in order to enable the image-text comparison model to learn the ability of the alignment processing, the model parameters can be adjusted based on the three alignment scores to obtain the target image-text comparison model that meets the training stop conditions according to the adjustment results.

[0156] Among them, the training stop condition specifically refers to the condition for stopping the training of the image-text comparison model, which includes but is not limited to the loss value comparison condition, the number of iterations condition, and the validation set verification condition. Among them, the loss value comparison condition specifically refers to determining whether the loss value is less than the loss value threshold by calculating the loss value after training the model. If it is not less than, continue to obtain new samples for training until it is less than the loss value threshold, and use it as the target image-text comparison model. The number of iterations condition specifically refers to training the image-text comparison model according to the preset number of iterations until the number of iterations meets the iteration threshold, then the training can be stopped, and the model finally trained can be used as the target image-text comparison model. The validation set verification condition specifically refers to verifying it through the verification samples in the validation set in each training stage, and after the verification result matches the label a set number of times, it can be used as the target image-text comparison model.

[0157] Furthermore, when training the model, considering that a large number of sample texts and sample videos are required in actual application scenarios, and the sample texts and sample videos will constitute different types of sample pairs, such as positive / negative sample pairs, the loss value can be combined when adjusting the model parameters. In this embodiment, the specific implementation is as follows:

[0158] Based on the first alignment score, the second alignment score and the third alignment score, a target alignment score between the sample text and the sample video is calculated and output through the image-text comparison model; when multiple target alignment scores are obtained from the image-text comparison model, a target loss value is calculated based on the multiple target alignment scores; and the image-text comparison model is adjusted based on the target loss value until a target image-text comparison model that meets the training stop condition is obtained.

[0159] Specifically, the target alignment score refers to the score calculation result obtained by summarizing the first alignment score, the second alignment score and the third alignment score. Correspondingly, the target loss value refers to the multiple target alignment scores obtained after combining a large number of sample texts and sample videos for multiple target alignment score calculations and combining them with the loss function calculation.

[0160] Based on this, after the sample video and sample text are aligned in different dimensions through the above scheme, the target alignment score between the sample text and the sample video can be calculated based on the first alignment score, the second alignment score and the third alignment score, and output through the image-text comparison model; in this process, samples will be continuously obtained to repeatedly execute the above processing process, and after one round of iteration is completed, multiple target alignment scores output by the image-text comparison model will be obtained. On this basis, in order to accurately adjust the model parameters, the target loss value can be calculated based on multiple target alignment scores; then the image-text comparison model is adjusted based on the target loss value until the target image-text comparison model that meets the training stop conditions is obtained.

[0161] In practical applications, after calculating the alignment scores corresponding to the three granularities through the above scheme, the above three scores can be combined to calculate the overall matching score between the sample text and the sample video. The calculation result can be obtained by the following formula:

[0162] s=λ1s w-p +2s p-t +3s s-v

[0163] Wherein, λ1, λ2, and λ3 are three alignment weight hyperparameters. It should be noted that the meanings of letters not explained here can refer to the same description in this embodiment, and will not be repeated here.

[0164] During the training phase of the image-text comparison model, assuming B videos and B texts are given, B×B candidate video-text pairs can be formed. After obtaining B×B target alignment scores using the above method, the InfoNCE loss can be used to maximize the overall matching score of paired video-text pairs and minimize the score of unpaired pairs. After calculating the loss value, the model parameters are adjusted until the target image-text comparison model that meets the usage requirements is trained. The loss value can be calculated using the following formula:

[0165]

[0166] Among them, L represents the loss value, s i,jis the overall matching score between the i-th video and the j-th text, and τ is the temperature hyperparameter. It should be noted that the meanings of letters not explained here can refer to the same descriptions in this embodiment and will not be repeated here.

[0167] After completing the model training described above, during the testing phase, given a text, the image-text comparison model can be used to calculate the overall alignment score between the given text and candidate videos. After sorting the overall alignment scores, a set number of target videos can be selected based on the sorting results as retrieval results for the given text. Alternatively, given a video, the image-text comparison model can be used to calculate the overall alignment score between the given video and candidate texts. After sorting the overall alignment scores, a set number of target texts can be selected based on the sorting results as retrieval results for the given video.

[0168] Continuing with the above example, after obtaining the alignment scores corresponding to different granularities, the total score can be calculated by combining the alignment scores of different dimensions. The loss function can then be calculated by combining the total scores corresponding to a large number of samples. This allows the model to be adjusted in each iteration cycle to obtain a picture-text comparison model that meets usage requirements. This allows the model to complete retrieval from multiple granularities when performing text-to-video or video-to-text retrieval, thereby improving retrieval accuracy.

[0169] The model training method provided in this embodiment can achieve higher retrieval accuracy in video text retrieval scenarios. After obtaining sample text and sample video, they can be input into the image-text comparison model to be trained. First, the text encoder in the image-text comparison model extracts the word unit features and long sentence features corresponding to the sample text, and the video encoder extracts the block features and video frame features of the sample video. Based on this, short sentence features, trajectory features, and video features are constructed. This allows the extraction of multimodal features corresponding to the sample text and sample video, facilitating the subsequent establishment of cross-modal relationships. Secondly, the word unit features and block features are aligned to obtain a first alignment score, the short sentence features and trajectory features are aligned to obtain a second alignment score, and the long sentence features and video features are aligned to obtain a third alignment score. Starting from the smallest granularity, the dependency relationship between features at different granularities is captured step by step from the bottom up. Finally, the image-text comparison model is trained based on the above alignment scores to obtain the target image-text comparison model that meets actual usage requirements. This achieves higher retrieval performance and ensures retrieval accuracy in video text retrieval scenarios.

[0170] The following combined Figure 3 , taking the application of the model training method provided in this specification in the video matching text scenario as an example, the model training method is further explained. Figure 3A flowchart of the processing process of a model training method provided by an embodiment of this specification is shown, which specifically includes the following steps.

[0171] Step S302: Obtain sample text and sample video, and input them into the image-text comparison model.

[0172] Step S304: extract word unit features and long sentence features of the sample text through the text encoder in the image-text comparison model.

[0173] Step S306 , sampling the sample video through the video encoder in the image comparison model to obtain a video frame sequence composed of video frames, and extracting block features and video frame features from the video frames included in the video frame sequence through the video encoder.

[0174] When using the text encoder and video encoder provided by the image-text comparison pre-training model to extract basic features from the input data, for the text side, the text encoder will extract word unit features from the input sample text. and long sentence features s ∈R d ; Where T represents the matrix of sample text features, its superscript w represents word, that is, the sample text word unit; correspondingly, t represents the vector of word unit features, N w Indicates the number of word units, d t The dimension representing the word unit features. s The vector representing the long sentence feature, s represents sentence, and d is the feature dimension shared by the long sentence feature and the video frame feature.

[0175] For the video side, the video encoder will uniformly sample F frames of the input sample video and extract the block features and and video frame features Among them, V represents the matrix of video features, its superscript p represents patch, and j represents the jth frame in the video; N p Indicates the number of tiles in each frame, d v Represents the dimension of the tile features, A vector representing the features of the j-th video frame.

[0176] In step S308, the word unit features are input into the temporal convolutional network unit in the image-text comparison model to construct sentences, and short sentence features are generated and output according to the sentence construction results.

[0177] Step S310 , clustering the sub-block features included in the block features, and obtaining a block feature cluster according to the clustering result.

[0178] In step S312 , the sub-block features in the block feature cluster are input into the first feature construction unit in the image-text comparison model to perform trajectory construction, and a trajectory feature is generated according to the trajectory construction result.

[0179] Considering that short sentences can express the state of an object in text, and trajectories can express the position change of an object in a video, aligning short sentence features with trajectory features can ensure that the model can more accurately learn to align cross-modal information. For the text side, a one-dimensional temporal convolutional network (CNN) with a step size of 2h+1 can be embedded in the image-text comparison model to construct short sentence features based on word unit features. The short sentence features are i represents the i-th word unit extracted from N word units.

[0180] For the video side, in order to accurately express the position change of the same object through the image blocks, all the image block features corresponding to the sample video can be clustered first to find the image blocks that may contain the same object across time and space. For the i-th cluster obtained by clustering, the Transformer in the image-text comparison model is used to perform spatiotemporal modeling on the image blocks in the cluster and take the average as the trajectory feature, where the trajectory feature is

[0181] In step S314, the video frame features are input into the second feature construction unit in the image-text comparison model to perform video construction, and video features are generated according to the video construction results.

[0182] In order to achieve the alignment of video and long sentences in a coarse-grained dimension, the Transformer embedded in the image-text comparison model can be used to perform temporal modeling on the frame features of the entire video, and finally the average is taken to integrate the video features; among them, the video features are

[0183] Step S316: splicing the sub-block features contained in the block feature to obtain a block feature sequence.

[0184] In step S318, the image block feature sequence is input into the multi-layer perceptron network unit in the image-text comparison model to calculate the importance, and the importance calculation result is processed by the normalization unit in the image-text comparison model.

[0185] Step S320 , determining the importance of the block corresponding to the sub-block feature in the block feature according to the processing result.

[0186] Step S322 , based on the tile importance, target sub-tile features are filtered from the tile features to form target tile features, and based on a preset many-to-many calculation strategy, a fine-grained feature score between the word unit feature and the target tile feature is calculated as a first alignment score.

[0187] Considering that the sample video contains a large number of tiles, the information described by a large number of tiles may be irrelevant to the text semantics, and there may also be noise. Therefore, in order to align the basic feature dimensions and the word unit-tiles before aligning them, a tile selection module can be added to the image-text comparison model to predict the importance of each tile, so as to retain important tiles with rich information according to the screening results, while discarding unimportant tiles.

[0188] Specifically, for all the tiles on a given video frame, the features of the video frame are broadcasted in the spatial dimension and the tile features are spliced. The splicing results are then input into the multi-layer perceptron network (MLP) in the image-text comparison model. After being processed by the multi-layer perceptron network, the Softmax layer in the image-text comparison model is used to predict the importance of the tile, and the tile is obtained. Importance in, represents the importance matrix expression corresponding to the j-th tile, and [] refers to splicing on the channel dimension.

[0189] On this basis, after obtaining the importance corresponding to all tiles, the importance of all tiles compared to the sample video can be represented. In order to be able to screen out tiles with higher importance than the sample video and align them with word unit features, the top-k tiles can be selected according to the importance sorting results as the k tiles retained in each video frame.

[0190] Finally, when aligning the tile features and word unit features obtained above, the matching score of this stage is actually obtained through a many-to-many calculation method. That is, the matching score between each tile and all word unit features is calculated in turn as the sub-fine-grained feature score, and summarized after all calculations are completed to obtain the first alignment score between the tile features and the word unit features, so as to achieve the smallest granularity semantic information alignment that can be considered in retrieval.

[0191] The first alignment score can be calculated using the following formula:

[0192]

[0193] Among them, s w-p The first alignment score is obtained by aligning word unit features and image patch features, as can be determined from the superscripts; wp refers to word (word unit) and patch (image patch) alignment. It should be noted that the meanings of letters not explained here can be referred to the same descriptions in this embodiment and will not be repeated here.

[0194] Step S324: Determine the sub-sentence features contained in the sentence features and the sub-trajectory features contained in the trajectory features. Calculate the sub-medium-granularity feature scores between the sub-sentence features in the sentence features and the sub-trajectory features in the trajectory features based on a preset many-to-many calculation strategy. Generate a medium-granularity feature score based on the sub-medium-granularity feature scores and use it as the second alignment score.

[0195] When aligning short sentence features and trajectory features, a many-to-many calculation method can be used to obtain the matching score at this stage. That is, the matching score between each sub-trajectory feature and sub-sentence feature is calculated in turn as the sub-medium-granularity feature score. After all calculations are completed, they are summarized to obtain the second alignment score between the trajectory feature and the short sentence feature, so that medium-granularity semantic information alignment can be considered in retrieval.

[0196] The second alignment score can be calculated using the following formula:

[0197]

[0198] Among them, s p-t Indicates the second alignment score. The superscript indicates that it is obtained by aligning the trajectory features and the short sentence features. pt refers to the alignment of phrases and trajectory. p and N t It should be noted that the meanings of the letters not explained here can refer to the same descriptions in this embodiment, and will not be repeated here.

[0199] In step S326 , a coarse-grained feature score between the long sentence feature and the short sentence feature is calculated according to a preset one-to-one alignment calculation strategy, and the coarse-grained feature score is used as a third alignment score.

[0200] When aligning long sentence features and video features, a one-to-one calculation method can be used to obtain the matching score of this stage. That is, the matching score of the two can be directly calculated, and the obtained coarse-grained feature score can be used as the third alignment score to achieve coarse-grained semantic alignment in retrieval, that is, overall semantic alignment.

[0201] The third alignment score can be calculated using the following formula:

[0202] s s-v =(t s ) T v v

[0203] Among them, s s-vThe third alignment score is obtained by aligning video features with sentence features, as can be determined from the superscripts. "sv" refers to sentence and video alignment. It should be noted that the meanings of letters not explained here refer to the same descriptions in this embodiment and are not further elaborated here.

[0204] Step S328: Calculate the target alignment score between the sample text and the sample video based on the first alignment score, the second alignment score, and the third alignment score, and output it through the image-text comparison model.

[0205] In step S330, when multiple target alignment scores are obtained from the image-text comparison model output, a target loss value is calculated based on the multiple target alignment scores, and the image-text comparison model is adjusted based on the target loss value until a target image-text comparison model that meets the training stop condition is obtained.

[0206] After calculating the alignment scores corresponding to the three granularities through the above scheme, the above three scores can be combined to calculate the overall matching score between the sample text and the sample video. The calculation result can be obtained by the following formula:

[0207] s=λ1s w-p +2s p-t +3s s-v

[0208] Wherein, λ1, λ2, and λ3 are three alignment weight hyperparameters. It should be noted that the meanings of letters not explained here can refer to the same description in this embodiment, and will not be repeated here.

[0209] During the training phase of the image-text comparison model, assuming B videos and B texts are given, B×B candidate video-text pairs can be formed. After obtaining B×B target alignment scores using the above method, the InfoNCE loss can be used to maximize the overall matching score of paired video-text pairs and minimize the score of unpaired pairs. After calculating the loss value, the model parameters are adjusted until the target image-text comparison model that meets the usage requirements is trained. The loss value can be calculated using the following formula:

[0210]

[0211] Among them, L represents the loss value, s i,j is the overall matching score between the i-th video and the j-th text, and τ is the temperature hyperparameter. It should be noted that the meanings of letters not explained here can refer to the same descriptions in this embodiment and will not be repeated here.

[0212] After completing the model training described above, during the testing phase, given a text, the image-text comparison model can be used to calculate the overall alignment score between the given text and candidate videos. After sorting the overall alignment scores, a set number of target videos can be selected based on the sorting results as retrieval results for the given text. Alternatively, given a video, the image-text comparison model can be used to calculate the overall alignment score between the given video and candidate texts. After sorting the overall alignment scores, a set number of target texts can be selected based on the sorting results as retrieval results for the given video.

[0213] Step S332: obtain a video query request carrying user text, and determine the video set corresponding to the user text. Through the target image-text comparison model, calculate the candidate score between the user text and the candidate videos in the video set, and screen out at least one target video in the video set based on the candidate score as a response to the video query request.

[0214] Step S334: obtain a text query request carrying the user video, and determine the text set corresponding to the user video. Through the target image-text comparison model, calculate the candidate score between the user video and the candidate text in the text set, and filter out at least one target text in the text set based on the candidate score as a response to the text query request.

[0215] The model training method provided in this embodiment can achieve higher retrieval accuracy in video text retrieval scenarios. After obtaining sample text and sample video, they can be input into the image-text comparison model to be trained. First, the text encoder in the image-text comparison model extracts the word unit features and long sentence features corresponding to the sample text, and the video encoder extracts the block features and video frame features of the sample video. Based on this, short sentence features, trajectory features, and video features are constructed. This allows the extraction of multimodal features corresponding to the sample text and sample video, facilitating the subsequent establishment of cross-modal relationships. Secondly, the word unit features and block features are aligned to obtain a first alignment score, the short sentence features and trajectory features are aligned to obtain a second alignment score, and the long sentence features and video features are aligned to obtain a third alignment score. Starting from the smallest granularity, the dependency relationship between features at different granularities is captured step by step from the bottom up. Finally, the image-text comparison model is trained based on the above alignment scores to obtain the target image-text comparison model that meets actual usage requirements. This achieves higher retrieval performance and ensures retrieval accuracy in video text retrieval scenarios.

[0216] Corresponding to the above method embodiment, this specification also provides a model training device embodiment, Figure 4 FIG1 shows a schematic diagram of the structure of a model training device provided by an embodiment of this specification. Figure 4 As shown, the device includes:

[0217] An acquisition module 402 is configured to acquire sample text and sample video and input them into a text-image comparison model;

[0218] An extraction module 404 is configured to extract word unit features and long sentence features of the sample text through the text encoder in the image-text comparison model, and to extract image block features and video frame features of the sample video through the video encoder in the image-text comparison model;

[0219] A construction module 406 is configured to construct a short sentence feature based on the word unit feature, construct a trajectory feature based on the image block feature, and construct a video feature based on the video frame feature;

[0220] an alignment module 408 configured to align the word unit feature with the image block feature to obtain a first alignment score, align the short sentence feature with the trajectory feature to obtain a second alignment score, and align the long sentence feature with the video feature to obtain a third alignment score;

[0221] The training module 410 is configured to adjust the parameters of the image-text comparison model based on the first alignment score, the second alignment score, and the third alignment score until a target image-text comparison model that meets the training stop condition is obtained.

[0222] In an optional embodiment, the extraction module 404 is further configured to:

[0223] The sample video is sampled by the video encoder in the image comparison model to obtain a video frame sequence composed of video frames; and the block features and the video frame features are extracted from the video frames included in the video frame sequence by the video encoder.

[0224] In an optional embodiment, the construction module 406 is further configured to:

[0225] The word unit features are input into the temporal convolutional network unit in the image-text comparison model to construct sentences; the short sentence features are generated and output according to the sentence construction results; wherein the temporal convolutional network unit constructs sentences according to the set dimension and the set step size.

[0226] In an optional embodiment, the construction module 406 is further configured to:

[0227] Clustering the sub-block features contained in the block features to obtain a block feature cluster based on the clustering result; inputting the sub-block features in the block feature cluster into the first feature construction unit in the image-text comparison model to perform trajectory construction, and generating the trajectory feature based on the trajectory construction result.

[0228] In an optional embodiment, the construction module 406 is further configured to:

[0229] The video frame features are input into the second feature construction unit in the image-text comparison model to perform video construction; and the video features are generated according to the video construction results.

[0230] In an optional embodiment, the alignment module 408 is further configured to:

[0231] Calculate the tile importance corresponding to the sub-tile feature in the tile feature; filter the target sub-tile feature in the tile feature according to the tile importance to form a target tile feature; and calculate the fine-grained feature score between the word unit feature and the target tile feature according to a preset many-to-many calculation strategy as the first alignment score.

[0232] In an optional embodiment, the alignment module 408 is further configured to:

[0233] The sub-block features contained in the block feature are spliced to obtain a block feature sequence; the block feature sequence is input into the multi-layer perceptron network unit in the image-text comparison model to calculate the importance, and the importance calculation result is processed by the normalization unit in the image-text comparison model; and the block importance corresponding to the sub-block feature in the block feature is determined according to the processing result.

[0234] In an optional embodiment, the alignment module 408 is further configured to:

[0235] Determine a sub-sentence feature contained in the sentence feature and a sub-trajectory feature contained in the trajectory feature; calculate a sub-medium-granularity feature score between the sub-sentence feature in the sentence feature and the sub-trajectory feature in the trajectory feature according to a preset many-to-many calculation strategy; generate a medium-granularity feature score based on the sub-medium-granularity feature score and use it as the second alignment score.

[0236] In an optional embodiment, the alignment module 408 is further configured to:

[0237] According to a preset one-to-one alignment calculation strategy, a coarse-grained feature score between the long sentence feature and the short sentence feature is calculated; and the coarse-grained feature score is used as the third alignment score.

[0238] In an optional embodiment, the training module 410 is further configured to:

[0239] Based on the first alignment score, the second alignment score and the third alignment score, a target alignment score between the sample text and the sample video is calculated and output through the image-text comparison model; when multiple target alignment scores are obtained from the image-text comparison model, a target loss value is calculated based on the multiple target alignment scores; and the image-text comparison model is adjusted based on the target loss value until a target image-text comparison model that meets the training stop condition is obtained.

[0240] The model training device provided in this embodiment can achieve higher retrieval accuracy in video text retrieval scenarios. After obtaining sample text and sample video, they can be input into the image-text comparison model to be trained. First, the text encoder in the image-text comparison model extracts word unit features and long sentence features corresponding to the sample text, and the video encoder extracts image block features and video frame features of the sample video. Based on these features, short sentence features, trajectory features, and video features are constructed. This allows the extraction of multimodal features corresponding to the sample text and sample video, facilitating the subsequent establishment of cross-modal relationships. Secondly, the word unit features and image block features are aligned to obtain a first alignment score, the short sentence features and trajectory features are aligned to obtain a second alignment score, and the long sentence features and video features are aligned to obtain a third alignment score. Starting from the smallest granularity, the dependency relationship between features at different granularities is captured step by step from the bottom up. Finally, the image-text comparison model is trained based on the above alignment scores to obtain a target image-text comparison model that meets actual usage requirements. This achieves higher retrieval performance and ensures retrieval accuracy in video text retrieval scenarios.

[0241] The above is a schematic scheme of a model training device of this embodiment. It should be noted that the technical scheme of the model training device and the technical scheme of the above-mentioned model training method are of the same concept. For details not described in detail in the technical scheme of the model training device, please refer to the description of the technical scheme of the above-mentioned model training method.

[0242] Corresponding to the above method embodiment, Figure 5 A flowchart of a method for comparing images and texts according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0243] Step S502: Obtain a video query request carrying user text, and determine a video set corresponding to the user text.

[0244] Step S504 , calculating the candidate scores between the user text and the candidate videos in the video set using the target image-text comparison model in the above method.

[0245] Step S506: Filter out at least one target video from the video set based on the candidate scores as a response to the video query request.

[0246] In specific implementation, after receiving the video query request submitted by the front end, the video query request can be parsed first to obtain the user text carried in the request; then the video set corresponding to the user text can be determined first, that is, the video set that can hit the user text. The set can be a set corresponding to all videos stored in the video database of the server, or a set composed of videos that match the user text type. This embodiment does not impose any restrictions here.

[0247] Furthermore, after determining the user text and video collection, the matching score between the user text and each video in the video collection can be calculated using the target image-text comparison model trained in the above method as a candidate score; the candidate scores can then be sorted to select at least one target video based on the sorting results and feed it back to the front end as a response to the video query request.

[0248] It should be noted that the process of the target image-text comparison model calculating the candidate scores between the user text and each video can refer to the same or corresponding descriptions in the above embodiments, and this embodiment will not be described in detail here.

[0249] Corresponding to the above method embodiment, this specification also provides an embodiment of a picture-text comparison device, Figure 6 FIG1 shows a schematic diagram of the structure of a picture-text comparison device provided by an embodiment of this specification. Figure 6 As shown, the device includes:

[0250] The video request acquisition module 602 is configured to acquire a video query request carrying user text and determine a video set corresponding to the user text;

[0251] The candidate score calculation module 604 is configured to calculate the candidate scores between the user text and the candidate videos in the video set using the target image-text comparison model in the above method;

[0252] The target video screening module 606 is configured to screen out at least one target video from the video set based on the candidate scores as a response to the video query request.

[0253] The above is a schematic diagram of a device for comparing images and texts in accordance with this embodiment. It should be noted that the technical solution of the device for comparing images and texts is identical in concept to the technical solution of the aforementioned method for comparing images and texts. For details not described in detail in the technical solution of the device for comparing images and texts, please refer to the description of the technical solution of the aforementioned method for comparing images and texts.

[0254] Corresponding to the above method embodiment, Figure 7 A flowchart of another image-text comparison method provided according to an embodiment of this specification is shown, which specifically includes the following steps.

[0255] Step S702: Obtain a text query request carrying a user video, and determine a text set corresponding to the user video.

[0256] Step S704 , calculating the candidate scores between the user video and the candidate texts in the text set using the target image-text comparison model in the above method.

[0257] Step S706: Screen out at least one target text from the text collection based on the candidate scores as a response to the text query request.

[0258] In specific implementation, after receiving the text query request submitted by the front end, the text query request can be parsed first to obtain the user video carried in the request; then the text set corresponding to the user video can be determined first, that is, the text set that can hit the user video. The set can be a set corresponding to all texts stored in the text database of the server, or a set composed of texts matching the user video type. This embodiment does not impose any restrictions here.

[0259] Furthermore, after determining the user video and text collection, the matching score between the user video and each text in the text collection can be calculated using the target image-text comparison model trained in the above method as a candidate score; the candidate scores can then be sorted to filter out at least one target text based on the sorting results and feed it back to the front end as a response to the text query request.

[0260] It should be noted that the process of the target image-text comparison model calculating the candidate scores between the user video and each text can refer to the same or corresponding descriptions in the above embodiments, and this embodiment will not be elaborated on here.

[0261] Corresponding to the above method embodiment, this specification also provides another embodiment of an image-text comparison device, Figure 8 FIG. 1 shows a schematic diagram of another image-text comparison device provided by an embodiment of this specification. Figure 8 As shown, the device includes:

[0262] The text request acquisition module 802 is configured to acquire a text query request carrying a user video and determine a text set corresponding to the user video;

[0263] The candidate score calculation module 804 is configured to calculate the candidate scores between the user video and the candidate texts in the text set using the target image-text comparison model in the above method;

[0264] The target text screening module 806 is configured to screen out at least one target text from the text collection based on the candidate scores as a response to the text query request.

[0265] The above is a schematic diagram of another image-text comparison device of this embodiment. It should be noted that the technical solution of this image-text comparison device and the technical solution of the above-mentioned image-text comparison method are of the same concept. For details not described in detail in the technical solution of the image-text comparison device, please refer to the description of the technical solution of the above-mentioned image-text comparison method.

[0266] The above is a schematic diagram of a device for comparing images and texts in accordance with this embodiment. It should be noted that the technical solution of the device for comparing images and texts is identical in concept to the technical solution of the aforementioned method for comparing images and texts. For details not described in detail in the technical solution of the device for comparing images and texts, please refer to the description of the technical solution of the aforementioned method for comparing images and texts.

[0267] Corresponding to the above method embodiment, Figure 9 A flowchart of another model training method provided according to an embodiment of this specification is shown, which is applied to cloud-side devices and specifically includes the following steps.

[0268] Step S902: Obtain sample text and sample video submitted by the client device and input them into the image-text comparison model;

[0269] Step S904: extracting word unit features and long sentence features of the sample text through the text encoder in the image-text comparison model, and extracting image block features and video frame features of the sample video through the video encoder in the image-text comparison model;

[0270] Step S906, constructing a short sentence feature based on the word unit feature, constructing a trajectory feature based on the image block feature, and constructing a video feature based on the video frame feature;

[0271] Step S908: aligning the word unit feature with the image block feature to obtain a first alignment score, aligning the short sentence feature with the trajectory feature to obtain a second alignment score, and aligning the long sentence feature with the video feature to obtain a third alignment score;

[0272] Step S910: Adjusting parameters of the image-text comparison model based on the first alignment score, the second alignment score, and the third alignment score until a target image-text comparison model that meets a training stop condition is obtained;

[0273] Step S912: Send the target model parameters corresponding to the target image-text comparison model to the terminal device.

[0274] Corresponding to the above method embodiment, this specification also provides another model training device embodiment, Figure 10 FIG. 1 shows a schematic diagram of another model training device provided by an embodiment of this specification. Figure 10 As shown, the device is applied to a cloud-side device and includes:

[0275] The sample acquisition module 1002 is configured to acquire sample text and sample video submitted by the end-side device and input them into the image-text comparison model;

[0276] The feature extraction module 1004 is configured to extract word unit features and long sentence features of the sample text through the text encoder in the image-text comparison model, and to extract image block features and video frame features of the sample video through the video encoder in the image-text comparison model;

[0277] A feature construction module 1006 is configured to construct a short sentence feature based on the word unit feature, construct a trajectory feature based on the image block feature, and construct a video feature based on the video frame feature;

[0278] an alignment feature module 1008 configured to align the word unit feature with the image block feature to obtain a first alignment score, align the short sentence feature with the trajectory feature to obtain a second alignment score, and align the long sentence feature with the video feature to obtain a third alignment score;

[0279] The model training module 1010 is configured to adjust parameters of the image-text comparison model based on the first alignment score, the second alignment score, and the third alignment score until a target image-text comparison model that meets a training stop condition is obtained;

[0280] The parameter sending module 1012 is configured to send the target model parameters corresponding to the target image-text comparison model to the terminal device.

[0281] The above is a schematic scheme of another model training device of this embodiment. It should be noted that the technical scheme of the model training device and the technical scheme of the above-mentioned model training method are of the same concept. For details not described in detail in the technical scheme of the model training device, please refer to the description of the technical scheme of the above-mentioned model training method.

[0282] Figure 11 The following is a block diagram of a computing device 1100 according to one embodiment of the present disclosure. Components of the computing device 1100 include, but are not limited to, a memory 1110 and a processor 1120. The processor 1120 is connected to the memory 1110 via a bus 1130, and a database 1150 is used to store data.

[0283] The computing device 1100 also includes an access device 1140 that enables the computing device 1100 to communicate via one or more networks 1160. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1140 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0284] In one embodiment of the present application, the above components of the computing device 400 and Figure 11 Other components not shown in the figure may also be connected to each other, for example, via a bus. Figure 11 The computing device structure block diagram shown is for illustrative purposes only and is not intended to limit the scope of the present application. Those skilled in the art may add or replace other components as needed.

[0285] Computing device 1100 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 1100 may also be a mobile or stationary server.

[0286] Among them, the processor 1120 is used to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-mentioned model training method or image-text comparison method.

[0287] The above is a schematic diagram of a computing device according to this embodiment. It should be noted that the technical solution of this computing device is based on the same concept as the technical solution of the model training method or the image-text comparison method described above. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the model training method or the image-text comparison method described above.

[0288] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned model training method or image-text comparison method.

[0289] The above is a schematic diagram of a computer-readable storage medium of this embodiment. It should be noted that the technical solution of this storage medium is based on the same concept as the technical solution of the model training method or the image-text comparison method described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the model training method or the image-text comparison method described above.

[0290] One embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned model training method or image-text comparison method.

[0291] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of this computer program and the technical scheme of the above-mentioned model training method or image-text comparison method are based on the same concept. For details not described in detail in the technical scheme of the computer program, please refer to the description of the technical scheme of the above-mentioned model training method or image-text comparison method.

[0292] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0293] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0294] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0295] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0296] The preferred embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A model training method, comprising: Obtaining sample text and sample video and inputting them into a picture-text comparison model, wherein the picture-text comparison model is used to predict the matching score between the text and the video; Extracting word unit features and long sentence features of the sample text through the text encoder in the image-text comparison model, and extracting image block features and video frame features of the sample video through the video encoder in the image-text comparison model; Constructing a short sentence feature based on the word unit feature, constructing a trajectory feature based on the image block feature, and constructing a video feature based on the video frame feature, wherein the trajectory feature is used to reflect a vector expression of a trajectory corresponding to the same object in the sample video, and the trajectory expresses a position change of the object in the sample video; Aligning the word unit feature and the image block feature to obtain a first alignment score, aligning the short sentence feature and the trajectory feature to obtain a second alignment score, and aligning the long sentence feature and the video feature to obtain a third alignment score, wherein the first alignment score is a total matching score obtained after performing a many-to-many alignment of the word unit feature and the image block feature, the second alignment score is a total matching score obtained after performing a many-to-many alignment of the short sentence feature and the trajectory feature, and the third alignment score is a total matching score obtained after performing a one-to-one alignment of the long sentence feature and the video feature; Based on the first alignment score, the second alignment score, and the third alignment score, the image-text comparison model is adjusted until a target image-text comparison model that meets a training stop condition is obtained.

2. The method according to claim 1, wherein extracting the image block features and video frame features of the sample video by the video encoder in the image-text comparison model comprises: Sampling the sample video by the video encoder in the image-text comparison model to obtain a video frame sequence consisting of video frames; The video encoder extracts the image block features and the video frame features from the video frames included in the video frame sequence.

3. The method according to claim 1, wherein constructing a short sentence feature based on the word unit feature comprises: Inputting the word unit features into the temporal convolutional network unit in the image-text comparison model to construct sentences; The short sentence features are generated and output according to the sentence construction results; wherein the temporal convolutional network unit constructs the sentence according to the set dimension and the set step size.

4. The method according to claim 1, wherein constructing the trajectory feature according to the block feature comprises: Clustering the sub-block features contained in the block features, and obtaining a block feature cluster according to the clustering result; Inputting the sub-block features in the block feature cluster into the first feature construction unit in the image-text comparison model to perform trajectory construction, and generating the trajectory feature according to the trajectory construction result; Accordingly, constructing video features according to the video frame features includes: Inputting the video frame features into the second feature construction unit in the image-text comparison model to perform video construction; The video features are generated according to the video construction result.

5. The method according to any one of claims 1 to 4, wherein the step of aligning the word unit feature and the image block feature to obtain a first alignment score comprises: Calculating the tile importance corresponding to the sub-tile feature in the tile feature; Filtering target sub-block features from the block features according to the block importance to form target block features; According to a preset many-to-many calculation strategy, a fine-grained feature score between the word unit feature and the target tile feature is calculated as the first alignment score.

6. The method according to claim 5, wherein the step of calculating the tile importance corresponding to the sub-tile feature in the tile feature comprises: splicing the sub-block features contained in the block features to obtain a block feature sequence; Inputting the block feature sequence into the multi-layer perceptron network unit in the image-text comparison model to calculate the importance, and processing the importance calculation result through the normalization unit in the image-text comparison model; The importance of the block corresponding to the sub-block feature in the block feature is determined according to the processing result.

7. The method according to any one of claims 1 to 4, wherein the aligning the short sentence feature and the trajectory feature to obtain a second alignment score comprises: determining a sub-sentence feature included in the sentence feature, and a sub-trajectory feature included in the trajectory feature; Calculating, according to a preset many-to-many calculation strategy, a sub-granularity feature score between the sub-sentence feature in the sentence feature and the sub-trajectory feature in the trajectory feature; A medium-granularity feature score is generated according to the sub-medium-granularity feature score and used as the second alignment score.

8. The method according to any one of claims 1 to 4, wherein aligning the long sentence feature and the video feature to obtain a third alignment score comprises: Calculating the coarse-grained feature score between the long sentence feature and the short sentence feature according to a preset one-to-one alignment calculation strategy; The coarse-grained feature score is used as the third alignment score.

9. The method according to any one of claims 1 to 4, wherein the adjusting parameters of the image-text comparison model based on the first alignment score, the second alignment score, and the third alignment score until a target image-text comparison model that meets a training stop condition is obtained comprises: Calculating a target alignment score between the sample text and the sample video based on the first alignment score, the second alignment score, and the third alignment score, and outputting the result through the image-text comparison model; When a plurality of target alignment scores output by the image-text comparison model are obtained, a target loss value is calculated according to the plurality of target alignment scores; The image-text comparison model is adjusted based on the target loss value until a target image-text comparison model that meets the training stop condition is obtained.

10. A method for comparing images and texts, comprising: Obtaining a video query request carrying user text, and determining a video set corresponding to the user text; Calculating candidate scores between the user text and candidate videos in the video set using the target image-text comparison model in the method according to any one of claims 1 to 9; At least one target video is selected from the video collection based on the candidate scores as a response to the video query request.

11. A method for comparing images and text, comprising: Obtaining a text query request carrying a user video, and determining a text set corresponding to the user video; Calculating a candidate score between the user video and the candidate text in the text set by using the target image-text comparison model in the method according to any one of claims 1 to 9; At least one target text is screened out from the text collection based on the candidate scores as a response to the text query request.

12. A model training method, applied to a cloud-side device, comprising: Obtaining sample text and sample video submitted by the client device and inputting them into a text-to-image comparison model, wherein the text-to-image comparison model is used to predict a matching score between the text and the video; Extracting word unit features and long sentence features of the sample text through the text encoder in the image-text comparison model, and extracting image block features and video frame features of the sample video through the video encoder in the image-text comparison model; Constructing a short sentence feature based on the word unit feature, constructing a trajectory feature based on the image block feature, and constructing a video feature based on the video frame feature, wherein the trajectory feature is used to reflect a vector expression of a trajectory corresponding to the same object in the sample video, and the trajectory expresses a position change of the object in the sample video; Aligning the word unit feature and the image block feature to obtain a first alignment score, aligning the short sentence feature and the trajectory feature to obtain a second alignment score, and aligning the long sentence feature and the video feature to obtain a third alignment score, wherein the first alignment score is a total matching score obtained after performing a many-to-many alignment of the word unit feature and the image block feature, the second alignment score is a total matching score obtained after performing a many-to-many alignment of the short sentence feature and the trajectory feature, and the third alignment score is a total matching score obtained after performing a one-to-one alignment of the long sentence feature and the video feature; Adjusting parameters of the image-text comparison model based on the first alignment score, the second alignment score, and the third alignment score until a target image-text comparison model that meets a training stop condition is obtained; The target model parameters corresponding to the target image-text comparison model are sent to the terminal side device.

13. A computing device comprising: memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method according to any one of claims 1 to 12 are implemented.

14. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 12.

Images