Cross-modal retrieval model processing method, device, equipment, product and medium

By performing image frame fusion and text concatenation on the samples of the cross-modal retrieval model, fused training samples are generated, which solves the problem of difficulty in obtaining training samples and improves the retrieval accuracy and efficiency of the model.

CN115994243BActive Publication Date: 2024-11-12BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310074339.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-01-13
Publication Date
2024-11-12
Estimated Expiration
2043-01-13

AI Technical Summary

Technical Problem

Cross-modal retrieval models require a large number of training samples during training, resulting in high acquisition costs and low efficiency. Existing augmentation methods lead to semantic inconsistencies, affecting retrieval accuracy.

Method used

By fusing image frames and concatenating text between two randomly selected original training samples, a fused training sample is generated, maintaining semantic consistency, and used for training a cross-modal retrieval model.

Benefits of technology

This improved the quantity and quality of training samples, enhanced the model's retrieval accuracy and efficiency, and ensured the matching degree between the target content and the query content.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115994243B_ABST
    Figure CN115994243B_ABST
Patent Text Reader

Abstract

The present disclosure provides a cross-modal retrieval model processing method, device, equipment, product and medium, which relates to the field of artificial intelligence technology, specifically computer vision, image processing, deep learning and other technical fields, and can be applied to scenarios such as smart security and short videos. The specific implementation scheme is: obtaining a sample pair of a cross-modal retrieval model to be trained, the sample pair including two randomly determined original training samples; performing sample fusion processing on the two original training samples in the sample pair to obtain a fused training sample; training the cross-modal retrieval model according to the original training samples of the cross-modal retrieval model and the fused training samples to obtain a target retrieval model; wherein the target retrieval model is used to query target content that matches the content to be queried, and the modalities of the content to be queried and the target content are different.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically computer vision, image processing, deep learning and other technical fields, which can be applied to scenarios such as smart security and short videos, and in particular to a cross-modal retrieval model processing method, device, equipment, product and medium. Background Art

[0002] With the rapid development of information technology, the demand for video playback is getting higher and higher. In many applications such as video playback programs, content retrieval programs, transaction programs, security programs, social programs, etc., there may be a retrieval demand for videos using text or images. This retrieval model with different modalities of retrieval results and query modalities can be called a cross-modal retrieval model.

[0003] However, the current cross-modal retrieval model training requires a large number of training samples, which are generally composed of videos and text information related to the videos. The cost of acquiring training samples for the cross-modal retrieval model is high, which makes it difficult to obtain a large number of training samples. The efficiency of acquiring training samples for the cross-modal retrieval model is low, which in turn leads to low retrieval accuracy of the cross-modal retrieval model. Summary of the invention

[0004] The present disclosure provides a cross-modal retrieval model processing method, device, equipment, product and medium.

[0005] According to a first aspect of the present disclosure, a method for training a cross-modal retrieval model is provided, comprising:

[0006] Obtaining a sample pair of a cross-modal retrieval model to be trained, where the sample pair includes two randomly determined original training samples;

[0007] Perform sample fusion processing on two original training samples in the sample pair to obtain a fusion training sample;

[0008] The cross-modal retrieval model is trained according to the original training samples and the fusion training samples of the cross-modal retrieval model to obtain a target retrieval model;

[0009] Among them, the target retrieval model is used to query the target content that matches the content to be queried, and the modalities of the content to be queried and the target content are different.

[0010] According to a second aspect of the present disclosure, a query method of a cross-modal retrieval model is provided, comprising:

[0011] receiving query content sent by a user terminal, where the query content belongs to a first modality;

[0012] Inputting the query content into the trained target retrieval model to obtain target content matching the query content, wherein the target retrieval model is trained based on the training method of the cross-modal retrieval model provided in the first aspect, and the target content belongs to the second modality;

[0013] The target content is sent to the user terminal of the user, and the target content is displayed by the user terminal.

[0014] According to a third aspect of the present disclosure, a training device for a cross-modal retrieval model is provided, comprising:

[0015] An acquisition unit, used to acquire a sample pair of a cross-modal retrieval model to be trained, wherein the sample pair includes two randomly determined original training samples;

[0016] A fusion unit, used for performing sample fusion processing on two original training samples in the sample pair to obtain a fused training sample;

[0017] A training unit, used for training the cross-modal retrieval model according to the original training samples and the fusion training samples of the cross-modal retrieval model to obtain a target retrieval model;

[0018] Among them, the target retrieval model is used to query the target content that matches the content to be queried, and the modalities of the content to be queried and the target content are different.

[0019] According to a fourth aspect of the present disclosure, a query device for a cross-modal retrieval model is provided, comprising:

[0020] A receiving unit, configured to receive query content sent by a user terminal, where the query content belongs to a first mode;

[0021] A query unit, configured to input query content into a trained target retrieval model to obtain target content matching the query content, wherein the target retrieval model is trained based on the training method of the cross-modal retrieval model provided in the first aspect, and the target content belongs to the second modality;

[0022] The sending unit is used to send the target content to the user terminal of the user, and the target content is displayed by the user terminal.

[0023] According to a fifth aspect of the present disclosure, a computer program product is provided, the computer program product comprising: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the method of the first aspect.

[0024] According to the technology disclosed in the present invention, a sample pair of a cross-modal retrieval model to be trained is obtained, and the sample pair may include two randomly determined original training samples. A new training sample is obtained by fusing two samples, and the obtained fused training sample may contain relevant information of the two original training samples, and there is no need to modify the original training samples, so that the semantic consistency of the fused training sample remains intact. Therefore, the cross-modal retrieval model is trained using the original training samples and the fused training samples, and the training of the cross-modal retrieval model using more training samples is realized, so that the cross-modal retrieval model can learn the characteristics of more training samples, and improve the retrieval accuracy of the target retrieval model obtained by training. When the target retrieval model is used to query the target content that matches the content to be queried, the obtained target content has higher accuracy and is more matched with the content to be queried. The modalities of the content to be queried and the target content may be different, which improves the efficiency and accuracy of cross-modal retrieval.

[0025] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0027] Figure 1 is a schematic diagram of a system architecture of a retrieval method of a cross-modal retrieval model provided according to an embodiment of the present disclosure;

[0028] Figure 2 is a flowchart of a method for training a cross-modal retrieval model according to an embodiment of the present disclosure;

[0029] Figure 3 is a flowchart of a method for training a cross-modal retrieval model according to an embodiment of the present disclosure;

[0030] Figure 4 is an example diagram of a video frame and text fusion provided according to an embodiment of the present disclosure;

[0031] Figure 5 is a flowchart of a method for training a cross-modal retrieval model according to an embodiment of the present disclosure;

[0032] Figure 6 is a flowchart of a query method of a cross-modal retrieval model provided according to an embodiment of the present disclosure;

[0033] Figure 7 is a schematic diagram of the structure of a training device for a cross-modal retrieval model provided according to an embodiment of the present disclosure;

[0034] Figure 8is a schematic diagram of the structure of a query device of a cross-modal retrieval model provided according to an embodiment of the present disclosure;

[0035] Fig. 9 It is a block diagram of an electronic device used to implement the training and query methods of the cross-modal retrieval model of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0037] The present disclosure provides a training and query method, device, equipment, medium and product for a cross-modal retrieval model, which relates to the field of artificial intelligence technology, specifically computer vision, image processing, deep learning and other technical fields, and can be applied to scenarios such as smart security and short videos to achieve the purpose of improving the training accuracy of the cross-modal retrieval model.

[0038] At present, the cross-modal retrieval model can refer to a content query model in which the modality of the retrieval result is different from the modality of the query. The input of the cross-modal retrieval model can be the content to be queried, and the output can be the query result. The modalities of the content to be queried and the query result are different. At present, the cross-modal retrieval model is generally obtained through training with training samples. The training samples can generally include videos and text description information of the videos. The content contained is relatively large and the acquisition method is relatively complex. In practical applications, the acquisition of cross-modal retrieval models requires a large number of training samples, resulting in high acquisition costs and low efficiency of training samples. In order to solve the problem of the small number of training samples for the cross-modal retrieval model, the cross-modal retrieval model can be subjected to sample augmentation processing, that is, based on the original samples, the original samples are expanded to obtain new samples. Specific expansion methods include text replacement, random insertion of words into the text description content, adding noise to the image in the video, cutting out the image, etc. When the augmentation processing is performed on a single sample, the label of the original sample can be used or recalibrated. However, this method of augmenting the original samples makes it difficult to keep the video and text consistent, that is, the correspondence between the newly obtained samples and labels is not accurate. When inaccurate samples and labels are involved in the training of the cross-modal retrieval model, the cross-modal retrieval model will learn inaccurate retrieval characteristics, resulting in a decrease in the accuracy of the cross-modal retrieval model.

[0039] In order to achieve accurate training of the cross-modal retrieval model, the disclosed embodiment takes into account that, for the cross-modal retrieval model, it is more important to maintain the consistency of the video and the text description information of the video. Therefore, in order to obtain a video with high semantic consistency and the text description information of the video, different videos can be fused for image frames, and the text description information corresponding to each video can be spliced, so that the fused sample contains information of the two images, and the splicing of the text description information can intuitively reflect the accurate text description of the two videos used for fusion, thereby obtaining accurate fused video and fused text, so that the fused video and text can be involved in the training process of the cross-modal retrieval model. Using training samples with higher semantic consistency can improve the training accuracy of the cross-modal retrieval model and obtain a more accurate target retrieval model.

[0040] Therefore, the technical solution disclosed in the present invention can obtain a sample pair of a cross-modal retrieval model to be trained, and the sample pair may include two randomly determined original training samples. By fusing the two original training samples of the sample pair, a fused training sample can be obtained. That is, the present invention adopts a method of fusing two samples to obtain a new training sample, and the obtained fused training sample can contain relevant information of the two original training samples, and there is no need to modify the original training samples, so that the semantic consistency of the fused training sample remains intact. Therefore, the cross-modal retrieval model is trained using the original training samples and the fused training samples, and the training of the cross-modal retrieval model using more training samples is realized, so that the cross-modal retrieval model can learn the characteristics of more training samples, and improve the retrieval accuracy of the target retrieval model obtained by training. When the target retrieval model is used to query the target content that matches the content to be queried, the obtained target content has higher accuracy and is more matched with the content to be queried. The modality of the content to be queried and the target content can be different, which improves the efficiency and accuracy of cross-modal retrieval.

[0041] The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0042] like Figure 1 As shown, a schematic diagram of a system architecture of a retrieval method of a cross-modal retrieval model provided by an embodiment of the present disclosure is provided. The system architecture may include a user terminal 1 and an electronic device 2. The user terminal 1 may establish a communication connection with the electronic device 2.

[0043] Among them, the electronic device 2 can be configured with the training method of the cross-modal retrieval model disclosed in the present invention. When training the cross-modal retrieval model, the electronic device 2 can obtain a sample pair from the sample database 3 of the cross-modal retrieval model. The sample pair may include two randomly determined original training samples. By performing sample fusion processing on the two original training samples in the sample pair, a fused training sample can be obtained. The sample is augmented, and then the cross-modal retrieval model is trained based on the original training samples and the fused training samples of the cross-modal retrieval model to obtain a target retrieval model. The target retrieval model can learn the features of more samples with higher accuracy.

[0044] User terminal 1 can send query content to electronic device 2, and the query content belongs to the first modality. Electronic device 2 can receive the query content sent by user terminal 1, and query the target content matching the content to be queried through the target retrieval model, which can make the obtained target content more accurate. Afterwards, electronic device 2 can feed back the target content to user terminal 2. After receiving the target content, user terminal 2 can display the target content.

[0045] Figure 2 A flowchart of an embodiment of a method for training a cross-modal retrieval model provided in an embodiment of the present disclosure, the method may include the following steps:

[0046] Step 201: Obtain a sample pair of a cross-modal retrieval model to be trained, where the sample pair includes two randomly determined original training samples.

[0047] Optionally, the original training sample may be a sample already existing in the sample database. The original training sample may include a video and text description information of the video.

[0048] The cross-modal retrieval model may be a content retrieval model for a cross-modal scenario built by a machine learning model, a neural network, etc. The sample pair may be randomly extracted from a sample database. In practical applications, the sample pair may include multiple ones, and each sample pair may perform step 202 to achieve the fusion of the two original training samples in each sample pair, and obtain the fused training samples corresponding to each sample.

[0049] Step 202: Perform sample fusion processing on two original training samples in the sample pair to obtain a fused training sample.

[0050] The fused training samples may be a direct fusion of two original training samples. During the fusion process, no sample transformation is performed on the original training samples, such as cropping, adding noise, deleting or adding text, or other sample transformation operations.

[0051] Step 203: Train the cross-modal retrieval model according to the original training samples and the fused training samples of the cross-modal retrieval model to obtain a target retrieval model.

[0052] Among them, the target retrieval model is used to query the target content that matches the content to be queried, and the modalities of the content to be queried and the target content are different.

[0053] The technical solution disclosed in the present invention can obtain a sample pair of a cross-modal retrieval model to be trained, and the sample pair may include two randomly determined original training samples. By fusing the two original training samples of the sample pair, a fused training sample can be obtained. That is, the present invention adopts a two-sample fusion method to obtain a new training sample, and the obtained fused training sample can contain relevant information of the two original training samples, and there is no need to modify the original training samples, so that the semantic consistency of the fused training sample remains intact. Therefore, the cross-modal retrieval model is trained using the original training samples and the fused training samples, and the training of the cross-modal retrieval model using more training samples is realized, so that the cross-modal retrieval model can learn the characteristics of more training samples, and improve the retrieval accuracy of the target retrieval model obtained by training. When the target retrieval model is used to query the target content that matches the content to be queried, the obtained target content has higher accuracy and is more matched with the content to be queried. The modalities of the content to be queried and the target content can be different, which improves the efficiency and accuracy of cross-modal retrieval.

[0054] In order to enable readers to more deeply understand the implementation principle of the present disclosure, the following Figure 3-Figure 5 right Figure 2 The illustrated embodiment is further refined.

[0055] Further, optionally, based on the above embodiment, the training sample includes a video and text description information of the video. Figure 3 FIG. 2 is a flowchart of another embodiment of a method for training a cross-modal retrieval model provided by an embodiment of the present disclosure, which is different from the above-mentioned embodiment in that step 202: performing sample fusion processing on two original training samples in a sample pair to obtain a fused training sample, including:

[0056] Step 301: Perform video fusion processing on the videos corresponding to the two original training samples in the sample pair to obtain a fused video.

[0057] Step 302: Perform text fusion processing on the text description information corresponding to the two original training samples in the sample pair to obtain fused text.

[0058] Alternatively, the two original training samples can be represented as {V i , T i} and {V j , Tj}, where V and V j For video, T i and T j is the text description information of the corresponding video, i and j are the sample numbers or identifiers of the original training samples respectively.

[0059] Video fusion may refer to adding the video frames of two videos with the same time stamp, that is, adding the coordinate values ​​of each coordinate point to obtain a fused target image.

[0060] Step 303: Determine a fusion training sample according to the fusion video and the fusion text.

[0061] The fusion training samples may be composed of fusion videos and fusion texts.

[0062] In the disclosed embodiment, the sample is augmented by using a sample fusion method. Specifically, the videos corresponding to the two original training samples can be subjected to video fusion processing to obtain a fused video. The fused video can retain the videos corresponding to the two original training samples, and the video content is not lost. The text description information corresponding to the two original samples can also be subjected to text fusion processing to obtain a fused text. The fused text can retain the text description information corresponding to the two original training samples, and the text description information is also not lost. Therefore, the fused video and the fused text can be used as fused training samples to achieve the fusion of the fused training samples in terms of video and text, and obtain a more accurate and comprehensive fused training sample.

[0063] Further, based on any of the embodiments, performing video fusion processing on the videos corresponding to the two original training samples in the sample pair to obtain a fused video includes:

[0064] According to the preset video frame sampling frequency, the target sampling time is determined.

[0065] The video frames corresponding to the target sampling moments are extracted from the videos corresponding to the two original training samples of the sample pair, so as to obtain the video frames corresponding to the two original training samples at the target sampling moments.

[0066] Perform image fusion on the video frames corresponding to the two original training samples collected at the target sampling time to obtain the target image corresponding to the target sampling time;

[0067] Generate a fused video based on the target image corresponding to the target sampling moment.

[0068] Optionally, weights corresponding to two original training samples may be determined, wherein the sum of the weights corresponding to the two training samples is 1. Assume that the weight of one original training sample is α, and the weight of the other original training sample is (1-α), and α may be set to 0.5, for example.

[0069] The image fusion steps of the video frames corresponding to the two original training samples may include:

[0070] The pixel values ​​at the same coordinate position of the video frames corresponding to the two original training samples are weighted according to their respective weights to obtain the target element value corresponding to each coordinate position, so as to determine the target image composed of the target element values ​​corresponding to each coordinate position.

[0071] Specifically, the fused video V q It can be calculated by the following fusion formula:

[0072] in, is the kth image frame in the fused video.

[0073] It can be obtained by combining the following formulas:

[0074]

[0075] in, is a video V of an original training sample from the sample i Extract the obtained k-th image frame. is the video V of the other original training sample from the sample pair j Extract the obtained k-th image frame.

[0076] The target sampling time may include multiple, and video frames can be extracted at each target sampling time to obtain K video frames corresponding to the video of each original training sample. i The extracted image frame can be expressed as: From the video V of another original training sample of the sample pair j The extracted image frame can be expressed as:

[0077] In the disclosed embodiment, the target sampling moment can be determined according to a preset video frame sampling frequency. The target sampling moment can be the sampling moment of the sample pair, and then the video frames corresponding to the target sampling moment can be extracted from the videos corresponding to the two original training samples of the sample pair, so as to achieve synchronous extraction of the videos of the two original training samples, and ensure the consistency of the extraction time of the two original training samples. By performing image fusion on the video frames corresponding to the two original training samples collected at the target sampling moment, the target image at the target sampling moment can be obtained, and the fusion sampling of the image at the target sampling moment can be achieved. Through the target image corresponding to the target sampling moment, a fused video can be generated, that is, the target image after fusion at the target sampling moment is used to achieve accurate extraction of the fused video, so that the target image in the fused video corresponds to the target sampling moment, ensuring the time consistency of the fused video, achieving effective fusion, and obtaining a more accurate fused video.

[0078] Further, based on any of the above embodiments, text fusion processing is performed on the text description information corresponding to the two original training samples in the sample pair to obtain a fused text, including:

[0079] The text description information corresponding to the two original training samples in the sample pair is concatenated to obtain the fused text.

[0080] Optionally, a text concatenation function may be used to concatenate text description information corresponding to two original samples in a sample pair to obtain a fused text.

[0081] Among them, the text concatenation function can be expressed as:

[0082] T k =concat(T i , T j )

[0083] Among them, T i , T j They are the text description information corresponding to the two original training samples respectively.

[0084] In the embodiment of the present disclosure, the text description information corresponding to the two original training samples in the sample pair can be fused by text splicing, so that the fused text is composed based on the text description information of each of the original training samples, and the text description information of each of the two original training samples is not modified, thereby ensuring the content relevance of the fused text and the fused video, and improving the efficiency and accuracy of obtaining the fused text.

[0085] For ease of understanding, Figure 4 An example diagram showing the fusion of video frames and text. Figure 4, assuming that there are video frames 401 and 402 collected at the target sampling time in two original training samples, and text description information 403 and 404 collected at the target sampling time in two original training samples. Among them, the video frame 401 and the video frame 402 can be image fused to obtain the target image 405. The text description information 403 and the text description information 404 can be text spliced ​​to obtain the fused text 406. Of course, Figure 4 The target sampling moment shown in is only one, which is only for illustrating a specific fusion example of image frames and text description information, and should not constitute a limitation on the number of sampling times. In practical applications, there can be multiple target sampling moments, that is, the video frames of the two original training samples collected at each target sampling moment can be fused, and multiple target images can be obtained to generate a fused video based on the multiple target images.

[0086] like Figure 5 As shown, it is a flow chart of another embodiment of a training method for a cross-modal retrieval model provided by an embodiment of the present disclosure, which is different from the above-mentioned embodiment in that the cross-modal retrieval model is trained according to the original training samples and the fused training samples of the cross-modal retrieval model to obtain a target retrieval model, including:

[0087] Step 501: Determine a target training sample for participating in the cross-modal retrieval model training based on the original training sample and the fused training sample of the cross-modal retrieval model to obtain a target video and text description information of the target video in the target training sample.

[0088] Optionally, step 501 may include: directly using the original training samples and fused training samples of the cross-modal retrieval model as target training samples to participate in the training of the cross-modal retrieval model. Of course, in practical applications, the original training samples and / or fused training samples may be screened, and the original training samples and / or fused training samples that meet the sample use conditions may be used as target training samples to ensure the validity of the target training samples. The training samples meeting the sample use conditions may include, for example, that there is no lack of video or text description information corresponding to the video, that the video length meets the length requirement, or that the number of characters in the text description information meets the word count requirement. The video length meets the length requirement, for example, that the video length cannot be too long or too short. The number of characters in the text description information meets the word count requirement, for example, that the number of characters in the text description information cannot be too large or too small.

[0089] Step 502: Determine model parameters of the cross-modal retrieval model.

[0090] Step 503: extracting video semantic features corresponding to the target video in the target training sample and text semantic features corresponding to the text description information of the target video according to the cross-modal retrieval model corresponding to the model parameters.

[0091] Step 504: Determine the target loss value of the cross-modal retrieval model corresponding to the model parameters according to the semantic difference between the video semantic features and the text semantic features.

[0092] Step 505: If it is determined that the target loss value satisfies the loss condition, then the cross-modal retrieval model corresponding to the model parameters when the loss condition is satisfied is determined as the target retrieval model.

[0093] Optionally, step 502 may include: initializing model parameters of the cross-modal retrieval model, and specifically, reference may be made to the parameter initialization process of the machine learning model and the neural network model.

[0094] Optionally, the target loss value satisfies the loss condition, which may mean that the target loss value is less than a loss threshold. The target loss value does not satisfy the loss condition, which may mean that the target loss value is greater than or equal to the loss threshold.

[0095] In the disclosed embodiment, the original training samples and the fused training samples are used to determine the target training samples of the cross-modal retrieval model, and the number of target training samples is larger and the samples are more sufficient. Therefore, in the cross-modal retrieval model training process, accurate extraction of video semantic features and text semantic features can be achieved. Then, semantic difference analysis is performed using video semantic features and text semantic features to obtain the target loss value of the cross-modal retrieval model. And by constraining the target loss value through the loss condition, the satisfaction detection of the loss condition can be realized, so that the target loss value corresponding to the obtained target retrieval model can meet the loss condition and have higher accuracy.

[0096] Further, based on any of the embodiments, according to the semantic difference between the video semantic features and the text semantic features, determining the target loss value of the cross-modal retrieval model corresponding to the model parameters includes:

[0097] According to the semantic constraint formula, the semantic related information between the video semantic features and the text semantic features is calculated;

[0098] According to the semantic related information, the target loss value of the cross-modal retrieval model corresponding to the model parameters is determined.

[0099] Optionally, the semantic constraint formula may be a loss constraint function such as InfoNCE loss (info Noise ContrastiveEstimation loss, a loss function based on contrast).

[0100] Semantically related information can be expressed using the following formula:

[0101]

[0102] Referring to the above formula, s(vi , t j ) is semantically relevant information. j It is the semantic feature of text. is the video semantic feature. That is, the semantic related information can be the quotient of the product of the transposition of the text semantic feature and the video semantic feature and the product of the norm of the text semantic feature and the video semantic norm, to obtain the semantic related information.

[0103] In the disclosed embodiment, the semantically related information between the video semantic features and the text semantic features is calculated through a semantic constraint formula, and the video semantic features and the text semantic features are differentially expressed through the semantically related information. Then, based on the semantically related information, the target loss value of the cross-modal retrieval model corresponding to the model parameters is determined. Using the semantically related information of the video semantic features and the text semantic features as the calculation basis for the target loss value of the cross-modal retrieval model can enable the target loss value to better characterize the difference between the video and the text, improve the degree of training constraints of the cross-modal retrieval model, and obtain a higher accuracy cross-modal retrieval model obtained through training.

[0104] Further, based on any embodiment, determining a target loss value of the cross-modal retrieval model according to semantic related information includes:

[0105] Inputting semantic related information into a first loss function and calculating a first loss value, where the first loss function is a loss function for retrieving text based on a video;

[0106] Inputting semantic related information into a second loss function to calculate a second loss value, where the second loss function is a loss function for retrieving videos based on text;

[0107] The sum of the first loss value and the second loss value is calculated to obtain the target loss value of the cross-modal retrieval model corresponding to the model parameters.

[0108] Optionally, the first loss function and the second loss function may be constructed by an exponential function exp and a logarithmic function log.

[0109] Optionally, the first loss function can be expressed using the following formula:

[0110]

[0111] The second loss function can be expressed using the following formula:

[0112]

[0113] Among them, s(v i , t i ) can be a kind of semantic related information. j is the text semantic feature, v iis the video semantic feature. B is the number of text semantic features and video semantic features, that is, the number of target training samples.

[0114] Specifically, the calculation process of the first loss function can be: sum the exponential value of the exponential function of the semantically relevant information and the value of the exponential function of the semantically relevant information from each video to the text to obtain the exponential sum. Calculate the quotient of the exponential value and the exponential sum to obtain the exponential error, calculate the logarithm of each exponential error, and add the logarithm values ​​to obtain the first logarithmic sum, and then calculate the sum of the first logarithmic sum and The semantic information from video to text can be expressed as s(v i , t j ).

[0115] Similarly, the calculation process of the second loss function can be: sum the exponential value of the exponential function of the semantically relevant information and the value of the exponential function of the semantically relevant information from each text to the video to obtain the exponential sum. Calculate the quotient of the exponential value and the exponential sum to obtain the exponential error, calculate the logarithm of each exponential error, and add the logarithm values ​​to obtain the second logarithmic sum, and then calculate the second logarithmic sum and The semantic information from video to text can be expressed as s(v j , t i ).

[0116] In the disclosed embodiment, the first loss function and the second loss function can be used to respectively calculate the first loss value and the second loss value corresponding to the semantically relevant information. The first loss function is the loss function when retrieving text based on video, and the second loss function is the loss function when retrieving video based on text. The cross-modal retrieval model performs cross-modal loss calculation on the loss of text and video, so that the target loss value corresponding to the sum of the first loss value and the second loss value can be subject to the loss constraints of the two aspects of text to video and video text, so that the composition of the target loss value has a broader meaning and the loss representation of the cross-modal retrieval model is more accurate.

[0117] Further, based on any of the embodiments, according to the cross-modal retrieval model corresponding to the model parameters, extracting the video semantic features corresponding to the target video in the target training sample and the text semantic features corresponding to the text description information of the target video, including:

[0118] Determine the video semantic extraction sub-model and the text semantic extraction sub-model corresponding to the model parameters of the cross-modal retrieval model;

[0119] The video semantic extraction sub-model is used to extract the video semantic features of the target video in the target training sample;

[0120] The text semantic extraction sub-model is used to extract the text semantic features of the text description information of the target video in the target training sample.

[0121] Optionally, the video semantic extraction sub-model may include a ViT (Vision Transformer) model and a Transformer (conversion) model. The Vit model can be used as an image feature extractor to extract a feature sequence of the target video, and then the feature sequence is input into the Transformer model for processing to obtain a vector of the CSI (classification) position output by the Transformer model as a video semantic feature.

[0122] Optionally, the text semantic extraction sub-model may include a neural network model such as a BERT (Bidirectional Encoder Representations from Transformer) model. The text semantic extraction sub-model can be used to obtain local features of the text at the word level, and text semantic features can be obtained after global average pooling of the local features.

[0123] In the disclosed embodiment, by determining the video semantic extraction sub-model and the text semantic extraction sub-model corresponding to the model parameters of the cross-modal retrieval model, the text semantics and the video semantics are processed separately. Then, the video semantic extraction sub-model is used to extract the video semantic features of the target video, and the text semantic extraction sub-model is used to extract the text semantic features of the text description information of the target video, so as to achieve accurate and efficient extraction of video semantic features and text semantic features.

[0124] Further, based on any of the above embodiments, the method may further include:

[0125] If it is determined that the target loss value does not meet the loss condition, the model parameters of the cross-modal retrieval model are updated, and the process returns to the cross-modal retrieval model corresponding to the model parameters, and the steps of extracting the video semantic features corresponding to the target video in the target training sample and the text semantic features corresponding to the text description information of the target video are continued.

[0126] Optionally, the target loss value is the training error of the cross-modal retrieval model corresponding to the model parameters. The target loss value can be used to evaluate the retrieval effect of the cross-modal retrieval model corresponding to the model parameters. The target loss value can be used to control the update of the cross-modal retrieval model to improve the update accuracy and efficiency of the cross-modal retrieval model.

[0127] Optionally, updating the model parameters of the cross-modal retrieval model may include: updating the model parameters of the cross-modal retrieval model by combining a target loss value with a gradient descent algorithm.

[0128] In the disclosed embodiment, during the training process of the cross-modal retrieval model, when the target loss value does not meet the loss condition, the cross-modal retrieval model can be updated to achieve an update of the cross-modal retrieval model, implement iterative training of the cross-modal retrieval model, and improve the training efficiency of the cross-modal retrieval model.

[0129] like Figure 6 As shown, it is a flowchart of an embodiment of a query method of a cross-modal retrieval model provided by an embodiment of the present disclosure, and the method may include the following steps:

[0130] Step 601: Receive query content sent by a user terminal, where the query content belongs to the first mode.

[0131] Step 602: Input the query content into the trained target retrieval model to obtain target content matching the query content, wherein the target retrieval model is trained based on the training method of the cross-modal retrieval model shown in any of the above embodiments, and the target content belongs to the second modality.

[0132] Step 603: Send the target content to the user terminal of the user, and the target content is displayed by the user terminal.

[0133] Optionally, the query content may include video, image or text. The target content may be content with a different modality from the query content. For example, when the query content is an image or text, the target content may be a video and the query content may be text or an image. Of course, in actual applications, the query content and the target content may be in different modalities.

[0134] In the disclosed embodiment, the electronic device can receive the query content sent by the user terminal, and input the query content into the target retrieval model obtained through training. The target content matching the query content is retrieved through the target retrieval model, so that the target content is sent to the user terminal of the user to realize content query and content feedback, and the query content belongs to the first modality, and the target content belongs to the second modality, so as to realize cross-modal content query. The target retrieval model is obtained through training based on the training method of the above-mentioned cross-modal retrieval model. Fusion training samples are used in the training process, the number of training samples is more sufficient, and the target retrieval model obtained through training has higher accuracy. Therefore, the content query accuracy through the target retrieval model with higher accuracy is higher, which improves the accuracy of content query.

[0135] like Figure 7 As shown, it is a structural schematic diagram of an embodiment of a cross-modal retrieval model training device provided by an embodiment of the present disclosure. The cross-modal retrieval model training device 700 may include the following units:

[0136] Acquisition unit 701: used to acquire a sample pair of a cross-modal retrieval model to be trained, where the sample pair includes two randomly determined original training samples.

[0137] Fusion unit 702: used to perform sample fusion processing on two original training samples in a sample pair to obtain a fused training sample.

[0138] Training unit 703: used to train the cross-modal retrieval model according to the original training samples and fused training samples of the cross-modal retrieval model to obtain a target retrieval model.

[0139] Among them, the target retrieval model is used to query the target content that matches the content to be queried, and the modalities of the content to be queried and the target content are different.

[0140] Further, based on any of the above embodiments, the training sample includes a video and text description information of the video, and the fusion unit includes:

[0141] A video fusion module is used to perform video fusion processing on the videos corresponding to the two original training samples in the sample pair to obtain a fused video;

[0142] A text fusion module is used to perform text fusion processing on the text description information corresponding to the two original training samples in the sample pair to obtain a fused text;

[0143] The sample determination module is used to determine the fusion training sample according to the fusion video and the fusion text.

[0144] Further, based on any of the above embodiments, the video fusion module includes:

[0145] The time determination submodule is used to determine the target sampling time according to the preset video frame sampling frequency;

[0146] An image sampling submodule is used to extract video frames corresponding to target sampling moments from the videos corresponding to the two original training samples of the sample pair, so as to obtain video frames corresponding to the two original training samples at the target sampling moments;

[0147] An image fusion submodule is used to fuse the video frames corresponding to the two original training samples collected at the target sampling time to obtain the target image corresponding to the target sampling time;

[0148] The video generation submodule is used to generate a fused video according to the target image corresponding to the target sampling moment.

[0149] Further, based on any of the above embodiments, the text fusion module includes:

[0150] The text splicing submodule is used to perform text splicing on the text description information corresponding to the two original training samples in the sample pair to obtain a fused text.

[0151] Further, based on any of the above embodiments, the training unit includes:

[0152] A sample determination module is used to determine a target training sample participating in the cross-modal retrieval model training according to the original training sample and the fusion training sample of the cross-modal retrieval model, so as to obtain a target video and text description information of the target video in the target training sample;

[0153] A parameter determination module, used to determine model parameters of the cross-modal retrieval model;

[0154] A semantic extraction module, used to extract video semantic features corresponding to a target video in a target training sample and text semantic features corresponding to text description information of the target video according to a cross-modal retrieval model corresponding to model parameters;

[0155] A loss calculation module, used to determine a target loss value of a cross-modal retrieval model corresponding to a model parameter according to a semantic difference between a video semantic feature and a text semantic feature;

[0156] The target determination module is used to determine that the cross-modal retrieval model corresponding to the model parameters is the target retrieval model if it is determined that the target loss value meets the loss condition.

[0157] Further, based on any of the above embodiments, the loss calculation module includes:

[0158] A semantic correlation submodule, used to calculate semantic correlation information between video semantic features and text semantic features according to a semantic constraint formula;

[0159] The loss calculation submodule is used to determine the target loss value of the cross-modal retrieval model corresponding to the model parameters based on semantic related information.

[0160] Further, based on any of the above embodiments, the loss calculation submodule is specifically used for:

[0161] Inputting semantic related information into a first loss function and calculating a first loss value, where the first loss function is a loss function for retrieving text based on a video;

[0162] Inputting semantic related information into a second loss function and calculating a second loss value, where the second loss value is a loss function when retrieving a video based on text;

[0163] The first loss value and the second loss value are added to obtain the target loss value of the cross-modal retrieval model corresponding to the model parameters.

[0164] Further, based on any of the above embodiments, the semantic extraction module includes:

[0165] A model determination submodule, used to determine a video semantic extraction submodel and a text semantic extraction submodel corresponding to model parameters of a cross-modal retrieval model;

[0166] A video extraction submodule, used to extract video semantic features of a target video in a target training sample using a video semantic extraction submodel;

[0167] The text extraction submodule is used to extract text semantic features of text description information of the target video in the target training sample by using the text semantic extraction submodel.

[0168] Further, based on any of the above embodiments, the training unit further includes:

[0169] The parameter updating module is used to update the model parameters of the cross-modal retrieval model if it is determined that the target loss value does not meet the loss condition, return to the cross-modal retrieval model corresponding to the model parameters, and continue to extract the video semantic features corresponding to the target video in the target training sample and the text semantic features corresponding to the text description information of the target video.

[0170] like Figure 8 As shown, it is a structural schematic diagram of an embodiment of a query device of a cross-modal retrieval model provided by an embodiment of the present disclosure, and the device may include the following units:

[0171] Receiving unit 801: used to receive query content sent by a user terminal, where the query content belongs to the first modality.

[0172] Query unit 802: used to input the query content into the trained target retrieval model to obtain target content matching the query content, wherein the target retrieval model is trained based on the training method of the cross-modal retrieval model shown in any of the above embodiments, and the target content belongs to the second modality.

[0173] The sending unit 803 is used to send the target content to the user terminal of the user, and the target content is displayed by the user terminal.

[0174] The device disclosed in the present invention can be used to execute the above method. For the specific execution content of each device, please refer to the description of the relevant method, which will not be repeated here.

[0175] It should be noted that the cross-modal retrieval model in this embodiment is not a head model for a specific user and cannot reflect the personal information of a specific user. It should be noted that the training samples in this embodiment are from a public data set.

[0176] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0177] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0178] The electronic device may include at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided in any of the above embodiments.

[0179] Among them, a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions can be used to enable a computer to execute the method provided by any of the above embodiments.

[0180] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0181] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0182] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0183] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0184] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as a training method or a query method for a cross-modal retrieval model. For example, in some embodiments, the training method or the query method for a cross-modal retrieval model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the training method or the query method for the cross-modal retrieval model described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute a training method or a query method for a cross-modal retrieval model in any other appropriate manner (eg, by means of firmware).

[0185] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0186] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0187] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0188] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0189] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0190] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0191] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0192] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for training a cross-modal retrieval model, comprising: Obtaining a sample pair of a cross-modal retrieval model to be trained, wherein the sample pair includes two randomly determined original training samples; Performing video fusion processing on the videos corresponding to the two original training samples in the sample pair to obtain a fused video, wherein the video fusion refers to adding and calculating video frames of the two videos with the same time stamp, wherein the videos of the original training samples are not modified during the video fusion processing; Performing text fusion processing on the text description information corresponding to the two original training samples in the sample pair to obtain a fused text, wherein the text description information of each of the original training samples is not modified during the text fusion processing; Determining a fusion training sample according to the fusion video and the fusion text; Determining target training samples participating in the cross-modal retrieval model training according to the original training samples and the fused training samples of the cross-modal retrieval model, wherein the target training samples include the original training samples and the fused training samples, or include the original training samples and / or the fused training samples that meet the sample use condition; Training the cross-modal retrieval model according to the target training sample to obtain a target retrieval model; The target retrieval model is used to search for target content that matches the content to be queried, and the content to be queried and the target content have different modalities; The step of performing video fusion processing on the videos corresponding to the two original training samples in the sample pair to obtain a fused video includes: Determine at least one target sampling time according to a preset video frame sampling frequency; Extracting video frames corresponding to the at least one target sampling moment from the videos corresponding to the two original training samples of the sample pair, respectively, to obtain video frames corresponding to the at least one target sampling moment respectively by the two original training samples; The video frames corresponding to the two original training samples collected at the at least one target sampling moment are respectively fused to obtain at least one target image corresponding to the at least one target sampling moment; and the fused video is generated according to the at least one target image corresponding to the at least one target sampling moment.

2. The method according to claim 1, wherein: The step of performing text fusion processing on the text description information respectively corresponding to the two original training samples in the sample pair to obtain fused text includes: The text description information corresponding to the two original training samples in the sample pair is concatenated to obtain the fused text.

3. The method according to claim 1, wherein: The target training sample includes a target video and text description information of the target video, and the cross-modal retrieval model is trained according to the target training sample to obtain a target retrieval model, including: Determining model parameters of the cross-modal retrieval model; Extracting video semantic features corresponding to the target video in the target training sample and text semantic features corresponding to the text description information of the target video according to the cross-modal retrieval model corresponding to the model parameters; Determining a target loss value of a cross-modal retrieval model corresponding to the model parameter according to a semantic difference between the video semantic feature and the text semantic feature; If it is determined that the target loss value satisfies the loss condition, then the cross-modal retrieval model corresponding to the model parameters when the loss condition is satisfied is determined to be the target retrieval model.

4. The method according to claim 3, wherein: The step of determining the target loss value of the cross-modal retrieval model corresponding to the model parameter according to the semantic difference between the video semantic feature and the text semantic feature includes: Calculating semantically related information between the video semantic features and the text semantic features according to a semantic constraint formula; Determine a target loss value of a cross-modal retrieval model corresponding to the model parameters according to the semantically relevant information.

5. The method according to claim 4, wherein: Determining the target loss value of the cross-modal retrieval model according to the semantic related information includes: Inputting the semantic related information into a first loss function to calculate a first loss value, where the first loss function is a loss function for retrieving text based on a video; Inputting the semantic related information into a second loss function to calculate a second loss value, where the second loss function is a loss function for retrieving videos based on text; The sum of the first loss value and the second loss value is calculated to obtain a target loss value of the cross-modal retrieval model corresponding to the model parameters.

6. The method according to any one of claims 3 to 5, wherein: The step of extracting video semantic features corresponding to the target video and text semantic features corresponding to text description information of the target video in the target training sample according to the cross-modal retrieval model corresponding to the model parameters includes: Determine a video semantic extraction sub-model and a text semantic extraction sub-model of the cross-modal retrieval model corresponding to the model parameters; Extracting video semantic features of a target video in the target training sample using a video semantic extraction sub-model; The text semantic extraction sub-model is used to extract text semantic features of the text description information of the target video in the target training sample.

7. The method according to any one of claims 3 to 5, further comprising: If it is determined that the target loss value does not meet the loss condition, the model parameters of the cross-modal retrieval model are updated, and the step of returning to the cross-modal retrieval model corresponding to the model parameters and extracting the video semantic features corresponding to the target video in the target training sample and the text semantic features corresponding to the text description information of the target video is continued.

8. A query method for a cross-modal retrieval model, comprising: Receiving query content sent by a user terminal, where the query content belongs to a first modality; Inputting the query content into a trained target retrieval model to obtain target content matching the query content, wherein the target retrieval model is trained based on the training method of the cross-modal retrieval model of claims 1 to 7, and the target content belongs to the second modality; The target content is sent to the user terminal of the user, and the target content is displayed by the user terminal.

9. A training device for a cross-modal retrieval model, comprising: An acquisition unit, used to acquire a sample pair of a cross-modal retrieval model to be trained, wherein the sample pair includes two randomly determined original training samples; A fusion unit, used for performing sample fusion processing on two original training samples in the sample pair to obtain a fused training sample; A training unit, configured to train the cross-modal retrieval model according to the original training samples of the cross-modal retrieval model and the fused training samples to obtain a target retrieval model; The target retrieval model is used to search for target content that matches the content to be queried, and the content to be queried and the target content have different modalities; The training sample includes a video and text description information of the video, and the fusion unit includes: A video fusion module, used for performing video fusion processing on the videos corresponding to the two original training samples in the sample pair to obtain a fused video, wherein the video fusion refers to adding and calculating video frames of the two videos with the same time stamp, wherein the videos of the original training samples are not modified during the video fusion processing; A text fusion module, used for performing text fusion processing on the text description information corresponding to the two original training samples in the sample pair to obtain a fused text, wherein the text description information of each of the original training samples is not modified during the text fusion processing; A sample determination module, used for determining the fusion training sample according to the fusion video and the fusion text; The training unit is specifically used to determine target training samples participating in the training of the cross-modal retrieval model according to the original training samples and the fused training samples of the cross-modal retrieval model, wherein the target training samples include the original training samples and the fused training samples, or include the original training samples and / or the fused training samples that meet the sample use conditions; train the cross-modal retrieval model according to the target training samples to obtain a target retrieval model; Wherein, the video fusion module includes: A time determination submodule, used to determine at least one target sampling time according to a preset video frame sampling frequency; An image sampling submodule, configured to extract video frames corresponding to the at least one target sampling moment from the videos corresponding to the two original training samples of the sample pair, and obtain video frames corresponding to the at least one target sampling moment of the two original training samples; An image fusion submodule, used to perform image fusion on the video frames corresponding to the two original training samples collected at the at least one target sampling moment, to obtain at least one target image corresponding to the at least one target sampling moment; The video generation submodule is used to generate the fused video according to at least one target image corresponding to the at least one target sampling moment.

10. The device according to claim 9, wherein: The text fusion module comprises: The text splicing submodule is used to perform text splicing on the text description information corresponding to the two original training samples in the sample pair to obtain the fused text.

11. The device according to claim 9, wherein: The training unit comprises: A sample determination module, used to determine a target training sample participating in the training of the cross-modal retrieval model according to the original training sample of the cross-modal retrieval model and the fused training sample, so as to obtain a target video and text description information of the target video in the target training sample; A parameter determination module, used to determine model parameters of the cross-modal retrieval model; A semantic extraction module, used to extract video semantic features corresponding to the target video in the target training sample and text semantic features corresponding to the text description information of the target video according to the cross-modal retrieval model corresponding to the model parameters; A loss calculation module, used to determine a target loss value of the cross-modal retrieval model corresponding to the model parameter according to the semantic difference between the video semantic feature and the text semantic feature; A target determination module is used to determine that the cross-modal retrieval model corresponding to the model parameters is the target retrieval model if it is determined that the target loss value satisfies the loss condition.

12. The device according to claim 11, wherein The loss calculation module comprises: A semantic correlation submodule, used for calculating semantic correlation information between the video semantic features and the text semantic features according to a semantic constraint formula; The loss calculation submodule is used to determine the target loss value of the cross-modal retrieval model corresponding to the model parameters based on the semantic related information.

13. The device according to claim 12, wherein: The loss calculation submodule is specifically used for: Inputting semantic related information into a first loss function to calculate a first loss value, wherein the first loss function is a loss function for retrieving text based on a video; Inputting the semantic related information into a second loss function to calculate a second loss value, where the second loss value is a loss function when retrieving a video based on text; The first loss value and the second loss value are added to obtain a target loss value of the cross-modal retrieval model corresponding to the model parameters.

14. The device according to any one of claims 11 to 13, wherein: The semantic extraction module comprises: A model determination submodule, used to determine the video semantic extraction submodel and the text semantic extraction submodel of the cross-modal retrieval model corresponding to the model parameters; A video extraction submodule, used to extract video semantic features of a target video in the target training sample using a video semantic extraction submodel; The text extraction submodule is used to extract text semantic features of the text description information of the target video in the target training sample by using the text semantic extraction submodel.

15. The device according to any one of claims 11 to 13, wherein: The training unit further includes: A parameter updating module is used to update the model parameters of the cross-modal retrieval model if it is determined that the target loss value does not meet the loss condition, return to the cross-modal retrieval model corresponding to the model parameters, and continue to extract the video semantic features corresponding to the target video in the target training sample and the text semantic features corresponding to the text description information of the target video.

16. A query device for a cross-modal retrieval model, comprising: A receiving unit, configured to receive query content sent by a user terminal, wherein the query content belongs to a first mode; A query unit, configured to input the query content into a trained target retrieval model to obtain target content matching the query content, wherein the target retrieval model is trained based on the training method of the cross-modal retrieval model of claims 1 to 7, and the target content belongs to the second modality; A sending unit is used to send the target content to the user terminal of the user, and the target content is displayed by the user terminal.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7 or 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7 or 8.