A training sample generation method, model training method and device

Abstract

Embodiments of the present application provide a training sample generation method and device, and a model training method, relating to the technical field of artificial intelligence, comprising: obtaining a plurality of original training samples; for each original training sample, obtaining video features and text features of the original training sample; determining a plurality of target image regions and a plurality of target graphemes in the original training sample based on the video features and the text features of the original training sample; calculating quality data of the original training sample based on the video features of the plurality of target image regions and the text features of the plurality of target graphemes; and selecting each original training sample based on a first number of original training samples with quality data less than a first preset threshold to obtain a target training sample. The embodiments can improve the training effect of the cross-modal model, save time and labor costs, improve the generation efficiency of the training sample, and further improve the training efficiency of the cross-modal model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a training sample generation method and a model training method and device. BACKGROUND

[0002] Based on the cross-modal model, resources containing the same target object as the target resource of another modal can be determined from multiple resources of one modal. For example, the cross-modal model is a video-text cross-modal model, the electronic device obtains a target video containing a target object, and then the similarity between the target video and multiple texts can be calculated through the video-text cross-modal model, and the text with the maximum similarity to the target video is determined to obtain the text containing the target object.

[0003] Before determining the text containing the target object through the video-text cross-modal model, the electronic device can use a preset training set to train the initial structure of the video-text cross-modal model. The preset training set contains multiple original training samples. In related technologies, the quality of each original training sample is judged by technical personnel according to experience, and based on the quality of each original training sample, a target training sample with higher quality is determined from the preset training set. Then, based on the target training sample, the initial structure of the video-text cross-modal model is trained.

[0004] However, the quality of each original training sample is judged by technical personnel according to experience, and a target training sample with higher quality is determined from the preset training set, which requires a large time cost and labor cost, resulting in low efficiency of generating training samples in related technologies, and further resulting in low efficiency of training the video-text cross-modal model. SUMMARY

[0005] The purpose of the embodiment of the present application is to provide a training sample generation method, a model training method and device to improve the generation efficiency of training samples and further improve the efficiency of training the cross-modal model. The specific technical solutions are as follows:

[0006] In the first aspect of the present application, a training sample generation method is first provided, which comprises:

[0007] obtaining multiple original training samples; wherein one original training sample comprises a sample video and a sample text;

[0008] For each original training sample, the video features of each target video frame of the sample video in the original training sample and the text features of the sample text are obtained; wherein the video features of each target video frame represent the first object contained in the target video frame; the text features of the sample text represent the second object described in the sample text;

[0009] determine, based on the video features of the target video frames, target image regions containing the first object from the target video frames, to obtain a plurality of target image regions of the sample video, and determine, based on the text features of the sample text, target morphemes for describing the second object from the sample text, to obtain a plurality of target morphemes of the sample text;

[0010] calculate, based on the video features of the plurality of target image regions of the sample video in the original training sample and the text features of the plurality of target morphemes of the sample text in the original training sample, a similarity between the sample video and the sample text in the original training sample as quality data of the original training sample;

[0011] obtain a first number of original training samples whose quality data is less than a first preset threshold, and select each original training sample based on the first number to obtain a target training sample.

[0012] Optionally, the video feature of a target video frame is a response heat map of the target video frame.

[0013] The determining, based on the video features of the target video frames, target image regions containing the first object from the target video frames, to obtain a plurality of target image regions of the sample video, comprises:

[0014] For each target video frame, determine, from the target video frame, an image region composed of each pixel point in the response heat map of the target video frame whose response value is greater than a second preset threshold, as a key image region.

[0015] Determine the minimum circumscribed rectangle of each key image region in the target video frame to obtain a target image region of the target video frame.

[0016] Select target image regions of a second number of target video frames from the target image regions of the target video frames to obtain a plurality of target image regions of the sample video in the original training sample.

[0017] Optionally, the text feature of the sample text in the original training sample is a response heat map of the sample text.

[0018] The determining, based on the text features of the sample text, target morphemes for describing the second object from the sample text, to obtain a plurality of target morphemes of the sample text, comprises:

[0019] Determine, from the sample text in the original training sample, a morpheme in the response heat map of the sample text whose response value is greater than a third preset threshold, as a key morpheme.

[0020] Based on the response values ​​of each key morpheme in the response heatmap of the sample text from high to low, select the top three key morphemes from each key morpheme to obtain multiple target morphemes of the sample text in the original training sample.

[0021] Optionally, the calculation of the similarity between the sample video and sample text in the original training sample, based on the video features of multiple target image regions of the sample video in the original training sample and the text features of multiple target morphemes of the sample text in the original training sample, as the quality data of the original training sample, includes:

[0022] Each target image region of the sample video is combined with each target morpheme of the sample text to obtain multiple initial mapping combinations; each initial mapping combination contains one target image region and one target morpheme.

[0023] For each initial mapping combination, the similarity between the video features of the target image region and the text features of the target morpheme is calculated, and this similarity is used as the similarity of the initial mapping combination.

[0024] Based on the similarity corresponding to each initial mapping combination, the quality data of the original training sample is determined.

[0025] Optionally, determining the quality data of the original training sample based on the similarity corresponding to each initial mapping combination includes:

[0026] From each initial mapping combination, determine the initial mapping combination with a similarity greater than the fourth preset threshold as the candidate mapping combination;

[0027] Based on the similarity of each alternative mapping combination, the quality data of the original training sample is determined.

[0028] Optionally, determining the quality data of the original training sample based on the similarity corresponding to each candidate mapping combination includes:

[0029] For each target image region, select one alternative mapping combination from the alternative mapping combinations that contain the target image region as the target mapping combination for that target image region; wherein, the target morphemes included in the target mapping combinations of each target image region are different;

[0030] The target mappings, which include each target image region, are combined and defined as a cross-modal combination;

[0031] Calculate the sum of similarities for each target mapping combination in the cross-modal combination to obtain the quality data of the cross-modal combination;

[0032] The maximum value among the quality data of each cross-modal combination is determined as the quality data of the original training sample.

[0033] In a second aspect of the present invention, a model training method is also provided, the method comprising:

[0034] Obtain target training samples including sample videos and sample texts, and obtain sample similarity between the sample videos and the sample texts; wherein, the target training samples are determined based on any of the training sample generation methods described in the first aspect above;

[0035] The sample video and the sample text are input into the cross-modal model with the initial structure to obtain the predicted similarity between the sample video and the sample text;

[0036] Calculate the loss function value representing the difference between the predicted similarity and the sample similarity;

[0037] The model parameters of the cross-modal model of the initial structure are adjusted based on the calculated loss function value until the preset convergence condition is met, thus obtaining the trained cross-modal model.

[0038] In a third aspect of the invention, a training sample generation apparatus is also provided, the apparatus comprising:

[0039] The original training sample acquisition module is used to acquire multiple original training samples; one original training sample includes: a sample video and a sample text.

[0040] The feature acquisition module is used to acquire, for each original training sample, the video features of each target video frame in the sample video of the original training sample, and the text features of the sample text; wherein, the video features of each target video frame represent the first object contained in the target video frame; and the text features of the sample text represent the second object described by the sample text.

[0041] The determination module is used to determine, based on the video features of each target video frame, the target image region containing the first object from each target video frame, to obtain multiple target image regions of the sample video, and to determine, based on the text features of the sample text, the target morphemes used to describe the second object from the sample text, to obtain multiple target morphemes of the sample text.

[0042] The quality data calculation module is used to calculate the similarity between the sample video and the sample text in the original training sample based on the video features of multiple target image regions of the sample video in the original training sample and the text features of multiple target morphemes of the sample text in the original training sample, and to use the similarity as the quality data of the original training sample.

[0043] The filtering module is used to obtain a first number of original training samples whose quality data is less than a first preset threshold, and to filter each original training sample based on the first number to obtain the target training sample.

[0044] Optionally, the video features of a target video frame are the response heatmap of that target video frame;

[0045] The determining module is specifically used for:

[0046] For each target video frame, the image region composed of each pixel whose response value in the response heatmap of the target video frame is greater than the second preset threshold is determined as the key image region.

[0047] Determine the minimum bounding rectangle of each key image region in the target video frame to obtain the target image region of the target video frame;

[0048] From the target image regions of each target video frame, select the target image regions of a second number of target video frames to obtain multiple target image regions of the sample video in the original training sample.

[0049] Optionally, the text features of the sample text in the original training samples are the response heatmap of the sample text;

[0050] The determining module is specifically used for:

[0051] From the sample text in the original training sample, identify the morphemes whose response values ​​in the response heatmap of the sample text are greater than a third preset threshold, and use them as key morphemes.

[0052] Based on the response values ​​of each key morpheme in the response heatmap of the sample text from high to low, select the top three key morphemes from each key morpheme to obtain multiple target morphemes of the sample text in the original training sample.

[0053] Optionally, the quality data calculation module is specifically used for:

[0054] Each target image region of the sample video is combined with each target morpheme of the sample text to obtain multiple initial mapping combinations; each initial mapping combination contains one target image region and one target morpheme.

[0055] For each initial mapping combination, the similarity between the video features of the target image region and the text features of the target morpheme is calculated, and this similarity is used as the similarity of the initial mapping combination.

[0056] Based on the similarity corresponding to each initial mapping combination, the quality data of the original training sample is determined.

[0057] Optionally, the quality data calculation module is specifically used for:

[0058] From each initial mapping combination, determine the initial mapping combination with a similarity greater than the fourth preset threshold as the candidate mapping combination;

[0059] Based on the similarity of each alternative mapping combination, the quality data of the original training sample is determined.

[0060] Optionally, the quality data calculation module is specifically used for:

[0061] For each target image region, select one alternative mapping combination from the alternative mapping combinations that contain the target image region as the target mapping combination for that target image region; wherein, the target morphemes included in the target mapping combinations of each target image region are different;

[0062] The target mappings, which include each target image region, are combined and defined as a cross-modal combination;

[0063] Calculate the sum of similarities for each target mapping combination in the cross-modal combination to obtain the quality data of the cross-modal combination;

[0064] The maximum value among the quality data of each cross-modal combination is determined as the quality data of the original training sample.

[0065] In a fourth aspect of the invention, a model training apparatus is also provided, the apparatus comprising:

[0066] A sample acquisition module is used to acquire target training samples including sample videos and sample text, and to acquire the sample similarity between the sample videos and the sample text; wherein the target training samples are determined based on any of the training sample generation methods described in the first aspect above;

[0067] The prediction similarity acquisition module is used to input the sample video and the sample text into the cross-modal model of the initial structure to obtain the predicted similarity between the sample video and the sample text;

[0068] The loss function value calculation module is used to calculate the loss function value representing the difference between the predicted similarity and the sample similarity;

[0069] The model training module is used to adjust the model parameters of the cross-modal model of the initial structure based on the calculated loss function value until the preset convergence condition is reached, so as to obtain the trained cross-modal model.

[0070] In a fifth aspect of the present invention, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0071] Memory, used to store computer programs;

[0072] When a processor executes a program stored in memory, it implements the training sample generation method steps described in any of the first aspects above, or the model training method steps described in any of the second aspects above.

[0073] In a sixth aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when executed by a processor, the computer program implements the training sample generation method described in any of the first aspects above, or the model training method described in any of the second aspects above.

[0074] In another aspect of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the training sample generation method described in any of the first aspects above, or the model training method described in any of the second aspects above.

[0075] The training sample generation method provided in this embodiment of the invention obtains multiple original training samples. Each original training sample includes a sample video and a sample text. For each original training sample, the video features of each target video frame of the sample video and the text features of the sample text are obtained. The video features of each target video frame represent a first object contained in the target video frame; the text features of the sample text represent a second object described by the sample text. Based on the video features of each target video frame, target image regions containing the first object are determined from each target video frame, resulting in multiple target image regions of the sample video. Based on the text features of the sample text, target morphemes used to describe the second object are determined from the sample text, resulting in multiple target morphemes of the sample text. Based on the video features of the multiple target image regions of the sample video in the original training sample and the text features of the multiple target morphemes of the sample text in the original training sample, the similarity between the sample video and the sample text in the original training sample is calculated as the quality data of the original training sample. A first number of original training samples with quality data less than a first preset threshold is obtained, and each original training sample is filtered based on the first number to obtain target training samples.

[0076] Based on the above processing, since the video features of the target video frame can represent the first object contained in the target video frame, the target image region determined based on the video features of the target video frame is the image region containing the first object. The text features of the sample text can represent the second object described by the sample text, and the target morphemes determined based on the text features of the sample text are the morphemes used to describe the second object. Furthermore, based on the video features of the target image region of the sample video and the text features of the target morphemes of the sample text, the similarity between the sample video and the sample text is calculated as the quality data of the original training sample. The quality data of the original training sample can then represent the similarity between the first object contained in the sample video and the second object described in the sample text. Furthermore, based on the number of original training samples with quality data less than a first preset threshold, each original training sample is filtered to obtain the target training sample. This avoids the problem of low accuracy in cross-modal models due to a high proportion of low-quality original training samples, which leads to difficulties in cross-modal model convergence. This can improve the training effect of cross-modal models. Furthermore, it eliminates the need for technicians to judge the quality of each original training sample based on experience and to manually determine the target training sample from each original training sample, which can save time and labor costs, improve the efficiency of training sample generation, and thus improve the efficiency of training cross-modal models. Attached Figure Description

[0077] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0078] Figure 1 This is a first flowchart of a training sample generation method provided in an embodiment of the present invention;

[0079] Figure 2 This is a second flowchart of the training sample generation method provided in an embodiment of the present invention;

[0080] Figure 3 This is a third flowchart of the training sample generation method provided in the embodiments of the present invention;

[0081] Figure 4 This is a fourth flowchart of the training sample generation method provided in the embodiments of the present invention;

[0082] Figure 5 A flowchart of a model training method provided in an embodiment of the present invention;

[0083] Figure 6 A fifth flowchart of the training sample generation method provided in the embodiments of the present invention;

[0084] Figure 7A structural diagram of a training sample generation device provided in an embodiment of the present invention;

[0085] Figure 8 A structural diagram of a model training device provided in an embodiment of the present invention;

[0086] Figure 9 This is a structural diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0087] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0088] Cross-modal models can identify resources containing the same target object as a target resource in another modality from multiple resources in one modality. Before determining the text containing the target object using a video-text cross-modal model, an electronic device can train an initial-structure video-text cross-modal model using a pre-set training set containing multiple original training samples. In related technologies, technicians judge the quality of each original training sample based on experience, and based on the quality of each original training sample, determine high-quality target training samples from the pre-set training set. Then, based on the target training samples, the initial-structure video-text cross-modal model is trained. However, judging the quality of each original training sample based on experience and determining high-quality target training samples from the pre-set training set requires significant time and labor costs, resulting in low efficiency in generating training samples in related technologies, and consequently, low efficiency in training the initial-structure video-text cross-modal model.

[0089] To address the aforementioned problems, embodiments of the present invention provide a training sample generation method applied to an electronic device. The electronic device can acquire multiple original training samples and, according to the training sample generation method provided in this embodiment, obtain quality data for each original training sample. Furthermore, the electronic device can filter the original training samples based on the number of original training samples whose quality data is less than a first preset threshold to obtain target training samples. This eliminates the need for technicians to judge the quality of each original training sample based on experience and to manually determine the target training sample from among the original training samples, saving time and labor costs, improving the efficiency of training sample generation, and thus improving the efficiency of training cross-modal models.

[0090] See Figure 1 , Figure 1 This is a first flowchart of a training sample generation method provided in an embodiment of the present invention. The method may include the following steps:

[0091] S101: Obtain multiple original training samples.

[0092] One original training sample consists of one sample video and one sample text.

[0093] S102: For each original training sample, obtain the video features of each target video frame in the sample video of the original training sample, as well as the text features of the sample text.

[0094] In this context, the video features of each target video frame represent the first object contained in that target video frame; and the text features of the sample text represent the second object described by that sample text.

[0095] S103: Based on the video features of each target video frame, determine the target image region containing the first object from each target video frame to obtain multiple target image regions of the sample video.

[0096] S104: Based on the text features of the sample text, determine the target morphemes used to describe the second object from the sample text, and obtain multiple target morphemes of the sample text.

[0097] S105: Based on the video features of multiple target image regions in the sample video in the original training sample and the text features of multiple target morphemes in the sample text in the original training sample, calculate the similarity between the sample video and the sample text in the original training sample, and use it as the quality data of the original training sample.

[0098] S106: Obtain the first number of original training samples whose quality data is less than the first preset threshold, and filter each original training sample based on the first number to obtain the target training sample.

[0099] Based on the training sample generation method provided in this embodiment of the invention, since the video features of the target video frame can represent the first object contained in the target video frame, the target image region determined based on the video features of the target video frame is the image region containing the first object. The text features of the sample text can represent the second object described by the sample text, and the target morpheme determined based on the text features of the sample text is the morpheme used to describe the second object. Furthermore, based on the video features of the target image region of the sample video and the text features of the target morpheme of the sample text, the similarity between the sample video and the sample text is calculated as the quality data of the original training sample. The quality data of the original training sample can represent the similarity between the first object contained in the sample video and the second object described by the sample text. Furthermore, based on the number of original training samples with quality data less than a first preset threshold, each original training sample is screened to obtain the target training sample. This can avoid the problem that a high proportion of low-quality original training samples leads to difficulty in cross-modal model convergence, resulting in low accuracy of the cross-modal model, and can improve the training effect of the cross-modal model. Furthermore, it eliminates the need for technicians to judge the quality of each original training sample based on experience and to manually determine the target training sample from each original training sample, which can save time and labor costs, improve the efficiency of training sample generation, and thus improve the efficiency of training cross-modal models.

[0100] For step S101, the electronic device acquires a preset training set containing multiple original training samples.

[0101] An original training sample consists of one sample video and one sample text. The original training sample can include both positive and negative samples.

[0102] For each positive sample, if the first object contained in the sample video is the same as the second object described in the sample text, then the sample video and sample text in the positive sample have a high similarity, and the quality data of the positive sample is also high. For example, if the first object contained in the sample video is "dog", then the second object described in the sample text is also "dog", such as text containing the text "dog" or text containing the text "dog".

[0103] For each negative sample, the first object contained in the sample video is different from the second object described in the sample text. In other words, the similarity between the sample video and the sample text in this negative sample is low, and the quality data of this negative sample is not high. For example, if the first object contained in the sample video is "dog", then the second object described in the sample text can be "cat", or it can be "wolf", etc., such as text containing the text "cat" and text containing the text "wolf", etc.

[0104] Regarding step S102, the target video frame in a sample video can be all the video frames contained in the sample video, or the target video frame in a sample video can be a portion of the video frames contained in the sample video.

[0105] When the target video frame is a portion of the sample video, the electronic device can determine the target video frame of the sample video in the following manner.

[0106] Method 1:

[0107] Electronic devices can extract frames from a sample video at equal intervals to determine the target video frames. For example, the electronic device can extract frames from the sample video at specified intervals (e.g., 1 second or 30 milliseconds) to obtain multiple target video frames. Alternatively, the electronic device can extract frames from the sample video at specified intervals (e.g., 10 or 20) to obtain multiple target video frames.

[0108] Method 2:

[0109] Electronic devices can use lens detection technology to identify the first and last video frames captured by the same shot in a sample video, and then identify multiple video frames between the first and last video frames as a group of video frames captured by that shot. In other words, the video frames in the sample video are grouped into multiple video frame groups. Furthermore, based on these multiple video frame groups, the target video frame of the sample video is determined.

[0110] In one implementation, for each group of video frames, the electronic device can determine the first and last video frames in the group as the target video frames of the sample video.

[0111] In another implementation, for each video frame group, the electronic device can obtain the optical flow map corresponding to that video frame group based on the motion trajectory of the moving object from its position in the first video frame to its position in the last video frame of that video frame group. Then, based on the optical flow map corresponding to each video frame group, the electronic device determines the target video frame of the sample video from each video frame group.

[0112] The video features of each target video frame can represent the first object contained in that target video frame. For example, the video features of a target video frame can be a response heatmap. For instance, if the target video frame includes pedestrians and the background environment in which the pedestrians are located, the electronic device can extract the response heatmap of the pedestrians in the target video frame to obtain the video features of the target video frame. Alternatively, the electronic device can also perform convolution kernel visualization processing on the target video frame to obtain the video features of the target video frame, but is not limited to this.

[0113] The text features of the sample text can represent the second object described by the sample text. These text features can be a response heatmap, or they can be obtained by visualizing the sample text using convolutional kernels, but are not limited to these methods.

[0114] Sample text can include multiple morphemes, which are the smallest granular units of language with both semantic and phonetic properties; for example, a morpheme can be a word. The multiple morphemes included in the sample text are equivalent to the multiple words included in the sample text. For example, the sample text includes the morphemes "autumn," "wheat," and "sun."

[0115] Electronic devices can use feature extraction models to extract video features from each target video frame of a sample video, as well as text features from sample text. The feature extraction model can be a CLIP (Contrastive Language-Image Pre-training) model.

[0116] For each target video frame, the target video frame is input into the CLIP model. The CLIP model processes the target video frame based on the pixel values ​​of each pixel in the target video frame to obtain the video features of the target video frame.

[0117] The sample text is input into the CLIP model, which processes the sample text to obtain its text features.

[0118] For step S103, for each target video frame, the target image region of the target video frame is the image region in the target video frame that contains the first object. For example, if the target video frame includes a pedestrian and the background environment in which the pedestrian is located, then the target image region in the target video frame can be the image region in the target video frame to which the pedestrian belongs.

[0119] Furthermore, multiple target image regions are determined based on multiple target video frames, which are the image regions in the sample video that contain the first object.

[0120] In some embodiments, the video features of a target video frame can be a response heatmap of that target video frame. Figure 1 Based on this, see Figure 2 Step S103 may include the following steps:

[0121] S1031: For each target video frame, determine the image region composed of each pixel whose response value in the response heatmap of the target video frame is greater than the second preset threshold, and use it as the key image region.

[0122] S1032: Determine the minimum bounding rectangle of each key image region in the target video frame to obtain the target image region of the target video frame.

[0123] S1033: Select the target image regions of a second number of target video frames from the target image regions of each target video frame to obtain multiple target image regions of the sample video in the original training sample.

[0124] For each target video frame, the response values ​​of pixels in different image regions of the target video frame are different in the response heatmap of the target video frame. The response value of a pixel in an image region in the response heatmap of the target video frame represents the probability that the image region contains the first object.

[0125] For each image region in the target video frame, the higher the response value of the pixels in the image region in the response heatmap of the target video frame, the higher the probability that the image region contains the first object; the lower the response value of the pixels in the image region in the response heatmap of the target video frame, the lower the probability that the image region contains the first object.

[0126] Since the higher the response value of the pixels in the image region in the response heatmap of the target video frame, the higher the probability that the image region contains the first object, when the response value of the pixels in the image region in the response heatmap of the target video frame is higher than the second preset threshold, the electronic device can use the image region as the key image region of the target video frame. For example, the range of the response value of the pixels in the image region in the response heatmap of the target video frame can be [0, 1], and the second preset threshold can be expressed as th2 (Threshold 2), which can be 0.3.

[0127] The image region composed of pixels with higher response values ​​is the image region that stands out more and has clearer semantic information than other image regions in the target video frame. Therefore, the probability that the image region composed of pixels with higher response values ​​contains the first object is relatively high. Thus, the image region composed of pixels with higher response values ​​can be determined as the key image region.

[0128] For each target video frame, since the response values ​​of pixels in different image regions within the target video frame are different in the response heatmap of the target video frame, there may be multiple image regions in the target video frame whose pixel response values ​​are greater than the second preset threshold. In other words, the electronic device can identify multiple key image regions in the target video frame. Furthermore, the electronic device can obtain the image region containing the minimum bounding rectangle of these multiple key image regions in the target video frame, and determine the image region containing the minimum bounding rectangle as the target image region of the target video frame.

[0129] After the electronic device determines the target image region of each target video frame, it can determine a second number of target image regions from the target image regions of each target video frame, thereby obtaining multiple target image regions of the sample video in the original training sample.

[0130] The second number can be set according to actual needs, and can also be represented as k2. To reduce the number of target image regions that need to be processed and improve the efficiency of training sample generation, a smaller k2 can be set, for example, k2 can be 3; to enrich the content of the target image regions and improve the accuracy of training sample generation, a larger k2 can be set, for example, k2 can be 5.

[0131] Alternatively, the electronic device may determine a second number based on the number of target video frames in the sample video. For example, the second number may be 10% of the number of target video frames in the sample video, or 20% of the number of target video frames in the sample video, but is not limited thereto.

[0132] In one implementation, the electronic device can determine the area of ​​the target image region in each target video frame. The smaller the area of ​​the target image region in a target video frame, the less content that the target image region can contain, and the lower the probability that the determined target image region of that target video frame contains the first object. Therefore, the electronic device can select the target image regions of the second-to-last number of target video frames from the target image regions of each target video frame in descending order of area, thus obtaining multiple target image regions of the sample video in the original training sample. The target image regions of the second-to-last number of target video frames are, in other words, the target image regions of each target video frame with a higher probability of containing the first object.

[0133] In another implementation, for each target video frame, the electronic device can calculate the statistical values ​​(e.g., maximum value, average value, etc.) of the response values ​​of pixels in each key image region of the target video frame in the response heatmap of the target video frame, thus obtaining the response value of the target image region of the target video frame. Since the key image regions are determined by the electronic device based on the response values ​​of pixels in each image region in the response heatmap of the target video frame, correspondingly, the larger the statistical value of the response values ​​of pixels in each key image region of the target video frame in the response heatmap of the target video frame, that is, the larger the response value of the target image region of the target video frame, the higher the probability that the target image region of the target video frame contains the first object; conversely, the smaller the statistical value of the response values ​​of pixels in each key image region of the target video frame in the response heatmap of the target video frame, that is, the smaller the response value of the target image region of the target video frame, the lower the probability that the target image region of the target video frame contains the first object.

[0134] Therefore, the electronic device can select the target image regions of the second-to-last number of target video frames from the target image regions of each target video frame, in descending order of response values, to obtain multiple target image regions of the sample video in the original training sample. The target image regions of the second-to-last number of target video frames are, in other words, the target image regions of each target video frame with a higher probability of containing the first object.

[0135] Based on the above processing, the electronic device can determine the target image region of the target video frame according to the response value in the response heatmap of the target video frame, thus improving the accuracy of the determined target image region. Subsequently, the quality data of the original training sample is calculated based on multiple target image regions of the sample video and multiple target morphemes in the sample text, thereby improving the accuracy of the calculated quality data of the original training sample.

[0136] For step S104, for each target morpheme in the sample text, the target morpheme is the morpheme in the sample text used to describe the second object. For example, if the sample text includes nouns and adverbs, then the noun can be the target morpheme in the sample text.

[0137] Furthermore, multiple target morphemes are determined based on the sample text, which are the morphemes in the sample text used to describe the second object.

[0138] In some embodiments, the textual features of the sample text are the response heatmap of the sample text. Figure 1 Based on this, see Figure 3 Step S104 may include the following steps:

[0139] S1041: From the sample text in the original training sample, identify the morphemes whose response values ​​in the response heatmap of the sample text are greater than the third preset threshold, and use them as key morphemes.

[0140] S1042: According to the response values ​​of each key morpheme in the response heatmap of the sample text from high to low, select the top three number of key morphemes from each key morpheme to obtain multiple target morphemes of the sample text in the original training sample.

[0141] For each sample text, the electronic device can perform word segmentation and filter out stop words to obtain multiple words. Each word is a morpheme, meaning that multiple morphemes (which can be called text description morphemes) can be obtained for that sample text. The response heatmap of the sample text is the response heatmap corresponding to each text description morpheme in the sample text.

[0142] For each text description morpheme, the response value of the text description morpheme in the response heatmap of the sample text is different. The response value of the text description morpheme in the response heatmap of the sample text represents the probability that the text description morpheme in the sample text is used to describe the second object.

[0143] For each text description morpheme in the sample text, the higher the response value of the text description morpheme in the response heatmap of the sample text, the higher the probability that the text description morpheme is used to describe the second object; the lower the response value of the text description morpheme in the response heatmap of the sample text, the lower the probability that the text description morpheme is used to describe the second object.

[0144] Since a higher response value of the text description morpheme in the response heatmap of the sample text indicates a higher probability that the text description morpheme is used to describe the second object, when the response value of the text description morpheme in the response heatmap of the sample text is higher than a third preset threshold, the electronic device can identify the text description morpheme in the sample text as a key morpheme. For example, the response value of the text description morpheme in the response heatmap of the sample text can be in the range of [0, 1], and the third preset threshold can be represented as th3 (threshold 3), which can be 0.2.

[0145] The text description morpheme with a higher response value is the morpheme that stands out more than other morphemes in the sample text and has clearer semantic information. Therefore, the text description morpheme with a higher response value is more likely to be used to describe the second object. Thus, the text description morpheme with a higher response value can be identified as the target morpheme.

[0146] Since the response values ​​of each textual descriptive morpheme in the response heatmap of the sample text are different, there may be multiple textual descriptive morphemes in the sample text with corresponding response values ​​greater than the third preset threshold. This means the electronic device can identify multiple key morphemes in the sample text. For each key morpheme, since a higher response value in the response heatmap of the sample text indicates a higher probability that the key morpheme is used to describe the second object, the electronic device can select the top three key morphemes from the key morphemes in the response heatmap of the sample text, in descending order of their response values, to obtain multiple target morphemes in the sample text of the original training sample.

[0147] The third number can be set according to actual needs, and can also be represented as k3. To reduce the number of target morphemes that need to be processed and improve the efficiency of training sample generation, a smaller k3 can be set, for example, k3 can be 3; to enrich the content of target morphemes and improve the accuracy of training sample generation, a larger k3 can be set, for example, k3 can be 5.

[0148] Alternatively, the electronic device may determine a third number based on the number of text descriptive morphemes in the sample text. For example, the third number may be 10% of the number of text descriptive morphemes in the sample text, or 20% of the number of text descriptive morphemes in the sample text, but is not limited thereto.

[0149] Based on the above processing, the electronic device can determine the target morphemes of the sample text according to the response values ​​in the response heatmap of the sample text, thus improving the accuracy of the determined target morphemes. Subsequently, the quality data of the original training sample is calculated based on multiple target image regions of the sample video and multiple target morphemes in the sample text, thereby improving the accuracy of the calculated quality data of the original training sample.

[0150] In this embodiment, the execution order of steps S103 and S104 is not limited. Step S103 can be executed first, or step S104 can be executed first. For example, step S103 can be executed first, and then step S104 can be executed; or step S104 can be executed first, and then step S103 can be executed; or step S103 and step S104 can be executed simultaneously.

[0151] Regarding step S105, multiple target image regions of the sample video can represent the first object contained in the sample video, and multiple target morphemes of the sample text can represent the second object described in the sample text. Furthermore, based on the multiple target image regions of the sample video and the multiple target morphemes of the sample text, the similarity between the first object contained in the sample video and the second object described in the sample text can be calculated, that is, the similarity between the sample video and the sample text can be obtained.

[0152] The higher the similarity between the first object in the sample video and the second object described in the sample text, the higher the probability that the first object in the sample video and the second object described in the sample text are the same; the lower the similarity between the first object in the sample video and the second object described in the sample text, the lower the probability that the first object in the sample video and the second object described in the sample text are the same.

[0153] In some embodiments, Figure 1 Based on this, see Figure 4 Step S105 may include the following steps:

[0154] S1051: Combine each target image region of the sample video with each target morpheme of the sample text to obtain multiple initial mapping combinations.

[0155] Each initial mapping combination contains a target image region and a target morpheme.

[0156] S1052: For each initial mapping combination, calculate the similarity between the video features of the target image region and the text features of the target morpheme, and use this similarity as the similarity of the initial mapping combination.

[0157] S1053: Determine the quality data of the original training sample based on the similarity corresponding to each initial mapping combination.

[0158] For each target image region in the sample video, the electronic device can combine that target image region with each target morpheme in the sample text to obtain multiple initial mapping combinations containing that target image region. Therefore, based on the multiple target image regions in the sample video, multiple initial mapping combinations containing each target image region can be obtained.

[0159] For example, the target image regions of the sample video include target image region A, target image region B, and target image region C; the target morphemes of the sample text include target morpheme 1, target morpheme 2, and target morpheme 3. The electronic device can combine target image region A with target morpheme 1, target morpheme 2, and target morpheme 3 respectively to obtain three initial mapping combinations: mapping combination A1 (target image region A and target morpheme 1), mapping combination A2 (target image region A and target morpheme 2), and mapping combination A3 (target image region A and target morpheme 3). Similarly, the electronic device can combine target image region B with target morpheme 1, target morpheme 2, and target morpheme 3 respectively to obtain three initial mapping combinations: mapping combination B1 (target image region B and target morpheme 1), mapping combination B2 (target image region B and target morpheme 2), and mapping combination B3 (target image region B and target morpheme 3). The electronic device can also combine the target image region C with target morpheme 1, target morpheme 2 and target morpheme 3 respectively to obtain three initial mapping combinations. These three initial mapping combinations include: mapping combination C1 composed of target image region C and target morpheme 1, mapping combination C2 composed of target image region C and target morpheme 2, and mapping combination C3 composed of target image region C and target morpheme 3.

[0160] Furthermore, for each initial mapping combination, the electronic device can calculate the similarity between the video features of the target image region and the text features of the target morpheme, as the similarity corresponding to that initial mapping combination. Here, the video features of the target image region represent the first object contained within the target image region, and the text features of the target morpheme represent the second object described by the target morpheme.

[0161] For each initial mapping combination, the electronic device can use a similarity algorithm to calculate the similarity between the video features of the target image region and the text features of the target morpheme, which is then used as the similarity for that initial mapping combination. For example, the similarity algorithm can be cosine similarity, Pearson correlation coefficient, Euclidean distance, Manhattan distance, Mahalanobis distance, etc. The similarity value for the initial mapping combination can range from [0, 1].

[0162] After calculating the similarity corresponding to each initial mapping combination, the electronic device can determine the quality data of the original training sample based on the similarity corresponding to each initial mapping combination.

[0163] In some embodiments, step S1053 may include the following steps:

[0164] Step 1: From each initial mapping combination, determine the initial mapping combination with a similarity greater than the fourth preset threshold as the candidate mapping combination.

[0165] Step 2: Determine the quality data of the original training sample based on the similarity of each alternative mapping combination.

[0166] To improve the efficiency of training sample generation and the accuracy of the calculated original training samples, the electronic device can determine the corresponding initial mapping combination with higher similarity from each initial mapping combination as the alternative mapping combination.

[0167] In one implementation, the electronic device can determine initial mapping combinations with a similarity greater than a fourth preset threshold as candidate mapping combinations. For example, the fourth preset threshold can be 0.2.

[0168] For example, in the above embodiment, the initial mapping combinations include: mapping combination A1, mapping combination A2, mapping combination A3, mapping combination B1, mapping combination B2, mapping combination B3, mapping combination C1, mapping combination C2, and mapping combination C3. The fourth preset threshold is 0.2.

[0169] The quality data for mapping combination A1 is 0.4, the quality data for mapping combination A2 is 0.3, the quality data for mapping combination A3 is 0.2, the quality data for mapping combination B1 is 0.5, the quality data for mapping combination B2 is 0.4, the quality data for mapping combination B3 is 0.3, the quality data for mapping combination C1 is 0.4, the quality data for mapping combination C2 is 0.5, and the quality data for mapping combination C3 is 0.6.

[0170] Therefore, the alternative mapping combinations include: mapping combination A1, mapping combination A2, mapping combination B1, mapping combination B2, mapping combination B3, mapping combination C1, mapping combination C2, and mapping combination C3.

[0171] In another implementation, the electronic device can select the top four-number initial mapping combinations as candidate mapping combinations, based on their similarity from highest to lowest. For example, the fourth number can be 4, or it can be 50% of the number of initial mapping combinations.

[0172] After obtaining multiple alternative mapping combinations, the electronic device can determine the quality data of the original training sample based on the corresponding similarity of each alternative mapping combination.

[0173] In some embodiments, step 2 above may include the following steps:

[0174] Step 21: For each target image region, select one alternative mapping combination from the alternative mapping combinations that contain the target image region, and use it as the target mapping combination for the target image region.

[0175] The target morphemes included in the target mapping combinations of each target image region are different.

[0176] Step 22: Combine the target mappings that include each target image region into a cross-modal combination.

[0177] Step 23: Calculate the sum of similarities for each target mapping combination in the cross-modal combination to obtain the quality data of the cross-modal combination.

[0178] Step 24: Determine the maximum value among the quality data of each cross-modal combination, and use it as the quality data of the original training sample.

[0179] After obtaining multiple alternative mapping combinations, for each target image region, the electronic device can determine the target mapping combination for that target image region from the alternative mapping combinations containing that target image region. Since different target morphemes may describe the second object from different perspectives, in order to improve the accuracy of the similarity between the first object contained in the calculated sample video and the second object described in the sample text, for each target image region, the electronic device can determine the target mapping combination for that target image region from the alternative mapping combinations containing that target image region, and the target morphemes included in the target mapping combinations of each target image region are different.

[0180] For example, for the above embodiments, the alternative mapping combinations include: mapping combination A1, mapping combination A2, mapping combination B1, mapping combination B2, mapping combination B3, mapping combination C1, mapping combination C2, and mapping combination C3.

[0181] For a target image region A, the alternative mapping combinations containing target image region A are mapping combination A1 and mapping combination A2. The electronic device can arbitrarily select one of the alternative mapping combinations A1 and A2 as the target mapping combination for the target image region. This embodiment uses the selection of mapping combination A1 as the target mapping combination for target image region A as an example for explanation.

[0182] For target image region B, the alternative mapping combinations that include target image region B are: mapping combination B1, mapping combination B2, and mapping combination B3. Since the target mapping combination for target image region A is mapping combination A1, which already includes target morpheme 1, the electronic device can arbitrarily select one of the alternative mapping combinations for target image region B that does not include target morpheme 1, namely mapping combination B2 and mapping combination B3, as the target mapping combination for that target image region. In this embodiment, the selection of mapping combination B2 as the target mapping combination for target image region B is used as an example for explanation.

[0183] For target image region C, the alternative mapping combinations containing target image region C are: mapping combination C1, mapping combination C2, and mapping combination C3. Since the target mapping combination for target image region A is mapping combination A1, meaning it already includes target morpheme 1; and the target mapping combination for target image region B is mapping combination B2, meaning it already includes target morpheme 2, the electronic device can select one alternative mapping combination from the alternative mapping combinations for target image region C that does not include target morpheme 1 or target morpheme 2, as the target mapping combination for that target image region. That is, mapping combination C3 is selected as the target mapping combination for target image region C.

[0184] After selecting the target mapping combination for each target image region, the electronic device can determine the target mapping combination including each target image region as a cross-modal combination.

[0185] For example, in the above embodiments, the electronic device can determine mapping combination A1, mapping combination B2 and mapping combination C3 as a cross-modal combination, as cross-modal combination 1.

[0186] Correspondingly, the electronic device can also obtain, based on the above method, a cross-modal combination 2 including mapping combination A1, mapping combination B3 and mapping combination C2; a cross-modal combination 3 including mapping combination A2, mapping combination B1 and mapping combination C3; and a cross-modal combination 4 including mapping combination A2, mapping combination B3 and mapping combination C1.

[0187] After obtaining multiple cross-modal combinations, for each cross-modal combination, the electronic device can calculate the sum of similarities corresponding to each target mapping combination in that cross-modal combination, and obtain the quality data of that cross-modal combination.

[0188] For each target mapping combination, the similarity corresponding to the target mapping combination can represent the similarity between the first object in the target video frame to which the target image region in the target mapping combination belongs and the second object described by the target morpheme in the target mapping combination.

[0189] The higher the similarity between the first object in the target video frame to which the target image region belongs in the target mapping combination and the second object described in the sample text, the greater the similarity of the target mapping combination; the lower the similarity between the first object in the target video frame to which the target image region belongs in the target mapping combination and the second object described in the sample text, the smaller the similarity of the target mapping combination.

[0190] Since the similarity corresponding to each target mapping combination represents the similarity between the first object in the target video frame to which the target image region in the target mapping combination belongs and the second object described by the target morpheme in the target mapping combination, for a cross-modal combination, the sum of the similarities corresponding to each target mapping combination in the cross-modal combination is calculated, and the resulting quality data of the cross-modal combination can represent the similarity between the first object contained in the sample video and the second object described by the sample text.

[0191] Accordingly, after determining the quality data for each cross-modal combination, the electronic device can determine the quality data of the original training sample based on the statistical values ​​in the quality data of each cross-modal combination. For example, to improve the accuracy of the quality data of the original training sample, the maximum value in the quality data of each cross-modal combination can be determined as the quality data of the original training sample.

[0192] Based on the above processing, since the target image region is an image region containing the first object, the video features of the target image region can also represent the first object; the target morpheme is a morpheme used to describe the second object, so the text features of the target morpheme can also represent the second object. Therefore, the similarity between the calculated video features of the target image region and the text features of the target morpheme can represent the similarity between the first object in the target video frame to which the target image region belongs in the target mapping combination and the second object described in the sample text. Calculating the sum of the similarities corresponding to each target mapping combination in the cross-modal combination yields the quality data of the cross-modal combination, which can represent the similarity between the first object contained in the sample video and the second object described in the sample text. Furthermore, determining the maximum value of the quality data from multiple cross-modal combinations as the quality data of the original training sample can improve the accuracy of the calculated quality data of the original training sample.

[0193] Regarding step S106, the quality data of the original training samples is low, that is, the similarity between the first object contained in the sample video of the original training samples and the second object described in the sample text is low. The negative sample is the original training sample in which the sample video and the sample text have low similarity. In other words, the negative sample is the original training sample with low quality data.

[0194] If the proportion of low-quality original training samples in the preset training set is high, that is, if the proportion of negative samples is high, it will lead to difficulties in cross-modal model convergence, and thus low accuracy of the cross-modal model. Therefore, after calculating the quality data of each original training sample, the electronic device can determine the target training sample from multiple original training samples.

[0195] The electronic device can acquire the number of original training samples in the preset training set whose quality data is less than a first preset threshold (i.e., the first number), and then determine whether the preset training set meets the preset filtering conditions based on the first number, that is, to filter each original training sample based on the first number. The value range of the quality data of the original training samples can be [0, 10], and the first preset threshold can be 1.5.

[0196] The preset filtering condition can be: the ratio of the first number to the number of original training samples in the preset training set is greater than the fifth preset threshold. For example, the fifth preset threshold can be 80%.

[0197] If the preset training set does not meet the preset filtering conditions, it means that the proportion of low-quality original training samples in the preset training set is not high, that is, the proportion of negative samples is not high, which means that the quality of the preset training set is high. The electronic device can directly determine all the original training samples in the preset training set as target training samples.

[0198] If the preset training set meets the preset filtering conditions, it indicates that the proportion of low-quality original training samples in the preset training set is relatively high, that is, the proportion of negative samples is relatively high, meaning the quality of the preset training set is low. If a cross-modal model is trained based on a low-quality preset training set, it will lead to difficulties in cross-modal model convergence, and consequently, low accuracy of the cross-modal model. Therefore, the electronic device can clean the preset training set based on the number of original training samples with lower quality data, that is, based on a first number of original training samples with quality data less than a first preset threshold, the original training samples are filtered to obtain the target training samples.

[0199] In one implementation, when the preset training set meets the preset filtering conditions, that is, when the ratio of the first number to the number of original training samples in the preset training set is greater than the fifth preset threshold, the electronic device can determine the original training samples (which can be called the training samples to be processed) with quality data lower than the first preset threshold from multiple original training samples, and determine the other original training samples besides the training samples to be processed as the target training samples.

[0200] In some embodiments, if the number of original training samples other than the training samples to be processed in the preset training set is small, that is, the number of target training samples is small, the training effect of the cross-modal model will be poor. Therefore, for each training sample to be processed, technicians can also judge the quality of each training sample to be processed based on experience, and determine the better quality training samples from each training sample to be processed as target training samples, so as to increase the number of target training samples and thus improve the training effect of the cross-modal model.

[0201] Based on the above processing, the proportion of low-quality original training samples in the preset training set can be reduced, which means reducing the proportion of negative samples and thus improving the quality of the preset training set. Correspondingly, training the cross-modal model based on the high-quality preset training set can avoid the problem of low accuracy caused by the high proportion of low-quality original training samples, which leads to difficulties in convergence of the cross-modal model. This can improve the training effect of the cross-modal model.

[0202] See Figure 5 , Figure 5 A flowchart of a model training method provided in an embodiment of the present invention, the method may include the following steps:

[0203] S501: Obtain target training samples including sample videos and sample texts, and obtain sample similarity between sample videos and sample texts.

[0204] The target training sample is determined based on any of the training sample generation methods in the foregoing embodiments.

[0205] S502: Input the sample video and sample text into the cross-modal model of the initial structure to obtain the predicted similarity between the sample video and sample text.

[0206] S503: Calculate the loss function value representing the difference between predicted similarity and sample similarity.

[0207] S504: Adjust the model parameters of the cross-modal model of the initial structure based on the calculated loss function value until the preset convergence condition is met, and obtain the trained cross-modal model.

[0208] Cross-modal models can be CLIP models, CNN (Convolutional Neural Network) models, RNN (Recurrent Neural Network) models, or FNN (Fully Connected Neural Network) models, but are not limited to these.

[0209] After the electronic device inputs the sample video and sample text into the cross-modal model of the initial structure, it extracts the video features of each target video frame in the sample video and the text features of each text descriptive morpheme in the sample text through the feature extraction layer (e.g., convolutional layer) in the cross-modal model.

[0210] Then, the text features of each extracted text descriptive morpheme can be fused through the first feature fusion layer (e.g., a fully connected layer) in the cross-modal model to obtain the text fusion features of the sample text. The video features of each extracted target video frame can be fused through the second feature fusion layer in the cross-modal model to obtain the video fusion features of the sample video. The second feature fusion layer is a temporal fusion structure; by fusing the video features of each extracted target video frame through the second feature fusion layer, the individual video features of each target video frame can be fused to obtain the video fusion features of the sample video with good temporal comprehensive expressive ability.

[0211] Furthermore, the similarity prediction layer in the cross-modal model can calculate the similarity between the video fusion features of the sample video and the text fusion features of the sample text to obtain the predicted similarity of the target training data. The electronic device can also calculate a loss function value representing the difference between the predicted similarity and the sample similarity; for example, the loss function can be the cross-entropy loss function. Based on the calculated loss function value, the electronic device adjusts the model parameters of the initial cross-modal model structure, for example, by adjusting the model parameters of the initial cross-modal model structure using gradient descent until a preset convergence condition is reached, resulting in a trained cross-modal model.

[0212] The preset convergence condition can be: the number of training iterations reaches a preset number of training iterations. For example, the preset number of training iterations can be 50, or it can be 100, but it is not limited to these. Alternatively, the preset convergence condition can also be: the difference between the loss function value calculated in this iteration and the loss function value calculated in the previous preset number of iterations is less than a sixth preset threshold. For example, the preset number of iterations can be 20, and the sixth preset threshold can be 0.01.

[0213] Based on the above processing, the electronic device filters each original training sample based on the number of original training samples with quality data less than a first preset threshold. After obtaining the target training sample, it can train the cross-modal model of the initial structure based on the target training sample, which can improve the efficiency of training the cross-modal model of the initial structure.

[0214] Furthermore, since the model parameters of the cross-modal model with the initial structure are adjusted based on the calculated loss function value during model training, the impact of feature loss caused by the temporal fusion structure is taken into account when training the cross-modal model. This allows the trained cross-modal model to have high accuracy even when there is feature loss in video frames due to the temporal fusion structure, thus improving the accuracy of the trained cross-modal model.

[0215] Furthermore, after obtaining a trained cross-modal model, the electronic device can also use the trained cross-modal model. For example, if the cross-modal model is a video-text cross-modal model, the electronic device can acquire a target video containing the target object, and then use the video-text cross-modal model to calculate the similarity between the target video and multiple texts, and determine the text with the highest similarity to the target video, thus obtaining the text containing the target object.

[0216] See Figure 6 , Figure 6 A sixth flowchart of a training sample generation method provided in an embodiment of the present invention. The method may include the following steps:

[0217] S601: Acquire video-text cross-modal training data.

[0218] The video-text cross-modal training data is the training data used to train the cross-modal model with the initial structure, i.e., the preset training set containing multiple original training samples in the aforementioned embodiments. Each video-text cross-modal training data includes: one sample video and one sample text.

[0219] S602: Acquire keyframe images from the sample video.

[0220] For each video-text cross-modal training data set, the electronic device can acquire keyframe images from the sample videos within that set. These keyframe images are the target video frames described in the aforementioned embodiments. In other words, for each original training sample, the electronic device can acquire the target video frames from the sample videos within that original training sample.

[0221] S603: Cross-modal feature visualization.

[0222] For each original training sample, cross-modal feature visualization is performed on the original training sample. That is, for the sample video of the original training sample, the visualization features of each target video frame of the sample video are obtained. The visualization feature of a target video frame can be the response heatmap of the target video frame in the aforementioned embodiment. For the sample text in the original training sample, the visualization features of the sample text are obtained. The visualization features of the sample text can be the response heatmap of the sample text in the aforementioned embodiment.

[0223] S604: Visually salient region extraction.

[0224] The visually salient region is the target image region in the aforementioned embodiment. After the electronic device acquires the response heatmaps of each target video frame of the sample video, for each target video frame, it can determine the key image regions whose response values ​​in the response heatmap of the target video frame are greater than a second preset threshold, and determine the minimum bounding rectangle of each key image region in the target video frame to obtain the target image region of the target video frame. Furthermore, the electronic device can select target image regions of a second number of target video frames from the target image regions of each target video frame to obtain multiple target image regions of the sample video in the original training sample.

[0225] S605: Extraction of salient regions in text.

[0226] The salient regions of the text are the target morphemes in the aforementioned embodiments. After acquiring the response heatmap of the sample text, the electronic device can determine the key morphemes whose response values ​​in the response heatmap of the sample text are greater than a third preset threshold from the sample text in the original training sample. Furthermore, the electronic device can select the top three number of key morphemes from the key morphemes according to their response values ​​in the response heatmap of the sample text from high to low, thereby obtaining multiple target morphemes of the sample text in the original training sample.

[0227] S606: Cross-modal feature extraction.

[0228] Electronic devices can use the CLIP model to extract video features of multiple target image regions from the sample video in the original training sample to obtain a set of visually highly expressive features, which includes video features of multiple target image regions from the sample video; and extract text features of multiple target morphemes from the sample text in the original training sample to obtain a set of textually highly expressive features, which includes text features of multiple target morphemes from the sample text.

[0229] S607: Feature similarity calculation.

[0230] After acquiring the visual high-expressive feature set and the text high-expressive feature set, the electronic device can calculate the similarity between the visual high-expressive feature set and the text high-expressive feature set using a set similarity calculation method to obtain the quality score of the original training sample. This quality score is the same as the quality data of the original training sample in the aforementioned embodiment. In other words, the quality data of the original training sample is calculated based on the video features of each target image region of the sample video and the text features of each target morpheme of the sample text.

[0231] Based on the above processing, electronic devices can determine the target image regions and target morphemes in the original training samples through feature visualization. By analyzing the similarity between the target image regions and target morphemes, the quality data of the original training samples can be obtained, enabling the assessment of the quality of original training samples containing cross-modal resources (i.e., sample videos and sample text). Based on the number of original training samples with quality data below a first preset threshold, each original training sample is filtered to obtain the target training samples, which means a high-quality preset training set can be constructed. Furthermore, training the cross-modal model based on the high-quality preset training set can improve the efficiency and accuracy of modal model training.

[0232] Based on the same inventive concept as the training sample generation method described above, this invention also provides a training sample generation apparatus, see [link to apparatus]. Figure 7 The device includes:

[0233] The original training sample acquisition module 701 is used to acquire multiple original training samples; wherein, an original training sample includes: a sample video and the sample text corresponding to the sample video;

[0234] The feature acquisition module 702 is used to acquire, for each original training sample, the video features of each target video frame of the sample video in the original training sample, and the text features of the sample text; wherein, the video features of each target video frame represent the first object contained in the target video frame; and the text features of the sample text represent the second object described by the sample text.

[0235] The determining module 703 is used to determine, based on the video features of each target video frame, a target image region containing the first object from each target video frame, to obtain multiple target image regions of the sample video, and to determine, based on the text features of the sample text, a target morpheme used to describe the second object from the sample text, to obtain multiple target morphemes of the sample text.

[0236] The quality data calculation module 704 is used to calculate the similarity between the sample video and the sample text in the original training sample based on the video features of multiple target image regions of the sample video in the original training sample and the text features of multiple target morphemes of the sample text in the original training sample, and to use the similarity as the quality data of the original training sample.

[0237] The filtering module 705 is used to obtain a first number of original training samples whose quality data is less than a first preset threshold, and to filter each original training sample based on the first number to obtain the target training sample.

[0238] Optionally, the video features of a target video frame are the response heatmap of that target video frame;

[0239] The determining module 703 is specifically used for:

[0240] For each target video frame, the image region composed of each pixel whose response value in the response heatmap of the target video frame is greater than the second preset threshold is determined as the key image region.

[0241] Determine the minimum bounding rectangle of each key image region in the target video frame to obtain the target image region of the target video frame;

[0242] From the target image regions of each target video frame, select the target image regions of a second number of target video frames to obtain multiple target image regions of the sample video in the original training sample.

[0243] Optionally, the text features of the sample text in the original training samples are the response heatmap of the sample text;

[0244] The determining module 703 is specifically used for:

[0245] From the sample text in the original training sample, identify the morphemes whose response values ​​in the response heatmap of the sample text are greater than a third preset threshold, and use them as key morphemes.

[0246] Based on the response values ​​of each key morpheme in the response heatmap of the sample text from high to low, select the top three key morphemes from each key morpheme to obtain multiple target morphemes of the sample text in the original training sample.

[0247] Optionally, the quality data calculation module 704 is specifically used for:

[0248] Each target image region of the sample video is combined with each target morpheme of the sample text to obtain multiple initial mapping combinations; each initial mapping combination contains one target image region and one target morpheme.

[0249] For each initial mapping combination, the similarity between the video features of the target image region and the text features of the target morpheme is calculated, and this similarity is used as the similarity of the initial mapping combination.

[0250] Based on the similarity corresponding to each initial mapping combination, the quality data of the original training sample is determined.

[0251] Optionally, the quality data calculation module 704 is specifically used for:

[0252] From each initial mapping combination, determine the initial mapping combination with a similarity greater than the fourth preset threshold as the candidate mapping combination;

[0253] Based on the similarity of each alternative mapping combination, the quality data of the original training sample is determined.

[0254] Optionally, the quality data calculation module 704 is specifically used for:

[0255] For each target image region, select one alternative mapping combination from the alternative mapping combinations that contain the target image region as the target mapping combination for that target image region; wherein, the target morphemes included in the target mapping combinations of each target image region are different;

[0256] The target mappings, which include each target image region, are combined and defined as a cross-modal combination;

[0257] Calculate the sum of similarities for each target mapping combination in the cross-modal combination to obtain the quality data of the cross-modal combination;

[0258] The maximum value among the quality data of each cross-modal combination is determined as the quality data of the original training sample.

[0259] Based on the training sample generation apparatus provided in this embodiment of the invention, since the video features of the target video frame can represent the first object contained in the target video frame, the target image region determined based on the video features of the target video frame is the image region containing the first object. The text features of the sample text can represent the second object described by the sample text, and the target morpheme determined based on the text features of the sample text is the morpheme used to describe the second object. Furthermore, based on the video features of the target image region of the sample video and the text features of the target morpheme of the sample text, the similarity between the sample video and the sample text is calculated as the quality data of the original training sample. The quality data of the original training sample can then represent the similarity between the first object contained in the sample video and the second object described in the sample text. Furthermore, based on the number of original training samples with quality data less than a first preset threshold, each original training sample is filtered to obtain the target training sample. This avoids the problem that a high proportion of low-quality original training samples leads to difficulty in cross-modal model convergence, resulting in low accuracy of the cross-modal model, and can improve the training effect of the cross-modal model. Furthermore, it eliminates the need for technicians to judge the quality of each original training sample based on experience and to manually determine the target training sample from each original training sample, which can save time and labor costs, improve the efficiency of training sample generation, and thus improve the efficiency of training cross-modal models.

[0260] Based on the same inventive concept as the model training method described above, this invention also provides a model training apparatus, see [link to previous document]. Figure 8 The device includes:

[0261] The sample acquisition module 801 is used to acquire target training samples including sample videos and sample text, and to acquire the sample similarity between the sample videos and the sample text; wherein the target training samples are determined based on any of the training sample generation methods described in the first aspect above;

[0262] The prediction similarity acquisition module 802 is used to input the sample video and the sample text into the cross-modal model of the initial structure to obtain the prediction similarity between the sample video and the sample text;

[0263] The loss function value calculation module 803 is used to calculate the loss function value representing the difference between the predicted similarity and the sample similarity;

[0264] The model training module 804 is used to adjust the model parameters of the cross-modal model of the initial structure based on the calculated loss function value until the preset convergence condition is reached, so as to obtain the trained cross-modal model.

[0265] Based on the model training device provided in this embodiment of the invention, the electronic device filters each original training sample based on the number of original training samples with quality data less than a first preset threshold. After obtaining the target training sample, it can train the cross-modal model of the initial structure based on the target training sample, which can improve the efficiency of training the cross-modal model of the initial structure.

[0266] Furthermore, since the model parameters of the cross-modal model with the initial structure are adjusted based on the calculated loss function value during model training, the impact of feature loss caused by the temporal fusion structure is taken into account when training the cross-modal model. This allows the trained cross-modal model to have high accuracy even when there is feature loss in video frames due to the temporal fusion structure, thus improving the accuracy of the trained cross-modal model.

[0267] This invention also provides an electronic device, such as... Figure 9 As shown, it includes a processor 901, a communication interface 902, a memory 903, and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904.

[0268] Memory 903 is used to store computer programs;

[0269] When the processor 901 executes the program stored in the memory 903, it implements the steps of any training sample generation method in the above embodiments, or implements the steps of any model training method in the above embodiments.

[0270] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0271] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0272] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0273] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0274] In another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements any of the training sample generation methods in the above embodiments, or implements any of the model training methods in the above embodiments.

[0275] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the training sample generation methods in the above embodiments, or to implement any of the model training methods in the above embodiments.

[0276] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0277] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0278] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0279] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for generating training samples, characterized in that, The method includes: Obtain multiple original training samples; where each original training sample includes: a sample video and a sample text. For each original training sample, the video features of each target video frame in the sample video and the text features of the sample text are obtained; wherein, the video features of each target video frame represent the first object contained in the target video frame; and the text features of the sample text represent the second object described by the sample text. Based on the video features of each target video frame, a target image region containing the first object is determined from each target video frame to obtain multiple target image regions of the sample video. Based on the text features of the sample text, a target morpheme used to describe the second object is determined from the sample text to obtain multiple target morphemes of the sample text. Based on the video features of multiple target image regions in the sample videos of the original training samples, and the text features of multiple target morphemes in the sample text of the original training samples, the similarity between the sample videos and sample texts in the original training samples is calculated and used as the quality data of the original training samples. Obtain a first number of original training samples whose quality data is less than a first preset threshold, and filter each original training sample based on the first number to obtain the target training sample; The video features of a target video frame are represented by the response heatmap of that target video frame; The step of determining target image regions containing the first object from each target video frame based on its respective video features, thereby obtaining multiple target image regions of the sample video, includes: For each target video frame, the image region composed of each pixel whose response value in the response heatmap of the target video frame is greater than the second preset threshold is determined as the key image region. Determine the minimum bounding rectangle of each key image region in the target video frame to obtain the target image region of the target video frame; From the target image regions of each target video frame, select the target image regions of a second number of target video frames to obtain multiple target image regions of the sample video in the original training sample. The text features of the sample text in the original training samples are the response heatmaps of the sample text; The text features based on the sample text are used to determine the target morphemes used to describe the second object from the sample text, resulting in multiple target morphemes of the sample text, including: From the sample text in the original training sample, identify the morphemes whose response values ​​in the response heatmap of the sample text are greater than a third preset threshold, and use them as key morphemes. Based on the response values ​​of each key morpheme in the response heatmap of the sample text from high to low, select the top three key morphemes from each key morpheme to obtain multiple target morphemes of the sample text in the original training sample.

2. The method according to claim 1, characterized in that, The similarity between the sample videos and sample text in the original training samples is calculated based on the video features of multiple target image regions from the sample videos in the original training samples and the text features of multiple target morphemes from the sample text in the original training samples. This similarity serves as the quality data for the original training samples, including: Each target image region of the sample video is combined with each target morpheme of the sample text to obtain multiple initial mapping combinations; each initial mapping combination contains one target image region and one target morpheme. For each initial mapping combination, the similarity between the video features of the target image region and the text features of the target morpheme is calculated, and this similarity is used as the similarity of the initial mapping combination. Based on the similarity corresponding to each initial mapping combination, the quality data of the original training sample is determined.

3. The method according to claim 2, characterized in that, The process of determining the quality data of the original training sample based on the similarity corresponding to each initial mapping combination includes: From each initial mapping combination, determine the initial mapping combination with a similarity greater than the fourth preset threshold as the candidate mapping combination; Based on the similarity of each alternative mapping combination, the quality data of the original training sample is determined.

4. The method according to claim 3, characterized in that, The process of determining the quality data of the original training sample based on the similarity corresponding to each candidate mapping combination includes: For each target image region, select one alternative mapping combination from the alternative mapping combinations that contain the target image region as the target mapping combination for that target image region; wherein, the target morphemes included in the target mapping combinations of each target image region are different; The target mappings, which include each target image region, are combined and defined as a cross-modal combination; Calculate the sum of similarities for each target mapping combination in the cross-modal combination to obtain the quality data of the cross-modal combination; The maximum value among the quality data of each cross-modal combination is determined as the quality data of the original training sample.

5. A model training method, characterized in that, The method includes: Obtain target training samples including sample videos and sample texts, and obtain sample similarity between the sample videos and the sample texts; wherein the target training samples are determined based on the training sample generation method according to any one of claims 1 to 4; The sample video and the sample text are input into the cross-modal model with the initial structure to obtain the predicted similarity between the sample video and the sample text; Calculate the loss function value representing the difference between the predicted similarity and the sample similarity; The model parameters of the cross-modal model of the initial structure are adjusted based on the calculated loss function value until the preset convergence condition is met, thus obtaining the trained cross-modal model.

6. A training sample generation device, characterized in that, The device includes: The original training sample acquisition module is used to acquire multiple original training samples; one original training sample includes: a sample video and a sample text. The feature acquisition module is used to acquire, for each original training sample, the video features of each target video frame in the sample video of the original training sample, and the text features of the sample text; wherein, the video features of each target video frame represent the first object contained in the target video frame; and the text features of the sample text represent the second object described by the sample text. The determination module is used to determine, based on the video features of each target video frame, the target image region containing the first object from each target video frame, to obtain multiple target image regions of the sample video, and to determine, based on the text features of the sample text, the target morphemes used to describe the second object from the sample text, to obtain multiple target morphemes of the sample text. The quality data calculation module is used to calculate the similarity between the sample video and the sample text in the original training sample based on the video features of multiple target image regions of the sample video in the original training sample and the text features of multiple target morphemes of the sample text in the original training sample, and to use the similarity as the quality data of the original training sample. The filtering module is used to obtain a first number of original training samples whose quality data is less than a first preset threshold, and to filter each original training sample based on the first number to obtain the target training sample. The video features of a target video frame are represented by the response heatmap of that target video frame; The determining module is specifically used for: for each target video frame, determining the image region composed of each pixel whose response value is greater than a second preset threshold in the response heatmap of the target video frame, as a key image region; determining the minimum bounding rectangle of each key image region in the target video frame to obtain the target image region of the target video frame; and selecting the target image regions of a second number of target video frames from the target image regions of each target video frame to obtain multiple target image regions of the sample video in the original training sample. The text features of the sample text in the original training samples are the response heatmaps of the sample text; The determining module is specifically used to: determine, from the sample text in the original training sample, the morphemes whose response values ​​in the response heatmap of the sample text are greater than a third preset threshold, as key morphemes; and select the first third number of key morphemes from each key morpheme in the response heatmap of the sample text in descending order of their response values, to obtain multiple target morphemes of the sample text in the original training sample.

7. A model training device, characterized in that, The device includes: A sample acquisition module is used to acquire target training samples including sample videos and sample text, and to acquire the sample similarity between the sample videos and the sample text; wherein the target training samples are determined based on the training sample generation method according to any one of claims 1 to 4; The prediction similarity acquisition module is used to input the sample video and the sample text into the cross-modal model of the initial structure to obtain the predicted similarity between the sample video and the sample text; The loss function value calculation module is used to calculate the loss function value representing the difference between the predicted similarity and the sample similarity; The model training module is used to adjust the model parameters of the cross-modal model of the initial structure based on the calculated loss function value until the preset convergence condition is reached, so as to obtain the trained cross-modal model.

8. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the steps of the method according to any one of claims 1-4, or claim 5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-4, or claim 5.

Images