Data labeling method and device, electronic equipment, storage medium and program product

Abstract

The invention discloses a data labeling method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of computer data process.The method comprises the steps that a video to be processed is labeled, a data set containing at least one data pair is constructed, and the data pair comprises text description and video frames; dividing the data set into a first data pair set and a second data pair set; wherein the semantics between the text description and the video frames in the data pairs contained in the first data pair set are consistent, and the semantics between the text description and the video frames in the data pairs contained in the second data pair set are inconsistent; adding corresponding text description to video frames contained in the second data pair set to obtain a third data pair set; and determining the first data pair set and the third data pair set as annotated data. According to the method and the device, the semantic alignment degree of the description text and the picture of the training data in the text graph model can be improved, and cross-modal semantic consistency is realized.

Description

Technical Field

[0001] The present application relates to the field of computer data processing technology, and in particular to a data annotation method, device, electronic device, storage medium and program product. Background Art

[0002] Currently, automatic multimodal data annotation methods typically use deep neural networks for feature extraction and fusion. Deep neural networks are trained using a small amount of labeled data and a large amount of unlabeled data to generate pseudo-labels based on the inherent structure of the data. Cross-modal representations are learned by comparing data from different modalities and fine-tuned using large pre-trained models, thereby improving the efficiency and accuracy of automatic multimodal data annotation. Related art text-based graph models use convolutional neural networks to extract visual features from images and Transformer models to extract semantic features from text. New image pairs are then generated through self-supervised learning, while retaining the original text descriptions. Alternatively, contrastive learning uses images and corresponding text as positive samples and randomly pairs them as negative samples to train the model to distinguish between positive and negative samples. Finally, the text-based graph model is fine-tuned using a pre-trained multimodal model to automatically generate new image-text pairs. However, due to the cross-modality of the data, the training data in the text-based graph model can suffer from semantic misalignment between the descriptive text and the image, leading to cross-modal semantic inconsistency. Summary of the Invention

[0003] The embodiments of the present application provide a data annotation method, apparatus, electronic device, storage medium, and program product, which can improve the semantic alignment of descriptive text and images of training data in a text-based graph model and achieve cross-modal semantic consistency.

[0004] In a first aspect, an embodiment of the present application provides a data labeling method, comprising:

[0005] Annotate the video to be processed and construct a data set containing at least one data pair, wherein the data pair includes a text description and a video frame;

[0006] Dividing the data set into a first data pair set and a second data pair set; wherein the semantics of the text descriptions and the video frames in the data pairs included in the first data pair set are consistent, and the semantics of the text descriptions and the video frames in the data pairs included in the second data pair set are inconsistent;

[0007] adding corresponding text descriptions to the video frames included in the second data pair set to obtain a third data pair set;

[0008] The first data pair set and the third data pair set are determined as labeled data.

[0009] Optionally, the step of labeling the video to be processed and constructing a data set containing at least one data pair includes:

[0010] Converting the voice data in the video to be processed into text information;

[0011] Splitting the sentences in the text information to obtain at least one text description, wherein each text description in the at least one text description corresponds to a starting node and an ending node in the video to be processed;

[0012] Extracting at least one video frame corresponding to each text description in the at least one text description from the video to be processed based on a start node and an end node corresponding to each text description in the at least one text description;

[0013] Based on the at least one text description and at least one video frame corresponding to each text description in the at least one text description, at least one data pair is constructed to obtain the data set.

[0014] Optionally, dividing the data set into a first data pair set and a second data pair set includes:

[0015] Obtaining a target text description and a target video frame in a first target data pair, where the target video frame is any one of at least one video frame corresponding to the target text description, and the first target data pair is any one of the data sets;

[0016] Splitting the target text description to obtain a first description word set, where the first description word set includes at least one description word corresponding to the target text description;

[0017] Based on a preset knowledge graph, the description words in the first description word set are expanded to obtain a second description word set;

[0018] Inputting the target video frame into a first preset model, and obtaining a prompt word set corresponding to the target video frame output by the first preset model;

[0019] Inputting the second description word set and the prompt word set into a second preset model, and obtaining a first feature vector corresponding to the second description word set and a second feature vector corresponding to the prompt word set output by the second preset model;

[0020] Calculating the Euclidean distance between the first eigenvector and the second eigenvector;

[0021] When the Euclidean distance is less than or equal to a first preset threshold, determining semantic consistency between the target text description and the target video frame, and determining the first target data pair as a data pair in the first data pair set;

[0022] When the Euclidean distance is greater than the first preset threshold, it is determined that there is semantic inconsistency between the target text description and the target video frame, and the first target data pair is determined as a data pair in the second data pair set.

[0023] Optionally, any first data pair in the first data pair set includes a first text description, a first word set, a first video frame corresponding to the first text description, and a first result; wherein the first word set includes a description word set obtained by splitting the first text description, and a prompt word set corresponding to the first video frame, and the first result is used to indicate semantic consistency between the first text description and the first video frame;

[0024] Any second data pair in the second data pair set includes a second text description, a second word set, a second video frame corresponding to the second text description, and a second result; wherein the second word set includes a prompt word set corresponding to the second video frame, and the second result is used to indicate the semantic inconsistency between the second text description and the second video frame.

[0025] Optionally, the step of adding corresponding text descriptions to the video frames included in the second data pair set to obtain a third data pair set includes:

[0026] Obtaining a target prompt word set corresponding to the third video frame in a second target data pair, where the second target data pair is any one data pair in the second data pair set, and the target prompt word set is a set consisting of at least one prompt word contained in the third video frame extracted by a preset model;

[0027] Searching for video frames containing at least one prompt word in the target prompt word set from the first data pair set to obtain a similar video frame set;

[0028] Determine a video frame in the similar video frame set that has the highest similarity to the video frame in the second target data pair as a target similar video frame;

[0029] When the target similarity corresponding to the target similar video frame is greater than or equal to a second preset threshold, the descriptive word set corresponding to the target similar video frame is added to the second target data pair to obtain a third target data pair;

[0030] Constructing the third data pair set based on the third target data pair;

[0031] The third target data pair in the third data pair set includes the third video frame, a description word set corresponding to the text description in the second target data pair, and a description word set corresponding to the target similar video frame.

[0032] Optionally, searching the first data pair set for a video frame containing at least one cue word in the target cue word set to obtain a similar video frame set includes:

[0033] Searching for a video frame containing at least one cue word in the target cue word set from the first data pair set to obtain at least one similar video frame;

[0034] sorting the at least one similar video frame according to the number of the prompt words contained in each of the at least one similar video frame to obtain a similar video frame sequence;

[0035] Based on the number of the prompt words included in the similar video frame sequence, a preset number of video frames are sequentially selected from the similar video frame sequence to obtain the similar video frame set.

[0036] Optionally, determining a video frame in the set of similar video frames having the highest similarity to a video frame in the second target data pair as a target similar video frame includes:

[0037] Extracting a first image and a second image from each similar video frame in the set of similar video frames based on a third preset model; wherein the first image is an action skeleton image of the target object in the video frame, and the second image is a depth image corresponding to the video frame;

[0038] Obtaining a first similarity between the text description word in the second target data pair and the text description word corresponding to each similar video frame;

[0039] calculating a second similarity between the third video frame and the first image in each of the similar video frames, and calculating a third similarity between the third video frame and the second image in each of the similar video frames;

[0040] determining a target similarity between the third video frame and each of the similar video frames based on the first similarity, the second similarity, and the third similarity;

[0041] The similar video frame with the highest target similarity in the similar video frame set is determined as the target similar video frame.

[0042] Optionally, the method further includes:

[0043] When the target similarity corresponding to the target similar video frame is less than the second preset threshold, expanding at least one target description word obtained by splitting the text description in the second target data pair to obtain a fourth data pair set;

[0044] The fourth data pair set includes the video frame in the second target data pair, the at least one target descriptor, and at least one expanded descriptor corresponding to the target descriptor.

[0045] Optionally, the method further includes:

[0046] Inputting all the descriptive words in the third target data pair into a fourth preset model to obtain a first category set and a second category set of the descriptive words in the third target data pair;

[0047] Inputting all descriptive words in a fourth target data pair into a fourth preset model to obtain a first category set and a second category set of descriptive words in the fourth target data pair, wherein the fourth target data pair is any data pair in the third data pair set;

[0048] Obtaining a degree of overlap between the description words in the third target data pair and the description words in the fourth target data pair;

[0049] When the overlap degree is greater than or equal to a third preset threshold, the descriptive words in the second category set in the third target data pair are added to the fourth target data pair, and the descriptive words in the second category set in the fourth target data pair are added to the third target data pair, to obtain a supplemented third target data pair and a supplemented fourth target data pair;

[0050] The descriptive words in the first category set are words used to represent clear entities or behaviors, and the descriptive words in the second category set are words used to express emotions, metaphors or abstract concepts.

[0051] Optionally, the method further includes:

[0052] using the supplemented third target data pair and the supplemented fourth target data pair as positively labeled data;

[0053] combining the descriptive words corresponding to different video frames in the third data pair set to obtain negatively labeled data;

[0054] A preset text-generated graph model is trained based on the positively labeled data and the negatively labeled data to obtain an optimized text-generated graph model, wherein the preset text-generated graph model is a model for converting text descriptions into corresponding images.

[0055] In a second aspect, an embodiment of the present application further provides a data labeling device, comprising:

[0056] Annotation module, configured to annotate the video to be processed and construct a data set comprising at least one data pair, wherein the data pair comprises a text description and a video frame;

[0057] a classification module, configured to divide the data set into a first data pair set and a second data pair set; wherein the semantics of the text descriptions and the video frames in the data pairs included in the first data pair set are consistent, and the semantics of the text descriptions and the video frames in the data pairs included in the second data pair set are inconsistent;

[0058] an expansion module, configured to add corresponding text descriptions to the video frames included in the second data pair set to obtain a third data pair set;

[0059] A determination module is configured to determine the first data pair set and the third data pair set as labeled data.

[0060] In a third aspect, an embodiment of the present application provides an electronic device comprising: a processor, a memory, and a program stored on the memory and executable on the processor, wherein when the program is executed by the processor, the steps of the data labeling method as described in any one of the first aspects are implemented.

[0061] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data labeling method as described in any one of the first aspects are implemented.

[0062] In a fifth aspect, an embodiment of the present application further provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the data labeling method as described in any one of the first aspects.

[0063] In an embodiment of the present application, the video to be processed is annotated to construct a data set containing at least one data pair, and then the data set is divided into a first data pair set and a second data pair set. The semantics between the text descriptions and the video frames in the data pairs included in the first data pair set are consistent, and the semantics between the text descriptions and the video frames in the data pairs included in the second data pair set are inconsistent. Thus, the embodiment of the present application divides the data pairs into semantically consistent and inconsistent sets, ensuring that the annotated data in the first data pair set are accurately aligned, and avoiding annotation errors caused by semantic inconsistency. Moreover, corresponding text descriptions are added to the video frames included in the second data pair set to obtain a third data pair set, and the first data pair set and the third data pair set are determined as annotated data. In this way, the embodiment of the present application can also supplement the corresponding text descriptions for inconsistent video frames to improve the semantic alignment of the description text and pictures of the training data in the text graph model, and achieve cross-modal semantic consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0065] Figure 1 This is one of the flow charts of a data annotation method in an embodiment of the present application;

[0066] Figure 2 is a schematic diagram of similar video frames, corresponding first images, and second images in an embodiment of the present application;

[0067] Figure 3 This is the second flow chart of a data labeling method in an embodiment of the present application;

[0068] Figure 4 is a schematic diagram of a data labeling device in an embodiment of the present application;

[0069] Figure 5 It is a schematic diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0071] The embodiments of the present application provide a data annotation method, device, electronic device, storage medium and program product. The embodiments of the present application are described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0072] See Figure 1 , Figure 1 This is one of the flow charts of a data annotation method in an embodiment of the present application, which specifically includes the following steps:

[0073] Step 101: annotate the video to be processed and construct a data set containing at least one data pair, wherein the data pair includes a text description and a video frame.

[0074] It should be noted that the video to be processed can be an audio video containing text descriptions such as commentary and subtitles, such as a sports event video with commentary. The commentary content in the video to be processed can correspond to the image, making it easier to convert the commentary into text descriptions and match them with the image video frames, thereby forming data pairs containing text descriptions and video frames.

[0075] In the above steps, video data containing commentary can be used as the raw input. Using speech transcription and voice activity detection (VAD) model sentence segmentation techniques, the commentary text is paired with video frames to generate an initial data set. For example, the input data format is {event video file ID, commentary text, start time, end time}. By extracting video frames to form paired data of text descriptions and video frames, multiple data pairs are generated to construct a data set.

[0076] Step 102: Divide the data set into a first data pair set and a second data pair set; wherein the semantics between the text descriptions and the video frames in the data pairs included in the first data pair set are consistent, and the semantics between the text descriptions and the video frames in the data pairs included in the second data pair set are inconsistent.

[0077] In this embodiment of the present application, a cross-modal semantic consistency determination method can be used to classify data into two categories by comparing the semantic relevance of the commentary and the video frames. For example, the commentary is expanded using a knowledge graph and semantic similarity is calculated to determine the semantic consistency between the text and the video frame. The output result is {commentary, text description set, single video frame, determination result}, where a determination result of 1 indicates consistency and a determination result of 0 indicates inconsistency.

[0078] Step 103: Add corresponding text descriptions to the video frames included in the second data pair set to obtain a third data pair set.

[0079] In the above steps, text descriptions can be supplemented for semantically inconsistent video frames. Specifically, candidate images containing similar prompt words can be searched from the semantically consistent first data pair set. Similarity can then be calculated by combining multiple dimensions, such as text description words, action characteristics, and depth of field. Based on a similarity threshold, abstract concepts or concrete elements can be selectively supplemented, thereby converting the semantically inconsistent data into valid annotations, resulting in a third data pair set.

[0080] Step 104: Determine the first data pair set and the third data pair set as labeled data.

[0081] The annotated data finally obtained by the embodiment of the present application consists of two parts: semantically consistent data pairs (i.e., the first data pair set), and semantically aligned data pairs after expansion processing (i.e., the third data pair set). For example, the output format is {text description word set, picture}, where the text description word set contains specific elements and abstract concepts to ensure the integrity and diversity of the annotated data. In this way, the embodiment of the present application clearly distinguishes the semantic consistency between text descriptions and video frames, ensures that the annotated data in the first data pair set are accurately aligned, avoids annotation errors caused by semantic inconsistency, and improves the semantic alignment of the descriptive text and pictures of the training data in the text-based graph model, thereby achieving cross-modal semantic consistency.

[0082] Optionally, the step of labeling the video to be processed and constructing a data set containing at least one data pair includes:

[0083] Converting the voice data in the video to be processed into text information;

[0084] Splitting the sentences in the text information to obtain at least one text description, wherein each text description in the at least one text description corresponds to a starting node and an ending node in the video to be processed;

[0085] Extracting at least one video frame corresponding to each text description in the at least one text description from the video to be processed based on a start node and an end node corresponding to each text description in the at least one text description;

[0086] Based on the at least one text description and at least one video frame corresponding to each text description in the at least one text description, at least one data pair is constructed to obtain the data set.

[0087] In some embodiments, the voice data in the video can be converted into text information through speech transcription technology (such as a speech recognition model). For example, corresponding text information is obtained by using video data containing commentary (such as sports commentary), such as the commentary "The angle of Person A's volley shot is too straight". The VAD model can be used to sentence-process the text information and split the long text into multiple independent sentences. The text can be divided into multiple independent text descriptions based on the duration and semantic integrity of the commentary. For example, the commentary "The angle of Person A's volley shot is too straight, but the defensive player blocked it in time." is split into: text description 1, "The angle of Person A's volley shot is too straight", and text description 2: "But the defensive player blocked it in time".

[0088] According to the starting node (starting time point) and ending node (ending time point) corresponding to each text description, the corresponding video frames are extracted from the video. Specifically, video frames can be extracted proportionally according to the video frame rate (e.g., 24 frames per second) and the duration of the commentary. For example, if the commentary duration is 10 seconds and the frame rate is 24, 24 video frames are extracted. Thus, a set of video frames corresponding to each text description is obtained (e.g., {frame 1, frame 2, ..., frame n}). Subsequently, each text description can be combined with the corresponding video frame to form a data pair ({text description, video frame}). For example, the text description is "The angle of Person A's volley shot is too straight", and its corresponding video frames are {frame 1, frame 2, frame 3, frame 4}. The obtained data pairs include {"The angle of Person A's volley shot is too straight", {frame 1}}, {"The angle of Person A's volley shot is too straight", {frame 2}}, {"The angle of Person A's volley shot is too straight", {frame 3}} and {"The angle of Person A's volley shot is too straight", {frame 4}}.

[0089] For example, a sports event video is split into a data set consisting of multiple commentary texts and corresponding video frames. Taking a football match as an example, a data pair in the data set can be represented as {Team A vs. Team B Quarterfinal, "Character A's volley was too sharp," 11:23:40, 11:43:21}, where "Euro Cup Portugal vs. France Quarterfinal" represents the video to be processed, "Character A's volley was too sharp" represents a line of commentary, and "11:23:40, 11:43:21" represent the start and end nodes of the video to be processed. The commentary can be processed at the line granularity, with each line of commentary treated as a sentence and split into multiple text descriptions. Combined with the multiple video frames corresponding to the sentence, a data pair is formed.

[0090] In this way, the above-mentioned embodiments of the present application reduce the workload of manual labeling through automated speech transcription and video frame extraction, significantly improve data construction efficiency, and can also use timestamp and VAD model sentence segmentation technology to ensure the precise correspondence between text descriptions and video frames in terms of timing and semantics, thereby achieving efficient and accurate multimodal data labeling, and providing a high-quality original data foundation for subsequent semantic consistency judgment and data expansion.

[0091] Optionally, dividing the data set into a first data pair set and a second data pair set includes:

[0092] Obtaining a target text description and a target video frame in a first target data pair, where the target video frame is any one of at least one video frame corresponding to the target text description, and the first target data pair is any one of the data sets;

[0093] Splitting the target text description to obtain a first description word set, where the first description word set includes at least one description word corresponding to the target text description;

[0094] Based on a preset knowledge graph, the description words in the first description word set are expanded to obtain a second description word set;

[0095] Inputting the target video frame into a first preset model, and obtaining a prompt word set corresponding to the target video frame output by the first preset model;

[0096] Inputting the second description word set and the prompt word set into a second preset model, and obtaining a first feature vector corresponding to the second description word set and a second feature vector corresponding to the prompt word set output by the second preset model;

[0097] Calculating the Euclidean distance between the first eigenvector and the second eigenvector;

[0098] When the Euclidean distance is less than or equal to a first preset threshold, determining semantic consistency between the target text description and the target video frame, and determining the first target data pair as a data pair in the first data pair set;

[0099] When the Euclidean distance is greater than the first preset threshold, it is determined that there is semantic inconsistency between the target text description and the target video frame, and the first target data pair is determined as a data pair in the second data pair set.

[0100] In some embodiments, a data pair (i.e., the first target data) can be randomly selected from a data set, and the first target data pair includes a target text description (e.g., "The angle of Person A's volley shot is too straight"), and a target video frame (e.g., a frame of a football game). The target video frame can be any one of the at least one video frames corresponding to the text description, for example, a key frame in the video corresponding to the commentary. The target text description is segmented to obtain a first set of description words. Input the text description: "The angle of Person A's volley shot is too straight", and you can get the first set of description words after segmentation, such as {"Person A", "Volley", "Angle", "Straight"}.

[0101] Specifically, the first set of descriptive words can be expanded using a preset domain knowledge graph (e.g., a knowledge graph related to football matches) to generate a second set of descriptive words. For example, the original descriptive words can be {"person A", volley", "angle", "positive"}, and after expansion, {"person A", "shot", "volley", "attack", "football match", "goal", "goalkeeper", "angle positive"}. Thus, the domain knowledge related to the original text is supplemented by the knowledge graph, enhancing the semantic coverage.

[0102] Furthermore, the target video frame can be input into a first preset model (such as the CLIP model of Stable Diffusion) to generate a prompt word set for the video frame. For example, the shooting scene in a football match is used as a video frame in an embodiment of the present application, and a prompt word set {"shooting", "attack", "football match", "goal", "goalkeeper"} can be obtained to generate a feature vector and calculate the similarity. Subsequently, the second description word set and the prompt word set can be input into a second preset model (such as a Transformer model) to generate two feature vectors, namely a first feature vector (semantic features of the second description word set) and a second feature vector (semantic features of the prompt word set). Wherein, if the Euclidean distance between the first feature vector and the second feature vector is less than or equal to a first preset threshold value (such as 0.5), it is determined that the text description is semantically consistent with the video frame, and the data pair can be included in the first data pair set. If the Euclidean distance>the first preset threshold value, it is determined that the text description is semantically inconsistent with the video frame, and the data pair can be included in the second data pair set.

[0103] In this way, the embodiment of the present application accurately determines the semantic consistency between the text description and the video frame through knowledge graph expansion and feature vector calculation, avoiding the annotation errors caused by semantic deviation in traditional methods. After classifying the semantically inconsistent data, effective annotations can be further generated through a hierarchical expansion method (such as supplementing abstract concepts or specific elements) to expand the data coverage. Thus, the embodiment of the present application realizes the accurate classification of multimodal data, solves the problems of insufficient semantic alignment and poor data diversity in traditional annotation methods, and provides a high-quality data foundation for subsequent automatic annotation and model training.

[0104] Optionally, any first data pair in the first data pair set includes a first text description, a first word set, a first video frame corresponding to the first text description, and a first result; wherein the first word set includes a description word set obtained by splitting the first text description, and a prompt word set corresponding to the first video frame, and the first result is used to indicate semantic consistency between the first text description and the first video frame;

[0105] Any second data pair in the second data pair set includes a second text description, a second word set, a second video frame corresponding to the second text description, and a second result; wherein the second word set includes a prompt word set corresponding to the second video frame, and the second result is used to indicate the semantic inconsistency between the second text description and the second video frame.

[0106] It should be noted that the description word set may be the result of word segmentation of the first text description, such as "person A", "volley", "angle", and "positive". The prompt word set may be the text description corresponding to the video frame extracted by a preset model (such as the CLIP model), such as "shoot", "attack", "football match", and "goal".

[0107] In some specific embodiments, the first text description is segmented to obtain a set of description words, such as "Person A" and "Volley". The description words are expanded based on the domain knowledge graph (such as the football game knowledge graph) to supplement the relevant domain knowledge, such as "shooting", "attack", and "goal". The first video frame can be input into a preset model to generate a set of prompt words, such as "shooting", "attack", and "football game". Subsequently, the expanded set of description words and the set of prompt words can be input into the Transformer model to generate a feature vector. By calculating the Euclidean distance of the feature vector, if it is less than or equal to a preset threshold (for example, 0.5), it is determined to be semantically consistent, and a first data pair set is generated.

[0108] In some other embodiments, the second text description may be the original text description (for example, "The shot just now by Person A was not handled well"). The second word set only includes the prompt word set corresponding to the second video frame (for example, "pass", "defense", "football match"). The second video frame may be a video frame corresponding to the second text description (for example, a defensive scene in a football match). Specifically, the second text description may be segmented to obtain a description word set (for example, "Person A", "shot", "not handled well"). The description words may also be expanded based on the knowledge graph (for example, "defense", "offense", "football match"). During the video frame prompt word extraction process, the second video frame may be input into a preset model to generate a prompt word set (for example, "pass", "defense", "football match"). The expanded description word set and the prompt word set are input into the Transformer model to generate a feature vector. The Euclidean distance of the feature vector is calculated. If it is greater than a preset threshold, it is determined to be semantically inconsistent, and a second data pair set is generated.

[0109] In this way, the embodiment of the present application ensures that the text description in the first data pair set is semantically consistent with the video frame through knowledge graph expansion and feature vector calculation, avoiding annotation errors caused by semantic deviation in traditional methods, and the semantically inconsistent data in the second data pair set is converted into effective annotations through a hierarchical expansion method (such as supplementing abstract concepts or specific elements), thereby expanding the data coverage. Therefore, the embodiment of the present application achieves accurate classification of multimodal data by dividing the data set into semantically consistent and inconsistent subsets, and combining technologies such as knowledge graph expansion and feature vector calculation, solving the problems of insufficient semantic alignment and poor data diversity in traditional annotation methods, and providing a high-quality data foundation for subsequent automatic annotation and model training.

[0110] Optionally, the video frames in the second data pair set are added with text descriptions to obtain a third data pair set, comprising:

[0111] Obtaining a target prompt word set corresponding to the third video frame in a second target data pair, where the second target data pair is any one data pair in the second data pair set, and the target prompt word set is a set consisting of at least one prompt word contained in the third video frame extracted by a preset model;

[0112] Searching for video frames containing at least one prompt word in the target prompt word set from the first data pair set to obtain a similar video frame set;

[0113] Determine a video frame in the similar video frame set that has the highest similarity to the video frame in the second target data pair as a target similar video frame;

[0114] When the target similarity corresponding to the target similar video frame is greater than or equal to a second preset threshold, the descriptive word set corresponding to the target similar video frame is added to the second target data pair to obtain a third target data pair;

[0115] Constructing the third data pair set based on the third target data pair;

[0116] The third target data pair in the third data pair set includes the third video frame, a description word set corresponding to the text description in the second target data pair, and a description word set corresponding to the target similar video frame.

[0117] In some embodiments, a data pair (i.e., a second target data pair) is selected from the second data pair set, comprising a second text description (e.g., "Person A's shot just now was not handled well") and a second video frame (e.g., a defensive scene in a football match). The target prompt word set may be a prompt word set (e.g., {"pass", "defense", "football match"}) extracted from the second video frame through a preset model. Video frames containing at least one prompt word in the target prompt word set are searched from the first data pair set to generate a similar video frame set. Exemplarily, the target prompt word set may be {"pass", "defense", "football match"}, and the video frame prompt word set in the first data pair set may be {"shoot", "attack", "football match", "goal"}, then the similar video frame set may include video frames related to prompt words such as "football match" and "defense" (e.g., an offensive scene in a football match).

[0118] Specifically, the comprehensive similarity between each frame in the similar video frame set and the video frame in the second target data pair can be calculated, and the video frame with the highest similarity can be selected as the target similar video frame. The text description word similarity can be a semantic similarity comparison between the target prompt word set and the similar video frame prompt word set (such as using the BERT model). The action feature similarity can be obtained by extracting the action skeleton graph through the ControlNet model and calculating the Euclidean distance. The above-mentioned depth of field feature similarity can be obtained by extracting the depth of field graph through the Depth model and calculating the Euclidean distance. The above-mentioned comprehensive similarity can be a weighted average of the three similarities (for example, text similarity accounts for 40%, action similarity accounts for 30%, and depth of field similarity accounts for 30%).

[0119] If the comprehensive similarity of the target similar video frame is greater than or equal to the second preset threshold (such as 0.7), the following operations are performed: supplement the description words, and the description word set corresponding to the target similar video frame, such as {"shooting", "offense", "football match"}, can be supplemented to the second target data pair; update the data pair, and a third target data pair can be generated, including a video frame (a video frame in the second target data pair), a description word set (the description word set of the original second target data pair, such as {"person A", "shooting", "poorly handled"}, and the description word set of the target similar video frame, such as {"shooting", "offense", "football match"}). For example, the original second target data pair, i.e., {"person A's shot just now was not handled well", {"pass", "defense", "football match"}, video frame, 0}, is supplemented to obtain {"person A's shot just now was not handled well", {"pass", "defense", "football match", "shooting", "offense"}, video frame, 0}. In this way, a third data pair set can be constructed. Specifically, all updated third target data pairs are combined into a third data pair set, where each data pair contains: a video frame (a video frame in the second target data pair), a description word set (the original description word set and the supplemented description word set), and the result label is still "semantically inconsistent" (0), but the description word set has been enhanced.

[0120] In this way, the embodiment of the present application supplements the descriptive word set, repairs the semantically inconsistent data, makes it closer to the video content, improves the accuracy of the labeled data, uses the semantically consistent data in the first data pair set to supplement the second data pair set, expands the coverage of the labeled data, and combines the knowledge graph, feature vector calculation and similarity threshold judgment to achieve accurate expansion of multimodal data, providing a high-quality data foundation for subsequent automatic labeling and model training.

[0121] Optionally, searching the first data pair set for a video frame containing at least one cue word in the target cue word set to obtain a similar video frame set includes:

[0122] Searching for a video frame containing at least one cue word in the target cue word set from the first data pair set to obtain at least one similar video frame;

[0123] sorting the at least one similar video frame according to the number of the prompt words contained in each of the at least one similar video frame to obtain a similar video frame sequence;

[0124] Based on the number of the prompt words included in the similar video frame sequence, a preset number of video frames are sequentially selected from the similar video frame sequence to obtain the similar video frame set.

[0125] In some embodiments, all data pairs in the first data pair set are traversed to check whether their prompt word sets include at least one prompt word in the target prompt word set. The target prompt word set may be prompt words corresponding to video frames extracted from the second data pair set, such as {"pass", "defense", "football game"}. The first data pair set may include semantically consistent data pairs, and the prompt word set for each data pair has been generated using a preset model (such as a CLIP model). The aforementioned similar video frame set may include all video frames that meet the conditions, that is, video frames that include at least one target prompt word in the prompt word set.

[0126] For each video frame in the similar video frame set, the number of overlaps between its prompt word set and the target prompt word set can be counted, and the words can be sorted from high to low according to the number of overlaps. For example, the target prompt word set can be {"pass", "defense", "football match"}, and the prompt word set of similar video frame 1 can be {"pass", "defense", "football match", "shoot"}, and the number of overlaps between these two sets is 3. The prompt word set of similar video frame 2 can be {"defense", "football match"}, and the number of overlaps between them is 2. After sorting, similar video frame 1 is larger than similar video frame 2, and a preset number of video frames can be selected. For example, from the sorted similar video frame sequence, the preset number of video frames are selected in turn to form a final similar video frame set. In this way, it can be ensured that the semantic relevance between the selected video frames and the target prompt word set is the strongest, avoiding redundant or low-correlation data.

[0127] In one specific embodiment, if the prompt word set corresponding to a video frame in the second data pair set is {"pass," "defense," "football game"}, to search for similar video frames, the first data pair set can be searched for video frames containing "pass," "defense," or "football game." A video frame in the first data pair set with the prompt word set of {"shoot," "attack," "football game"} is selected. If there are three video frames in the similar video frame set, and the number of overlaps is 3, 2, and 1, respectively, the first two video frames are selected in that order as the final result.

[0128] In this way, the embodiment of the present application quickly selects video frames that are highly correlated with the target prompt word by counting the number of prompt word overlaps, avoiding the inefficient operation of traversing all data pairs. The sorting mechanism ensures that the video frames with the strongest correlation with the target prompt word are preferentially selected, improving the accuracy of subsequent steps. As a result, the embodiment of the present application can efficiently select video frames that are highly correlated with the target prompt word from the first data pair set, ensuring the accuracy of subsequent steps. Combined with the prompt word overlap statistics and sorting mechanism, the efficiency and quality of multimodal data annotation are significantly improved.

[0129] Optionally, determining a video frame in the set of similar video frames having the highest similarity to a video frame in the second target data pair as a target similar video frame includes:

[0130] Extracting a first image and a second image from each similar video frame in the set of similar video frames based on a third preset model; wherein the first image is an action skeleton image of the target object in the video frame, and the second image is a depth image corresponding to the video frame;

[0131] Obtaining a first similarity between the text description word in the second target data pair and the text description word corresponding to each similar video frame;

[0132] calculating a second similarity between the third video frame and the first image in each of the similar video frames, and calculating a third similarity between the third video frame and the second image in each of the similar video frames;

[0133] determining a target similarity between the third video frame and each of the similar video frames based on the first similarity, the second similarity, and the third similarity;

[0134] The similar video frame with the highest target similarity in the similar video frame set is determined as the target similar video frame.

[0135] In some embodiments, such as Figure 2 As shown, the action skeleton image and depth image in the video frame can be extracted. Specifically, the following two images can be extracted from each video frame in the set of similar video frames using a third preset model (such as the OpenPose model and Depth model of ControlNet):

[0136] First image (action skeleton): The action skeleton (such as the joint positions of the arms and legs) of the target object (such as the player) in the video frame is extracted through the OpenPose model.

[0137] Second image (depth map): The depth information of the video frame (such as the distance relationship of objects) is extracted through the Depth model.

[0138] For example, the video frame may be a shot scene in a football game, the obtained motion skeleton image may show the player's leg movements (such as a kicking posture), and the depth image may show the distance relationship between the player and the goal.

[0139] The target text description words can be the text description words in the second target data pair, such as "Person A's shot just now was not handled well". The text description words of similar video frames can be extracted from the description word sets corresponding to similar video frames, such as "pass", "defense", and "football game". For similarity calculation, a pre-trained model (such as BERT) can be used to calculate the semantic similarity (such as cosine similarity) between the target text description words and the text description words of similar video frames. For example, the target text description words: {"Person A", "shot", "not handled well"}, the similarity of similar video frames: {"pass", "defense", "football game"}, the similarity can be calculated by the BERT model, and the similarity value is 0.6.

[0140] Furthermore, the action skeleton graph of the target video frame and the action skeleton graph of the similar video frame can be feature vectorized (for example, using the joint coordinates extracted by the OpenPose model) to calculate the Euclidean distance. The depth map of the target video frame and the depth map of the similar video frame can also be feature vectorized (such as the depth value extracted by the Depth model) to calculate the Euclidean distance. For example, the feature vector of the action skeleton graph of the target video frame is: [x1, y1, z1, ...], and the feature vector of the action skeleton graph of the similar video frame is: [x2, y2, z2, ...], and the action similarity (Euclidean distance is 0.3) and the depth similarity (Euclidean distance is 0.2) are obtained.

[0141] In a specific embodiment, the video frame of the second target data pair is a defensive scene in a football game. The action skeleton diagram (showing the player's defensive posture) is extracted through the OpenPose model, and the depth map (showing the distance between the player and the goal) is extracted through the Depth model. A video frame in the similar video frame set is an offensive scene in a football game. The similarity between its action skeleton diagram and the skeleton diagram of the target video frame is 0.3, the similarity between the depth map is 0.2, the similarity between the text description words is 0.6, and the comprehensive target similarity is 0.39. If there are multiple video frames in the similar video frame set, after sorting by comprehensive similarity, the video frame with the highest similarity is selected as the target similar video frame.

[0142] In this way, the embodiment of the present application comprehensively evaluates the semantic and visual consistency of the video frame by combining the similarity of text description words, the similarity of action skeleton graphs and the similarity of depth maps, avoids the deviation of single-dimensional judgment, selects video frames that are highly similar to the target video frame as a supplementary source, and ensures that the supplementary description words (such as "shoot" and "attack") are highly relevant to the original video content, avoiding the introduction of irrelevant information. At the same time, the introduction of action skeleton graphs and depth maps makes up for the shortcomings of traditional methods that rely only on text description words, and more accurately captures the visual features of video frames. It can be seen that the embodiment of the present application extracts action skeleton graphs and depth maps, and combines the comprehensive calculation of text description word similarity, action similarity and depth similarity. This solution realizes the accurate screening of similar video frames, which not only improves the accuracy of multimodal data alignment, but also provides a reliable basis for subsequent annotation expansion, and significantly optimizes the efficiency and quality of data annotation.

[0143] Optionally, the method further includes:

[0144] When the target similarity corresponding to the target similar video frame is less than the second preset threshold, expanding at least one target description word obtained by splitting the text description in the second target data pair to obtain a fourth data pair set;

[0145] The fourth data pair set includes the video frame in the second target data pair, the at least one target descriptor, and at least one expanded descriptor corresponding to the target descriptor.

[0146] In some embodiments, when the comprehensive similarity of the target similar video frame (such as 0.39) is less than the second preset threshold (such as 0.7), it is determined that the text description of the current second target data pair has insufficient semantic relevance to the video frame and needs to be expanded. The target descriptive words can be split, and the text description in the second target data pair can be segmented to obtain a set of target description words. For example, the original text description can be "The shot that Person A just now was not handled well", and after splitting, {"Person A", "Shot", "Handled poorly"} is obtained. For expansion based on the knowledge graph, a knowledge graph related to the video content (such as a football game knowledge graph) can be used. The expansion rule can be to use the target descriptive word as a query condition, retrieve relevant field knowledge from the knowledge graph, and generate an expanded description word set. For example, if the target descriptive word is "Person A", the knowledge graph query result is {"football match", "attack", "midfield", "shoot"}. If the target descriptive word is "shoot", the knowledge graph query result can be {"attack", "goal", "goalkeeper"}.

[0147] Thus, a fourth data pair set can be generated, and the expanded descriptive word set is combined with the original video frame to form a fourth data pair set. The data pair structure includes: video frames (i.e., video frames in the second target data pair), target descriptive words (i.e., the descriptive word set after the original split (such as {"Character A", "shooting", "not handled well"})), expanded descriptive words (i.e., the descriptive word set after the knowledge graph is expanded (such as {"football match", "offense", "midfield"})). Exemplarily, the fourth data pair set may include: video frames (defensive scenes in football matches), target descriptive words ({"Character A", "shooting", "not handled well"}), expanded descriptive words ({"football match", "offense", "midfield"}).

[0148] In this way, the embodiment of the present application supplements domain knowledge through the knowledge graph, enhances the semantic relevance between text descriptions and video frames, avoids labeling errors caused by semantic deviations, and at the same time, expands the description words to include abstract concepts (such as "attack" and "midfield"), making up for the shortcomings of traditional methods in insufficient understanding of abstract content and improving the diversity of labeled data. It can be seen that the embodiment of the present application solves the problem of insufficient semantic relevance by using the domain knowledge graph to expand the target descriptive words when the similarity is insufficient. The embodiment of the present application significantly improves the accuracy and diversity of multimodal data annotation by combining knowledge graph expansion and automated processing, providing a high-quality data foundation for subsequent model training.

[0149] Optionally, the method further includes:

[0150] Inputting all the descriptive words in the third target data pair into a fourth preset model to obtain a first category set and a second category set of the descriptive words in the third target data pair;

[0151] Inputting all descriptive words in a fourth target data pair into a fourth preset model to obtain a first category set and a second category set of descriptive words in the fourth target data pair, wherein the fourth target data pair is any data pair in the third data pair set;

[0152] Obtaining a degree of overlap between the description words in the third target data pair and the description words in the fourth target data pair;

[0153] When the overlap degree is greater than or equal to a third preset threshold, the descriptive words in the second category set in the third target data pair are added to the fourth target data pair, and the descriptive words in the second category set in the fourth target data pair are added to the third target data pair, to obtain a supplemented third target data pair and a supplemented fourth target data pair;

[0154] The descriptive words in the first category set are words used to represent clear entities or behaviors, and the descriptive words in the second category set are words used to express emotions, metaphors or abstract concepts.

[0155] In some embodiments, all the description words in the third target data pair and the fourth target data pair are input into a fourth preset model (such as a BERT model) and divided into two categories:

[0156] The first category collection: specific content (such as "goal", "shooting", "defense").

[0157] The second category collection: abstract content (such as "earth-shattering", "a glimpse", "a wonderful story").

[0158] For example, the descriptive word set for the third target data pair may be {"person A", "shot", "poorly handled"}. Here, the first category set may be {"shot"}, and the second category set may be {"person A", "poorly handled"}. And the descriptive word set for the fourth target data pair may be {"person B", "pass", "football match"}. Here, the first category set may be {"pass", "football match"}, and the second category set may be {"person B"}.

[0159] For the specific content overlap, the overlap of the first category set in the third target data pair and the fourth target data pair is calculated (such as using Jaccard similarity). For the abstract content overlap, the overlap of the second category set in the third target data pair and the fourth target data pair is calculated. Exemplarily, the first category set of the third target data pair can be {"shooting"}, and the first category set of the fourth target data pair can be {"pass", "football match"}, and the resulting overlap is 0 (no overlap). The second category set of the third target data pair can be {"person A", "not handled well"}, and the second category set of the fourth target data pair can be {"person B"}, and the resulting overlap is 0 (no overlap).

[0160] Furthermore, if the overlap degree is greater than or equal to a third preset threshold (such as 0.5), the following operations are performed:

[0161] To supplement the abstract content, the descriptive words in the second category set of the third target data pair (such as "stone-breaking") can be added to the fourth target data pair;

[0162] To supplement the abstract content, the descriptive words in the second category set of the fourth target data pair (such as "wonderful transmission") can be added to the third target data pair.

[0163] For example, the second category set of the third target data pair may be {"groundbreaking"}, and the second category set of the fourth target data pair may be {"excellent pass"}. Thus, the third target data pair obtained after supplementation may be {"shot", "groundbreaking", "excellent pass"}, and the fourth target data pair may be {"pass", "football match", "groundbreaking", "excellent pass"}.

[0164] In a specific embodiment, the descriptive word set for the third target data pair is {"person A", "shot", "earth-shattering"}, and the descriptive word set for the fourth target data pair is {"person B", "pass", "excellent pass"}. After classification by the BERT model:

[0165] The third target data pair: the first category set {"shot"}, the second category set {"person A", "stone-breaking"};

[0166] The fourth target data pair: the first category set {"pass"}, the second category set {"person B", "good pass"};

[0167] If the overlap is 0, but domain knowledge shows that "earth-shattering" and "wonderful transmission" are both abstract contents, they can be directly added to the other party's data pair.

[0168] To calculate the overlap, you can use Jaccard similarity to calculate the overlap of specific content:

[0169] The first category set of the third target data pair: {“shot”};

[0170] The first category set of the fourth target data pair: {“pass”};

[0171] Overlap: 0 (no overlap).

[0172] In this way, the embodiment of the present application improves the abstract semantic expression ability of the data pair by exchanging abstract descriptive words in the second category set (such as "stone-breaking" and "wonderful transmission"), which can make up for the defect of traditional methods in insufficient understanding of abstract content. Therefore, the embodiment of the present application effectively improves the abstract semantic expression ability of multimodal data annotation by dividing descriptive words into specific content and abstract content, and exchanging and supplementing them based on the degree of overlap. By combining BERT model classification and domain knowledge, it optimizes the semantic integrity of the data pair and provides a higher quality data foundation for subsequent model training.

[0173] Optionally, the method further includes:

[0174] using the supplemented third target data pair and the supplemented fourth target data pair as positively labeled data;

[0175] combining the descriptive words corresponding to different video frames in the third data pair set to obtain negatively labeled data;

[0176] A preset text-generated graph model is trained based on the positively labeled data and the negatively labeled data to obtain an optimized text-generated graph model, wherein the preset text-generated graph model is a model for converting text descriptions into corresponding images.

[0177] In some embodiments, the supplemented third and fourth target data pairs can be used as positive annotation data. The third target data pair can include a video frame and a set of descriptive words (including specific and abstract content). For example, the third target data pair can be {"shot", {"shot", "offense", "groundbreaking"}, video frame}, and the fourth target data pair can be {"pass", {"pass", "defense", "excellent pass"}, video frame}. This ensures that the positive data pair ensures semantic consistency between the text description and the video frame, and is used to train the model to learn the correspondence between text and image.

[0178] Furthermore, the descriptive words of different video frames can be combined, for example, the descriptive word sets of two video frames are randomly selected from the third data pair set to generate mismatched text-image pairs. If the difference between the two descriptive word sets is large (e.g., the number of inconsistent descriptive words exceeds half), they are replaced with negative data. For example, the negative annotation data can be:

[0179] The descriptive word set of video frame A: {“shoot”, “attack”};

[0180] The descriptive word set of video frame B: {"pass", "defense"};

[0181] After combination, negative data is generated: {"shot", {"pass", "defense"}, video frame}.

[0182] Therefore, negatively labeled data pairs help the model learn the non-corresponding relationship between text and images, thereby improving the model's discrimination ability.

[0183] Finally, the pre-set text-to-image model is trained using positively labeled data (text-image pairs) and negatively labeled data (mismatched text-image pairs) to optimize its generative capabilities. For example, if the positive data is {"shot", {"shot", "offense"}, video frame} and the negative data is {"shot", {"pass", "defense"}, video frame}, the trained model can accurately generate images corresponding to "shot" while rejecting images corresponding to "pass".

[0184] In this way, in the embodiment of the present application, the positive data ensures that the model learns the correct text-image correspondence, and the negative data helps the model identify mismatches, significantly improving the accuracy and robustness of the model. Among them, the negative data covers more scenarios (such as the mismatch between abstract concepts and concrete elements), so that the model can adapt to complex and diverse inputs, and can also be trained through balanced positive and negative data. The text-graph model can more accurately generate images that conform to the text description and reduce generation errors. Thus, the automatic generation of positive and negative data replaces manual annotation, reducing dependence on manual intervention, and is particularly suitable for large-scale data training. In the embodiment of the present application, by generating positive / negative annotation data and training the text-graph model, accurate learning of text-image correspondence is achieved, positive data ensures that the model understands semantic consistency, and negative data enhances the discrimination ability of the model, ultimately improving the generation quality and versatility of the model.

[0185] like Figure 3 As shown, in a specific embodiment of the present application, the following steps are included:

[0186] Step 201: annotate the video to be processed and construct a data set containing at least one data pair, where the data pair includes a text description and a video frame;

[0187] Step 202: Obtain a target text description and a target video frame in a first target data pair, where the target video frame is any one of at least one video frame corresponding to the target text description, and the first target data is any one of the data pair set;

[0188] Step 203: Split the target text description to obtain a first description word set, where the first description word set includes at least one description word corresponding to the target text description;

[0189] Step 204: Based on the preset knowledge graph, expand the description words in the first description word set to obtain a second description word set;

[0190] Step 205: Input the target video frame into the first preset model, and obtain a prompt word set corresponding to the target video frame output by the first preset model;

[0191] Step 206: Input the second description word set and the prompt word set into the second preset model, and obtain a first feature vector corresponding to the second description word set and a second feature vector corresponding to the prompt word set output by the second preset model;

[0192] Step 207: Calculate the Euclidean distance between the first eigenvector and the second eigenvector;

[0193] Step 208: Determine whether the calculated Euclidean distance is less than or equal to a first preset threshold; if yes, execute step 209; if no, execute step 210;

[0194] Step 209: Determine semantic consistency between the target text description and the target video frame, and determine the first target data pair as a data pair in the first data pair set; then proceed to step 211;

[0195] Step 210: Determine that there is semantic inconsistency between the target text description and the target video frame, and determine the first target data pair as a data pair in the second data pair set; then proceed to step 212;

[0196] Step 211: Search the first data pair set for video frames containing at least one prompt word in the target prompt word set to obtain a similar video frame set;

[0197] Step 212: Obtain a target prompt word set corresponding to the third video frame in the second target data pair, where the second target data pair is any data pair in the second data pair set, and the target prompt word set is a set consisting of at least one prompt word contained in the third video frame extracted by a preset model;

[0198] Step 213: Determine a video frame in the similar video frame set that has the highest similarity to the video frame in the second target data pair as a target similar video frame;

[0199] Step 214: If the target similarity corresponding to the target similar video frame is greater than or equal to the second preset threshold, add the descriptive word set corresponding to the target similar video frame to the second target data pair to obtain a third target data pair; and construct a third data pair set based on the third target data pair.

[0200] Step 215: Input all the descriptive words in the third target data pair into the fourth preset model to obtain a first category set and a second category set of the descriptive words in the third target data pair;

[0201] Step 216: Input all the descriptive words in the fourth target data pair into the fourth preset model to obtain a first category set and a second category set of the descriptive words in the fourth target data pair, where the fourth target data pair is any data pair in the third data pair set.

[0202] Step 217: Obtain the degree of overlap between the description words in the third target data pair and the description words in the fourth target data pair;

[0203] Step 218: If the degree of overlap is greater than or equal to a third preset threshold, the descriptive words in the second category set in the third target data pair are added to the fourth target data pair, and the descriptive words in the second category set in the fourth target data pair are added to the third target data pair, to obtain a supplemented third target data pair and a supplemented fourth target data pair; wherein the descriptive words in the first category set are words used to represent specific entities or behaviors, and the descriptive words in the second category set are words used to express emotions, metaphors, or abstract concepts;

[0204] Step 219: Use the supplemented third target data pair and the supplemented fourth target data pair as positive labeled data; combine the descriptive words corresponding to different video frames in the third data pair set to obtain negative labeled data;

[0205] Step 220 : Training a preset text-to-graph model based on the positively labeled data and the negatively labeled data to obtain an optimized text-to-graph model. The preset text-to-graph model is a model that converts text descriptions into corresponding images.

[0206] The data annotation method provided in the above embodiment of the present application can perform the above Figure 1 The implementation principle and technical effects of the method embodiment shown are similar, and reference may be made to the relevant descriptions in the aforementioned embodiments, which will not be repeated here.

[0207] Please refer to Figure 4 , Figure 4 FIG. 1 is a schematic diagram of a data labeling device according to an embodiment of the present application. The data labeling device 300 specifically includes:

[0208] Annotation module 301 is used to annotate the video to be processed and construct a data set containing at least one data pair, wherein the data pair includes a text description and a video frame;

[0209] A classification module 302 is configured to divide the data set into a first data pair set and a second data pair set; wherein the first data pair set includes data pairs whose text descriptions and video frames have semantic consistency, and the second data pair set includes data pairs whose text descriptions and video frames have semantic inconsistency;

[0210] An expansion module 303 is configured to add corresponding text descriptions to the video frames included in the second data pair set to obtain a third data pair set;

[0211] The determination module 304 is configured to determine the first data pair set and the third data pair set as labeled data.

[0212] Optionally, the marking module 301 includes:

[0213] A first conversion unit, configured to convert the voice data in the video to be processed into text information;

[0214] A first splitting unit is configured to split the sentences in the text information to obtain at least one text description, wherein each text description in the at least one text description corresponds to a starting node and an ending node in the video to be processed;

[0215] a data extraction unit, configured to extract, from the to-be-processed video, at least one video frame corresponding to each of the at least one text description based on a start node and an end node corresponding to each of the at least one text description;

[0216] The data construction unit is used to construct at least one data pair based on the at least one text description and at least one video frame corresponding to each text description in the at least one text description to obtain the data set.

[0217] Optionally, the classification module 302 includes:

[0218] a first acquiring unit, configured to acquire a target text description and a target video frame in a first target data pair, wherein the target video frame is any one of at least one video frame corresponding to the target text description, and the first target data pair is any one of the data sets;

[0219] a second splitting unit, configured to split the target text description to obtain a first description word set, wherein the first description word set includes at least one description word corresponding to the target text description;

[0220] an expansion unit, configured to expand the description words in the first description word set based on a preset knowledge graph to obtain a second description word set;

[0221] a second acquiring unit, configured to input the target video frame into a first preset model, and acquire a prompt word set corresponding to the target video frame output by the first preset model;

[0222] a third acquiring unit, configured to input the second description word set and the prompt word set into a second preset model, and acquire a first feature vector corresponding to the second description word set and a second feature vector corresponding to the prompt word set output by the second preset model;

[0223] a first calculating unit, configured to calculate the Euclidean distance between the first eigenvector and the second eigenvector;

[0224] a first determining unit, configured to, when the Euclidean distance is less than or equal to a first preset threshold, determine semantic consistency between the target text description and the target video frame, and determine the first target data pair as a data pair in the first data pair set;

[0225] A second determination unit is configured to determine, when the Euclidean distance is greater than the first preset threshold, a semantic inconsistency between the target text description and the target video frame, and determine the first target data pair as a data pair in the second data pair set.

[0226] Optionally, any first data pair in the first data pair set includes a first text description, a first word set, a first video frame corresponding to the first text description, and a first result; wherein the first word set includes a description word set obtained by splitting the first text description, and a prompt word set corresponding to the first video frame, and the first result is used to indicate semantic consistency between the first text description and the first video frame;

[0227] Any second data pair in the second data pair set includes a second text description, a second word set, a second video frame corresponding to the second text description, and a second result; wherein the second word set includes a prompt word set corresponding to the second video frame, and the second result is used to indicate the semantic inconsistency between the second text description and the second video frame.

[0228] Optionally, the expansion module 303 includes:

[0229] a fourth acquisition unit, configured to acquire a target prompt word set corresponding to the third video frame in a second target data pair, wherein the second target data pair is any one data pair in the second data pair set, and the target prompt word set is a set consisting of at least one prompt word contained in the third video frame extracted by a preset model;

[0230] a search unit, configured to search the first data pair set for a video frame containing at least one cue word in the target cue word set, to obtain a similar video frame set;

[0231] a third determining unit, configured to determine a video frame in the similar video frame set that has the highest similarity to the video frame in the second target data pair as a target similar video frame;

[0232] a first supplementing unit, configured to supplement the descriptive word set corresponding to the target similar video frame into the second target data pair when the target similarity corresponding to the target similar video frame is greater than or equal to a second preset threshold, thereby updating to obtain a third target data pair;

[0233] A first constructing unit, configured to construct the third data pair set based on the third target data pair;

[0234] The third target data pair in the third data pair set includes the third video frame, a description word set corresponding to the text description in the second target data pair, and a description word set corresponding to the target similar video frame.

[0235] Optionally, the searching unit is configured to:

[0236] Searching for a video frame containing at least one cue word in the target cue word set from the first data pair set to obtain at least one similar video frame;

[0237] sorting the at least one similar video frame according to the number of the prompt words contained in each of the at least one similar video frame to obtain a similar video frame sequence;

[0238] Based on the number of the prompt words included in the similar video frame sequence, a preset number of video frames are sequentially selected from the similar video frame sequence to obtain the similar video frame set.

[0239] Optionally, the third determining unit is configured to:

[0240] Extracting a first image and a second image from each similar video frame in the set of similar video frames based on a third preset model; wherein the first image is an action skeleton image of the target object in the video frame, and the second image is a depth image corresponding to the video frame;

[0241] Obtaining a first similarity between the text description word in the second target data pair and the text description word corresponding to each similar video frame;

[0242] calculating a second similarity between the third video frame and the first image in each of the similar video frames, and calculating a third similarity between the third video frame and the second image in each of the similar video frames;

[0243] determining a target similarity between the third video frame and each of the similar video frames based on the first similarity, the second similarity, and the third similarity;

[0244] The similar video frame with the highest target similarity in the similar video frame set is determined as the target similar video frame.

[0245] Optionally, the data labeling device 300 is further configured to:

[0246] When the target similarity corresponding to the target similar video frame is less than the second preset threshold, expanding at least one target description word obtained by splitting the text description in the second target data pair to obtain a fourth data pair set;

[0247] The fourth data pair set includes the video frame in the second target data pair, the at least one target descriptor, and at least one expanded descriptor corresponding to the target descriptor.

[0248] Optionally, the data labeling device 300 is further configured to:

[0249] Inputting all the descriptive words in the third target data pair into a fourth preset model to obtain a first category set and a second category set of the descriptive words in the third target data pair;

[0250] Inputting all descriptive words in a fourth target data pair into a fourth preset model to obtain a first category set and a second category set of descriptive words in the fourth target data pair, wherein the fourth target data pair is any data pair in the third data pair set;

[0251] Obtaining a degree of overlap between the description words in the third target data pair and the description words in the fourth target data pair;

[0252] When the overlap degree is greater than or equal to a third preset threshold, the descriptive words in the second category set in the third target data pair are added to the fourth target data pair, and the descriptive words in the second category set in the fourth target data pair are added to the third target data pair, to obtain a supplemented third target data pair and a supplemented fourth target data pair;

[0253] The descriptive words in the first category set are words used to represent clear entities or behaviors, and the descriptive words in the second category set are words used to express emotions, metaphors or abstract concepts.

[0254] Optionally, the data labeling device 300 is further configured to:

[0255] using the supplemented third target data pair and the supplemented fourth target data pair as positively labeled data;

[0256] combining the descriptive words corresponding to different video frames in the third data pair set to obtain negatively labeled data;

[0257] A preset text-generated graph model is trained based on the positively labeled data and the negatively labeled data to obtain an optimized text-generated graph model, wherein the preset text-generated graph model is a model for converting text descriptions into corresponding images.

[0258] The data annotation device 300 provided in the embodiment of the present application can perform the above Figure 1 The implementation principle and technical effects of the method embodiment shown are similar, and will not be described in detail in this embodiment.

[0259] The present application also provides an electronic device. Since the principle of solving the problem by the electronic device is similar to the image detection model training method in the present application, the implementation of the electronic device can refer to Figure 1 The implementation of the method shown in FIG. 1 is omitted for repetition. Figure 5 As shown, the electronic device of the embodiment of the present application includes: a processor 410, which is used to read the program in the memory 420 and execute the following process:

[0260] Annotate the video to be processed and construct a data set containing at least one data pair, wherein the data pair includes a text description and a video frame;

[0261] Dividing the data set into a first data pair set and a second data pair set; wherein the semantics of the text descriptions and the video frames in the data pairs included in the first data pair set are consistent, and the semantics of the text descriptions and the video frames in the data pairs included in the second data pair set are inconsistent;

[0262] adding corresponding text descriptions to the video frames included in the second data pair set to obtain a third data pair set;

[0263] The first data pair set and the third data pair set are determined as labeled data.

[0264] Among them, Figure 5 In the embodiment of the present invention, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits such as one or more processors represented by processor 410 and memory represented by memory 420. The bus architecture may also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and are not further described herein. The bus interface provides an interface.

[0265] Optionally, the processor 410 is further configured to read a program in the memory 420 and execute the following steps:

[0266] Converting the voice data in the video to be processed into text information;

[0267] Splitting the sentences in the text information to obtain at least one text description, wherein each text description in the at least one text description corresponds to a starting node and an ending node in the video to be processed;

[0268] Extracting at least one video frame corresponding to each text description in the at least one text description from the video to be processed based on a start node and an end node corresponding to each text description in the at least one text description;

[0269] Based on the at least one text description and at least one video frame corresponding to each text description in the at least one text description, at least one data pair is constructed to obtain the data set.

[0270] Optionally, the processor 410 is further configured to read a program in the memory 420 and execute the following steps:

[0271] Obtaining a target text description and a target video frame in a first target data pair, where the target video frame is any one of at least one video frame corresponding to the target text description, and the first target data pair is any one of the data sets;

[0272] Splitting the target text description to obtain a first description word set, where the first description word set includes at least one description word corresponding to the target text description;

[0273] Based on a preset knowledge graph, the description words in the first description word set are expanded to obtain a second description word set;

[0274] Inputting the target video frame into a first preset model, and obtaining a prompt word set corresponding to the target video frame output by the first preset model;

[0275] Inputting the second description word set and the prompt word set into a second preset model, and obtaining a first feature vector corresponding to the second description word set and a second feature vector corresponding to the prompt word set output by the second preset model;

[0276] Calculating the Euclidean distance between the first eigenvector and the second eigenvector;

[0277] When the Euclidean distance is less than or equal to a first preset threshold, determining semantic consistency between the target text description and the target video frame, and determining the first target data pair as a data pair in the first data pair set;

[0278] When the Euclidean distance is greater than the first preset threshold, it is determined that there is semantic inconsistency between the target text description and the target video frame, and the first target data pair is determined as a data pair in the second data pair set.

[0279] Optionally, any first data pair in the first data pair set includes a first text description, a first word set, a first video frame corresponding to the first text description, and a first result; wherein the first word set includes a description word set obtained by splitting the first text description, and a prompt word set corresponding to the first video frame, and the first result is used to indicate semantic consistency between the first text description and the first video frame;

[0280] Any second data pair in the second data pair set includes a second text description, a second word set, a second video frame corresponding to the second text description, and a second result; wherein the second word set includes a prompt word set corresponding to the second video frame, and the second result is used to indicate the semantic inconsistency between the second text description and the second video frame.

[0281] Optionally, the processor 410 is further configured to read a program in the memory 420 and perform the following steps:

[0282] Obtaining a target prompt word set corresponding to the third video frame in a second target data pair, where the second target data pair is any one data pair in the second data pair set, and the target prompt word set is a set consisting of at least one prompt word contained in the third video frame extracted by a preset model;

[0283] Searching for video frames containing at least one prompt word in the target prompt word set from the first data pair set to obtain a similar video frame set;

[0284] Determine a video frame in the similar video frame set that has the highest similarity to the video frame in the second target data pair as a target similar video frame;

[0285] When the target similarity corresponding to the target similar video frame is greater than or equal to a second preset threshold, the descriptive word set corresponding to the target similar video frame is added to the second target data pair to obtain a third target data pair;

[0286] Constructing the third data pair set based on the third target data pair;

[0287] The third target data pair in the third data pair set includes the third video frame, a description word set corresponding to the text description in the second target data pair, and a description word set corresponding to the target similar video frame.

[0288] Optionally, the processor 410 is further configured to read a program in the memory 420 and execute the following steps:

[0289] Searching for a video frame containing at least one cue word in the target cue word set from the first data pair set to obtain at least one similar video frame;

[0290] sorting the at least one similar video frame according to the number of the prompt words contained in each of the at least one similar video frame to obtain a similar video frame sequence;

[0291] Based on the number of the prompt words included in the similar video frame sequence, a preset number of video frames are sequentially selected from the similar video frame sequence to obtain the similar video frame set.

[0292] Optionally, the processor 410 is further configured to read a program in the memory 420 and execute the following steps:

[0293] Extracting a first image and a second image from each similar video frame in the set of similar video frames based on a third preset model; wherein the first image is an action skeleton image of the target object in the video frame, and the second image is a depth image corresponding to the video frame;

[0294] Obtaining a first similarity between the text description word in the second target data pair and the text description word corresponding to each similar video frame;

[0295] calculating a second similarity between the third video frame and the first image in each of the similar video frames, and calculating a third similarity between the third video frame and the second image in each of the similar video frames;

[0296] determining a target similarity between the third video frame and each of the similar video frames based on the first similarity, the second similarity, and the third similarity;

[0297] The similar video frame with the highest target similarity in the similar video frame set is determined as the target similar video frame.

[0298] Optionally, the processor 410 is further configured to read a program in the memory 420 and perform the following steps:

[0299] When the target similarity corresponding to the target similar video frame is less than the second preset threshold, expanding at least one target description word obtained by splitting the text description in the second target data pair to obtain a fourth data pair set;

[0300] The fourth data pair set includes the video frame in the second target data pair, the at least one target descriptor, and at least one expanded descriptor corresponding to the target descriptor.

[0301] Optionally, the processor 410 is further configured to read a program in the memory 420 and perform the following steps:

[0302] Inputting all the descriptive words in the third target data pair into a fourth preset model to obtain a first category set and a second category set of the descriptive words in the third target data pair;

[0303] Inputting all descriptive words in a fourth target data pair into a fourth preset model to obtain a first category set and a second category set of descriptive words in the fourth target data pair, wherein the fourth target data pair is any data pair in the third data pair set;

[0304] Obtaining a degree of overlap between the description words in the third target data pair and the description words in the fourth target data pair;

[0305] When the overlap degree is greater than or equal to a third preset threshold, the descriptive words in the second category set in the third target data pair are added to the fourth target data pair, and the descriptive words in the second category set in the fourth target data pair are added to the third target data pair, to obtain a supplemented third target data pair and a supplemented fourth target data pair;

[0306] The descriptive words in the first category set are words used to represent clear entities or behaviors, and the descriptive words in the second category set are words used to express emotions, metaphors or abstract concepts.

[0307] Optionally, the processor 410 is further configured to read a program in the memory 420 and perform the following steps:

[0308] using the supplemented third target data pair and the supplemented fourth target data pair as positively labeled data;

[0309] combining the descriptive words corresponding to different video frames in the third data pair set to obtain negatively labeled data;

[0310] A preset text-generated graph model is trained based on the positively labeled data and the negatively labeled data to obtain an optimized text-generated graph model, wherein the preset text-generated graph model is a model for converting text descriptions into corresponding images.

[0311] The electronic device 400 provided in the embodiment of the present application can perform the above Figure 1 The implementation principle and technical effects of the method embodiment shown are similar, and will not be described in detail in this embodiment.

[0312] The present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned Figure 1The various processes of the data annotation method embodiment in the embodiment can achieve the same technical effect, so they are not repeated here to avoid repetition. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0313] The embodiment of the present application further provides a computer program / program product, which is stored in a storage medium and is executed by at least one processor to implement the above Figure 1 The various processes of the data annotation method embodiment can achieve the same technical effect. To avoid repetition, they will not be described here.

[0314] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection of some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0315] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may be physically included separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0316] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute some steps of the sending and receiving methods described in various embodiments of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program code.

[0317] The above is a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles described in the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data annotation method, characterized in that: include: Annotate the video to be processed and construct a data set containing at least one data pair, wherein the data pair includes a text description and a video frame; Dividing the data set into a first data pair set and a second data pair set; wherein the semantics of the text descriptions and the video frames in the data pairs included in the first data pair set are consistent, and the semantics of the text descriptions and the video frames in the data pairs included in the second data pair set are inconsistent; adding corresponding text descriptions to the video frames included in the second data pair set to obtain a third data pair set; The first data pair set and the third data pair set are determined as labeled data.

2. The method according to claim 1, characterized in that The step of labeling the video to be processed and constructing a data set containing at least one data pair includes: Converting the voice data in the video to be processed into text information; Splitting the sentences in the text information to obtain at least one text description, wherein each text description in the at least one text description corresponds to a starting node and an ending node in the video to be processed; Extracting at least one video frame corresponding to each text description in the at least one text description from the video to be processed based on a start node and an end node corresponding to each text description in the at least one text description; Based on the at least one text description and at least one video frame corresponding to each text description in the at least one text description, at least one data pair is constructed to obtain the data set.

3. The method according to claim 1, characterized in that The dividing the data set into a first data pair set and a second data pair set comprises: Obtaining a target text description and a target video frame in a first target data pair, where the target video frame is any one of at least one video frame corresponding to the target text description, and the first target data pair is any one of the data sets; Splitting the target text description to obtain a first description word set, where the first description word set includes at least one description word corresponding to the target text description; Based on a preset knowledge graph, the description words in the first description word set are expanded to obtain a second description word set; Inputting the target video frame into a first preset model, and obtaining a prompt word set corresponding to the target video frame output by the first preset model; Inputting the second description word set and the prompt word set into a second preset model, and obtaining a first feature vector corresponding to the second description word set and a second feature vector corresponding to the prompt word set output by the second preset model; Calculating the Euclidean distance between the first eigenvector and the second eigenvector; When the Euclidean distance is less than or equal to a first preset threshold, determining semantic consistency between the target text description and the target video frame, and determining the first target data pair as a data pair in the first data pair set; When the Euclidean distance is greater than the first preset threshold, it is determined that there is semantic inconsistency between the target text description and the target video frame, and the first target data pair is determined as a data pair in the second data pair set.

4. The method according to claim 1 or 3, characterized in that Any first data pair in the first data pair set includes a first text description, a first word set, a first video frame corresponding to the first text description, and a first result; wherein the first word set includes a description word set obtained by splitting the first text description, and a prompt word set corresponding to the first video frame, and the first result is used to indicate semantic consistency between the first text description and the first video frame; Any second data pair in the second data pair set includes a second text description, a second word set, a second video frame corresponding to the second text description, and a second result; wherein the second word set includes a prompt word set corresponding to the second video frame, and the second result is used to indicate the semantic inconsistency between the second text description and the second video frame.

5. The method according to claim 1, wherein The step of adding corresponding text descriptions to the video frames included in the second data pair set to obtain a third data pair set includes: Obtaining a target prompt word set corresponding to the third video frame in a second target data pair, where the second target data pair is any one data pair in the second data pair set, and the target prompt word set is a set consisting of at least one prompt word contained in the third video frame extracted by a preset model; Searching for video frames containing at least one prompt word in the target prompt word set from the first data pair set to obtain a similar video frame set; Determine a video frame in the similar video frame set that has the highest similarity to the video frame in the second target data pair as a target similar video frame; When the target similarity corresponding to the target similar video frame is greater than or equal to a second preset threshold, the descriptive word set corresponding to the target similar video frame is added to the second target data pair to obtain a third target data pair; Constructing the third data pair set based on the third target data pair; The third target data pair in the third data pair set includes the third video frame, a description word set corresponding to the text description in the second target data pair, and a description word set corresponding to the target similar video frame.

6. The method according to claim 5, characterized in that The step of searching the first data pair set for a video frame containing at least one prompt word in the target prompt word set to obtain a similar video frame set includes: Searching for a video frame containing at least one cue word in the target cue word set from the first data pair set to obtain at least one similar video frame; sorting the at least one similar video frame according to the number of the prompt words contained in each of the at least one similar video frame to obtain a similar video frame sequence; Based on the number of the prompt words included in the similar video frame sequence, a preset number of video frames are sequentially selected from the similar video frame sequence to obtain the similar video frame set.

7. The method according to claim 5, characterized in that The determining a video frame in the set of similar video frames that has the highest similarity to the video frame in the second target data pair as a target similar video frame includes: Extracting a first image and a second image from each similar video frame in the set of similar video frames based on a third preset model; wherein the first image is an action skeleton image of the target object in the video frame, and the second image is a depth image corresponding to the video frame; Obtaining a first similarity between the text description word in the second target data pair and the text description word corresponding to each similar video frame; calculating a second similarity between the third video frame and the first image in each of the similar video frames, and calculating a third similarity between the third video frame and the second image in each of the similar video frames; determining a target similarity between the third video frame and each of the similar video frames based on the first similarity, the second similarity, and the third similarity; The similar video frame with the highest target similarity in the similar video frame set is determined as the target similar video frame.

8. The method according to claim 5, characterized in that The method further comprises: When the target similarity corresponding to the target similar video frame is less than the second preset threshold, expanding at least one target description word obtained by splitting the text description in the second target data pair to obtain a fourth data pair set; The fourth data pair set includes the video frame in the second target data pair, the at least one target descriptor, and at least one expanded descriptor corresponding to the target descriptor.

9. The method according to any one of claims 5 to 8, characterized in that The method further comprises: Inputting all the descriptive words in the third target data pair into a fourth preset model to obtain a first category set and a second category set of the descriptive words in the third target data pair; Inputting all descriptive words in a fourth target data pair into a fourth preset model to obtain a first category set and a second category set of descriptive words in the fourth target data pair, wherein the fourth target data pair is any data pair in the third data pair set; Obtaining a degree of overlap between the description words in the third target data pair and the description words in the fourth target data pair; When the overlap degree is greater than or equal to a third preset threshold, the descriptive words in the second category set in the third target data pair are added to the fourth target data pair, and the descriptive words in the second category set in the fourth target data pair are added to the third target data pair, to obtain a supplemented third target data pair and a supplemented fourth target data pair; The descriptive words in the first category set are words used to represent clear entities or behaviors, and the descriptive words in the second category set are words used to express emotions, metaphors or abstract concepts.

10. The method according to claim 9, characterized in that The method further comprises: using the supplemented third target data pair and the supplemented fourth target data pair as positively labeled data; combining the descriptive words corresponding to different video frames in the third data pair set to obtain negatively labeled data; A preset text-generated graph model is trained based on the positively labeled data and the negatively labeled data to obtain an optimized text-generated graph model, wherein the preset text-generated graph model is a model for converting text descriptions into corresponding images.

11. A data labeling device, characterized in that: Applied to a Redis server, the device includes: Annotation module, configured to annotate the video to be processed and construct a data set comprising at least one data pair, wherein the data pair comprises a text description and a video frame; a classification module, configured to divide the data set into a first data pair set and a second data pair set; wherein the semantics of the text descriptions and the video frames in the data pairs included in the first data pair set are consistent, and the semantics of the text descriptions and the video frames in the data pairs included in the second data pair set are inconsistent; an expansion module, configured to add corresponding text descriptions to the video frames included in the second data pair set to obtain a third data pair set; A determination module is configured to determine the first data pair set and the third data pair set as labeled data.

12. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the data labeling method according to any one of claims 1 to 10.

13. A computer-readable storage medium for storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the data labeling method according to any one of claims 1 to 10 are implemented.

14. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps in the data labeling method according to any one of claims 1 to 10.

Images