Video marking method and device, computer equipment and storage medium

Abstract

The embodiment of the invention provides a video marking method and device, computer equipment and a storage medium, belongs to the technical field of video processing, and is suitable for the fields of financial science and technology, medical health and the like. The method comprises the following steps: acquiring a target video feature and a target audio feature of target audio and video data; obtaining and performing feature extraction on the query text to obtain query text features; wherein the query text features represent key contents in the query text; performing feature fusion on the target video feature and the target audio feature to obtain a target audio and video feature; performing feature map construction according to the target audio and video features to obtain an audio and video feature map; wherein the audio and video feature map comprises candidate fragment features with a time sequence; performing video clip searching on the target audio and video data according to the query text feature and the candidate clip feature to obtain a target video clip; and marking the target video clip in the target audio and video data. According to the embodiment of the invention, the video marking manpower can be reduced, and the video marking efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and is applicable to fields such as financial technology and medical health, and in particular to a video marking method and apparatus, computer equipment, and storage medium. Background Art

[0002] With the rapid development of science and technology, in order to make data analysis more intuitive, it is necessary to locate key events in a large amount of data. For example, in the financial field, when the financial market fluctuates greatly, in order to analyze market-related news reports, audio of market data, and stock price charts / videos, it is necessary to mark the relevant audio and video data in the financial field in combination with news reports to mark video clips that reflect the fluctuations in the financial market, making financial analysis easier. In related technologies, video tagging based on query text mainly relies on manual completion, and relevant personnel are required to watch the video in full before tagging it, which not only consumes a lot of manpower, but also affects the efficiency of video tagging. Therefore, how to reduce the manpower of video tagging and improve the efficiency of video tagging has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to provide a video tagging method and apparatus, a computer device and a storage medium, aiming to reduce the manpower for video tagging and improve the efficiency of video tagging.

[0004] To achieve the above object, a first aspect of an embodiment of the present application provides a video marking method, the method comprising:

[0005] Obtain target video features and target audio features of target audio and video data;

[0006] Get the query text;

[0007] Extracting features from the query text to obtain query text features; wherein the query text features are used to characterize key content in the query text;

[0008] Performing feature fusion on the target video feature and the target audio feature to obtain the target audio and video feature;

[0009] Constructing a feature graph according to the target audio and video features to obtain an audio and video feature graph; wherein the audio and video feature graph includes candidate segment features with a time sequence;

[0010] Searching for a video segment of the target audio and video data according to the query text feature and the candidate segment feature to obtain a target video segment;

[0011] The target video segment in the target audio and video data is marked.

[0012] In some embodiments, the step of fusing the target video feature with the target audio feature to obtain the target audio and video feature includes:

[0013] Performing attention interaction on the target video feature and the target audio feature according to a preset bottleneck feature to obtain an attention interaction feature;

[0014] According to the attention interaction feature, the target video feature and the target audio feature are subjected to feature fusion to obtain a preliminary audio and video feature;

[0015] The preliminary audio and video features are averaged and pooled to obtain the target audio and video features.

[0016] In some embodiments, performing attention interaction on the target video feature and the target audio feature according to the preset bottleneck feature to obtain the attention interaction feature includes:

[0017] converting the bottleneck feature into a query feature;

[0018] Converting the target video features into key features;

[0019] Converting the target audio feature into a value feature;

[0020] Perform attention interaction according to the query feature, the key feature, and the value feature to obtain attention interaction data;

[0021] Feature aggregation processing is performed on the target video features and the target audio features according to the attention interaction data to obtain the attention interaction features.

[0022] In some embodiments, constructing a feature map according to the target audio and video features to obtain an audio and video feature map includes:

[0023] Performing pooling processing on the target audio and video features to obtain updated audio and video features;

[0024] Splitting the updated audio and video features to obtain candidate segment features;

[0025] Obtaining the start and end time information of each candidate segment feature;

[0026] The candidate segment features are arranged according to the start and end time information to obtain the audio and video feature graph.

[0027] In some embodiments, the target audio and video data includes: candidate video segments and candidate video frame images; the step of searching for video segments on the target audio and video data according to the query text features and the candidate segment features to obtain the target video segment includes:

[0028] Performing frame processing on the candidate segment features to obtain candidate frame features;

[0029] Performing relevance evaluation on the candidate video segments according to the query text features and the candidate segment features to obtain segment relevance evaluation data;

[0030] Performing relevance evaluation on the candidate video frame images according to the query text features and the candidate frame features to obtain frame relevance evaluation data;

[0031] splicing the segment correlation evaluation data and the frame correlation evaluation data to obtain target correlation evaluation data;

[0032] The target audio and video data is searched for a video segment according to the target relevance evaluation data to obtain the target video segment.

[0033] In some embodiments, marking the target video segment in the target audio and video data includes:

[0034] Acquire the time information of the target video clip to obtain target time information;

[0035] The target audio and video data is marked according to the target time information.

[0036] In some embodiments, the step of obtaining target video features and target audio features of target audio and video data includes:

[0037] Performing frame processing on the target audio and video data to obtain a target video frame image;

[0038] Extracting audio data from the target audio and video data to obtain target audio data;

[0039] Extracting features from the target video frame image to obtain preliminary video features;

[0040] Performing global context processing on the preliminary video features to obtain the target video features;

[0041] Extracting features from the target audio data to obtain preliminary audio features;

[0042] The preliminary audio features are subjected to contextual global processing to obtain the target audio features.

[0043] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a video marking device, the device comprising:

[0044] A feature acquisition module, used to acquire target video features and target audio features of target audio and video data;

[0045] A text acquisition module, used to acquire query text;

[0046] A feature extraction module is used to extract features from the query text to obtain query text features; wherein the query text features are used to characterize key content in the query text;

[0047] A feature fusion module, used to fuse the target video feature and the target audio feature to obtain the target audio and video feature;

[0048] A feature map construction module, used to construct a feature map according to the target audio and video features to obtain an audio and video feature map; wherein the audio and video feature map includes candidate segment features with time sequence;

[0049] A video segment search module, used to search for video segments of the target audio and video data according to the query text features and the candidate segment features, to obtain a target video segment;

[0050] The marking module is used to mark the target video segment in the target audio and video data.

[0051] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes a computer device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0052] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0053] The video tagging method and device, computer equipment and storage medium proposed in the present application first fuse the target video features and target audio features of the target audio and video data into multimodal target audio and video features, and construct the multimodal target audio and video features into an audio and video feature graph including candidate segment features with a time sequence. Feature extraction of the query text is performed to obtain query text features, and the target video segments are searched out in the target audio and video data in combination with the query text features and the candidate segment features, so as to accurately find the target video segments related to the query text content, and finally mark the target video segments in the target audio and video data. Therefore, by automatically constructing multimodal candidate segment features with a time sequence, the target video segment tagging of the target audio and video data is automatically completed with the candidate segment features and the query text features, which can not only accurately mark the video segments related to the query text in the target audio and video data, but also make the marking operation of the video segments automatic, save manpower, and improve the efficiency of video tagging. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flow chart of a video marking method provided in an embodiment of the present application;

[0055] Figure 2 yes Figure 1 Flow chart of step S101 in FIG.

[0056] Figure 3 yes Figure 1 Flow chart of step S104 in FIG.

[0057] Figure 4 yes Figure 3 Flow chart of step S301 in FIG.

[0058] Figure 5 is a system architecture diagram of the video marking method provided in an embodiment of the present application;

[0059] Figure 6 yes Figure 1 Flow chart of step S105 in FIG.

[0060] Figure 7 yes Figure 1 Flow chart of step S106 in FIG.

[0061] Figure 8 yes Figure 1 Flow chart of step S107 in FIG.

[0062] Fig. 9 is an overall flow chart of the video marking method provided in an embodiment of the present application;

[0063] Fig.10 is a schematic diagram of the structure of a video marking device provided in an embodiment of the present application;

[0064] Fig.11 It is a schematic diagram of the hardware structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0066] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0068] First, some nouns involved in this application are analyzed:

[0069] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0070] 2D-TAN (Two-Dimensional Temporal Attention Network): is a deep learning network architecture for detecting target objects in videos. The network detects target objects by combining spatial and temporal attention mechanisms to improve detection performance.

[0071] Unimodal encoder: An encoder that accepts only a single type of data (such as text, image, or audio) as input. A unimodal encoder converts the input data into a low-dimensional vector representation that captures the key features and information of the input data. Different unimodal encoders use different architectures, such as convolutional neural networks (CNNs) for images, recurrent neural networks (RNNs) or Transformers for text, and convolutional neural networks or recurrent neural networks for audio.

[0072] Cross-modal encoder: A model that can encode data of different modalities (such as text, images, audio, etc.) into a unified vector representation. The cross-modal encoder learns the intrinsic connections between data of different modalities and maps different types of data into the same feature space, thereby achieving tasks such as cross-modal retrieval, understanding, and generation.

[0073] Fragment Generator: refers to a model or algorithm that generates text fragments in natural language processing. Fragment generators can generate relevant text fragments based on input prompts or topics, and are used in application areas such as text summarization, text generation, and dialogue systems.

[0074] Contrastive learning module: is a machine learning method that learns representations by comparing pairs of samples. Contrastive learning modules are often used to learn representations of images, text, or other modal data, with the goal of making the representations of samples from the same category or source close to each other, and the representations of samples from different categories or sources far away from each other.

[0075] Bottleneck token: It is a common technology in natural language processing. Its main function is to convert the input sequence into a vector of fixed dimension for subsequent processing and analysis.

[0076] Multiple detection heads: In an object detection model, multiple detection heads are used to simultaneously predict different types of objects or different levels of features. This can improve the detection accuracy and recall of the model compared to a single detection head model, especially when dealing with multi-category objects or objects with large scale variations. Each detection head can focus on a specific type of object or feature scale, thereby reducing interference between different objects and improving the robustness of the model.

[0077] In the era of big data, extracting valuable information from multi-source and multi-modal data and performing multiple tasks has become one of the major challenges in the fields of finance, medicine, and technology. For example, in the financial field, in order to analyze the current financial situation, it is necessary to jointly analyze news reports related to the financial market, sound recordings of market data, and video data related to stocks, and then find the clips in the sound recordings and video data that match the current financial situation. In related technologies, video clips related to financial analysis texts are manually found and marked. Therefore, relevant personnel need to fully review the entire video data and sound recordings before marking them in a targeted manner, which not only consumes a lot of manpower, but also affects the efficiency of video marking.

[0078] Based on this, the embodiments of the present application provide a video tagging method and apparatus, a computer device and a storage medium, which are intended to first extract the query text features of the query text by acquiring the query text, and fuse the target video features and target audio features of the target audio and video data into the target audio and video features to achieve multimodal feature construction. Then, the target audio and video features are constructed into an audio and video feature graph in time sequence to obtain the candidate segment features sorted by time. Then, when finding the video segment related to the query text, the relevant target video segment can be found in the target audio and video data according to each candidate segment feature and the query text segment to mark the target video segment in the target audio and video data. Therefore, the present application automatically finds the video segment related to the query text in a feature matching manner, and automatically marks the video segment related to the query text, which not only saves manpower for video tagging, but also improves the efficiency of video tagging.

[0079] The video marking method and apparatus, computer equipment, and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the video marking method in the embodiments of the present application is described.

[0080] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0081] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0082] The video marking method provided in the embodiment of the present application relates to the field of video processing technology and is applicable to the fields of financial technology and medical health. The video marking method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the video marking method, etc., but is not limited to the above forms.

[0083] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0084] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0085] Figure 1 is an optional flowchart of the video marking method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S107.

[0086] Step S101, obtaining target video features and target audio features of target audio and video data;

[0087] Step S102, obtaining a query text;

[0088] Step S103, extracting features from the query text to obtain query text features; wherein the query text features are used to represent key content in the query text;

[0089] Step S104, performing feature fusion on the target video feature and the target audio feature to obtain the target audio and video feature;

[0090] Step S105, constructing a feature graph according to the target audio and video features to obtain an audio and video feature graph; wherein the audio and video feature graph includes candidate segment features with a time sequence;

[0091] Step S106, searching for a video segment in the target audio and video data according to the query text features and the candidate segment features to obtain the target video segment;

[0092] Step S107: marking the target video segment in the target audio and video data.

[0093] Steps S101 to S107 shown in the embodiment of the present application realize multimodal feature construction by fusing the target video features and target audio features of the target audio and video data into target audio and video features, and construct an audio and video feature graph according to the time sequence of the target audio and video features, and the audio and video feature graph includes candidate segment features with time sequence. Then, the query text features are extracted to obtain the query text features, and the target video segments related to the query text are extracted from the target audio and video data according to the query text features and the candidate segment features, so as to realize automatic extraction of related video segments, and mark the target video segments in the target audio and video data. Therefore, in the video marking process, the features are automatically extracted, and the search for related video segments is completed by feature comparison, which can not only automatically complete the extraction of video segments, but also accurately find the video segments related to the query text, and the video segment marking operation is automated and accurate, which not only saves manpower for video marking, but also improves the efficiency of video marking.

[0094] See also Figure 2 In some embodiments, step S101 may include but is not limited to steps S201 to S206:

[0095] Step S201, performing frame processing on target audio and video data to obtain target video frame images;

[0096] Step S202, extracting audio data from the target audio and video data to obtain target audio data;

[0097] Step S203, extracting features from the target video frame image to obtain preliminary video features;

[0098] Step S204, performing global context processing on the preliminary video features to obtain target video features;

[0099] Step S205, extracting features from the target audio data to obtain preliminary audio features;

[0100] Step S206: Perform global context processing on the preliminary audio features to obtain target audio features.

[0101] In some embodiments, in step S201 to step S202, the target audio and video data includes target video data and target audio data, and the target audio data is audio data corresponding to the target video data. Specifically, the target video data and the target audio data are obtained by separating the target audio and video data, and the target video data is video data containing multiple target video frame images, and the number of target video frame images can be defined as N. V , and define the target video data as V, and define the target audio data as A. It should be noted that the target audio and video data is processed by framing, that is, the target video data is decomposed into separate preliminary video frame images, and each preliminary video frame image represents an image at a time point. In order to make the target video features more representative of the target video data, it is necessary to first perform a clarity evaluation on the preliminary video frame images to obtain clarity evaluation data, and use the preliminary video frame images whose clarity evaluation data is greater than a preset clarity threshold as the target video frame images to filter out preliminary video frame images with insufficient image clarity. Therefore, feature extraction is performed through the target video frame images to obtain target video features that represent the target video data more accurately.

[0102] It should be noted that the video tagging method can be applied to the fields of financial technology, medical health, etc. For example, if it is applied to the scenario of securities sales analysis in the field of financial technology, by obtaining the securities sales explanation video related to the securities sales data, the target audio data is the sound recording corresponding to the securities sales explanation video. Therefore, by jointly analyzing the securities sales explanation video and the corresponding sound recording, the securities sales situation can be analyzed.

[0103] In step S203 of some embodiments, preliminary video features are obtained by inputting the target video frame image into a feature extraction model for feature extraction. It should be noted that the feature extraction model in this embodiment is a pre-trained VIT model, and other embodiments may also be an RNN model, a CNN model, a Transformer, etc. This embodiment does not impose specific restrictions on the feature extraction model. Specifically, feature extraction is performed on the target video frame image through the pre-trained VIT model to obtain a feature representation of the target video frame image, and then the feature representations of multiple target video frame images are spliced ​​into preliminary video features. It should be noted that the preliminary video features are defined as

[0104] In step S204 of some embodiments, in order to better capture the contextual relationship between video features and construct a target video feature with a contextual relationship, the preliminary video feature needs to be input into a unimodal encoder for global context processing. Specifically, the unimodal encoder includes: a first fully connected layer and a Transformer model, wherein the feature representation and position embedding information of each target video frame image in the preliminary video feature are fused through the fully connected layer to obtain a candidate video feature, and the candidate video feature is contextually associated through the Transformer model to obtain the target video feature. Among them, the Transformer model includes: a multi-head attention layer and a second fully connected layer, wherein the candidate video features are residually connected through the multi-head attention layer, and then the second fully connected layer normalizes the residually connected candidate video features to obtain the target video feature.

[0105] It should be noted that the process of encoding the preliminary video features through a single-mode encoder to obtain the target video features is shown in formula (1):

[0106] X V '=Transformer(X V ) (1)

[0107] In the formula, X V is the preliminary video feature, X V ' is the target video feature.

[0108] In step S205 of some embodiments, the extraction operation of the preliminary audio features is the same as the extraction operation of the preliminary video features, and the preliminary audio features are obtained by inputting the target audio data into the pre-trained VIT model for feature extraction. It should be noted that the extraction of the preliminary audio features can also be completed by other feature extraction models, and this embodiment does not specifically limit the model used for feature extraction.

[0109] In step S206 of some embodiments, the preliminary audio features are subjected to global context processing, which also captures the contextual relationship between the preliminary audio features to construct target audio features with contextual relationships. It should be noted that the global context processing of the preliminary audio features is similar to the global context processing of the preliminary video features, which also fuses the preliminary audio features and the position embedding information through a unimodal encoder, and then performs residual connection and normalization processing to obtain the target audio features. The process of generating the target audio features will not be described in detail here.

[0110] It should be noted that the process of obtaining the target audio features by performing global context processing on the preliminary audio features through the unimodal encoder is shown in formula (2):

[0111] X A ′=Transformer(X A ) (2)

[0112] In the formula, X A is the preliminary audio feature, and X A ′ is the target audio feature.

[0113] In steps S201 to S206 shown in the present embodiment, the target audio and video data is first split into target audio data and target video data, and the target video data is framed into target video frame images. The target video features are obtained by extracting preliminary video features from the target video frame images, and then the preliminary video features are subjected to contextual global processing. Then, preliminary audio features are extracted from the target audio data, and then the preliminary audio features are subjected to contextual global processing to obtain the target audio features. Therefore, by extracting the target video features and target audio features with contextual relationships, it is more accurate to search for video clips based on the target video features and target audio features later.

[0114] In step S102 of some embodiments, the query text is used to query the text of the relevant video clip in the target audio and video data. If applied to the scenario of securities sales analysis in the financial field, the query text can be text related to securities sales, and the query text can be an explanation text analyzing the current securities sales situation, or a data text related to securities sales.

[0115] In step S103 of some embodiments, the query text is feature extracted by using the pre-trained BERT model to obtain query text features. Specifically, the query text is first segmented to obtain query phrases, and then each query phrase is encoded by the BERT model to obtain phrase feature representation, and then multiple phrase feature representations are concatenated into query text features, and the query text features are defined as X Q ∈R dIt should be noted that the query text features can represent the key content in the query text, and after the BERT model outputs each phrase feature representation, it also contextualizes multiple phrase feature representations through a unimodal encoder and then concatenates them to obtain the query text features.

[0116] In step S104 of some embodiments, the target video features and the target audio features are feature fused by a cross-modal encoder to generate audio-visual multimodal target audio and video features. It should be noted that the target audio and video features help to understand the relationship between multimodal data, and the cross-modal encoder uses a bottleneck Transformer model, so the feature fusion process is divided into two stages: feature compression and expansion.

[0117] See also Figure 3 In some embodiments, step S104 may include but is not limited to steps S301 to S303:

[0118] Step S301, performing attention interaction on target video features and target audio features according to preset bottleneck features to obtain attention interaction features;

[0119] Step S302, fusion of target video features and target audio features according to the attention interaction features to obtain preliminary audio and video features;

[0120] Step S303, performing average pooling processing on the preliminary audio and video features to obtain target audio and video features.

[0121] In step S301 of some embodiments, the bottleneck feature is also called bottleneck tokens. In the bottleneck Transformer model, each token is independent and equal, so setting the bottleneck tokens to participate in the feature processing of the bottleneck Transformer model can affect the feature processing effect of the bottleneck Transformer model. The bottleneck feature is defined as bottleneck tokensY, and bottleneck tokensY∈R d′ The feature dimension of the bottleneck tokensY is d′, which is much smaller than the feature dimension d of the target video features and the target audio features. Therefore, the bottleneck Transformer model and the bottleneck tokensY can interact with the target video features and the target audio features to achieve the purpose of feature compression.

[0122] In step S302 of some embodiments, as disclosed above, after the attention interaction between the target video feature and the target audio feature is completed, the attention interaction feature is obtained, and the attention interaction feature can not only characterize the relationship between the features, but also compress the target video feature and the target audio feature. Therefore, the target video feature and the target audio feature are feature fused according to the attention interaction feature to obtain a preliminary audio and video feature with a closer feature relationship and compression.

[0123] In step S303 of some embodiments, in order to achieve feature expansion of the preliminary audio and video features, the preliminary audio and video features are subjected to average pooling processing through an average pooling layer to obtain target audio and video features. It should be noted that average pooling is to add each feature in the preliminary audio and video features to the adjacent features, and divide the addition result by the number of features to obtain a new audio and video feature as the target audio and video feature. Specifically, the fused preliminary audio and video features are defined as X C =X A ′+X V ′, the final target audio and video features are defined as

[0124] In steps S301 to S303 shown in this embodiment, the attention interaction feature is obtained by first performing attention interaction on the target video feature and the target audio feature, and the attention interaction feature can reflect the relationship between the target video feature and the target audio feature, and can also achieve feature compression. Therefore, the target video feature and the target audio feature are fused into a preliminary audio and video feature according to the attention interaction feature, and then the preliminary audio and video feature is averaged and pooled to obtain the target audio and video feature, so as to obtain the target audio and video feature that represents a closer relationship between the video and the audio, so that the video clip search using the target audio and video feature is more accurate.

[0125] See also Figure 4 In some embodiments, step S301 may include but is not limited to steps S401 to S405:

[0126] Step S401, converting bottleneck features into query features;

[0127] Step S402, converting the target video features into key features;

[0128] Step S403, converting the target audio feature into a value feature;

[0129] Step S404, performing attention interaction according to the query feature, the key feature and the value feature to obtain attention interaction data;

[0130] Step S405, performing feature aggregation processing on the target video features and the target audio features according to the attention interaction data to obtain attention interaction features.

[0131] In some embodiments, in step S401 to step S403, during attention interaction, key-value features and query features need to be set. Therefore, the bottleneck feature is first used as the query feature, the target video feature is used as the key feature, and the target audio feature is used as the value feature.

[0132] In step S404 of some embodiments, attention interaction is performed between query features, key features, and value features, that is, the bottleneck feature is used as the content to be searched, the target video feature is used as the information in the dictionary, and the target audio feature is also used as the corresponding information in the dictionary, and the features that meet the relationship between the target video feature and the target audio feature are continuously searched as the updated bottleneck feature. It should be noted that the attention interaction data is the updated bottleneck feature, and represents the feature correlation between the target video feature and the target audio feature.

[0133] In step S405 of some embodiments, as disclosed above, the attention interaction data is an updated bottleneck feature, so the process of aggregating the target video feature and the target audio feature according to the attention interaction data is to use the updated bottleneck feature as the key feature, and the target video feature and the target audio feature as the query feature, respectively, and then obtain the aggregated attention interaction feature.

[0134] In steps S401 to S405 shown in this embodiment, attention interaction is performed by converting the target video feature into a key feature, converting the target audio feature into a value feature, and converting the bottleneck feature into a query feature to obtain attention interaction data. The attention interaction data can characterize the relationship between the target video feature and the target audio feature. Then, based on the attention interaction data, the target video feature and the target audio feature are aggregated into an attention interaction feature, so that the target video feature and the target audio feature can be better fused according to the attention interaction feature, and the target audio and video feature that can accurately characterize the key content in the target audio and video data is obtained.

[0135] As disclosed above, Figure 5As shown, the preliminary video features are input into the unimodal encoder to obtain the target video features, and the preliminary audio features are also input into the unimodal encoder to obtain the target audio features, and the target video features and the target audio features are input into the cross-modal encoder. The target audio and video features output by the cross-modal encoder are input into the segment generator. It should be noted that the segment generator refers to the two-dimensional time-domain neighboring network 2D-TAN model to locate the target video segment in the two-dimensional time graph. Traditional video segment positioning mainly presets some candidate video segments, and then predicts the front and back positions of each candidate video segment to achieve the sorting of the candidate video segments, but the position prediction of each candidate video segment is performed independently, without considering the dependency between the candidate video segments, resulting in inaccurate generated audio and video feature graphs. The segment generator of this embodiment refers to the two-dimensional time-domain neighboring network 2D-TAN model, and the two-dimensional time-domain neighboring network 2D-TAN model will consider the dependency between video judgments, and can construct an audio and video feature graph that sorts the video segments more accurately.

[0136] See also Figure 6 In some embodiments, step S105 may include but is not limited to steps S601 to S604:

[0137] Step S601, performing pooling processing on the target audio and video features to obtain updated audio and video features;

[0138] Step S602, splitting the updated audio and video features to obtain candidate segment features;

[0139] Step S603, obtaining the start and end time information of each candidate segment feature;

[0140] Step S604, arranging the candidate segment features according to the start and end time information to obtain the audio and video feature graph.

[0141] In step S601 of some embodiments, the target audio and video features are first split into target audio and video sub-features corresponding to each target video frame image, and each target audio and video sub-feature is pooled within the time span, and this embodiment adopts the maximum pooling processing method to select the target audio and video feature with the largest feature dimension as the updated audio and video feature. Among them, the operation of performing the maximum pooling operation on the target audio and video feature to obtain the updated audio and video feature is shown in formula (3):

[0142]

[0143] Among them, a and b represent the index of the start and end of the video respectively, and X a,b To update audio and video features, X R is the target audio and video feature.

[0144] In steps S602 to S603 of some embodiments, the updated audio and video features are split and processed to obtain candidate segment features corresponding to each candidate video segment, and the start and end time information corresponding to the candidate segment features is obtained, that is, the start and end time information of each candidate video segment is obtained, and the candidate video segments are combined to form target audio and video data.

[0145] In step S604 of some embodiments, according to the start and end time information of each candidate segment feature, that is, according to the start and end time points of each candidate video segment, the candidate segment features corresponding to all candidate video segments are arranged into a two-dimensional feature map to obtain an audio and video feature map. V ×N V Therefore, the candidate segment features arranged in time order can be determined through the audio and video feature graph, and each candidate segment feature includes the audio feature and video feature of each candidate video segment.

[0146] In steps S601 to S604 shown in this embodiment, the target audio and video features are first pooled to obtain updated audio and video features, and the updated audio and video features are split into candidate segment features corresponding to each candidate video segment, and the start and end time information of the candidate segment features is obtained, so that the candidate segment features are arranged into an audio and video feature graph according to the start and end time information. Therefore, by displaying the candidate segment features sorted in chronological order through the audio and video feature graph, subsequent video segment searches can be made more convenient and accurate.

[0147] It should be noted that if Figure 5 As shown, the segment generator is connected to the contrastive learning module, and the contrastive learning module is also connected to the unimodal encoder that outputs the query text features. Among them, the contrastive learning operation of the contrastive learning module includes segment-level contrastive learning operation and frame-level contrastive learning operation, and the segment-level contrastive learning operation is to determine the candidate video segment that is highly relevant to the query text as the target video segment, and the frame-level contrastive learning operation is to determine the candidate video frame image that is highly relevant to the query text as the target video frame image. Specifically, after finding the target video segment, the target video frame image located in the target video segment is used as a positive sample, and the target video frame image not located in the target video segment is used as a negative sample. The contrastive learning module is trained by calculating the contrast loss between the positive sample and the negative sample, so that the contrastive learning module can identify the target video segment that is more accurate to the query text.

[0148] See also Figure 7 In some embodiments, the target audio and video data includes: candidate video segments and candidate video frame images; step S106 includes but is not limited to steps S701 to S705:

[0149] Step S701, performing frame processing on the candidate segment features to obtain candidate frame features;

[0150] Step S702, performing relevance evaluation on the candidate video segments according to the query text features and the candidate segment features to obtain segment relevance evaluation data;

[0151] Step S703, performing correlation evaluation on the candidate video frame images according to the query text features and the candidate frame features to obtain frame correlation evaluation data;

[0152] Step S704, concatenating the segment correlation evaluation data and the frame correlation evaluation data to obtain target correlation evaluation data;

[0153] Step S705 , searching for a video segment of the target audio and video data according to the target relevance evaluation data to obtain a target video segment.

[0154] In step S701 of some embodiments, in order to achieve more precise video segment search, the search will be performed at the video frame level, so the candidate segment features need to be split into candidate frame features at the video frame level, and each candidate frame feature represents the audio and video features of each target video frame image.

[0155] In step S702 of some embodiments, the candidate video segments are evaluated for relevance based on the query text features and the candidate segment features. Specifically, the query text features and the candidate segment features are compared to obtain first feature comparison information, and segment relevance evaluation data of each candidate video segment is determined based on the first feature comparison information. It should be noted that the segment relevance evaluation data represents the degree of semantic association between each candidate video segment and the query text.

[0156] In step S703 of some embodiments, in addition to the segment-level relevance evaluation, the query text features and the candidate frame features are feature-compared to obtain second feature comparison information, and frame relevance evaluation data of each candidate video frame image is determined based on the second feature comparison information. It should be noted that the frame relevance evaluation data represents the semantic association process between the candidate video frame image and the query text.

[0157] Specifically, the first feature comparison information represents the feature matching between the query text feature and the candidate segment feature, and the higher the matching degree represented by the first feature comparison information, the higher the segment relevance evaluation data of the candidate video segment. Similarly, if the second feature comparison information represents the higher feature matching degree between the query text feature and the candidate frame feature, the higher the frame relevance evaluation data of the candidate video frame image.

[0158] In step S704 of some embodiments, the segment correlation evaluation data and the frame correlation evaluation data may be spliced ​​in a weighted summation, summation, and averaging manner, and this embodiment does not impose any specific restrictions on the splicing manner. For example, if this embodiment adopts a weighted summation manner, the weights corresponding to the segment correlation evaluation data and the frame correlation evaluation data are first determined, and then the segment correlation evaluation data and the frame correlation evaluation data are spliced ​​into target correlation evaluation data according to the weights. Specifically, there are multiple candidate video frame images in each candidate video segment, and the candidate video frame images in each candidate video segment are first extracted as selected video frame images, and then the segment correlation evaluation data corresponding to the candidate video segment and the frame correlation evaluation data corresponding to the multiple selected video frame images are weighted and spliced ​​into target correlation evaluation data. Therefore, the target correlation evaluation data characterizes the degree of semantic association between each candidate video segment and the query text.

[0159] In step S705 of some embodiments, the target relevance evaluation data are sorted from large to small, and a preset number of target relevance evaluation data before sorting are used as selected relevance evaluation data, and then the candidate video segments corresponding to the multiple selected relevance evaluation data are used as target video segments to find out multiple video segments that are highly relevant to the query text. It should be noted that the maximum relevance evaluation data can also be extracted from the target relevance evaluation data, and the candidate video segment corresponding to the maximum relevance evaluation data can be used as the target video segment. Therefore, a video segment that is highly relevant to the query text in the target audio and video data can be found, and multiple highly relevant video segments can also be found, so as to achieve accurate search of video segments.

[0160] For example, if it is applied to the scenario of securities sales analysis in the financial field, the query text analyzes the explanation text of the current securities sales situation, and the target audio and video data is the securities sales situation explanation video, which will be divided into the opening segment, securities introduction segment, current sales situation explanation segment, future sales trend speculation segment and closing segment. Through the feature extraction of the query text, the target audio and video features of the target audio and video data are constructed, and then the current sales situation explanation segment can be extracted from the securities sales situation explanation video by combining the query text features and the target audio and video features, and the current sales situation explanation segment can be marked as the key segment of the video, so that securities analysts can quickly locate the current sales situation explanation segment and conduct target securities sales analysis.

[0161] In steps S701 to S705 shown in this embodiment, relevance evaluation is performed on each candidate video segment at the video segment level and the video frame level to find a target video segment that is highly relevant to the query text, making the search for the target video segment more accurate.

[0162] See also Figure 8 In some embodiments, step S107 may include but is not limited to steps S801 to S802:

[0163] Step S801, obtaining time information of a target video clip to obtain target time information;

[0164] Step S802: Mark the target audio and video data according to the target time information.

[0165] It should be noted that if Figure 5 As shown, the contrastive learning module inputs the target video segment into a multi-detection head, and the multi-detection head includes at least two detection heads, so as to complete the labeling of the target audio and video segment through the at least two detection heads.

[0166] In step S801 to step S802 of some embodiments, the target time information is the time point of the target video segment in the target audio and video data, so the target audio and video data is marked according to the target time information, that is, the time point where the target video segment in the target audio and video data is located is marked. It should be noted that the target audio and video data can be marked by highlighting or labeling. This embodiment does not specifically limit the marking operation of the target audio and video data, and the marking method can be set according to user needs to facilitate users to quickly find target video segments related to the query text.

[0167] For example, if the target video clip is a clip explaining the current sales situation, the time point of the clip explaining the current sales situation is marked in the video explaining the securities sales situation, and a corresponding label is set at the time point. It should be noted that the label can be "Securities Sales Situation Description", so that securities personnel can quickly slide to the current sales situation explanation clip when viewing the securities sales situation explanation video, reducing the search for other irrelevant content, making the analysis of securities sales more efficient.

[0168] In steps S801 to S802 shown in this embodiment, the target video segment in the target audio and video data is marked according to the target time information of the target video segment, so as to quickly locate the target video segment and facilitate viewing of the target video segment.

[0169] Please refer to Figure 5 and Fig. 9, the embodiment of the present application obtains target audio and video data, splits the target audio and video data into target video data and target audio data, and obtains query text at the same time, and the query text is used to query the text of relevant content in the target audio and video data. The target video frame image is obtained by frame processing in the target video data, and the target video frame image is input into the pre-trained VIT model for feature extraction to obtain preliminary video features, and then the preliminary video features are input into the unimodal encoder for position embedding information fusion, residual connection and normalization processing to obtain the target video features. At the same time, the target audio data is input into the pre-trained VIT model for feature extraction to obtain preliminary audio features, and then the preliminary audio features are input into another unimodal encoder for position embedding information fusion, residual connection and normalization processing to obtain the target audio features. The query text is segmented to obtain a query phrase, and the query phrase is input into the pre-trained BERT model for feature extraction to obtain the query phrase features, and then the query phrase features are input into another unimodal encoder for context global processing to obtain the query text features. The target video features and the target audio features are input into the cross-modal encoder, and the target video features and the target audio features are fused into the target audio and video features through the cross-modal encoder. Specifically, the cross-modal encoder uses the target video features and the target audio features as key values, and the bottleneck features as queries, and performs attention interaction to obtain updated bottleneck features. Then, according to the updated bottleneck features, the target video features and the target audio features are fused into attention interaction features to achieve multimodal feature compression. Then, the attention interaction features are used as queries, and the updated bottleneck features are used as key values, and attention processing is performed again to obtain preliminary audio and video features after feature fusion, and then the preliminary audio and video features are averaged and pooled to obtain the target audio and video features.

[0170] The cross-modal encoder outputs the target audio and video features to the segment generator, so that the segment generator performs maximum pooling on the target audio and video features to obtain updated audio and video features, and then splits the updated audio and video features into candidate segment features, and arranges the candidate segment features into an audio and video feature graph according to the start and end time information of the candidate segment features. Therefore, the audio and video feature graph includes candidate segment features with time sequence, and the candidate segment features are features obtained by fusion of the target video features and the target audio features.

[0171] In order to find the target video segment related to the query text, the candidate segment features are split into candidate frame features, and the feature matching degree between the query text features and the candidate segment features is calculated to determine the segment relevance evaluation data of each candidate video segment. Then, the feature matching degree between the query text features and the candidate frame features is calculated to determine the frame relevance evaluation data of each candidate video frame image, and the segment relevance evaluation data and the frame relevance evaluation data are spliced ​​into the target relevance evaluation data of each candidate video segment. Each candidate video segment is sorted in descending order according to the target relevance evaluation data, and a preset number of candidate video segments before sorting are used as target video segments to complete the automatic search of the target video segment.

[0172] Finally, the target time information of the target video segment is obtained, and the target video segment in the target audio and video data is highlighted according to the target time information, so that the user can quickly identify the video segment related to the query text, which is conducive to video content analysis.

[0173] In summary, by integrating visual, audio and text features into an audio-visual multimodal feature space and using the multimodal features to complete the search and labeling of video clips, the video labeling operation is accurate and automated, which saves manpower and improves the efficiency of video labeling.

[0174] See also Fig.10 The embodiment of the present application further provides a video marking device, which can implement the above-mentioned video marking method, and the device includes:

[0175] A feature acquisition module 1001 is used to acquire target video features and target audio features of target audio and video data;

[0176] A text acquisition module 1002 is used to acquire a query text;

[0177] The feature extraction module 1003 is used to extract features from the query text to obtain query text features; wherein the query text features are used to represent key content in the query text;

[0178] The feature fusion module 1004 is used to fuse the target video feature and the target audio feature to obtain the target audio and video feature;

[0179] The feature graph construction module 1005 is used to construct a feature graph according to the target audio and video features to obtain an audio and video feature graph; wherein the audio and video feature graph includes candidate segment features with a time sequence;

[0180] The video segment search module 1006 is used to search for video segments in the target audio and video data according to the query text features and the candidate segment features to obtain the target video segment;

[0181] The marking module 1007 is used to mark the target video segment in the target audio and video data.

[0182] The specific implementation of the video marking device is substantially the same as the specific implementation of the above-mentioned video marking method, and will not be described in detail here.

[0183] The embodiment of the present application also provides a computer device, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the above video marking method when executing the computer program. The computer device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0184] See also Fig.11 , Fig.11 The hardware structure of a computer device according to another embodiment is shown, and the computer device includes:

[0185] The processor 1101 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0186] The memory 1102 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1102 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1102, and the processor 1101 calls and executes the video marking method of the embodiment of this application;

[0187] Input / output interface 1103, used to implement information input and output;

[0188] The communication interface 1104 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0189] A bus 1105 that transmits information between various components of the device (e.g., the processor 1101, the memory 1102, the input / output interface 1103, and the communication interface 1104);

[0190] The processor 1101 , the memory 1102 , the input / output interface 1103 and the communication interface 1104 are connected to each other in communication within the device via the bus 1105 .

[0191] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned video marking method is implemented.

[0192] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0193] The video tagging method and apparatus, computer equipment and storage medium provided in the embodiments of the present application first fuse the target video features and target audio features of the target audio and video data into multimodal target audio and video features, and construct the multimodal target audio and video features into an audio and video feature graph including candidate segment features with a time sequence. Feature extraction of the query text is performed to obtain query text features, and the target video segments are searched out in the target audio and video data in combination with the query text features and the candidate segment features, so as to accurately find the target video segments related to the query text content, and finally mark the target video segments in the target audio and video data. Therefore, by automatically constructing multimodal candidate segment features with a time sequence, the target video segment tagging of the target audio and video data is automatically completed with the candidate segment features and the query text features, which can not only accurately mark the video segments related to the query text in the target audio and video data, but also make the marking operation of the video segments automatic, save manpower, and improve the efficiency of video tagging.

[0194] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0195] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0196] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0197] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0198] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0199] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0200] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0201] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0202] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0203] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0204] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A video marking method, characterized in that: The method comprises: Obtain target video features and target audio features of target audio and video data; Get the query text; Extracting features from the query text to obtain query text features; wherein the query text features are used to characterize key content in the query text; Performing feature fusion on the target video feature and the target audio feature to obtain the target audio and video feature; Constructing a feature graph according to the target audio and video features to obtain an audio and video feature graph; wherein the audio and video feature graph includes candidate segment features with a time sequence; Searching for a video segment of the target audio and video data according to the query text feature and the candidate segment feature to obtain a target video segment; The target video segment in the target audio and video data is marked.

2. The method according to claim 1, characterized in that The step of fusing the target video feature and the target audio feature to obtain the target audio and video feature includes: Performing attention interaction on the target video feature and the target audio feature according to a preset bottleneck feature to obtain an attention interaction feature; According to the attention interaction feature, the target video feature and the target audio feature are subjected to feature fusion to obtain a preliminary audio and video feature; The preliminary audio and video features are averaged and pooled to obtain the target audio and video features.

3. The method according to claim 2, characterized in that The performing attention interaction on the target video feature and the target audio feature according to the preset bottleneck feature to obtain the attention interaction feature includes: converting the bottleneck feature into a query feature; Converting the target video features into key features; Converting the target audio feature into a value feature; Perform attention interaction according to the query feature, the key feature, and the value feature to obtain attention interaction data; Feature aggregation processing is performed on the target video features and the target audio features according to the attention interaction data to obtain the attention interaction features.

4. The method according to claim 1, characterized in that: The step of constructing a feature graph according to the target audio and video features to obtain an audio and video feature graph includes: Performing pooling processing on the target audio and video features to obtain updated audio and video features; Splitting the updated audio and video features to obtain candidate segment features; Obtaining the start and end time information of each candidate segment feature; The candidate segment features are arranged according to the start and end time information to obtain the audio and video feature graph.

5. The method according to any one of claims 1 to 4, characterized in that: The target audio and video data includes: candidate video segments and candidate video frame images; the step of searching the target audio and video data for video segments according to the query text features and the candidate segment features to obtain the target video segment includes: Performing frame processing on the candidate segment features to obtain candidate frame features; Performing relevance evaluation on the candidate video segments according to the query text features and the candidate segment features to obtain segment relevance evaluation data; Performing relevance evaluation on the candidate video frame images according to the query text features and the candidate frame features to obtain frame relevance evaluation data; splicing the segment correlation evaluation data and the frame correlation evaluation data to obtain target correlation evaluation data; The target audio and video data is searched for a video segment according to the target relevance evaluation data to obtain the target video segment.

6. The method according to any one of claims 1 to 4, characterized in that: The marking of the target video segment in the target audio and video data includes: Acquire the time information of the target video clip to obtain target time information; The target audio and video data is marked according to the target time information.

7. The method according to any one of claims 1 to 4, characterized in that: The step of obtaining target video features and target audio features of target audio and video data includes: Performing frame processing on the target audio and video data to obtain a target video frame image; Extracting audio data from the target audio and video data to obtain target audio data; Extracting features from the target video frame image to obtain preliminary video features; Performing global context processing on the preliminary video features to obtain the target video features; Extracting features from the target audio data to obtain preliminary audio features; The preliminary audio features are subjected to contextual global processing to obtain the target audio features.

8. A video marking device, characterized in that: The device comprises: A feature acquisition module, used to acquire target video features and target audio features of target audio and video data; A text acquisition module, used to acquire query text; A feature extraction module is used to extract features from the query text to obtain query text features; wherein the query text features are used to characterize key content in the query text; A feature fusion module, used to fuse the target video feature and the target audio feature to obtain the target audio and video feature; A feature map construction module, used to construct a feature map according to the target audio and video features to obtain an audio and video feature map; wherein the audio and video feature map includes candidate segment features with time sequence; A video segment search module, used to search for video segments of the target audio and video data according to the query text features and the candidate segment features, to obtain a target video segment; The marking module is used to mark the target video segment in the target audio and video data.

9. A computer device, characterized in that: The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the video marking method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the video marking method according to any one of claims 1 to 7 is implemented.

Images