Video processing method, video detection model training method, device and medium

Abstract

The application discloses a video processing method, a video detection model training method, a device and a medium, obtains initial query vector information, a plurality of initial query point information and video feature information, calculates video frame feature information obtained according to the initial query point information to obtain corresponding query point feature information, updates each initial query point information according to each query point feature information to obtain a plurality of target query point information, obtains target query vector information according to the initial query vector information and all query point feature information, identifies a target behavior category of a time sequence behavior according to the target query vector information, and determines a start time and an end time corresponding to the target behavior category according to a video frame time sequence of all target query point information. The embodiment of the application can improve the detection accuracy of the time sequence behavior in the video. The embodiment of the application can be applied to video processing technologies in various scenes such as video review, video editing, video material extraction, video labeling and the like.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a video processing method, a video detection model training method, an apparatus, and a medium. Background Technology

[0002] In video understanding tasks, it is often necessary to locate and identify temporal behaviors in videos, obtaining the start time, end time, and behavior category of each behavior. To achieve this, various implementation schemes have been proposed, such as temporal behavior detection based on video segment features and temporal behavior detection based on video point features. However, temporal behavior detection based on video segment features requires dense segmentation of the video, which significantly increases the complexity of the video detection model, and its accuracy in detecting the boundary information of temporal behaviors is relatively low. Temporal behavior detection based on video point features requires dense sampling of the video, which reduces the detection efficiency of temporal behaviors, and its accuracy in detecting the boundary information of temporal behaviors is also not high. Summary of the Invention

[0003] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0004] This invention provides a video processing method, a video detection model training method, an apparatus, and a medium that can improve the accuracy of detecting temporal behaviors in videos.

[0005] On one hand, embodiments of the present invention provide a video processing method, including the following steps:

[0006] Get the video to be processed;

[0007] Acquire initial query vector information, multiple initial query point information, and video feature information of the video to be processed, wherein the initial query vector information is used to identify the temporal behavior of the video to be processed, and the initial query point information corresponds to the temporal sequence of the video frames of the video to be processed;

[0008] Multiple video frame feature information is extracted from the video feature information based on multiple initial query point information. Query point feature information of each initial query point information is calculated based on each video frame feature information. The initial query point information is then updated based on the query point feature information to obtain multiple target query point information. The target query point information corresponds to the temporal sequence of the video frames in the video to be processed.

[0009] The initial query vector information is mixed with the feature information of all the query points to obtain the target query vector information;

[0010] Based on the target query vector information, the temporal behavior is classified to obtain the target behavior category;

[0011] The start and end times of the target behavior category in the video to be processed are determined based on the video frame timing corresponding to all the target query point information.

[0012] On the other hand, embodiments of the present invention also provide a video detection model training method, including the following steps:

[0013] Obtain video samples and tag information;

[0014] Obtain query vector samples, multiple query point samples, and video feature information of the video sample, wherein the query vector samples are used to identify the temporal behavior of the video sample, and the query point samples correspond to the video frame temporal sequence of the video sample;

[0015] The query vector sample, multiple query point samples, and video feature information are input into the video detection model to perform behavior prediction, and the prediction start time, prediction end time, and prediction behavior category are obtained.

[0016] The parameters of the video detection model are corrected based on the prediction start time, the prediction end time, the prediction behavior category, and the label information.

[0017] The process of behavior prediction performed by the video detection model includes:

[0018] Multiple video frame feature information is extracted from the video feature information based on multiple query point samples. Query point feature information of each query point sample is calculated based on each video frame feature information. Each query point sample is then updated based on the query point feature information to obtain multiple target query point information. The target query point information corresponds to the temporal sequence of the video frames in the video sample.

[0019] The query vector sample is combined with the feature information of all the query points to obtain the target query vector information;

[0020] Based on the target query vector information, the temporal behavior is classified to obtain the predicted behavior category;

[0021] The prediction start time and prediction end time of the prediction behavior category in the video sample are determined based on the video frame time sequence corresponding to all the target query point information.

[0022] On the other hand, embodiments of the present invention also provide a video processing apparatus, including:

[0023] The first acquisition unit is used to acquire the video to be processed;

[0024] The second acquisition unit is used to acquire initial query vector information, multiple initial query point information and video feature information of the video to be processed, wherein the initial query vector information is used to identify the temporal behavior of the video to be processed, and the initial query point information corresponds to the temporal sequence of the video frames of the video to be processed.

[0025] The first processing unit is configured to extract multiple video frame feature information from the video feature information based on multiple initial query point information, calculate query point feature information of each initial query point information based on each video frame feature information, and update each initial query point information based on each query point feature information to obtain multiple target query point information, wherein the target query point information corresponds to the temporal sequence of the video frames in the video to be processed.

[0026] The second processing unit is used to perform feature mixing between the initial query vector information and all the query point feature information to obtain the target query vector information;

[0027] A behavior recognition unit is used to perform category recognition on the temporal behavior based on the target query vector information to obtain the target behavior category;

[0028] The time determination unit is used to determine the start and end times of the target behavior category in the video to be processed based on the video frame timing corresponding to all the target query point information.

[0029] Optionally, the first processing unit is further configured to:

[0030] Based on the video frame timing corresponding to each of the initial query point information, obtain multiple timing offset values ​​corresponding to each of the initial query point information;

[0031] Based on the multiple temporal offset values ​​corresponding to each of the initial query point information, multiple video frame feature information corresponding to each of the initial query point information is extracted from the video feature information;

[0032] The query point feature information of each initial query point is obtained by weighting and calculating the feature information of multiple video frames corresponding to each initial query point.

[0033] Optionally, the first processing unit is further configured to:

[0034] Offset prediction is performed on the feature information of each query point to obtain the offset information of each initial query point;

[0035] The initial query point information is updated based on the offset information of each initial query point information to obtain multiple target query point information.

[0036] Optionally, the second processing unit is further configured to:

[0037] The initial query vector information and the feature information of each query point are subjected to variable convolution processing to obtain multiple intermediate feature information;

[0038] All the intermediate feature information is stacked to obtain the query point stacked feature information;

[0039] The stacked feature information of the query points is mixed with the initial query vector information to obtain cascaded feature information;

[0040] The concatenated feature information and the initial query vector information are residually joined to obtain the target query vector information.

[0041] Optionally, the second processing unit is further configured to:

[0042] The query point stacking feature information and the initial query vector information are mixed based on the first dimension to obtain the first mixed feature information;

[0043] The query point stacking feature information and the initial query vector information are mixed based on the second dimension to obtain the second mixed feature information;

[0044] The first mixed feature information and the second mixed feature information are concatenated and mapped to obtain concatenated feature information.

[0045] Optionally, the first processing unit is further configured to:

[0046] The initial query point information is updated based on the feature information of each query point to obtain multiple new initial query point information.

[0047] Based on the multiple new initial query point information, multiple new video frame feature information is extracted from the video feature information;

[0048] Multiple new query point feature information are calculated based on the new video frame feature information;

[0049] The information of each new initial query point is updated according to the feature information of each new query point until the number of updates reaches a preset value, thereby obtaining multiple target query point information.

[0050] Optionally, the second processing unit is further configured to:

[0051] The initial query vector information is mixed with the feature information of all the query points to obtain a new initial query vector information;

[0052] The new initial query vector information is mixed with all the new query point feature information until the number of feature mixing reaches the preset number value to obtain the target query vector information.

[0053] On the other hand, embodiments of the present invention also provide a video detection model training apparatus, comprising:

[0054] The third acquisition unit is used to acquire video samples and label information;

[0055] The fourth acquisition unit is used to acquire query vector samples, multiple query point samples, and video feature information of the video sample, wherein the query vector samples are used to identify the temporal behavior of the video sample, and the query point samples correspond to the video frame temporal sequence of the video sample.

[0056] The model prediction unit is used to input the query vector sample, multiple query point samples and the video feature information into the video detection model to perform behavior prediction and obtain the prediction start time, prediction end time and prediction behavior category.

[0057] The parameter correction unit is used to correct the parameters of the video detection model based on the prediction start time, the prediction end time, the prediction behavior category, and the label information.

[0058] The model prediction unit is further used for:

[0059] Multiple video frame feature information is extracted from the video feature information based on multiple query point samples. Query point feature information of each query point sample is calculated based on each video frame feature information. Each query point sample is then updated based on the query point feature information to obtain multiple target query point information. The target query point information corresponds to the temporal sequence of the video frames in the video sample.

[0060] The query vector sample is combined with the feature information of all the query points to obtain the target query vector information;

[0061] Based on the target query vector information, the temporal behavior is classified to obtain the predicted behavior category;

[0062] The prediction start time and prediction end time of the prediction behavior category in the video sample are determined based on the video frame time sequence corresponding to all the target query point information.

[0063] Optionally, the model prediction unit is further configured to:

[0064] Based on the video frame timing corresponding to each query point sample, obtain multiple timing offset values ​​corresponding to each query point sample;

[0065] Based on the multiple temporal offset values ​​corresponding to each query point sample, extract multiple video frame feature information corresponding to each query point sample from the video feature information;

[0066] The query point feature information of each query point sample is obtained by weighting and calculating the feature information of the multiple video frames corresponding to each query point sample.

[0067] Optionally, the model prediction unit is further configured to:

[0068] Offset prediction is performed on the feature information of each query point to obtain the offset information of each query point sample;

[0069] The offset information of each query point sample is used to update each query point sample to obtain multiple target query point information.

[0070] Optionally, the model prediction unit is further configured to:

[0071] The query vector sample is subjected to variable convolution with the feature information of each query point to obtain multiple intermediate feature information;

[0072] All the intermediate feature information is stacked to obtain the query point stacked feature information;

[0073] The stacked feature information of the query points is mixed with the query vector sample to obtain cascaded feature information;

[0074] The concatenated feature information is residually joined with the query vector sample to obtain the target query vector information.

[0075] Optionally, the model prediction unit is further configured to:

[0076] The query point stacked feature information and the query vector sample are mixed based on the first dimension to obtain the first mixed feature information;

[0077] The query point stacked feature information is mixed with the query vector sample based on the second dimension to obtain the second mixed feature information;

[0078] The first mixed feature information and the second mixed feature information are concatenated and mapped to obtain concatenated feature information.

[0079] Optionally, the model prediction unit is further configured to:

[0080] The query point samples are updated based on the feature information of each query point to obtain multiple new query point samples;

[0081] Based on the multiple new query point samples, extract multiple new video frame feature information from the video feature information;

[0082] Multiple new query point feature information are calculated based on the new video frame feature information;

[0083] The new query point samples are updated based on the feature information of each new query point until the number of updates reaches a preset value, thereby obtaining multiple target query point information.

[0084] Optionally, the model prediction unit is further configured to:

[0085] The query vector sample is feature-blended with all the query point feature information to obtain a new query vector sample;

[0086] The new query vector sample is mixed with all the new query point feature information until the number of feature mixing reaches the preset number value to obtain the target query vector information.

[0087] Optionally, the tag information includes a start time tag, an end time tag, and a behavior category tag; the parameter correction unit is further used for:

[0088] The positioning loss value is calculated based on the predicted start time, the predicted end time, the start time label, and the end time label;

[0089] The classification loss value is calculated based on the predicted behavior category and the behavior category label;

[0090] The parameters of the video detection model are adjusted based on the localization loss value and the classification loss value.

[0091] On the other hand, embodiments of the present invention also provide an electronic device, including:

[0092] At least one processor;

[0093] At least one memory for storing at least one program;

[0094] When at least one program is executed by at least one processor, it implements the video processing method described above, or the video detection model training method described above.

[0095] On the other hand, embodiments of the present invention also provide a computer-readable storage medium storing a processor-executable computer program, which, when executed by a processor, is used to implement the video processing method described above, or to implement the video detection model training method described above.

[0096] On the other hand, embodiments of the present invention also provide a computer program product, including a computer program or computer instructions, the computer program or computer instructions being stored in a computer-readable storage medium, the processor of an electronic device reading the computer program or computer instructions from the computer-readable storage medium, the processor executing the computer program or computer instructions, causing the electronic device to perform the video processing method as described above, or to perform the video detection model training method as described above.

[0097] The embodiments of the present invention include at least the following beneficial effects: acquiring a video to be processed, initial query vector information, and multiple initial query vector information, wherein each initial query point information corresponds to the video frame temporal sequence of the video to be processed; extracting multiple video frame feature information from the video feature information using the multiple initial query point information; calculating query point feature information for each initial query point information based on the video frame feature information; updating each initial query point information based on the query point feature information to adjust the video frame temporal sequence of each initial query point information; obtaining multiple target query point information; ensuring that the video frame temporal sequence corresponding to each target query point information corresponds to the temporal behavior in the video to be processed; thereby determining the start and end times of the target behavior category in the video to be processed; achieving in-depth mining of the relationship between the start and end boundaries of temporal behavior; and improving the edge detection of temporal behavior. The accuracy of boundary information detection is improved. Furthermore, the initial query vector information is mixed with the feature information of all query points to obtain the target query vector information. Since the query point feature information is calculated based on the feature information of multiple video frames extracted from the initial query point information, the target query vector information can learn the semantic features in the temporal behavior. Therefore, classifying the temporal behavior using the target query vector information can accurately determine the target behavior category within the temporal behavior, resulting in high accuracy. Combining the start and end times of the target behavior category in the video to be processed allows for accurate detection of the temporal behavior in the video. Additionally, since only the video frame feature information corresponding to each initial query point information needs to be extracted during the temporal behavior detection process, without sampling the entire video to be processed, the detection efficiency of temporal behavior is improved.

[0098] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the description, claims, and drawings. Attached Figure Description

[0099] The accompanying drawings are provided to further understand the technical solutions of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the technical solutions of the present invention, and do not constitute a limitation on the technical solutions of the present invention.

[0100] Figure 1 This is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0101] Figure 2 This is a schematic diagram of an application scenario provided by an embodiment of the present invention;

[0102] Figure 3 This is a schematic diagram of another application scenario provided by an embodiment of the present invention;

[0103] Figure 4 This is a flowchart of a video processing method provided in an embodiment of the present invention;

[0104] Figure 5 This is a flowchart of a video detection model training method provided in an embodiment of the present invention;

[0105] Figure 6 This is a schematic diagram of the workflow of a video detection model provided in an embodiment of the present invention;

[0106] Figure 7 This is a schematic diagram of the workflow of a multi-level interactive module provided in an embodiment of the present invention;

[0107] Figure 8 This is a schematic diagram of the structure of a video processing device provided in an embodiment of the present invention;

[0108] Figure 9 This is a schematic diagram of the structure of a video detection model training device provided in an embodiment of the present invention;

[0109] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0110] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. The described embodiments should not be considered as limitations on the present invention, and all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the present invention.

[0111] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0112] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0113] Before providing a further detailed description of the embodiments of the present invention, the nouns and terms involved in the embodiments of the present invention will be explained, and the nouns and terms involved in the embodiments of the present invention shall be interpreted as follows.

[0114] 1) Temporal behavior detection is the process of locating and identifying behavioral segments in a video that has not been segmented into video segments. The detection results include the start time, end time, and behavior category of each behavior in the video.

[0115] 2) Query vectors are embedded information representing potential behaviors in the video. Query vector information refers to the numerical information of the query vector. Initial query vector information refers to the numerical information of the query vector obtained through initial randomization, which can be obtained through a normal distribution. After inputting the initial query vector information into the video detection model, the model can learn the offset positions of multiple query points corresponding to the initial query vector information, and simultaneously learn a semantic feature to ultimately identify the category of the behavior. Furthermore, the target query vector information is the final query vector information obtained by the video detection model after processing the initial query vector information.

[0116] 3) Query points can be understood as a group of discrete points whose positions can be changed. Query point information refers to the numerical information of the query points, and initial query point information refers to the numerical information of the query points obtained through initial randomization. Each query vector corresponds to a set of query points, and each set of query points includes multiple query points. The initialized multiple query points will randomly correspond to multiple video frames in the video. After inputting a set of initial query point information into the video detection model, the video detection model can use the initial query vector information to distribute these query points to the start frame, end frame, and semantic keyframe of each behavior segment. At this time, the semantic features of the behavior can also be learned through the initial query vector information. Finally, multiple query vectors combined with multiple sets of query points can be used to learn the position and specific category of various temporal behaviors in the video. In addition, the target query point information is the final query point information obtained by the video detection model after processing the initial query point information.

[0117] 4) RGB is short for three-channel image feature. Each image is composed of multiple pixels, and each pixel can be represented by a combination of red, green and blue pixel values.

[0118] 5) L1 loss, also known as mean absolute error, is a commonly used regression loss. L1 loss is the sum of the absolute values ​​of the difference between the target value and the predicted value, representing the average error magnitude of the predicted value, without needing to consider the direction of the error.

[0119] 6) Cross-entropy loss. Cross-entropy is a concept in information theory used to estimate the average code length. For two given probability distributions p and q, the cross-entropy of p, expressed in terms of q, can be:

[0120]

[0121] Cross-entropy represents the distance between two probability distributions, where p represents the calibration value and q represents the predicted value. The smaller the cross-entropy, the closer the two probability distributions are.

[0122] 7) Intersection over Union (IoU) is commonly used in detection tasks such as object detection and behavior detection to measure the accuracy of the location information in the predicted results. IoU can be expressed as the ratio of the area of ​​the intersection of two regions to the area of ​​the union of the two regions.

[0123] 8) mAP (mean Average Precision) is an average value that is often used to measure the detection accuracy in a detection task. It is calculated by taking the average target and measuring the different AP (Average Precision) values ​​of multiple targets in the detection task.

[0124] 9) Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. AI technology is a comprehensive discipline involving a wide range of fields, encompassing both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision, speech processing, natural language processing, and machine learning / deep learning. With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0125] 10) Computer Vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and further processes the images to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0126] 11) Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0127] Currently, common methods for locating and recognizing temporal behaviors in videos include various implementation schemes such as temporal behavior detection based on video segment features and temporal behavior detection based on video point features. The video segment feature-based temporal behavior detection scheme is similar to image anchor point-based object detection in image-oriented systems. This scheme uses potential behavior segments as anchor points and performs further feature extraction and pooling operations, then detects multi-label behaviors based on the final features. However, this scheme requires dense video segmentation, significantly increasing model complexity during potential behavior segment extraction. Furthermore, it struggles to simultaneously learn the boundary and semantic information of multi-label behaviors within a temporal video sequence, resulting in low accuracy in detecting temporal behavior boundary information. On the other hand, the video point feature-based temporal behavior detection scheme primarily uses a bottom-up strategy to extract keyframe features from local features. However, this scheme requires dense video sampling, which not only reduces the efficiency of temporal behavior detection but also makes it difficult to learn the semantic features of behavior boundary information, leading to low accuracy in detecting temporal behavior boundary information. Furthermore, related technologies have proposed temporal behavior detection schemes based on multi-label classification of frame images. These schemes primarily rely on deep learning models to mine inter-frame temporal relationships, and then use these relationships to identify multi-label behaviors within a single frame, ultimately achieving the localization and recognition of multi-label temporal behaviors. However, these schemes lack the ability to mine the relationships between the start and end boundaries of behaviors, and also lack the extraction of temporal environmental information at the temporal semantic feature level. In other words, all the temporal behavior detection schemes provided in these related technologies suffer from low accuracy in detecting temporal behaviors in videos.

[0128] To improve the accuracy of detecting temporal behaviors in videos, this invention provides a video processing method, a video detection model training method, a video processing device, a video detection model training device, a computer-readable storage medium, and a computer program product. The method first acquires the video to be processed, initial query vector information, and multiple initial query vectors, where each initial query point corresponds to the temporal sequence of video frames in the video to be processed. Multiple video frame feature information is extracted from the video feature information using the multiple initial query point information. Query point feature information for each initial query point is calculated based on the video frame feature information. The initial query point information is then updated based on the query point feature information to adjust the temporal sequence of the video frames for each initial query point, resulting in multiple target query point information. This ensures that the temporal sequence of the video frames corresponding to each target query point corresponds to the temporal behaviors in the video to be processed, thereby enabling the determination of the start and end times of target behavior categories in the video to be processed. This approach enables in-depth analysis of the start and end boundaries of temporal behaviors, improving the accuracy of temporal behavior boundary information detection. Furthermore, it blends the initial query vector information with all query point feature information to obtain the target query vector information. Since the query point feature information is calculated based on multiple video frame feature information extracted from the initial query point information, the target query vector information can learn the semantic features of temporal behaviors. Therefore, classifying temporal behaviors using the target query vector information can accurately determine the target behavior category within the temporal behavior, resulting in high accuracy. Combining the start and end times of the target behavior category in the video to be processed allows for accurate detection of temporal behaviors within the video. Additionally, since the temporal behavior detection process only requires extracting the video frame feature information corresponding to each initial query point information, without sampling the entire video to be processed, it improves the efficiency of temporal behavior detection.

[0129] The solutions provided in the embodiments of the present invention involve technologies such as video analysis, video feature extraction, and temporal behavior recognition, which are specifically described through the following embodiments.

[0130] Figure 1 This is a schematic diagram of an implementation environment provided by an embodiment of the present invention. (Refer to...) Figure 1 The implementation environment includes a terminal 101 and a server 102, which are directly or indirectly connected via wired or wireless communication. The terminal 101 and the server 102 can be nodes in a blockchain, but this embodiment does not specifically limit this.

[0131] Terminal 101 may include, but is not limited to, smart devices with video shooting and / or video playback functions such as smartphones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, and aircraft; no specific limitation is made here.

[0132] Terminal 101 has at least the functions of sending video information to server 102 and receiving video information sent by server 102. For example, it can send video information to server 102 in response to a user's operation of publishing video information through a multimedia social platform client; or, in response to a user's operation of obtaining a video segment containing time-series behavior through a video editing application, it can send the corresponding video information to server 102, and after server 102 recognizes that the video information contains a video segment containing time-series behavior, it can receive and display the video segment containing time-series behavior sent by server 102.

[0133] Server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0134] Server 102 has at least the functions of acquiring video information, performing temporal actions on video information, and recognizing video segments containing such temporal actions. For example, after acquiring the video to be processed, it can acquire initial query vector information, multiple initial query point information, and video feature information of the video to be processed. Then, based on the multiple initial query point information, it can extract multiple video frame feature information from the video feature information, calculate the query point feature information of each initial query point information based on each video frame feature information, and update each initial query point information based on each query point feature information to obtain multiple target query point information. It can also perform feature mixing of the initial query vector information and all query point feature information to obtain target query vector information. Then, it can perform category recognition of the temporal actions based on the target query vector information to obtain the target action category, and determine the start and end times of the target action category in the video to be processed based on the temporal sequence of the video frames corresponding to all target query point information. After identifying the target action category and determining the start and end times of the target action category in the video to be processed, server 102 can perform subsequent processing such as video review, video segment editing, video material pre-extraction, or video annotation based on the target action category, start time, and end time.

[0135] Reference Figure 2As shown, in an application scenario, assuming terminal 101 is a smartphone and a multimedia social platform client is installed on terminal 101, when a user publishes video information through the multimedia social platform client, terminal 101 sends the video information to server 102. After receiving the video information, server 102 can obtain initial query vector information, multiple initial query point information, and video feature information of the video information. The initial query vector information is used to identify the temporal behavior of the video information, and the initial query point information corresponds to the temporal sequence of the video frames. Then, server 102 extracts multiple video frame feature information from the video feature information based on the multiple initial query point information, calculates the query point feature information of each initial query point information based on each video frame feature information, and updates each initial query point information based on the query point feature information to obtain multiple target query point information. The query point information corresponds to the temporal sequence of video frames in the video information. Next, the server 102 mixes the initial query vector information with all query point feature information to obtain the target query vector information. After obtaining the target query vector information and multiple target query point information, the server 102 performs category recognition on the temporal behavior based on the target query vector information to obtain the target behavior category, and determines the start and end times of the target behavior category in the video information based on the temporal sequence of the video frames corresponding to all target query point information. After obtaining the target behavior category, start time, and end time, the server 102 obtains the target video segment in the video information based on the start and end times, and performs video review based on the target video segment and the target behavior category to determine whether the target video segment and the target behavior category belong to the allowed content. If the video review is passed, the server 102 publishes the video information.

[0136] Reference Figure 3As shown, in another application scenario, assuming terminal 101 is a smartphone and a video editing application is installed on it, when a user extracts behavioral segments from a target video using the video editing application, and the user triggers the behavioral segment extraction button in the video editing application, terminal 101 sends the target video to server 102. After receiving the target video, server 102 can obtain initial query vector information, multiple initial query point information, and video feature information of the target video. The initial query vector information is used to identify the temporal behavior of the target video, and the initial query point information corresponds to the temporal sequence of the video frames of the target video. Then, server 102 extracts multiple video frame feature information from the video feature information based on the multiple initial query point information, calculates the query point feature information of each initial query point based on the video frame feature information, and performs various functions based on the query point feature information. The initial query point information is updated to obtain multiple target query point information, where the target query point information corresponds to the temporal sequence of video frames in the target video. Next, the server 102 performs feature mixing of the initial query vector information and all query point feature information to obtain target query vector information. After obtaining the target query vector information and multiple target query point information, the server 102 performs category recognition on the temporal behavior based on the target query vector information to obtain the target behavior category, and determines the start and end times of the target behavior category in the target video based on the temporal sequence of video frames corresponding to all target query point information. After obtaining the target behavior category, start time, and end time, the server 102 obtains the target video segment in the target video based on the start and end times, marks the target video segment with the target behavior category, and then sends the target video segment marked with the target behavior category to the terminal 101 for display.

[0137] The embodiments of the present invention can be applied to various scenarios that require temporal behavior recognition of videos, including but not limited to temporal behavior recognition scenarios in fields such as video review, video clip editing, video material pre-extraction, and video annotation.

[0138] Figure 4 This is a flowchart of a video processing method provided in an embodiment of the present invention. This video processing method can be executed by a terminal or a server, or jointly by both. In this embodiment, the method is described using an example of execution by a server. (Refer to...) Figure 4 The video processing method includes, but is not limited to, steps 410 to 460.

[0139] Step 410: Obtain the video to be processed.

[0140] In this step, the video to be processed can be a video stored on a server, such as a video uploaded to the server by a user or a video retrieved by the server via the Internet and stored on the server. In another possible implementation, the video to be processed can also be a video stored on a terminal, such as a video downloaded by a user from a server and stored in the terminal's memory, or a video taken by a user using a terminal (e.g., a mobile phone) and stored in the terminal's memory. In this step, the video to be processed can be automatically obtained according to the processing task, such as automatically obtaining the corresponding video to be processed based on a network link or video processing task, or it can be obtained according to the user's operation, such as being uploaded by the user, taken by the user, or selected and obtained by the user from a user interface such as a resource browser or photo album. The video to be processed can be a video segment with T consecutive frames, where T is a positive integer, meaning the video segment includes T frames of images, where each frame is an RGB image, a grayscale image, or a black and white image.

[0141] Step 420: Obtain initial query vector information, multiple initial query point information, and video feature information of the video to be processed. The initial query vector information is used to identify the temporal behavior of the video to be processed, and the initial query point information corresponds to the temporal sequence of the video frames of the video to be processed.

[0142] In this step, video feature information is obtained by performing video feature extraction on the video to be processed. For example, a video feature extraction device is used to perform the above-mentioned video feature extraction. In one possible embodiment, the video feature extraction device is a video encoder. The video encoder extracts multiple video frame feature information by extracting features from each frame of the video to be processed. Each video frame feature information corresponds to the temporal sequence of the video frames of the extracted image. These video frame feature information can be represented by a matrix X, where the dimension of matrix X is T×D, and can be represented as follows: T represents the number of video frames (i.e., the number of images) in the video to be processed, and D represents the dimension of the video frame feature information extracted from each image. In one possible implementation, this dimension D is 768. In another possible implementation, other video feature extraction models, such as convolutional neural networks or machine learning models, can also be used to extract video features from the video to be processed.

[0143] In one possible implementation, Temporal Action Localization (Inflated 3D ConvNet, I3D) is used as the video encoder, and the model is trained end-to-end using a loss for multi-label action detection. For ease of model deployment, in one embodiment, only the RGB values ​​of the images are used as model input.

[0144] In one possible implementation, the initial query vector information and multiple initial query point information are automatically obtained or generated by the server or terminal. For example, the initial query vector information or multiple initial query point information can be preset default information or initialization information, or they can be randomly generated or generated based on the number of frames or duration of the video to be processed. In another possible implementation, the initial query vector information or multiple initial query point information can be selected or input by the user, or obtained by the server or terminal via the Internet according to the user's instructions. Of course, the methods of obtaining the initial query vector information and initial query point information can be different; for example, the initial query vector information can be input by the user, while the initial query point information can be randomly generated by the server or terminal. The initial query vector information is used to identify the temporal behavior of the video to be processed. That is, an initial query vector information is used to query a temporal behavior in the video to be processed. The initial query point information corresponds to the temporal sequence of the video frames in the video to be processed. In other words, an initial query point information records the position information of one video frame in the video to be processed. It can also be understood that the initial query point information represents the temporal position of one video frame in the video to be processed, or the initial query point information represents the frame number of one video frame in the video to be processed. The corresponding video frame in the video to be processed can be located through the initial query point information. For example, if an initial query point information records frame number 20, then the video frame feature information of the image with frame number 20 can be located through the initial query point information. That is, the initial query point information can be represented as a query point corresponding to the temporal sequence of a certain video frame.

[0145] In one possible implementation, one initial query vector corresponds to Ns initial query points, which are distributed across different video frames in the video to be processed. In another possible implementation, the initial query points are randomly generated, randomly distributed across different video frames in the video to be processed. In yet another possible implementation, the initial query points are generated based on the number of frames or duration of the video to be processed, for example, the query points are normally distributed across the video frames. In one possible implementation, Ns < T, meaning the query points indicate which video frames' feature information to extract for processing. Since the number of query points corresponding to one initial query vector is smaller than the number of frames in the video to be processed, sparse querying can be achieved, reducing the amount of video feature information processed and improving video processing efficiency. In one possible implementation, multiple initial query vectors are obtained, for example, Nq initial query vectors are input, and each initial query vector corresponds to Ns initial query points, thus obtaining a set of query points. It means that t iThis represents the video frame temporal position of the i-th query point. Multiple initial query vectors can also be represented as a set of query vector information q, where each initial query vector has a dimension D. In one possible implementation, this dimension D is 768. A set of query vector information q includes Nq initial query vectors, meaning that multiple initial query vector information q can be represented by a matrix q, where... A set of initial query vector information q includes Nq×Ns initial query point information. Therefore, the obtained initial query vector information can be generated or input as a matrix q, and multiple initial query point information can also be generated or input as rectangles. Both the initial query vector information and the multiple initial query point information are learnable initial information. This will be updated during the video processing described later.

[0146] Step 430: Extract multiple video frame feature information from the video feature information based on multiple initial query point information; calculate the query point feature information of each initial query point information based on each video frame feature information; and update each initial query point information based on each query point feature information to obtain multiple target query point information, wherein the target query point information corresponds to the temporal sequence of the video frames in the video to be processed.

[0147] In this step, multiple video frame feature information is extracted from the video feature information based on multiple initial query point information, and query point feature information corresponding to each initial query point information is calculated. In one possible implementation, the video frame feature information corresponding to the temporal sequence of each initial query point information is obtained as the query point feature information. For example, for a query point set... The video frame feature information corresponding to each ti is obtained as the query point feature information x(ti). In another possible implementation, variable convolution is used to extract features from the local neighborhood temporal sequence of a query point. Based on the video frame temporal sequence corresponding to each initial query point information, multiple temporal offset values ​​corresponding to each initial query point information are obtained. Based on the multiple temporal offset values ​​corresponding to each initial query point information, multiple video frame information corresponding to each initial query point is extracted from the video feature information. The multiple video frame feature information corresponding to each initial query point information is weighted and calculated to obtain the query point feature information x(ti) of each initial query point information. For example, for a query point ti, based on the query point {t i The position of} and its corresponding weight {w j Predict multiple time-series offset vectors {o j The query point is used as a local center point and multiple temporal offset vectors {o}. jThe features of the sub-query points are summed to obtain multiple variable sub-query points, which are used to describe the characteristics of the query point in the local neighborhood. In one possible implementation, the features corresponding to the sub-query points can be obtained by bilinear interpolation and weighted by corresponding weights to obtain the query point feature information x(ti), i.e., x(t) i )=∑ j x(t i +o j )·w j .

[0148] In this step, the initial query point information is updated based on the feature information of each query point to obtain multiple target query point information. Since the initial query point information corresponds to the video frame time sequence of the video to be processed, updating the initial query point information is equivalent to changing each set of query points. The video frame timing of each ti is determined by changing the value of ti to correspond to different video frames. In one possible implementation, offset prediction can be performed on the feature information of each query point to obtain the offset information of each initial query point. Then, the initial query point information is updated according to the offset information of each initial query point to obtain multiple target query point information. Specifically, the feature information of each query point can be input into a pre-trained offset prediction model to predict the offset information of each initial query point. Since the query point feature information is obtained by extracting the feature information of multiple video frames corresponding to the query point, it carries the image information of the video frame corresponding to the query point. By learning and training the offset prediction model, the offset information of each initial query point predicted by the offset prediction model can identify the start time (or start frame), end time (or end frame), and timing (or key frame) corresponding to the key features of the timing behavior that are closer to each query point. The update process of each initial query point information can be represented as follows: in For the target query point information after one update, Δt i To predict the offset based on the feature information of the query point, s = max({t i})-min({ i}) is the scale parameter of the query point, which is the time difference between the maximum and minimum time series of the query points in the current group.

[0149] In one possible implementation, the target query point information can be obtained by updating the initial query point information multiple times. This allows each query point corresponding to the target query point information to be closer to the start time, end time, and keyframe of the temporal behavior. In another possible implementation, the target query point information can be obtained by iteratively updating the initial query point information multiple times. For example, each initial query point information is updated based on its feature information to obtain multiple new initial query point information. Then, multiple new video frame feature information is extracted from the video feature information based on the multiple new initial query point information, and multiple new query point feature information is calculated based on the multiple new video frame feature information. Each new initial query point information is updated based on its feature information until the number of updates reaches a preset value, resulting in multiple target query point information. A pre-trained offset prediction model with l layers can be used to iteratively update the initial query point information. The update of the query point set offset in the lth update can be expressed as: in, Let be the scale parameter of the query point in layer l. It can be seen that since the scale parameters differ between different layers and are related to the boundary conditions of the input query point, a scale-adaptive self-updating strategy can be implemented, improving the accuracy of the target query point information and matching temporal behavior. In one possible implementation, the offset prediction model is a pre-trained multilayer feedforward neural network (FFN) model.

[0150] Step 440: The initial query vector information is mixed with the feature information of all query points to obtain the target query vector information.

[0151] In this step, the initial query vector information is updated by feature mixing with all query point feature information. Since the query point feature information includes the feature information of the video frame corresponding to the query point, feature mixing can update the target query vector information according to the feature information of the video frame, so that the target query vector information can learn the semantic information in the video to be processed.

[0152] In one possible implementation, when multiple initial query vectors exist, each initial query vector is used to learn the temporal behavior category in the video to be processed, resulting in multiple target query vectors. To avoid the reduction in classification performance due to excessive temporal overlap between different temporal behaviors, this step proposes to adaptively fuse the video frame features and channel features of the video to be processed. Specifically, the initial query vectors are subjected to variable convolution with the aforementioned query point feature information to obtain multiple intermediate feature information. All intermediate feature information is then stacked to obtain query point stacked feature information. The query point feature information is x(t) in the above embodiment. i )=∑ j x(t i +o j )·w j By performing variable convolution on the initial query vector information and the query point feature information, we can obtain intermediate feature information. This intermediate feature information corresponds to one initial query point or one query point. Therefore, the Ns query points corresponding to the initial query vector have Ns intermediate feature information, each with a dimension of D. By stacking all the intermediate feature information corresponding to each group of query points, we obtain the query point stacked feature information. Among them, query point stacking feature information The dimension is Ns×D. Then, the stacked feature information of the above query points is mixed with the initial query vector information to obtain cascaded feature information. The cascaded feature information is then residually connected with the initial query vector information to obtain the target query vector information. First, a linear mapping is performed on the initial query vector information q to obtain...

[0153] In one possible implementation, two different mixing methods can be used to mix the stacked feature information and the initial query vector information. One feature mixing method involves mixing the query point stacked feature information and the initial query vector information based on a first dimension to obtain the first mixed feature information. In one embodiment, the initial query vector is linearly mapped to obtain the query vector mapping feature Mf, wherein... The query vector mapping feature Mf has a dimension of Ns×Ns. Then, the query points are stacked with feature information. The query vector mapping feature Mf is subjected to frame mixing processing, normalization processing, and activation function processing to obtain the relationship between each frame in the potential temporal behavior prediction interval. In one embodiment, the LayerNorm normalization model and ReLU function are used to obtain the first channel mixed feature representing the relationship between each frame in the temporal behavior prediction interval:

[0154] FrameMix(x)=ReLU(LayerNorm(x T M f In this process, the query point stacking feature information x needs to be transposed first so that its dimension is the same as the dimension of the query vector mapping feature Mf before matrix multiplication.

[0155] Another feature fusion approach involves combining the query point stacked feature information Mf with the initial query vector information q based on a second dimension to obtain second mixed feature information. In one embodiment, the query point stacked feature information... After transposing and linearly mapping the query vector to the feature Mf, channel mixing is performed, followed by normalization and activation function processing to obtain the relationship between each channel in the potential temporal behavior prediction interval. In one embodiment, two bottleneck structures are used to linearly map the initial query vector q to enhance the behavioral semantics, which can be obtained as follows:

[0156] M c,1 =Linear(q)∈R D×D′

[0157] M c,2 =Linear(q)∈R D′×D

[0158] Among them, for M c,1 First, map the initial query vector q to dimension D, then map it to dimension D×D′. For M... c,2 First, the initial query vector q is mapped to dimension D', then to dimension D′×D. Next, the query points are stacked with feature information. With M c,1 and M c,2 After cascaded normalization and activation function processing, the second mixed feature information representing the relationship between channels in the potential temporal behavior prediction interval is obtained:

[0159] ChannelMix(x)=ReLU(LayerNorm(ReLU(LayerNorm(xM c,1 ))M c,2 ))

[0160] By concatenating and mapping the first and second mixed feature information, concatenated feature information is obtained. The concatenated feature information is then residually joined with the initial query vector information q to obtain new initial query vector information. In one embodiment, the stacked feature information and the initial query vector information undergo a feature mixing process once, and the resulting new initial query vector information is the target query vector information. In another embodiment, the new initial query vector information and the extracted point stacked feature information Mf after updating the query point's temporal position can be iteratively mixed to obtain the target query vector information. For example, the new initial query vector information can be mixed with all the new query point feature information until the number of feature mixing operations reaches the preset value to obtain the target query vector information.

[0161] The above embodiments mention mixing the query point stacking feature information with the initial query vector information to obtain cascaded feature information, which can be processed by a multi-level interactive module. Specifically, the initial query vector information (or the iteratively updated initial query vector information), each initial query point information, and the video feature information of the video to be processed are input into the multi-level interactive module for processing, resulting in new initial query vector information and query point stacking feature information. The query point stacking feature information is then input into an offset prediction model for offset prediction, and the initial query point information is updated based on the predicted offset to obtain new initial query point information.

[0162] In one possible implementation, multiple initial query vectors are first processed by multi-head self-attention before being input into the aforementioned multi-level interaction module. Multi-head attention processing models the relationships between query vectors. In one embodiment, a multi-head self-attention (MHSA) module can be used to process multiple initial query vectors. Multi-head self-attention refers to using multiple self-attention mechanisms to capture relationships at different abstract levels from different perspectives.

[0163] When multi-level iterative processing is required, a multi-level interaction module with multiple layers can be trained for iterative processing. Since new query point positions need to be obtained to update the extracted video frame feature information during iterative processing, the number of layers of the multi-level interaction module is the same as the number of layers of the offset prediction model and the multi-head attention module that need to be iterated. That is, the multi-level interaction module, the multi-head attention module and the offset prediction model with corresponding layers are interconnected and perform iterative processing together until the preset number of times is reached. For example, if the multi-head attention module, the offset prediction model and the multi-level interaction module are all set with l layers, then l iterations are performed.

[0164] Step 450: Based on the target query vector information, perform category identification on the temporal behavior to obtain the target behavior category.

[0165] In this step, since the target query vector is obtained by feature mixing the initial query vector information and the feature information of all query points, the target query vector information can learn the semantic features in the temporal behavior. By performing category recognition on the temporal behavior through the target query vector information, the category of the target behavior in the temporal behavior can be accurately obtained. In one possible implementation, multiple initial query vector information are included, where each initial query vector information corresponds to learning a temporal behavior in the video to be processed. Therefore, when there are multiple initial query vector information, multiple target query vector information are also obtained. By performing category recognition on the temporal behavior of each target query vector information, the corresponding target behavior category can be obtained. For example, it includes two initial query vectors. The model described above uses multiple sets of query points to represent temporal behaviors, enabling the target query vectors to learn the temporal behaviors in the video to be processed. Therefore, two target query vectors are obtained. By performing category recognition on each target query vector, two target behavior categories are obtained. For example, a basketball video may simultaneously involve two temporal behaviors: shooting and blocking. Category recognition of the first target query vector results in the target behavior category of "shooting," while category recognition of the second target query vector results in the target behavior category of "blocking." In one embodiment, a pre-trained multilayer feedforward network (FFN) model can be used for category recognition.

[0166] Step 460: Determine the start and end times of the target behavior category in the video to be processed based on the video frame time sequence corresponding to all target query point information.

[0167] In this step, the start and end times of the target behavior category in the video to be processed can be determined based on the temporal sequence of the multiple target query point information obtained in step 430 above. Since the query points represented by the target query point information obtained in step 430 above are concentrated on the start time (or start frame), end time (or end frame) of the temporal behavior, and the temporal sequence (or key frame) corresponding to the key features of the temporal behavior, the start and end times of the target behavior category in the video to be processed can be obtained based on the boundary of the query point distribution. In one embodiment, the query points can also be converted into pseudo-video segments based on the temporal sequence of the multiple target query point information, and the start and end times of the temporal behavior can be determined based on the temporal boundary of the pseudo-video segments.

[0168] Step 450 above allows us to obtain the target behavior category C of the temporal behavior in the video to be processed, and step 460 above allows us to obtain the start time t of the temporal behavior. s and end time t e Therefore, the temporal behavior instance ψ can be represented as:

[0169] ψ=(t s ,t e (c) When there are Nq initial query vectors, Nq temporal behavior instances ψ can be obtained.

[0170] In this embodiment, a top-down approach is used to directly regress the keyframe features of the video to be processed based on the initial query vector information. Multiple initial query point information is used to learn, represent, and extract features of temporal behavior. Each initial query point information corresponds to the temporal sequence of video frames in the video to be processed. Multiple video frame features are extracted from the video feature information using multiple initial query point information. Since the features of temporal behavior can be directly extracted from each query point, it is not necessary to obtain them through pooling with irrelevant frames. Query point feature information for each initial query point is calculated based on the feature information of each video frame. The initial query point information is then updated based on the query point feature information to adjust the video frame temporal sequence of each initial query point, resulting in multiple target query point information. This ensures that the video frame temporal sequence corresponding to each target query point corresponds to the temporal behavior in the video to be processed, thereby determining the start and end times of the target behavior category in the video to be processed. This enables in-depth mining of the start and end boundary relationships of temporal behavior, improving the accuracy of temporal behavior boundary information detection. Furthermore, the initial query vector information is mixed with all query point feature information to obtain the target query vector information. Since the query point feature information... The target query vector is calculated based on the feature information of multiple video frames extracted from the initial query point information. Therefore, the target query vector information can learn the semantic features in the temporal behavior. Thus, by classifying the temporal behavior through the target query vector information, the target behavior category in the temporal behavior can be accurately obtained with high accuracy. Combining the start and end times of the target behavior category in the video to be processed, the temporal behavior in the video to be processed can be accurately detected. In addition, since only a small amount of video frame feature information corresponding to the initial query point information is needed in the process of temporal behavior detection, it is not necessary to sample the entire video to be processed. Therefore, the detection efficiency of temporal behavior can be improved, thereby greatly improving the effectiveness and ease of use of the model.

[0171] Figure 5This is a flowchart of a video detection model training method provided in an embodiment of the present invention. This video processing method can be executed by a terminal or a server, or jointly by both. In this embodiment, the method is described using an example where it can be executed by a server. (Refer to...) Figure 5 The training method for the video detection model includes, but is not limited to, steps 510 to 570.

[0172] Step 510: Obtain video samples and label information.

[0173] In this step, the video sample can be a video stored on a server, such as a video uploaded to the server by a user or a video obtained by the server via the Internet and stored on the server. In another possible implementation, the video to be processed can also be a video stored on a terminal, such as a video downloaded by the user from the server and stored in the terminal's memory, or a video taken by the user using the terminal (e.g., a mobile phone) and stored in the terminal's memory.

[0174] A video sample is a video segment with T consecutive frames, where T is a positive integer, meaning the video segment consists of T frames, each of which is an RGB image, grayscale image, or black and white image. Each video sample is assigned one or more tags. These tags include the start time, end time, and target behavior category of the temporal action within the video sample. For example, a basketball video sample might have two tags: one tag with target behavior category c as "shoot," corresponding to start time t1 and end time t2; and another tag with target behavior category c as "block," corresponding to start time t3 and end time t4. Therefore, the tag information can be represented as ψ = (t... s ,t e c), where t s Let t be the start time. e This is the end time.

[0175] Step 520: Obtain video feature information of query vector samples, multiple query point samples, and video samples. The query vector samples are used to identify the temporal behavior of the video samples, and the query point samples correspond to the video frames of the video samples in time sequence.

[0176] In this step, the number of query vector samples corresponds to the number of label information. Video feature information is obtained by performing video feature extraction processing on the video samples. For example, a video feature extraction device is used to perform the above video feature extraction. In one possible implementation, the video feature extraction device is a video encoder. The video encoder extracts multiple video frame feature information by performing feature extraction on each frame image of the video to be processed. Each video frame feature information corresponds to the temporal sequence of the video frames of the extracted image. These video frame feature information can be represented by a matrix X, where the dimension of matrix X is T×D, and can be represented as X∈R. T×D T represents the number of video frames (i.e., the number of images) in the video to be processed, and D represents the dimension of the video frame feature information extracted from each image. In one possible implementation, this dimension D is 768. In another possible implementation, other video feature extraction models, such as convolutional neural networks or machine learning models, can also be used to extract video features from the video to be processed.

[0177] In one possible implementation, the query vector sample and multiple query point samples are automatically acquired or generated by the server or terminal. For example, the query vector sample or multiple query point samples can be preset default information or initialization information, or they can be randomly generated or generated based on the number of frames or duration of the video to be processed. In another possible implementation, the query vector sample or multiple query point samples can be selected or input by the user, or acquired by the server or terminal via the Internet according to the user's instructions.

[0178] The query vector sample is used to identify the temporal behavior of the video to be processed. That is, a query vector sample is used to query a temporal behavior in the video to be processed. The query point sample corresponds to the temporal sequence of the video frame of the video to be processed. That is, a query point sample records the position information of one video frame in the video to be processed. It can also be understood that the query point sample represents the temporal position of one video frame in the video to be processed, or the query point sample represents the frame number of one video frame in the video to be processed. In addition, the query point sample can be represented as a query point corresponding to the temporal sequence of a certain video frame.

[0179] In one possible implementation, one query vector sample corresponds to Ns query point samples, and the query points corresponding to these query point samples are distributed across different video frames in the time sequence of the video to be processed. In another possible implementation, multiple query vector samples are obtained, for example, Nq query vector samples are input, and each query vector sample corresponds to Ns query point samples, thus obtaining a set of query point samples. It means that t iThis represents the temporal position of the video frame at the i-th query point. Multiple query vector samples can also be represented as a set of query vector samples q', where each initial query vector has a dimension of D, which in one possible implementation is 768. A set of query vector samples q' includes Nq query vector samples; that is, multiple query vector samples q' can be represented by a matrix q', where...

[0180] Step 530: Input the query vector sample, multiple query point samples and video feature information into the video detection model to predict behavior and obtain the prediction start time, prediction end time and prediction behavior category.

[0181] In one possible implementation, the processing steps of the video detection model include:

[0182] Multiple video frame feature information is extracted from the video feature information based on multiple query point samples. The query point feature information of each query point sample is calculated based on the feature information of each video frame. The query point samples are then updated based on the query point feature information to obtain multiple target query point information. The target query point information corresponds to the temporal sequence of the video frames in the video samples.

[0183] In this step, multiple video frame feature information is extracted from the video feature information based on multiple query point samples, and the query point feature information corresponding to each query point sample is calculated. In one possible implementation, the video frame feature information corresponding to the temporal sequence of each query point sample is obtained as the query point feature information. For example, for the query point sample set... The video frame feature information corresponding to each ti is obtained as the query point feature information x(ti). In another possible implementation, variable convolution is used to extract features from the local neighborhood temporal sequence of a query point. Based on the video frame temporal sequence corresponding to each query point sample, multiple temporal offset values ​​corresponding to each query point sample are obtained. Based on the multiple temporal offset values ​​corresponding to each query point sample, multiple video frame information corresponding to each query point sample is extracted from the video feature information. The multiple video frame feature information corresponding to each query point sample is weighted and calculated to obtain the query point feature information x(ti) of each query point sample. For example, for a query point ti, based on the query point {t i The position of} and its corresponding weight {w j Predict multiple time-series offset vectors {o j The query point is used as a local center point and multiple temporal offset vectors {o}. jThe features of the sub-query points are summed to obtain multiple variable sub-query points, which are used to describe the characteristics of the query point in the local neighborhood. In one possible implementation, the features corresponding to the sub-query points can be obtained by bilinear interpolation and weighted by corresponding weights to obtain the query point feature information x(ti), i.e., x(t) i )=∑ j x(t i +o j )·w j .

[0184] In one possible implementation, the target query point information can be obtained by updating the query point samples multiple times. This allows each query point corresponding to the target query point information to be closer to the start time, end time, and keyframes of the temporal behavior. In another possible implementation, the target query point information can be obtained by iteratively updating the query point samples multiple times. For example, each query point sample can be updated based on its feature information to obtain multiple new query point samples. Then, multiple new video frame feature information is extracted from the video feature information based on these new query point samples, and multiple new query point feature information is calculated based on these new video frame feature information. Each new query point sample is updated based on its new query point feature information until the number of updates reaches a preset value, resulting in multiple target query point information. The video detection model includes an offset prediction model, which can be pre-trained with an l-layer offset prediction model. The query point samples are iteratively updated using the offset prediction model at each layer. The update of the query point set offset in the l-th update can be expressed as: in, Let be the scale parameter of the query point in layer l. It can be seen that since the scale parameters differ between different layers and are related to the boundary conditions of the input query point, a scale-adaptive self-updating strategy can be implemented, improving the accuracy of the target query point information and matching temporal behavior. In one possible implementation, the offset prediction model is a pre-trained multi-layer feedforward network (FFN) model. Supervised regression loss can be used to train the query point on the offset prediction model. The training of the offset prediction model can generate pseudo-video segments based on the temporal extracted video frame feature information corresponding to the currently updated query point, and then the loss function is calculated using the pseudo-video segments. In one embodiment, a "partial min-max function" is used to generate the pseudo-video segment boundary. This scheme first selects a subset of query points and uses the "min-max function" on this subset to obtain the final pseudo-video segment:

[0185]

[0186] The “minimum-maximum function” can be understood as follows: for each set of query points, a subset of query points is randomly selected to form a local query point subset, and then the minimum frame number and maximum frame number are obtained from this subset of query points, thus obtaining the pseudo video segment of the behavior.

[0187] The query vector sample is mixed with the feature information of all query points to obtain the target query vector information.

[0188] In this step, the query vector samples are updated by feature mixing with all query point feature information. Since the query point feature information includes the feature information of the video frame corresponding to the query point, feature mixing can update the target query vector information according to the feature information of the video frame, so that the target query vector information can learn the semantic information in the video to be processed.

[0189] In one possible implementation, when multiple query vector samples exist, each initial query vector is used to learn the temporal behavior category in the video to be processed, resulting in multiple target query vector information. The query vector samples are then subjected to variable convolution with the aforementioned query point feature information to obtain multiple intermediate feature information. All intermediate feature information is then stacked to obtain query point stacked feature information. Here, the query point feature information is x(t) in the above embodiment. i )=∑ j x(t i +o j )·w j By performing variable convolution on the query vector samples and the query point feature information, an intermediate feature information can be obtained.

[0190] Then, the aforementioned feature mixing of the query point stacked feature information and the query vector sample yields cascaded feature information. This cascaded feature information is then residual-connected with the query vector sample to obtain the target query vector information. Specifically, a linear mapping is first performed on the query vector sample q' to obtain...

[0191] In one possible implementation, two different mixing methods can be used to perform feature mixing between the stacked feature information and the query vector samples. One feature mixing method involves performing feature mixing between the query point stacked feature information and the query vector samples based on a first dimension to obtain first mixed feature information. In one embodiment, the query vector mapping feature Mf is obtained by linearly mapping the initial query vector. The query vector mapping feature Mf has a dimension of Ns×Ns. Then, the query points are stacked with feature information. The query vector mapping feature Mf is subjected to frame mixing processing, normalization processing, and activation function processing to obtain the relationship between each frame in the potential temporal behavior prediction interval. In one embodiment, the LayerNorm normalization model and ReLU function are used to obtain the first channel mixed feature representing the relationship between each frame in the temporal behavior prediction interval:

[0192] FrameMix(x)=ReLU(LayerNorm(x T M f In this process, the query point stacking feature information x needs to be transposed first so that its dimension is the same as the dimension of the query vector mapping feature Mf before matrix multiplication.

[0193] Another feature mixing approach involves combining the query point stacked feature information Mf with the query vector sample q' based on a second dimension to obtain second mixed feature information. In one embodiment, the query point stacked feature information... After transposing and linearly mapping the query vector to the feature Mf, channel mixing is performed, followed by normalization and activation function processing to obtain the relationship between each channel in the potential temporal behavior prediction interval. In one embodiment, two bottleneck structures are used to linearly map the initial query vector q to enhance the behavioral semantics, which can be obtained as follows:

[0194] M c,1 =Linear(q)∈R D×D′ ,

[0195] M c,2 =Linear(q)∈R D′×D .

[0196] Among them, for M c,1 First, map the initial query vector q to dimension D, then map it to dimension D×D′. For M... c,2 First, the initial query vector q is mapped to dimension D', then to dimension D′×D. Next, the query points are stacked with feature information. With M c,1 and M c,2 After cascaded normalization and activation function processing, the second mixed feature information representing the relationship between channels in the potential temporal behavior prediction interval is obtained:

[0197] ChannelMix(x)=ReLU(LayerNorm(ReLU(LayerNorm(xM c,1 ))M c,2 ))

[0198] By concatenating and mapping the first and second mixed feature information, concatenated feature information is obtained. The concatenated feature information is then residually joined with the query vector sample q' to obtain a new query vector sample. In one embodiment, the stacked feature information and the query vector sample undergo a feature mixing process once, and the resulting new query vector sample is the target query vector information. In another embodiment, the new query vector sample and the extracted point stacked feature information Mf after updating the query point's temporal position can be iteratively mixed to obtain the target query vector information. For example, the new query vector sample can be feature-mixed with all the new query point feature information until the number of feature mixing operations reaches the preset value, thus obtaining the target query vector information.

[0199] The above embodiments mention mixing the query point stacked feature information with the query vector samples to obtain cascaded feature information. In one embodiment, the video detection model includes a multi-level interactive module. The query vector samples (or iteratively updated initial query vector information), each query point sample, and the video feature information of the video to be processed are input into the multi-level interactive module for processing, resulting in new query vector samples and query point stacked feature information. The query point stacked feature information is then input into an offset prediction model for offset prediction, and the query point samples are updated based on the predicted offsets to obtain new query point samples.

[0200] In one possible implementation, multiple query vector samples are first processed by multi-head self-attention before being input into the aforementioned multi-level interaction module. Multi-head attention processing enables the modeling of relationships between query vectors. In one embodiment, a multi-head self-attention (MHSA) module can be used to process multiple query vector samples. Multi-head self-attention refers to using multiple self-attention mechanisms to capture relationships at different abstract levels from different perspectives.

[0201] When multi-level iterative processing is required, a multi-level interaction module with multiple layers can be trained for iterative processing. Since new query point positions need to be obtained to update the extracted video frame feature information during iterative processing, the number of layers of the multi-level interaction module is the same as the number of layers of the offset prediction model and the multi-head attention module that need to be iterated. That is, the multi-level interaction module, the multi-head attention module and the offset prediction model with corresponding layers are interconnected and perform iterative processing together until the preset number of times is reached. For example, if the multi-head attention module, the offset prediction model and the multi-level interaction module are all set with l layers, then l iterations are performed.

[0202] Based on the target query vector information, the temporal behavior is classified to obtain the predicted behavior category.

[0203] In this step, since the target query vector is obtained by feature mixing of the initial query vector information and the feature information of all query points, the target query vector information can learn the semantic features in the temporal behavior. By performing category recognition on the temporal behavior through the target query vector information, the predicted behavior category in the temporal behavior can be accurately obtained. In one possible implementation, multiple initial query vector information are included, where each initial query vector information corresponds to learning a temporal behavior in the video to be processed. Therefore, when there are multiple initial query vector information, multiple target query vector information are also obtained. By performing category recognition on the temporal behavior of each target query vector information, the corresponding predicted behavior category can be obtained.

[0204] The prediction start and end times of the predicted behavior category in the video sample are determined based on the temporal sequence of the video frames corresponding to all target query point information.

[0205] In this step, the start and end times of the predicted behavior category in the video to be processed can be determined based on the temporal sequence of the multiple target query point information obtained above. Since the query points represented by the target query point information obtained in step 430 are concentrated on the start time (or start frame), end time (or end frame) of the temporal behavior, and the temporal sequence (or key frame) corresponding to the key features of the temporal behavior, the start and end times of the predicted behavior category in the video to be processed can be obtained based on the boundary of the query point distribution. In one embodiment, the query points can also be converted into pseudo-video segments based on the temporal sequence of the multiple target query point information, and the start and end times of the temporal behavior can be determined based on the temporal boundary of the pseudo-video segments.

[0206] Step 540: Correct the parameters of the video detection model based on the prediction start time, prediction end time, prediction behavior category, and label information.

[0207] In this step, the parameters of the video detection model are corrected by comparing the prediction results of the video detection model (including prediction start time, prediction end time, and prediction behavior category) with the label information in step 510.

[0208] In one possible implementation, a localization loss value is calculated based on the prediction start time, the prediction end time, the start time label, and the end time label; a classification loss value is calculated based on the prediction behavior category and the behavior category label; and the parameters of the video detection model are corrected based on the localization loss value and the classification loss value. In another possible implementation, since a video sample can correspond to multiple temporal behaviors (i.e., one video sample corresponds to multiple label information), and the video detection model may output multiple prediction results when processing video samples, it is necessary to group and associate these multiple prediction results with the multiple label information of the corresponding real video segments, i.e., to perform a label information allocation and combination σ(·). A matching algorithm can be used to allocate and combine the prediction results and label information. In one embodiment, the Hungarian matching algorithm can be used to match the prediction results with the label information. To select the Hungarian matching algorithm with the minimum matching loss, a label set for each video segment is... By adding a no-behavior label If a calibration set of size Nq is formed, the cost of the corresponding Hungarian matching method can be expressed as:

[0209]

[0210] Where α represents the weight of each loss function in the cost. T(P) n ) represents the prediction result, σ(n) represents the actual video segment, and c n The labels are behavioral tags. Based on the above analysis of costs, it can be seen that, in order to minimize matching loss, this embodiment of the application uses a bilateral matching Hungarian algorithm to allocate and combine training prediction results with label information.

[0211] During the training of the video detection model, the L1 loss function and the IoU loss function are used as the localization loss of the video detection model:

[0212]

[0213] Where, σ * (n) represents the temporal behavior segments in the video samples. This refers to the label information for the corresponding temporal behavior segment.

[0214] For the classification loss in predicting behavior categories, this scheme uses cross-entropy loss. To further improve frame-level accuracy, this scheme obtains dense classification scores by linearly mapping video features. The final classification loss can be expressed as:

[0215] in, λ represents the cross-entropy loss. The first term represents the classification loss for each behavior category of the segment assigned above, with a total of n categories. The second term represents the cross-entropy loss for each behavior category in each frame and the true behavior category in each frame. λ represents the weight of the dense classification.

[0216] Therefore, the overall loss function of the video detection model is expressed as:

[0217]

[0218] Where, λ loc λ represents the weight of the localization loss in the overall loss function. cls This represents the weight of the classification loss in the overall loss function. The parameters of the video detection model are corrected using the calculated overall loss function, and the video detection model of any of the above embodiments is trained.

[0219] The following is combined with Figure 6 , Figure 7 The following describes the working process of the video detection model:

[0220] See Figure 6 As shown, the server includes a video encoder and a video detection model. The video detection model has L layers. The acquired video to be processed includes T frames of RGB images. The video encoder extracts the video frame feature information of each RGB image to obtain video feature information. The video feature information is represented by a matrix with dimensions T×D, where D is the feature dimension of each video frame feature information. Additionally, Nq initial query vectors are acquired, each corresponding to Ns initial query points. These Nq initial query vectors are represented by a matrix with dimensions Nq×D. The initial query point information corresponding to the Nq initial query vectors can also be represented by a matrix with dimensions Nq×Ns, where each row of the initial query point information matrix represents a set of query points. In this process, multiple initial query point information and video feature information are also input into the multi-level interaction module, and multiple initial query vector information are processed by the multi-head attention module before being input into the multi-level interaction module.

[0221] See Figure 7 As shown, the multi-level interaction module includes a query point processing layer and an instance recognition layer. In the query point processing layer, each initial query vector information and its corresponding set of query points are used. Processing is performed on a query point ti, based on the query point {t}. i The position of}701 and its corresponding weight {w j Predict multiple time-series offset vectors {o jThe query point is used as a local center point and multiple temporal offset vectors {o}. j The sums of these sub-query points yield multiple variable sub-query points 702, which are used to describe the characteristics of the query point in this local domain. The features corresponding to the sub-query points can be obtained by bilinear interpolation and weighted by the corresponding weights to obtain the query point feature information x(ti), i.e., x(t). i )=∑ j x(t i +o j )·w j By performing variable convolution on the initial query vector information and the query point feature information, we can obtain intermediate feature information. This intermediate feature information corresponds to one initial query point or one query point. Therefore, the Ns query points corresponding to the initial query vector have Ns intermediate feature information, each with a dimension of D. By stacking all the intermediate feature information corresponding to each group of query points, we obtain the query point stacked feature information. Among them, query point stacking feature information The dimension is Ns×D. In one embodiment, the initial query vector is linearly mapped to obtain the query vector mapping feature Mf, and the dimension of the query vector mapping feature Mf is Ns×Ns. Then, the query point stacked feature information is used. The query vector mapping feature Mf is subjected to frame mixing processing, normalization, and activation function processing to obtain the relationship between each frame in the potential temporal behavior prediction interval. In one embodiment, a LayerNorm normalization model and ReLU function are used to obtain a first channel mixed feature 703 representing the relationship between each frame in the temporal behavior prediction interval. Another feature mixing operation involves mixing the query point stacked feature information Mf with the initial query vector information q based on a second dimension, normalizing, and then processing with an activation function to obtain the relationship between each channel in the potential temporal behavior prediction interval, resulting in a second mixed feature information 703. The first and second mixed feature information are concatenated and mapped to obtain concatenated feature information. The concatenated feature information is then residually joined with the initial query vector information q to obtain a new initial query vector information.

[0222] See Figure 6 As shown, the stacked feature information of each query point The data is input into the offset prediction model for offset prediction, resulting in Nq×Ns offsets for each query point. The initial query point information is then updated based on these offsets, yielding new initial query point information. This new initial query point information, along with the new initial query vector information, is input into the next layer of the video detection model for iterative processing. After L iterations, each target query vector and its corresponding multiple target query points are obtained. The transformation unit transforms the multiple target query point information to obtain the start and end times of the target behavior category in the video. Additionally, the FFN unit performs category recognition on the target query vector information to determine the target behavior category.

[0223] Referring to Table 1 below, Table 1 is a comparison table of the effects of using the video processing method of the present invention for temporal behavior detection and using related technologies for temporal behavior detection.

[0224] Table 1

[0225]

[0226] As shown in Table 1, the video processing method of this invention can obtain an optimal Det-mAP value that is more beneficial for evaluating the capabilities of temporal behavior detection models (e.g., video detection models). Furthermore, without additional optimization, it can obtain a Seg-mAP value comparable to other related technologies. It should be noted that MultiTHUMOS and Charades are commonly used benchmark sets in temporal behavior detection tasks. MultiTHUMOS includes 413 sports videos covering 65 behavior categories, while Charades includes 9848 indoor activity videos covering 157 behavior categories.

[0227] The following examples illustrate the application scenarios of the embodiments of the present invention.

[0228] It should be noted that the video processing method provided in this embodiment of the invention can be applied to different application scenarios such as video review, video clip editing, video material pre-extraction, and video annotation. The following description takes the video review scenario and the video clip editing scenario as examples. For the specific content of different application scenarios such as video material pre-extraction or video annotation, please refer to the description of the video review scenario or the video clip editing scenario, which will not be repeated here.

[0229] Scene 1

[0230] The video processing method provided in this invention can be applied to video review scenarios. For example, assuming a user publishes video information through a multimedia social platform client on a terminal (e.g., a smartphone), the terminal sends the video information to a server. After receiving the video information, the server first obtains initial query vector information, multiple initial query point information, and video feature information of the video information. The initial query vector information is used to identify the temporal behavior of the video information, and the initial query point information corresponds to the temporal sequence of the video frames. Then, based on the temporal sequence of the video frames corresponding to each initial query point information, the server obtains multiple temporal offset values ​​corresponding to each initial query point information. Next, based on the multiple temporal offset values ​​corresponding to each initial query point information, the server extracts multiple video frame feature information corresponding to each initial query point information from the video feature information. Then, the server performs weighted calculations on the multiple video frame feature information corresponding to each initial query point information to obtain the query point feature information of each initial query point information. After obtaining the information, the server performs offset prediction on the feature information of each query point to obtain the offset information of each initial query point. Based on the offset information, the server updates the initial query point information to obtain multiple new initial query point information. After obtaining multiple new initial query point information, the server obtains multiple temporal offset values ​​corresponding to each new initial query point information based on the video frame timing. Then, based on these multiple temporal offset values, the server extracts multiple video frame feature information corresponding to each new initial query point information from the video feature information. These multiple video frame feature information are then weighted and calculated to obtain multiple new query point feature information. The server then updates the initial query point information based on these new query point feature information. This process is repeated until the number of updates reaches a preset value, resulting in multiple target query point information. The target query point information corresponds to the temporal sequence of the video frames in the video information.In addition, during the process of acquiring multiple target query point information, the server simultaneously performs feature mixing between the initial query vector information and all query point feature information to obtain the target query vector information. Specifically, when mixing the initial query vector information with all query point feature information to obtain the target query vector information, the server performs variable convolution processing on the initial query vector information and each query point feature information to obtain multiple intermediate feature information. All intermediate feature information is then stacked to obtain query point stacked feature information. This stacked feature information is then mixed with the initial query vector information based on a first dimension to obtain first mixed feature information. Next, the stacked feature information is mixed with the initial query vector information based on a second dimension to obtain second mixed feature information. The first and second mixed feature information are then concatenated and mapped to obtain concatenated feature information. This concatenated feature information is then residually joined with the initial query vector information to obtain new initial query vector information. Finally, the server performs variable convolution processing on the new initial query vector information and each new query point feature information to obtain multiple new intermediate feature information. All new intermediate feature information is then stacked to obtain new query point stacked feature information. The feature information is mixed with the new initial query vector information based on the first dimension to obtain new first mixed feature information. The new query point stacked feature information is then mixed with the new initial query vector information based on the second dimension to obtain new second mixed feature information. The new first mixed feature information and the new second mixed feature information are concatenated and mapped to obtain new concatenated feature information. This concatenated feature information is then residually joined with the new initial query vector information. This process is repeated until the feature mixing reaches a preset number, resulting in the target query vector information. After obtaining the target query vector information and multiple target query point information, the server performs category identification on the temporal behavior based on the target query vector information to obtain the target behavior category. Based on the temporal sequence of the video frames corresponding to all target query point information, the server determines the start and end times of the target behavior category in the video information. After obtaining the target behavior category, start time, and end time, the server retrieves the target video segment from the video information based on the start and end times. Based on the target video segment and the target behavior category, the server performs video review to determine whether the target video segment and the target behavior category belong to permitted content. If the video review is passed, the server publishes the video information.

[0231] Scene 2

[0232] The video processing method provided in this embodiment of the invention can also be applied to video clip editing scenarios. For example, assuming a user extracts behavioral segments from a target video using a video editing application on a terminal (e.g., a smartphone), when the user triggers the behavioral segment extraction button in the video editing application, the terminal sends the target video to a server. After receiving the target video, the server first obtains initial query vector information, multiple initial query point information, and video feature information of the target video. The initial query vector information is used to identify the temporal behavior of the target video, and the initial query point information corresponds to the temporal sequence of the video frames of the target video. Then, based on the temporal sequence of the video frames corresponding to each initial query point information, the server obtains multiple temporal offset values ​​corresponding to each initial query point information. Next, based on the multiple temporal offset values ​​corresponding to each initial query point information, the server extracts multiple video frame feature information corresponding to each initial query point information from the video feature information. Then, the server performs a weighted calculation on the multiple video frame feature information corresponding to each initial query point information to obtain the query point feature information of each initial query point information. After obtaining the query point feature information of each initial query point, the server performs offset prediction on each query point feature information to obtain the offset information of each initial query point. Based on the offset information, the server updates each initial query point information to obtain multiple new initial query point information. After obtaining multiple new initial query point information, the server obtains multiple temporal offset values ​​corresponding to each new initial query point information based on the video frame temporal sequence. Then, based on the multiple temporal offset values, the server extracts multiple video frame feature information corresponding to each new initial query point information from the video feature information. These multiple video frame feature information are then weighted and calculated to obtain multiple new query point feature information. The server then updates each new initial query point information based on these new query point feature information. This process is repeated until the number of updates reaches a preset value, resulting in multiple target query point information. The target query point information corresponds to the temporal sequence of the video frames in the target video.In addition, during the process of acquiring multiple target query point information, the server simultaneously performs feature mixing between the initial query vector information and all query point feature information to obtain the target query vector information. Specifically, when mixing the initial query vector information with all query point feature information to obtain the target query vector information, the server performs variable convolution processing on the initial query vector information and each query point feature information to obtain multiple intermediate feature information. All intermediate feature information is then stacked to obtain query point stacked feature information. This stacked feature information is then mixed with the initial query vector information based on a first dimension to obtain first mixed feature information. Next, the stacked feature information is mixed with the initial query vector information based on a second dimension to obtain second mixed feature information. The first and second mixed feature information are then concatenated and mapped to obtain concatenated feature information. This concatenated feature information is then residually joined with the initial query vector information to obtain new initial query vector information. Finally, the server performs variable convolution processing on the new initial query vector information and each new query point feature information to obtain multiple new intermediate feature information. All new intermediate feature information is then stacked to obtain new query point stacks. The server performs feature mixing on a first-dimensional basis with the new initial query vector information to obtain new first mixed feature information. Then, it performs feature mixing on a second-dimensional basis with the new initial query vector information to obtain new second mixed feature information. The new first and second mixed feature information are then concatenated and mapped to obtain new concatenated feature information. Finally, the concatenated feature information is residually joined with the new initial query vector information. This process is repeated until the feature mixing reaches a preset number, resulting in the target query vector information. After obtaining the target query vector information and multiple target query point information, the server performs category identification on the temporal behavior based on the target query vector information to obtain the target behavior category. Based on the temporal sequence of the video frames corresponding to all target query point information, the server determines the start and end times of the target behavior category in the target video. After obtaining the target behavior category, start time, and end time, the server retrieves the target video segment from the target video based on the start and end times, labels the target video segment with the target behavior category, and then sends the labeled target video segment to the terminal for display.

[0233] It is understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0234] Reference Figure 8 The present invention also discloses a video processing apparatus 800, which is capable of implementing the video processing method as described in the preceding embodiments. The video processing apparatus 800 includes:

[0235] The first acquisition unit 810 is used to acquire the video to be processed;

[0236] The second acquisition unit 820 is used to acquire initial query vector information, multiple initial query point information and video feature information of the video to be processed. The initial query vector information is used to identify the temporal behavior of the video to be processed, and the initial query point information corresponds to the temporal sequence of the video frames of the video to be processed.

[0237] The first processing unit 830 is used to extract multiple video frame feature information from video feature information based on multiple initial query point information, calculate query point feature information of each initial query point information based on each video frame feature information, and update each initial query point information based on each query point feature information to obtain multiple target query point information, wherein the target query point information corresponds to the temporal sequence of video frames in the video to be processed.

[0238] The second processing unit 840 is used to perform feature mixing between the initial query vector information and the feature information of all query points to obtain the target query vector information.

[0239] The behavior recognition unit 850 is used to perform category recognition on temporal behavior based on target query vector information to obtain the target behavior category;

[0240] The time determination unit 860 is used to determine the start and end times of the target behavior category in the video to be processed based on the video frame timing corresponding to all target query point information.

[0241] In one embodiment, the first processing unit 830 is further configured to:

[0242] Based on the video frame timing corresponding to each initial query point, obtain multiple timing offset values ​​corresponding to each initial query point.

[0243] Based on the multiple temporal offset values ​​corresponding to each initial query point information, extract multiple video frame feature information corresponding to each initial query point information from the video feature information;

[0244] The feature information of multiple video frames corresponding to each initial query point is weighted and calculated to obtain the query point feature information of each initial query point.

[0245] In one embodiment, the first processing unit 830 is further configured to:

[0246] Offset prediction is performed on the feature information of each query point to obtain the offset information of each initial query point.

[0247] The information of each initial query point is updated based on the offset information of each initial query point to obtain multiple target query point information.

[0248] In one embodiment, the second processing unit 840 is further configured to:

[0249] The initial query vector information and the feature information of each query point are subjected to variable convolution to obtain multiple intermediate feature information.

[0250] All intermediate feature information is stacked to obtain the query point stacked feature information;

[0251] The stacked feature information of the query points is mixed with the initial query vector information to obtain cascaded feature information;

[0252] The concatenated feature information is residually joined with the initial query vector information to obtain the target query vector information.

[0253] In one embodiment, the second processing unit 840 is further configured to:

[0254] The query point stacked feature information and the initial query vector information are mixed based on the first dimension to obtain the first mixed feature information;

[0255] The query point stacked feature information and the initial query vector information are mixed based on the second dimension to obtain the second mixed feature information;

[0256] The first and second mixed feature information are concatenated and mapped to obtain concatenated feature information.

[0257] In one embodiment, the first processing unit 830 is further configured to:

[0258] The initial query point information is updated based on the feature information of each query point, resulting in multiple new initial query point information.

[0259] Extract multiple new video frame feature information from the video feature information based on multiple new initial query point information;

[0260] Multiple new query point feature information is calculated based on the feature information of multiple new video frames;

[0261] The information of each new initial query point is updated based on the feature information of each new query point until the number of updates reaches a preset value, thus obtaining information on multiple target query points.

[0262] In one embodiment, the second processing unit 840 is further configured to:

[0263] The initial query vector information is mixed with the feature information of all query points to obtain a new initial query vector information;

[0264] The new initial query vector information is mixed with all new query point feature information until the number of feature mixing reaches a preset value, and then the target query vector information is obtained.

[0265] It should be noted that since the video processing device 800 of this embodiment can implement the video processing method of the previous embodiment, the video processing device 800 of this embodiment and the video processing method of the previous embodiment have the same technical principle and the same beneficial effect. In order to avoid repetition, it will not be described again here.

[0266] Reference Figure 9 This invention also discloses a video detection model training device 900, which can implement the video detection model training method as described in the previous embodiments. The video detection model training device 900 includes:

[0267] The third acquisition unit 910 is used to acquire video samples and label information;

[0268] The fourth acquisition unit 920 is used to acquire video feature information of query vector samples, multiple query point samples and video samples. The query vector samples are used to identify the temporal behavior of the video samples, and the query point samples correspond to the video frames of the video samples in a temporal sequence.

[0269] The model prediction unit 930 is used to input query vector samples, multiple query point samples and video feature information into the video detection model to predict behavior and obtain the prediction start time, prediction end time and prediction behavior category.

[0270] The parameter correction unit 940 is used to correct the parameters of the video detection model based on the prediction start time, prediction end time, prediction behavior category and label information.

[0271] The model prediction unit 930 is also used for:

[0272] Multiple video frame feature information is extracted from the video feature information based on multiple query point samples. The query point feature information of each query point sample is calculated based on the feature information of each video frame. The query point samples are updated based on the feature information of each query point to obtain multiple target query point information. The target query point information corresponds to the temporal sequence of the video frame in the video sample.

[0273] The query vector sample is mixed with the feature information of all query points to obtain the target query vector information;

[0274] Based on the target query vector information, the temporal behavior is classified to obtain the predicted behavior category;

[0275] The prediction start and end times of the predicted behavior category in the video sample are determined based on the temporal sequence of the video frames corresponding to all target query point information.

[0276] In one embodiment, the model prediction unit 930 is further configured to:

[0277] Based on the video frame timing corresponding to each query point sample, obtain multiple timing offset values ​​corresponding to each query point sample;

[0278] Based on the multiple temporal offset values ​​corresponding to each query point sample, extract multiple video frame feature information corresponding to each query point sample from the video feature information;

[0279] The feature information of multiple video frames corresponding to each query point sample is weighted and calculated to obtain the query point feature information of each query point sample.

[0280] In one embodiment, the model prediction unit 930 is further configured to:

[0281] Offset prediction is performed on the feature information of each query point to obtain the offset information of each query point sample.

[0282] The offset information of each query point sample is used to update each query point sample, resulting in multiple target query point information.

[0283] In one embodiment, the model prediction unit 930 is further configured to:

[0284] The query vector samples are subjected to variable convolution with the feature information of each query point to obtain multiple intermediate feature information.

[0285] All intermediate feature information is stacked to obtain the query point stacked feature information;

[0286] The stacked feature information of the query points is mixed with the query vector samples to obtain cascaded feature information.

[0287] The cascaded feature information is residually joined with the query vector samples to obtain the target query vector information.

[0288] In one embodiment, the model prediction unit 930 is further configured to:

[0289] The query point stacked feature information and the query vector sample are mixed based on the first dimension to obtain the first mixed feature information;

[0290] The query point stacked feature information is mixed with the query vector sample based on the second dimension to obtain the second mixed feature information;

[0291] The first and second mixed feature information are concatenated and mapped to obtain concatenated feature information.

[0292] In one embodiment, the model prediction unit 930 is further configured to:

[0293] The query point samples are updated based on the feature information of each query point to obtain multiple new query point samples;

[0294] Extract multiple new video frame feature information from the video feature information based on multiple new query point samples;

[0295] Multiple new query point feature information is calculated based on the feature information of multiple new video frames;

[0296] The new query point samples are updated based on the feature information of each new query point until the number of updates reaches a preset value, thus obtaining information on multiple target query points.

[0297] In one embodiment, the model prediction unit 930 is further configured to:

[0298] The query vector sample is feature-blended with the feature information of all query points to obtain a new query vector sample;

[0299] The new query vector sample is mixed with the feature information of all new query points until the number of feature mixing reaches a preset value, and the target query vector information is obtained.

[0300] In one embodiment, the tag information includes a start time tag, an end time tag, and a behavior category tag; the parameter correction unit 940 is further configured to:

[0301] The positioning loss value is calculated based on the predicted start time, predicted end time, start time label, and end time label.

[0302] The classification loss value is calculated based on the predicted behavior category and the behavior category label;

[0303] The parameters of the video detection model are adjusted based on the localization loss value and the classification loss value.

[0304] It should be noted that since the video detection model training device 900 of this embodiment can implement the video detection model training method of the previous embodiment, the video detection model training device 900 of this embodiment and the video detection model training method of the previous embodiment have the same technical principle and the same beneficial effect. In order to avoid repetition, it will not be described again here.

[0305] Reference Figure 10 The present invention also discloses an electronic device, the electronic device 1000 comprising:

[0306] At least one processor 1001;

[0307] At least one memory 1002 is used to store at least one program;

[0308] When at least one program is executed by at least one processor 1001, it implements the video processing method as described above, or implements the video detection model training method as described above.

[0309] This invention also discloses a computer-readable storage medium storing a processor-executable computer program. When executed by a processor, the processor-executable computer program is used to implement the video processing method described above, or to implement the video detection model training method described above.

[0310] This invention also discloses a computer program product, including a computer program or computer instructions, which are stored in a computer-readable storage medium. The processor of an electronic device reads the computer program or computer instructions from the computer-readable storage medium and executes the computer program or computer instructions, causing the electronic device to perform the video processing method described above, or to perform the video detection model training method described above.

[0311] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0312] It should be understood that in this invention, "at least one (item)" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0313] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, apparatuses, or units, and may be electrical, mechanical, or other forms.

[0314] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0315] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0316] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0317] The step numbers in the above method embodiments are set only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

Claims

1. A video processing method, characterized in that, Includes the following steps: Get the video to be processed; The process involves acquiring initial query vector information, multiple initial query point information, and video feature information of the video to be processed. The initial query vector information refers to the numerical information of the query vector obtained through initial randomization, and the initial query point information refers to the numerical information of the query points obtained through initial randomization. The initial query vector information is selected or input by the user and is used to identify the temporal behavior of the video to be processed. The initial query point information is selected by the user, input by the user, randomly generated by the server, or randomly generated by the terminal. Based on the multiple initial query point information, multiple video frame feature information is extracted from the video feature information. Query point feature information of each initial query point information is calculated based on each video frame feature information. Offset prediction is performed on each query point feature information to obtain offset information of each initial query point information. The initial query point information is updated based on the offset information of each initial query point information to obtain multiple target query point information. The initial query vector information is mixed with the feature information of all the query points to obtain the target query vector information. The target query vector information is the final query vector information obtained by the video detection model after processing the initial query vector information. Based on the target query vector information, the temporal behavior is classified to obtain the target behavior category; The start and end times of the target behavior category in the video to be processed are determined based on the video frame timing corresponding to all the target query point information.

2. The video processing method according to claim 1, characterized in that, The step of extracting multiple video frame feature information from the video feature information based on multiple initial query point information, and calculating query point feature information for each of the initial query point information based on each of the video frame feature information, includes: Based on the video frame timing corresponding to each of the initial query point information, obtain multiple timing offset values ​​corresponding to each of the initial query point information; Based on the multiple temporal offset values ​​corresponding to each of the initial query point information, multiple video frame feature information corresponding to each of the initial query point information is extracted from the video feature information; The query point feature information of each initial query point is obtained by weighting and calculating the feature information of multiple video frames corresponding to each initial query point.

3. The video processing method according to claim 1, characterized in that, The step of mixing the initial query vector information with the feature information of all query points to obtain the target query vector information includes: The initial query vector information and the feature information of each query point are subjected to variable convolution processing to obtain multiple intermediate feature information; All the intermediate feature information is stacked to obtain the query point stacked feature information; The stacked feature information of the query points is mixed with the initial query vector information to obtain cascaded feature information; The concatenated feature information and the initial query vector information are residually joined to obtain the target query vector information.

4. The video processing method according to claim 3, characterized in that, The step of mixing the stacked feature information of the query points with the initial query vector information to obtain cascaded feature information includes: The query point stacking feature information and the initial query vector information are mixed based on the first dimension to obtain the first mixed feature information; The query point stacking feature information and the initial query vector information are mixed based on the second dimension to obtain the second mixed feature information; The first mixed feature information and the second mixed feature information are concatenated and mapped to obtain concatenated feature information.

5. The video processing method according to claim 1, characterized in that, The step of mixing the initial query vector information with the feature information of all query points to obtain the target query vector information includes: The initial query vector information is mixed with the feature information of all the query points to obtain a new initial query vector information; The new initial query vector information is mixed with all new query point feature information until the number of feature mixing reaches a preset value, and then the target query vector information is obtained.

6. A video detection model training method, characterized in that, Includes the following steps: Obtain video samples and tag information; The system acquires query vector samples, multiple query point samples, and video feature information of the video samples. The query vector samples refer to the numerical information of the query vectors obtained through initial randomization, and the query point samples refer to the numerical information of the query points obtained through initial randomization. The query vector samples are selected or input by the user and are used to identify the temporal behavior of the video samples. The query point samples are selected by the user, input by the user, randomly generated by the server, or randomly generated by the terminal. The query vector sample, multiple query point samples, and video feature information are input into the video detection model to perform behavior prediction, and the prediction start time, prediction end time, and prediction behavior category are obtained. The parameters of the video detection model are corrected based on the prediction start time, the prediction end time, the prediction behavior category, and the label information. The process of behavior prediction performed by the video detection model includes: Multiple video frame feature information is extracted from the video feature information based on multiple query point samples. Query point feature information of each query point sample is calculated based on each video frame feature information. Offset prediction is performed on each query point feature information to obtain offset information of each query point sample. Each query point sample is updated based on its offset information to obtain multiple target query point information. The query vector sample is mixed with the feature information of all the query points to obtain the target query vector information. The target query vector information is the final query vector information obtained by the video detection model after processing the initial query vector information. Based on the target query vector information, the temporal behavior is classified to obtain the predicted behavior category; The prediction start time and prediction end time of the prediction behavior category in the video sample are determined based on the video frame time sequence corresponding to all the target query point information.

7. The video detection model training method according to claim 6, characterized in that, The step of extracting multiple video frame feature information from the video feature information based on multiple query point samples, and calculating the query point feature information of each query point sample based on each video frame feature information, includes: Based on the video frame timing corresponding to each query point sample, obtain multiple timing offset values ​​corresponding to each query point sample; Based on the multiple temporal offset values ​​corresponding to each query point sample, extract multiple video frame feature information corresponding to each query point sample from the video feature information; The query point feature information of each query point sample is obtained by weighting and calculating the feature information of the multiple video frames corresponding to each query point sample.

8. The video detection model training method according to claim 6, characterized in that, The step of feature mixing the query vector sample with all the query point feature information to obtain the target query vector information includes: The query vector sample is subjected to variable convolution with the feature information of each query point to obtain multiple intermediate feature information; All the intermediate feature information is stacked to obtain the query point stacked feature information; The stacked feature information of the query points is mixed with the query vector sample to obtain cascaded feature information; The concatenated feature information is residually joined with the query vector sample to obtain the target query vector information.

9. The video detection model training method according to claim 8, characterized in that, The step of mixing the stacked feature information of the query points with the query vector samples to obtain cascaded feature information includes: The query point stacked feature information and the query vector sample are mixed based on the first dimension to obtain the first mixed feature information; The query point stacked feature information is mixed with the query vector sample based on the second dimension to obtain the second mixed feature information; The first mixed feature information and the second mixed feature information are concatenated and mapped to obtain concatenated feature information.

10. The video detection model training method according to claim 6, characterized in that, The step of feature mixing the query vector sample with all the query point feature information to obtain the target query vector information includes: The query vector sample is feature-blended with the feature information of all the query points to obtain a new query vector sample; The new query vector sample is mixed with all new query point feature information until the number of feature mixing reaches a preset value, and the target query vector information is obtained.

11. The video detection model training method according to claim 6, characterized in that, The tag information includes a start time tag, an end time tag, and a behavior category tag; The step of correcting the parameters of the video detection model based on the prediction start time, the prediction end time, the prediction behavior category, and the label information includes: The positioning loss value is calculated based on the predicted start time, the predicted end time, the start time label, and the end time label; The classification loss value is calculated based on the predicted behavior category and the behavior category label; The parameters of the video detection model are adjusted based on the localization loss value and the classification loss value.

12. A video processing apparatus, characterized in that, include: The first acquisition unit is used to acquire the video to be processed; The second acquisition unit is used to acquire initial query vector information, multiple initial query point information, and video feature information of the video to be processed. The initial query vector information refers to the numerical information of the query vector obtained by initial randomization, and the initial query point information refers to the numerical information of the query point obtained by initial randomization. The initial query vector information is selected or input by the user. The initial query vector information is used to identify the temporal behavior of the video to be processed. The initial query point information is selected by the user, input by the user, randomly generated by the server, or randomly generated by the terminal. The first processing unit is configured to extract multiple video frame feature information from the video feature information based on multiple initial query point information, calculate query point feature information of each initial query point information based on each video frame feature information, and perform offset prediction on each query point feature information to obtain offset information of each initial query point information; and update each initial query point information based on the offset information of each initial query point information to obtain multiple target query point information. The second processing unit is used to perform feature mixing between the initial query vector information and all the query point feature information to obtain target query vector information. The target query vector information is the final query vector information obtained by the video detection model after processing the initial query vector information. A behavior recognition unit is used to perform category recognition on the temporal behavior based on the target query vector information to obtain the target behavior category; The time determination unit is used to determine the start and end times of the target behavior category in the video to be processed based on the video frame timing corresponding to all the target query point information.

13. A video detection model training device, characterized in that, include: The third acquisition unit is used to acquire video samples and label information; The fourth acquisition unit is used to acquire query vector samples, multiple query point samples, and video feature information of the video sample. The query vector sample refers to the numerical information of the query vector obtained by initial randomization, and the query point sample refers to the numerical information of the query point obtained by initial randomization. The query vector sample is selected or input by the user and is used to identify the temporal behavior of the video sample. The query point sample is selected by the user, input by the user, randomly generated by the server, or randomly generated by the terminal. The model prediction unit is used to input the query vector sample, multiple query point samples and the video feature information into the video detection model to perform behavior prediction and obtain the prediction start time, prediction end time and prediction behavior category. The parameter correction unit is used to correct the parameters of the video detection model based on the prediction start time, the prediction end time, the prediction behavior category, and the label information. The model prediction unit is further used for: Multiple video frame feature information is extracted from the video feature information based on multiple query point samples. Query point feature information of each query point sample is calculated based on each video frame feature information. Offset prediction is performed on each query point feature information to obtain offset information of each query point sample. Each query point sample is updated based on its offset information to obtain multiple target query point information. The query vector sample is mixed with the feature information of all the query points to obtain the target query vector information. The target query vector information is the final query vector information obtained by the video detection model after processing the initial query vector information. Based on the target query vector information, the temporal behavior is classified to obtain the predicted behavior category; The prediction start time and prediction end time of the prediction behavior category in the video sample are determined based on the video frame time sequence corresponding to all the target query point information.

14. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When at least one of the programs is executed by at least one of the processors, it implements the video processing method as described in any one of claims 1 to 5, or the video detection model training method as described in any one of claims 6 to 11.

15. A computer-readable storage medium, characterized in that, It stores a processor-executable computer program, which, when executed by the processor, is used to implement the video processing method as described in any one of claims 1 to 5, or to implement the video detection model training method as described in any one of claims 6 to 11.

16. A computer program product, comprising a computer program or computer instructions, characterized in that, The computer program or the computer instructions are stored in a computer-readable storage medium. The processor of the electronic device reads the computer program or the computer instructions from the computer-readable storage medium. The processor executes the computer program or the computer instructions, causing the electronic device to perform the video processing method as described in any one of claims 1 to 5, or to implement the video detection model training method as described in any one of claims 6 to 11.

Images