Video processing method and device, terminal equipment and computer readable storage medium

Abstract

The invention provides a video processing method and apparatus, a terminal device and a computer readable storage medium. The method comprises the steps of obtaining a to-be-shielded target in a to-be-processed video; determining a target position of the to-be-shielded target in each video frame in the to-be-processed video; and performing shielding processing on the target position in each video frame to obtain a processed target video. According to the video processing method provided by the invention, the to-be-shielded target of the user for one video can be obtained, the position of the to-be-shielded target in the whole video is determined, and finally the to-be-shielded target in the whole video is shielded. Namely, the target specified by the user in the video can be shielded, the flexibility of video shielding is improved, and the shielding experience of the user can be better improved.

Description

Technical Field

[0001] The present application relates to computer technology, and in particular to a video processing method, apparatus, terminal device and computer-readable storage medium. Background Art

[0002] With the popularity of smart TVs and smart conference tablets, user demands such as video conversations and short video recording have gradually become standard. On the one hand, the camera function of large-screen displays such as TVs and conference tablets greatly facilitates online communication in daily life and work and social networking. On the other hand, users also need some functions to protect their own privacy information and hide some targets that are inconvenient to display in video previews.

[0003] Recently, with the development of artificial intelligence technology, some related technologies and functions have emerged, such as road scene segmentation for autonomous driving and portrait segmentation in live video. However, these semantic segmentation-based object separation / erasing technologies have obvious defects, that is, the target categories that need to be separated / erased, such as roads, portraits, etc., need to be determined in advance. In fact, the targets that users want to erase or separate vary from person to person and from scene to scene, and it is difficult to determine the user's true intention to erase or separate in advance. Summary of the invention

[0004] The present application provides a video processing method, which can shield targets matching the user's needs in the entire video according to the user's needs, thereby improving the flexibility of shielding.

[0005] In a first aspect, the present application provides a video processing method, the method comprising:

[0006] Obtain the target to be blocked in the video to be processed;

[0007] Determine the target position of the target to be shielded in each video frame of the video to be processed;

[0008] The target position in each of the video frames is shielded to obtain a processed target video.

[0009] In a second aspect, the present application further provides a video processing device, the device comprising:

[0010] An acquisition module, used for acquiring the target to be shielded in the video to be processed;

[0011] A determination module, used to determine the target position of the target to be shielded in each video frame of the video to be processed;

[0012] The processing module is used to perform masking processing on the target position in each of the video frames to obtain a processed target video.

[0013] In a third aspect, the present application also provides a terminal device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in any one of the video processing methods described.

[0014] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the steps in any one of the video processing methods.

[0015] The video processing method provided by the present application can obtain the target to be blocked by the user for a video, determine the position of the target to be blocked in the entire video, and finally block the target to be blocked in the entire video. That is, the present application can block the target specified by the user in the video, improve the flexibility of video blocking, and better improve the user's blocking experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 is a schematic diagram of a scenario of a video processing system provided in an embodiment of the present application;

[0018] Figure 2 This is a flow chart of an embodiment of a video processing method in an embodiment of the present application;

[0019] Figure 3 This is a schematic diagram of a model structure of an embodiment of a video processing method in an embodiment of the present application;

[0020] Figure 4 This is a schematic diagram of a model structure of an embodiment of a video processing method in an embodiment of the present application;

[0021] Figure 5 This is a schematic diagram of a model structure of an embodiment of a video processing method in an embodiment of the present application;

[0022] Figure 6 This is a schematic diagram of a model structure of an embodiment of a video processing method in an embodiment of the present application;

[0023] Figure 7 This is a schematic diagram of a model structure of an embodiment of a video processing method in an embodiment of the present application;

[0024] Figure 8 This is a schematic diagram of a model structure of an embodiment of a video processing method in an embodiment of the present application;

[0025] Fig. 9 This is a schematic diagram of a model structure of an embodiment of a video processing method in an embodiment of the present application;

[0026] Fig.10 is a schematic diagram of a functional module of a video processing device in an embodiment of the present application;

[0027] Fig.11 It is a schematic diagram of the structure of the terminal device in the embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0029] In the description of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the feature. In the description of the present application, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0030] In this application, the word "exemplary" is used to mean "used as an example, illustration or description". Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. At the same time, it is to be understood that in the specific implementation of this application, when user information, user data and other related data are involved, when the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data shall comply with relevant laws, regulations and standards of relevant countries and regions.

[0031] In order to enable any person skilled in the art to implement and use the present application, the following description is provided. In the following description, details are listed for the purpose of explanation. It should be understood that those of ordinary skill in the art will recognize that the present application can be implemented without using these specific details. In other examples, known structures and processes will not be elaborated in detail to avoid unnecessary details that make the description of the present application obscure. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest range of principles and features disclosed in the present application.

[0032] The present application provides a video processing method, apparatus, device and storage medium, which are described in detail below.

[0033] See also Figure 1 , Figure 1 Schematic diagram of a video processing system provided in an embodiment of the present application. The video processing system may include a terminal device 100 and a storage device 200. The storage device 200 may transmit data to the terminal device 100. Figure 1 The terminal device 100 in the storage device 200 can obtain the video data stored in the storage device 200 and the shielding information provided by the user to execute the video processing method in the present application.

[0034] In the embodiment of the present application, the terminal device 100 includes but is not limited to a desktop computer, a portable computer, a network server, a PDA (Personal Digital Assistant), a tablet computer, a wireless terminal device, an embedded device, etc.

[0035] In an embodiment of the present application, communication between the terminal device 100 and the storage device 200 can be achieved through any communication method, including but not limited to mobile communications based on the 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), and Worldwide Interoperability for Microwave Access (WiMAX), or computer network communications based on the TCP / IP Protocol Suite (TCP / IP) and User Datagram Protocol (UDP).

[0036] It should be noted that Figure 1 The scene diagram of the video processing system shown is only an example. The video processing system and scene described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the video processing system and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0037] like Figure 2 As shown, Figure 2 This is a flow chart of an embodiment of a video processing method in an embodiment of the present application. The video processing method may include the following steps 201 to 203:

[0038] 201. Obtain a target to be shielded in the video to be processed.

[0039] In an embodiment of the present application, the video to be processed can be any online video or any offline type of video, such as videos in formats such as avi, rmvb, mp4, etc. Among them, the method for obtaining the target to be shielded can be obtained in a method provided by the user. For example: the user can pause the video being played, and then enter the video shielding mode through the interactive area on the video or the interactive virtual button. After entering the shielding mode, the user can be prompted to select the target to be shielded on the paused video screen, including: the user uses the hand to outline a target object on the current video frame, so that the specific target to be shielded can be determined based on the target outlined by the user.

[0040] Specifically, assuming that a user outlines a picture in the currently paused video, at this time, the picture outlined by the user can be directly used as the target to be blocked.

[0041] 202. Determine a target position of the target to be shielded in each video frame of the video to be processed.

[0042] According to the above embodiment, the user specifies that the target to be shielded is the target to be shielded specified for the current video frame. In order to meet the user's shielding needs, it is also necessary to shield the target to be shielded in the entire video. Therefore, in the embodiment of the present application, the target to be shielded can be found in all video frames after the current video frame. For example, after determining the target to be shielded, the image feature extraction of the target to be shielded can be performed; however, the online video playback is to cache the video first and then play it, or if the video playback is a local video playback, the corresponding video file is stored locally. Therefore, based on the image features of the target to be shielded, the video frames subsequent to the current video frame can be searched in the cached video or the locally stored video for objects similar or identical to the target to be shielded; after finding similar or identical objects, the pixel coordinates of the similar or identical objects in each video frame are determined, and the target position of the target to be shielded in each video frame can be determined.

[0043] It should be noted that, in the embodiment of the present application, when extracting image features of the target to be shielded, any neural network model can be used, and it is only necessary to select it according to the actual situation. For example: if the terminal device is a lightweight device such as a mobile phone, the neural network model can be a lightweight model such as MobileNetV1. Afterwards, after the image features of the target to be shielded are obtained, the image features of different video areas in each subsequent video frame can be extracted to obtain the image features of different areas of each video frame, and then the image features of the target to be shielded are compared with the image features of different areas of each video frame to determine the similarity. Among them, the method for determining the similarity can include obtaining the similarity between features according to the Euclidean distance and cosine similarity. Next, the highest similarity among the similarities corresponding to each image area in each video frame is selected and compared with the preset similarity threshold. If the highest similarity does not exceed the similarity threshold, it is determined that there is no target to be shielded in the corresponding video frame; if the highest similarity exceeds the similarity threshold, the position of the corresponding image area in the corresponding video frame can be determined as the target position.

[0044] 203. Perform masking processing on the target position in each video frame to obtain a processed target video.

[0045] After the target position of the target to be shielded in each video frame is determined according to the above steps, the shielding processing method for each target position may include: adding mosaics at the target position, modifying the pixel color of the pixel point at the target position, etc., so as to obtain the processed target video.

[0046] The video processing method provided by the present application can obtain the target to be blocked by the user for a video, determine the position of the target to be blocked in the entire video, and finally block the target to be blocked in the entire video. That is, the present application can block the target specified by the user in the video, improve the flexibility of video blocking, and better improve the user's blocking experience.

[0047] In order to better implement the embodiments of the present application, in one embodiment of the present application, obtaining a target to be shielded in a video to be processed includes:

[0048] Obtain input shielding information of the video to be processed; parse the shielding information to obtain the target to be shielded.

[0049] In the above embodiment, a method is provided in which a user directly outlines the target to be shielded on the paused video screen.

[0050] However, in actual situations, the method of determining the target to be shielded includes not only the method of outlining, but also includes: the user clicks on the coordinates (one or more coordinate points); draws an image frame to select an area, or draws a circle (irregular outline), etc.; or, the interactive method is a gesture operation in the air or a remote control button operation, or the target position information obtained by the user's voice through voice recognition. The typical method of remote control operation is: divide the screen into grids, and the user selects the grid where the target is to be removed. The method of voice recognition can be: the user inputs voice information including "shield the painting in the upper right corner of the screen", and then the entire video can be divided into the upper right area, the lower right area, the upper left area and the lower left area according to the user's voice information, and then the upper right area is image recognized to find an object matching the painting. Therefore, the voice information provided by the user can be regarded as the shielding information provided by the user. The process of determining the target to be shielded based on the user's voice information is the process of parsing the shielding information and obtaining the target to be shielded.

[0051] However, compared with the outline outlining method, there are some errors in determining the target to be shielded that the user wants to shield according to the above-provided interactive method. For example: when the user selects the target to be shielded through the image frame, there are at least two situations, the first one: the user's image frame completely includes the target object and other video content; and the second one: the user's image frame does not completely include the target object that the user wants to shield. Therefore, for the above two situations, the terminal device can perform image recognition on the video content in the image frame to determine what kind of object the user's image frame includes. For example: image recognition is performed on the image content in the image frame, and then it can be determined which specific object the target object included in the image frame is, that is, the target to be shielded is obtained. Therefore, the image frame provided by the user is the shielding information provided by the user. The process of performing image recognition on the image content in the image frame is the process of parsing the shielding information and obtaining the target to be shielded. Therefore, if the interactive method adopted is different, the corresponding parsing method is also different, and it can be specifically set according to the actual situation, and the embodiment of the present application is not limited.

[0052] In order to better implement the embodiment of the present application, in one embodiment of the present application, the shielding information includes a selected area in a target video frame in a video frame to be processed, and the shielding information is parsed to obtain a target to be shielded, including:

[0053] Obtain coding features of a target video frame; perform label prediction on the coding features to obtain predicted labels; obtain the confidence of each mask in a mask image corresponding to the selected area based on the coding features, predicted labels, and the selected area; obtain the target to be shielded based on the confidence of each mask in the mask image and a preset first confidence threshold.

[0054] The above embodiment provides a solution for determining the target to be shielded by image recognition or by voice analysis. However, if the method of directly identifying the object in the image by image recognition is used, the effect may be poor. Therefore, in order to improve the accuracy of determining the target to be shielded according to the information provided by the user, the embodiment of the present application also provides a method, which specifically needs to pre-encode the video to be processed, and provide basic image features for the subsequent user-specified shielding target and the user-specified target removal in the video frame. Specifically, the embodiment of the present application can pre-encode the target video frame by any encoder, and the encoder is a trained encoder. Among them, the method of training the encoder can be trained in an unsupervised manner, for example, the encoder is trained unsupervised by clustering, including: K-means, hierarchical clustering, DBSCAN, spectral clustering and other methods. Specifically, K-means, hierarchical clustering, DBSCAN, spectral clustering and other methods are prior arts, which will not be repeated here. The encoder that has completed the training can classify each object in the target video frame, and then when combined with the shielding information provided by the user, it can more effectively determine the specific target to be shielded corresponding to the user shielding information.

[0055] The process of obtaining the encoding features through the trained encoder is as follows: For example, the input target video frame can be defined as X in , obtain Q, K, V through linear transformation FC, that is: Q, K, V = FC (X in ). After that, the local features are aggregated through the depth-wise separable convolution DWconv, that is: V a =DWcon(V). Since the proportion of the user-defined masked target relative to the video screen is small, this scene feature requires the network to have a strong local context perception ability. Therefore, an attention mechanism can be designed to strengthen the local features after aggregation, as shown in the following formula:

[0056] Q l =DWconv(Q)

[0057] K l =DWconv(K)

[0058] A t =FC(Switch(FC(Q l ⊙K l ))

[0059]

[0060] X l =V a ⊙Att

[0061] Among them, ⊙ is the multiplication of the corresponding elements of the tensor, d is the number of channels of the token, and X l To enhance the encoding features of the target video frame.

[0062] Specifically, Figure 3 As shown, Image Embedding (encoded features of the target video frame), Prompt Embedding (user prompt information, i.e., image frame) and Output Embedding (predicted label) are embedded in the new image and upsampled to the original image resolution through transposed convolution to generate an image mask (mask output by Dot product), and the confidence score (Iou score) of each mask is obtained through MLP (Multilayer Perceptron). Since the mask image includes various masks. Therefore, according to the target mask corresponding to the confidence greater than the preset first confidence threshold, the actual mask image of the selected area is determined to obtain the target to be shielded, which may include: filtering out the confidence less than the first confidence threshold through a preset first confidence threshold, thereby filtering out part of the mask in the mask image. Furthermore, the mask image that filters out part of the mask can represent the target to be shielded specified by the user.

[0063] It should be noted that the concept of confidence is: In statistics, the confidence interval of a probability sample is an interval estimate of a population parameter of the sample. The confidence interval shows the degree to which the true value of the parameter has a certain probability of falling around the measurement result. The confidence interval gives the range of the degree of credibility of the measured value of the measured parameter, that is, the "certain probability" required above. This probability is called confidence.

[0064] In order to better implement the embodiment of the present application, in one embodiment of the present application, obtaining the encoding features of the target video frame includes:

[0065] Obtain a preset target encoder; combine the target encoder with the attention convolution layer to encode each video frame of the video to be processed to obtain the encoding features of the target video frame.

[0066] The above embodiment provides a solution for obtaining the coding features corresponding to the target video frame, that is, the coding features corresponding to the currently paused current video frame, through an encoder. The embodiment of the present application also provides a structure of the encoder.

[0067] Specifically, in order to efficiently perform user-defined extraction of to-be-shielded targets on embedded terminals represented by smart TVs, it is necessary to design a computationally friendly image feature encoder. On embedded platforms, in order to reduce the huge amount of computation of Tansformer (processing model), MobileVIT (a lightweight general-purpose visualization Transformer for mobile devices) is usually selected as the image feature encoder. MobileVIT uses standard convolution operations as local feature extraction operators, and its performance in extracting local high-frequency information is unsatisfactory, resulting in the need to superimpose multiple VITBlocks, which significantly brings unnecessary computation. In extracting the target intention that the user wants to shield, it is necessary not only to pay attention to the positional relationship of the target in the whole image, but also to require local texture information to extract the precise boundary of the target. Therefore, an attention convolution module can be designed to replace the original VITBlock local information extraction structure, improve the ability to extract local features, reduce the number of subsequent vitblocks, and thus reduce the amount of computation.

[0068] Among them, the video encoding module used to obtain the encoding features of the target video frame is mainly composed of multiple groups of ViT Block. The user target extraction task requires the extraction of rich local features. In order to improve the local representation ability of the general ViT Block, the original shallow local feature extraction network structure is improved, that is, the Attention Conv (attention layer) is used instead of the original ordinary convolution to improve the learning capacity of the shallow network. Therefore, in the embodiment of the present application, the target encoder can be MobileVIT, and after the ordinary convolution layer in the MobileVIT model is replaced by Attention Conv, each video frame of the video to be processed can be encoded. Among them, Figure 4 As shown, Figure 4 The following figure shows a model structure of a target encoder in which an Attention Conv (attention layer) replaces the original ordinary convolution. The structure of the Attention Conv convolution layer is as follows: Figure 5 As shown; the network structure of ViT Block is as follows Figure 6 shown.

[0069] In order to better implement the embodiment of the present application, in one embodiment of the present application, determining the target position of the target to be shielded in each video frame in the video to be processed includes:

[0070] Generate a target mask image corresponding to the target to be shielded in each video frame; perform a first tracking on the target to be shielded in each video frame according to each target mask image, and obtain the initial target position feature of the target to be shielded in each video frame; perform a second tracking on the target to be shielded in each video frame according to the initial target position feature of the target to be shielded in each video frame, and obtain the target position of the target to be shielded in each video frame in the video to be processed.

[0071] The above embodiment provides a method of determining whether there is an image area identical or similar to the target to be shielded in each video frame by similarity to complete the shielding implementation method. In order to better determine the position of the target to be shielded in each video frame, the embodiment of the present application also provides a method of determining the position of the target to be shielded in each video frame by tracking.

[0072] According to the above embodiment, a target mask image can be generated for a video frame according to the user's shielding information, so as to determine the target to be shielded according to the mask image. At this time, the target to be shielded can be first tracked by the optical flow method, and the specific process includes: image preprocessing: preprocessing the input continuous video frame sequence, such as denoising, image enhancement, etc. Feature extraction: extract feature points, such as corner points, edges, etc. from the target to be shielded in the image. Feature matching: match feature points between adjacent frames to establish a corresponding relationship. Optical flow calculation: using the matched feature point pairs, calculate the displacement vector (optical flow vector) of the feature point through a certain optical flow estimation algorithm. Motion estimation and filtering: estimate the movement direction and speed in the object or scene according to the optical flow vector. The optical flow can be smoothed using a filtering method to reduce the influence of noise and outliers. Optical flow visualization: visualize the optical flow vector as an arrow, color coding, etc., to display the motion pattern in the object or scene, so as to perform motion analysis to achieve the first tracking. The result obtained by tracking according to the optical flow method is the initial target position feature.

[0073] Afterwards, the position and the corresponding bounding box corresponding to the initial position feature can be determined in the video frame by using a target detection algorithm, and then the movement of the initial target position between consecutive frames can be tracked using a tracking algorithm. Common algorithms for target detection and tracking include Kalman filters, particle filters, multi-target tracking, etc., which are not specifically limited here. The target detection and tracking algorithm is the second tracking of an embodiment of the present application, and the tracking result of the target detection and tracking algorithm is the target position of the target to be shielded in each video frame of the video to be processed in the embodiment of the present application.

[0074] In order to better implement the embodiment of the present application, in one embodiment of the present application, according to each target mask image, a first tracking is performed on the target to be shielded in each video frame to obtain the initial target position feature of the target to be shielded in each video frame, including:

[0075] Each video frame and the target mask image corresponding to each video frame are input into a preset tracking decoder to obtain the initial target position features of the target to be masked in each video frame.

[0076] The above embodiment provides an implementation method of implementing the first tracking by optical flow method. In order to better achieve the effect of the first tracking, the embodiment of the present application also provides a method of tracking by a tracking decoder.

[0077] Specifically, Figure 7 As shown, Figure 7 The architecture of the tracking decoder is shown. Figure 7 In the above example, object query is the target mask image corresponding to the video frame, and Frame Embedding is the video frame corresponding to the mask image. After the tracking decoder, the initial target position features can be output.

[0078] In order to better implement the embodiment of the present application, in one embodiment of the present application, according to the initial target position feature of the target to be shielded in each video frame, a second tracking is performed on the target to be shielded in each video frame to obtain the target position of the target to be shielded in each video frame in the video to be processed, including:

[0079] Input each video frame and the target mask image corresponding to each video frame into a preset tracking decoder to obtain the confidence of the target to be shielded in each video frame; filter the confidence of the target to be shielded in each video frame according to a preset second confidence threshold to obtain the target confidence of the target to be shielded in each video frame; determine the state information of the target to be shielded in each video frame; mark the target to be shielded in each video frame according to the target confidence of the target to be shielded in each video frame to obtain the marking information of the target to be shielded in each video frame; obtain the target position of the target to be shielded in each video frame in the video to be processed according to the state information of the target to be shielded in each video frame and the marking information of the target to be shielded in each video frame.

[0080] The above embodiment provides a solution for performing the first tracking according to the tracking decoder. However, according to the tracking decoder, three outputs can also be obtained. The first output is: classification, which is the target confidence score output by the classification subnetwork, that is, the confidence of the target to be masked in each video frame; the second output is: box regression, which is the output position coordinates; and the third output is: track query, track query is a new mask image obtained after processing based on the object query, specifically as follows Figure 8 shown.

[0081] After that, the track query needs to be input into the FM module, where the structure of the FM module is as follows Fig. 9 As shown. Among them, by inputting the track query into the FM module, the state information of the target to be shielded in each video frame can be determined, and the state information represents whether the target to be shielded is within or outside the field of view of the video frame. In this embodiment, the confidence of the target to be shielded in each video frame is filtered according to the preset second confidence threshold, and the reason for obtaining the target confidence of the target to be shielded in each video frame is the same as the purpose of filtering according to the first confidence in the above embodiment, and the specific details are not repeated here. The first confidence threshold and the second confidence threshold can be set according to actual conditions, and the two can also be the same, and the specific details are not limited here. Furthermore, the method of filtering out the target confidence according to the second confidence threshold can filter out an object with a larger predicted probability among the objects in the image frame as the target to be shielded, that is, according to the target to be shielded in each video frame, the target to be shielded in each video frame is marked, and the marking information of the target to be shielded in each video frame is obtained, and the marking information represents which specific object is the target to be shielded. When it is determined whether the target to be shielded is in the field of view or out of the field of view, and according to the specific label, the position of the target to be shielded can be determined, that is, according to the state information of the target to be shielded in each video frame and the label information of the target to be shielded in each video frame, the target position of the target to be shielded in each video frame in the video to be processed is obtained. In other words, if a target object is in the field of view and includes a label or a mark, then the object is a target to be shielded.

[0082] Among them, Fig. 9 The training of the FM module can be trained by the following loss function:

[0083]

[0084]

[0085] Among them, θ is a hyperparameter, L cls is the focal loss, L box is L1loss, L iou is the IOU loss of the associated box; N is the number of video frames of the video T to be processed; Y i and For the T i Prediction and label value of the frame. V i =V old +V new For the T i Target number of frames.

[0086] In summary, the overall embodiment process of the present application may include:

[0087] 1) User input prompt information;

[0088] 2) The mask decoder module extracts the target area;

[0089] 3) The user updates the prompt and jumps to 1), and the user completes the goal definition and proceeds to the next step;

[0090] 4) Generate a target frame based on the target mask as an object query input to the tracking decoder to track the subsequent target position;

[0091] 5) Using the tracked target position as a hint, the mask decoder is used to obtain the precise target area;

[0092] 6) Modify the target area image data to complete target shielding.

[0093] It should be noted that, as shown in 3), if the user has submitted an initial target to be shielded once, or after the user has submitted initial shielding information once, if the user has modified the shielding target or the shielding information on the initial target to be shielded or on the initial shielding information, then based on the user's latest target to be shielded or the latest information to be shielded, the target position is determined and the subsequent video shielding process is performed. In addition, this step 3) may not be a specific third step. If the user also provides the latest shielding information or shielding target during the final video shielding process, it is still necessary to return to the step of tracking the target. Therefore, the steps in this embodiment do not constitute a limitation on the overall process sequence.

[0094] In order to better implement the video processing method in the embodiment of the present application, in addition to the video processing method, the embodiment of the present application also provides a video processing device, such as Fig.10 As shown, the device 300 includes:

[0095] An acquisition module 301 is used to acquire a target to be shielded in a video to be processed;

[0096] A determination module 302 is used to determine the target position of the target to be shielded in each video frame of the video to be processed;

[0097] The processing module 303 is used to perform masking processing on the target position in each video frame to obtain a processed target video.

[0098] The video processing device provided by the present application can obtain the target to be shielded of a video by the user through the acquisition module 301, determine the position of the target to be shielded in the entire video through the determination module 302, and finally shield the target to be shielded in the entire video through the processing module 303. That is, the present application can shield the target specified by the user in the video, improve the flexibility of video shielding, and better improve the user's shielding experience.

[0099] In some embodiments of the present application, the acquisition module 301 is specifically used to:

[0100] Obtain input shielding information of the video to be processed;

[0101] The shielding information is parsed to obtain the target to be shielded.

[0102] In some embodiments of the present application, the acquisition module 301 is further used to:

[0103] Obtaining encoding features of the target video frame;

[0104] Perform label prediction on the encoded features to obtain the predicted labels;

[0105] According to the encoded features, the predicted labels and the selected area, the confidence of each mask in the mask image corresponding to the selected area is obtained;

[0106] According to the confidence of each mask in the mask image and a preset first confidence threshold, the target to be masked is obtained.

[0107] In some embodiments of the present application, the acquisition module 301 is further used to:

[0108] Get the preset target encoder;

[0109] The target encoder is combined with the attention convolution layer to encode each video frame of the video to be processed to obtain the encoding features of the target video frame.

[0110] In some embodiments of the present application, the determination module 302 is specifically used to:

[0111] Generate a target mask image corresponding to the target to be shielded in each video frame;

[0112] According to each target mask image, first tracking is performed on the target to be shielded in each video frame to obtain the initial target position feature of the target to be shielded in each video frame;

[0113] According to the initial target position feature of the target to be shielded in each video frame, the target to be shielded in each video frame is tracked for the second time to obtain the target position of the target to be shielded in each video frame in the video to be processed.

[0114] In some embodiments of the present application, the determination module 302 is further configured to:

[0115] Each video frame and the target mask image corresponding to each video frame are input into a preset tracking decoder to obtain the initial target position features of the target to be masked in each video frame.

[0116] In some embodiments of the present application, the determination module 302 is further configured to:

[0117] Input each video frame and the target mask image corresponding to each video frame into a preset tracking decoder to obtain the confidence of the target to be masked in each video frame;

[0118] According to a preset second confidence threshold, the confidence of the target to be shielded in each video frame is filtered to obtain the target confidence of the target to be shielded in each video frame;

[0119] Determine the status information of the target to be shielded in each video frame;

[0120] Marking the target to be shielded in each video frame according to the target confidence of the target to be shielded in each video frame to obtain marking information of the target to be shielded in each video frame;

[0121] According to the state information of the target to be shielded in each video frame and the marking information of the target to be shielded in each video frame, the target position of the target to be shielded in each video frame in the video to be processed is obtained.

[0122] The present application also provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the steps of any video processing method in the present application. The terminal device integrates any video processing method provided in the present application, such as Fig.11 As shown, it shows a schematic diagram of the structure of the terminal device involved in the embodiment of the present application, specifically:

[0123] The terminal device may include one or more processing core processors 401, one or more computer-readable storage media memories 402, a power supply 403, an input unit 404 and other components. Those skilled in the art will appreciate that Fig.11 The terminal device structure shown in the figure does not constitute a limitation on the terminal device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0124] The processor 401 is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device. By running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, the processor 401 executes various functions of the terminal device and processes data, thereby monitoring the terminal device as a whole. Optionally, the processor 401 may include one or more processing cores; the processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. Preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, the user interface and the application program, etc., and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0125] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0126] The terminal device also includes a power supply 403 for supplying power to each component. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, power consumption management and other functions through the power management system. The power supply 403 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0127] The terminal device may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0128] Although not shown, the terminal device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 401 in the terminal device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402, thereby realizing various functions, such as:

[0129] Obtain the target to be blocked in the video to be processed;

[0130] Determine the target position of the target to be shielded in each video frame of the video to be processed;

[0131] The target position in each video frame is shielded to obtain a processed target video.

[0132] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0133] To this end, the embodiment of the present application provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in any video processing method provided in the embodiment of the present application. For example, the computer program can be loaded by the processor to execute the following steps:

[0134] Obtain the target to be blocked in the video to be processed;

[0135] Determine the target position of the target to be shielded in each video frame of the video to be processed;

[0136] The target position in each video frame is shielded to obtain a processed target video.

[0137] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the detailed description of other embodiments above, and will not be repeated here.

[0138] In specific implementation, the above units or structures can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units or structures can refer to the previous method embodiments, which will not be repeated here.

[0139] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0140] The above is a detailed introduction to a video processing method and device provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A video processing method, characterized in that: The method comprises: Obtain the target to be blocked in the video to be processed; Determine the target position of the target to be shielded in each video frame of the video to be processed; The target position in each of the video frames is shielded to obtain a processed target video.

2. The video processing method according to claim 1, characterized in that: The step of obtaining the target to be shielded in the video to be processed includes: Obtaining input shielding information of the video to be processed; The shielding information is parsed to obtain the target to be shielded.

3. The video processing method according to claim 2, characterized in that: The shielding information includes a selected area in a target video frame in the video frame to be processed; The step of parsing the shielding information to obtain the target to be shielded includes: Obtaining encoding features of the target video frame; Performing label prediction on the encoding feature to obtain a predicted label; Obtaining, according to the encoding feature, the predicted label and the selected area, a confidence level of each mask in the mask image corresponding to the selected area; The target to be masked is obtained according to the confidence of each mask in the mask image and a preset first confidence threshold.

4. The video processing method according to claim 3, characterized in that: The obtaining the encoding feature of the target video frame includes: Get the preset target encoder; The target encoder is combined with an attention convolutional layer to encode each video frame of the video to be processed to obtain the encoding features of the target video frame.

5. The video processing method according to claim 1, characterized in that: The determining the target position of the target to be shielded in each video frame of the video to be processed includes: Generating a target mask image corresponding to the target to be shielded in each of the video frames; According to each of the target mask images, first tracking is performed on the target to be shielded in each of the video frames to obtain initial target position features of the target to be shielded in each of the video frames; According to the initial target position features of the target to be shielded in each of the video frames, a second tracking is performed on the target to be shielded in each of the video frames to obtain the target position of the target to be shielded in each of the video frames to be processed.

6. The video processing method according to claim 5, characterized in that: The first tracking of the target to be shielded in each of the video frames according to each of the target mask images to obtain the initial target position features of the target to be shielded in each of the video frames includes: Each of the video frames and the target mask image corresponding to each of the video frames are input into a preset tracking decoder to obtain the initial target position features of the target to be masked in each of the video frames.

7. The video processing method according to claim 5, characterized in that: The second tracking of the target to be shielded in each of the video frames is performed according to the initial target position feature of the target to be shielded in each of the video frames to obtain the target position of the target to be shielded in each of the video frames to be processed, including: Inputting each of the video frames and the target mask image corresponding to each of the video frames into a preset tracking decoder to obtain the confidence of the target to be masked in each of the video frames; According to a preset second confidence threshold, the confidence of the target to be shielded in each of the video frames is filtered to obtain the target confidence of the target to be shielded in each of the video frames; Determining the state information of the target to be shielded in each of the video frames; Marking the target to be shielded in each of the video frames according to the target confidence of the target to be shielded in each of the video frames to obtain marking information of the target to be shielded in each of the video frames; According to the state information of the target to be shielded in each of the video frames and the marking information of the target to be shielded in each of the video frames, the target position of the target to be shielded in each of the video frames to be processed is obtained.

8. A video processing device, characterized in that: The device comprises: An acquisition module, used for acquiring the target to be shielded in the video to be processed; A determination module, used to determine the target position of the target to be shielded in each video frame of the video to be processed; The processing module is used to perform masking processing on the target position in each of the video frames to obtain a processed target video.

9. A terminal device, characterized in that: The terminal device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the steps of the video processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in the video processing method according to any one of claims 1 to 7.

Images