Video processing method, device and equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses a video processing method and device, equipment and a computer readable storage medium. The method comprises the following steps: detecting a detection object contained in an image frame contained in video data to obtain an object detection frame containing the detection object; performing picture group grouping processing on the video data to obtain a plurality of continuous video frame groups; wherein each video frame group comprises a key frame and a plurality of non-key frames; cutting the non-key frame in each video frame group based on the object detection box to correspondingly obtain the cut non-key frame, and performing background filling on the cut non-key frame in the same video frame group based on the key frame to correspondingly obtain the processed video frame group; and coding based on the processed video frame group to obtain video coding data. Through the method, the data volume of the video coding data can be reduced.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a video processing method, apparatus, device and computer-readable storage medium. Background Art

[0002] With the continuous development of drone technology and artificial intelligence technology, the application scenarios of motion recognition through video data collected by drones are becoming more and more extensive, such as smart cities, public safety, unmanned retail and other fields.

[0003] At present, after the drone collects video data, it needs to encode the video data and send the video encoding data to the cloud for subsequent recognition processing. However, the amount of video data for action recognition tasks is very large, and the amount of video encoding data is also very large. The transmission process not only brings network load pressure, but also affects real-time performance and accuracy.

[0004] Therefore, how to reduce the amount of video encoding data is an urgent problem to be solved. Summary of the invention

[0005] To solve the above technical problems, the embodiments of the present application provide a video processing method, apparatus, device and computer-readable storage medium, which can reduce the data volume of video encoding data.

[0006] Among them, the technical solution adopted in this application is:

[0007] A video processing method, comprising:

[0008] Detecting a detection object contained in an image frame included in the video data, and obtaining an object detection frame containing the detection object;

[0009] Performing picture group processing on the video data to obtain a plurality of continuous video frame groups; wherein each video frame group includes a key frame and a plurality of non-key frames;

[0010] Cropping non-key frames in each video frame group based on the object detection frame to obtain cropped non-key frames, and performing background filling on the cropped non-key frames in the same video frame group based on the key frames to obtain a processed video frame group;

[0011] Encoding is performed based on the processed video frame group to obtain video encoding data.

[0012] A video processing device, comprising:

[0013] A processing unit, configured to detect a detection object contained in an image frame contained in the video data, and obtain an object detection frame containing the detection object;

[0014] A grouping unit, used for performing picture grouping processing on the video data to obtain a plurality of continuous video frame groups; wherein each video frame group includes a key frame and a plurality of non-key frames;

[0015] A cropping unit, configured to crop non-key frames in each video frame group based on the object detection frame to obtain cropped non-key frames, and to perform background filling on the cropped non-key frames in the same video frame group based on the key frames to obtain a processed video frame group;

[0016] The encoding unit is used to perform encoding based on the processed video frame group to obtain video encoding data.

[0017] In one embodiment of the present application, based on the aforementioned scheme, the processing unit is also used to extract background information from the key frame; use the background information to fill the background of the cropped non-key frame in the same video frame group, and obtain the filled non-key frame accordingly; wherein the image size of the filled non-key frame is the same as the image size of the key frame; and generate a processed video frame group based on the key frame and the filled non-key frame in the same video frame group.

[0018] In one embodiment of the present application, based on the aforementioned scheme, the processing unit is also used to generate a filling template based on the background information, and the image size of the filling template is the same as the image size of the background information; the background of the cropped non-key frame is filled by using the filling template to obtain a filled non-key frame accordingly.

[0019] In one embodiment of the present application, based on the aforementioned scheme, the processing unit is further used to connect the object detection frames in the same image frame to obtain the connected areas contained in the corresponding image frame; the cropping unit is further used to crop the non-key frames in each video frame group based on the connected areas contained in the image frame to obtain cropped non-key frames.

[0020] In one embodiment of the present application, based on the aforementioned scheme, the processing unit is also used to extract temporal redundant information from the connected area contained in the key frame; the encoding unit is also used to encode the padded non-key frames contained in the same video frame group based on the temporal redundant information to obtain encoded non-key frame data; the processing unit is also used to generate the video encoding data based on the encoded key frame data and the encoded non-key frame data.

[0021] In one embodiment of the present application, based on the aforementioned scheme, the processing unit is further used to extract spatial redundant information from the connected area contained in the filled non-key frame; the encoding unit is further used to encode the corresponding filled non-key frame based on the temporal redundant information and the spatial redundant information to obtain the encoded non-key frame data.

[0022] In one embodiment of the present application, based on the aforementioned scheme, the processing unit is also used to extract pixel information of the connected area contained in the filled non-key frame; the encoding unit is also used to encode the pixel information of the connected area based on the temporal redundant information and the spatial redundant information to obtain the encoded non-key frame data.

[0023] In one embodiment of the present application, based on the aforementioned scheme, the number of object detection frames in the same image frame is two; the processing unit is also used to determine two corner points on each object detection frame respectively; a polygonal frame is generated based on the determined corner points, and the image area where the polygonal frame is located is used as the connected area.

[0024] In one embodiment of the present application, based on the aforementioned scheme, there are multiple object detection frames in the same image frame; the processing unit is also used to detect the smallest circle area containing multiple object detection frames in the same image frame, and use the smallest circle area as the connected area.

[0025] In one embodiment of the present application, based on the aforementioned scheme, the processing unit is further used to perform feature detection on the detection object contained in the image frame in the video data to obtain the position information and body shape information of the detection object; determine the position and size of the object detection frame corresponding to the detection object based on the position information and the body shape information; and generate the object detection frame based on the position and the size.

[0026] A video processing system, the system comprising a first device and a second device, wherein:

[0027] The grouping unit in the first device is used to perform picture grouping processing on the video data to obtain a plurality of continuous video frame groups, each of which contains a key frame and a plurality of non-key frames, and the cropping unit is used to crop the non-key frames in each video frame group based on the object detection frame contained in the image frame in the video data to obtain cropped non-key frames accordingly, and the processing unit is used to perform background filling on the cropped non-key frames in the same video frame group based on the key frames to obtain processed video frame groups accordingly;

[0028] The encoding unit in the first device is used to encode based on the processed video frame group to obtain video encoding data, and send the video encoding data to the second device;

[0029] The second device receives the video encoding data, and the processing unit is further configured to decode the video encoding data to obtain video decoding data.

[0030] In one embodiment of the present application, based on the aforementioned scheme, the processing unit in the second device is also used to parse the cropped non-key frames, key frames and connected areas in the key frames contained in each video frame group from the video encoding data, and perform background filling on the cropped non-key frames in the same video frame group based on the key frames and the connected areas to obtain the video decoding data.

[0031] A video processing device includes a processor and a memory. The memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the above video processing method is implemented.

[0032] A computer-readable storage medium stores computer-readable instructions. When the computer-readable instructions are executed by a processor of a computer, the computer executes the above video processing method.

[0033] A computer program product includes computer-readable instructions, and when the computer-readable instructions are executed by a processor, the above video processing method is implemented.

[0034] In the above technical solution:

[0035] Cropping the non-key frames in each video frame group based on the object detection frame can remove the background information in the non-key frames, retain the image information in the object detection frame, reduce the amount of data encoded by the non-key frames, and further reduce the amount of video encoding data. And when encoding, the background of the cropped non-key frames in the same video frame group is filled based on the key frames, so that the image size of the non-key frames after the background filling can be consistent with the image size of the key frames, so that the size of each image frame in each video frame group is consistent, thereby meeting the size requirements of the encoding protocol for the images in the video frame group.

[0036] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0038] Figure 1 It is a schematic diagram of an implementation environment involved in this application;

[0039] Figure 2 is a flow chart of a video processing method according to an exemplary embodiment;

[0040] Figure 3 is a schematic diagram showing a method of generating an object detection frame according to an exemplary embodiment;

[0041] Figure 4 is a schematic diagram of a video frame group according to an exemplary embodiment;

[0042] Figure 5 is a schematic diagram of a process of performing connected region training based on a cropping model according to an exemplary embodiment;

[0043] Figure 6 is a flowchart of a video processing method according to another exemplary embodiment;

[0044] Figure 7a is a schematic diagram showing a method of cutting non-key frames based on connected areas according to another exemplary embodiment;

[0045] Figure 7b is a schematic diagram showing another method of cutting non-key frames based on connected areas according to another exemplary embodiment;

[0046] Figure 8a is a filled non-key frame shown according to an exemplary embodiment;

[0047] Figure 8b is another non-key frame after filling according to an exemplary embodiment;

[0048] Fig. 9 is a flowchart of a video processing method according to another exemplary embodiment;

[0049] Fig.10 is a flowchart of a video processing method according to another exemplary embodiment;

[0050] Fig.11 is a block diagram of a video processing device according to an exemplary embodiment;

[0051] Fig.12 The figure is a schematic diagram of the hardware structure of a video processing device according to an exemplary embodiment. DETAILED DESCRIPTION

[0052] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments that are the same as the present application. Instead, they are only examples of devices and methods that are the same as some aspects of the present application as detailed in the attached claims.

[0053] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0054] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, while some operations / steps can be integrated or partially integrated, so the actual execution order may change according to actual conditions.

[0055] It should be noted that the "multiple" mentioned in this application refers to two or more than two. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship.

[0056] It should be noted that in the specific implementation of this application, user-related data is involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0057] Before introducing the technical solutions of the embodiments of the present application, the technical terms involved in the embodiments of the present application are first introduced here.

[0058] GOP is a structure in video coding that is used to divide video frames into different groups and define the relationship and order between frames within a group. It is an important concept in video coding, which is used to control the encoding process, improve compression efficiency and decoding performance. In common video coding standards (such as H.264 / AVC, H.265 / HEVC), GOP is usually composed of a group of continuous video frames, including key frames (I frames), predicted frames (P frames) and bidirectional predicted frames (B frames). These frames are encoded and decoded in a specific order to achieve video compression and recovery. Among the typical GOP structures, the most common is GOP (Group of Pictures). GOP is a key frame-based GOP structure that contains one or more continuous predicted frames and / or bidirectional predicted frames, and a key frame as the start frame of the group. Key frames are usually key moments or important frames in a video sequence, which can be used as recovery points for independent decoding or random access. In addition to GOP, there are other picture grouping structures, such as Mini-GOP, Open-GOP, etc., which are divided or expanded on the basis of GOP in a more fine-grained manner to meet different needs and application scenarios. It should be noted that different video coding standards and implementations may have different definitions and regulations for picture grouping. Therefore, in actual application scenarios, the specific picture grouping structure and parameter settings can be selected and adjusted according to actual needs and system requirements. In the embodiment of the present application, the picture grouping is described as a GOP grouping method as an example, without limitation.

[0059] Temporal redundancy refers to the similarity or repetition between consecutive frames in a video sequence. In video coding, the image content between consecutive frames usually has some overlapping or similar areas, such as moving objects between adjacent frames or areas with little scene change. By taking advantage of this temporal correlation, only the changed parts can be encoded, and the repeated content or similar areas can be referenced or predicted, thereby reducing the amount of encoded data. This technology is usually used in the encoding of predicted frames (P frames) and bidirectional predicted frames (B frames). For example, a pixel in a P frame contains the corresponding temporal redundancy information in the same pixel in an I frame in the same video frame group.

[0060] Spatial redundancy refers to the similarity or redundancy within a video frame. In a single video frame, there may be a high degree of correlation between adjacent pixels or similar image regions, such as a smooth background, repeated textures, etc. By taking advantage of this spatial correlation, a more compact representation can be used to encode video frames, such as by using techniques such as transform, differential coding, and entropy coding to reduce redundant information. For example, for a pixel in an I frame, several pixels surrounding the pixel contain the corresponding spatial redundancy information.

[0061] YOLOX is an improvement and extension of the You Only Look Once (YOLO) series of algorithms. It is an efficient and accurate real-time target detection algorithm that uses a single-stage deep learning model and achieves fast and accurate target detection through innovative architecture design, feature fusion, and multi-scale prediction. The algorithm introduces CSPNet as the backbone network and further improves the performance of the model through pre-training strategies and a new loss function MegDet. YOLOX achieves a good balance between speed and accuracy and is suitable for a variety of real-time application scenarios.

[0062] The YOLOX algorithm is a method that can perform target recognition quickly and accurately. In the human body recognition module, the YOLOX algorithm can quickly detect the position of the human body in the video and provide key information such as the position, size, and confidence of each human body frame. In addition, the YOLOX algorithm has a high degree of accuracy and robustness, and can accurately identify the human body even in complex situations such as lighting changes, posture changes, and partial occlusion. Considering the limited computing resources of the camera equipment, the lightweight YOLOX version can achieve fast detection speed and high accuracy.

[0063] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0064] In related technologies, video data is often large, which makes the amount of video encoding data also large. During the transmission process, a large amount of transmission resources are often consumed, which brings load pressure to the network and affects real-time performance and accuracy.

[0065] Based on this, the embodiments of the present application respectively propose a video processing method, a video processing device, a video processing equipment, a computer-readable storage medium and a computer program product. In these embodiments, the non-key frames in each video frame group are cropped based on the object detection frame, so that the background information in the non-key frames can be removed, and the image information in the object detection frame is retained, thereby reducing the amount of data encoded by the non-key frames, and further reducing the amount of data encoded by the video.

[0066] See also Figure 1 , Figure 1It is a schematic diagram of an implementation environment involved in this application.

[0067] Figure 1 The implementation environment shown specifically includes a first device 110 and a second device 140. The first device 110 may include an image sensor 120, which may collect image data and video data. The second device 140 may be a cloud device, which is used to decode the video encoding data sent by the first device 110 and perform operations such as motion recognition on the video decoding data. A communication connection may be established between the first device 110 and the second device 140 through a transmission channel 130, that is, the first device 110 and the second device 140 may transmit data through the transmission channel 130.

[0068] The first device 110 may be a drone device, which may be equipped with a camera or multiple sensors, and collect image and video data by flying from a high altitude or moving to shoot. The drone device may be operated and the flight path may be controlled by a remote controller, an autonomous navigation system or a preset route to obtain the required image and video data.

[0069] The first device 110 may also be a traffic camera, which is usually installed on roads, intersections or other important areas to detect current traffic conditions and environments. These cameras may be fixed or adjustable in angle. They may use high-definition cameras or other dedicated sensors to collect image and video data through continuous shooting or real-time streaming.

[0070] like Figure 1 1 shows an example in which the first device 110 is a drone device, and the image sensor 120 therein collects an image frame of video data. The image frame includes pedestrians and background environment. The first device 110 also includes a processor, a memory and other modules, which can encode the collected video data to generate video encoding data, and then send the video encoding data to the second device 140 through the transmission channel 130.

[0071] The image sensor 120 is a component of the first device 110. It is a photosensitive device that can convert external light into electrical signals. It can capture images and videos in the environment and convert the captured image data into digital form for subsequent processing and transmission.

[0072] The second device 140 may be a cloud device, which is mainly used to decode the video encoding data sent from the first device 110 and perform operations such as motion recognition on the decoded video data. Cloud devices usually have more powerful computing and storage capabilities and can process large amounts of data.

[0073] The transmission channel 130 is a channel for establishing a communication connection between the first device 110 and the second device 140. This channel can be a wireless or wired communication link for transmitting image and video data. Through the transmission channel 130, the first device 110 can send the collected image and video data to the second device 140 for further processing and analysis.

[0074] It should be noted that, in addition to drone equipment and traffic cameras, the first device 110 can also be various terminal devices with image / video shooting capabilities, such as mobile phones, cameras, tablet computers, etc., and the specific type of the first device 110 is not limited here. These devices can shoot and record images and videos through their built-in cameras or external camera devices.

[0075] The second device 140 is not limited to cloud devices, but can also be other devices that can provide integrated services to users, such as servers, workstations, etc. These devices have powerful computing and storage capabilities, can receive, process and store image and video data collected from the first device 110, and provide corresponding services and functions.

[0076] For the convenience of description, the embodiment of the present application is described with the first device 110 being a drone device and the second device 140 being a cloud device. This choice is only for the convenience of description and explanation, and does not limit the specific device type. In fact, the technology can be applied to various types of first devices and second devices to meet the needs of different application scenarios.

[0077] See also Figure 2 , Figure 2 is a flowchart of a video processing method according to an exemplary embodiment. The method can be applied to Figure 1 The implementation environment shown is specifically executed by the first device 110. Of course, the method can also be applied to other implementation environments, and the execution subject of the method is not limited here.

[0078] The video processing method will be described in detail below by taking the first device 110 as an exemplary execution subject.

[0079] like Figure 2 As shown, in an exemplary embodiment, the method comprises at least the following steps:

[0080] S210, detecting a detection object contained in an image frame included in the video data, and obtaining an object detection frame containing the detection object.

[0081] The video data includes a plurality of image frames (also referred to as video frames), and each image frame is arranged in a certain order to form the video data.

[0082] Specifically, the first device can perform feature detection on the detection object contained in the image frame in the collected video data, so as to obtain the position information and body shape information of each detection object. The detection object can be a pedestrian, and the body shape information can be the height and body width of the detection object. Based on the position information and body shape information, the first device can determine the position and size of the object detection frame corresponding to each detection object, so that the object detection frame corresponding to each detection object can be generated according to the determined position and size.

[0083] Optionally, when detecting the position information and body shape information of each detection object, the first device may use a lightweight pedestrian detection algorithm, such as the YOLOX algorithm, to detect the position and body shape of each detection object, thereby determining an object detection box corresponding to each detection object.

[0084] See also Figure 3 , Figure 3 FIG. 1 is a schematic diagram of an exemplary object detection frame generation. Figure 3 As shown, the first device detects the detection object, detection object 2, and detection object 3 in the image frame, and can perform feature detection on the three detection objects respectively to obtain their position information and body shape information. Thus, the position and size of the object detection frame corresponding to each detection object are determined, and object detection frame 1, object detection frame 2, and object detection frame 3 are obtained.

[0085] It is worth noting that in this example image, part of the body of the detection object 3 is obscured by the tree, but after adopting the lightweight pedestrian detection algorithm, the position information and body shape information of the detection object 3 can still be detected and predicted, thereby determining the corresponding object detection frame 3. In this way, the accuracy of the object detection frame is improved, providing a more accurate data source for subsequent video encoding.

[0086] S220, performing picture group processing on the video data to obtain a plurality of continuous video frame groups; wherein each video frame group includes a key frame and a plurality of non-key frames.

[0087] In the embodiment of the present application, when grouping the picture groups, GOP grouping, Mini-GOP grouping, etc. can be used. The first device can determine which frames are key frames and which frames are non-key frames based on the collected video data, and then take a key frame and several non-key frames after it as a video frame group. After multiple such operations, multiple video frame groups can be obtained.

[0088] For example, Figure 4The figure shows a schematic diagram of a video frame group. The first device can determine an I frame from multiple image frames in the video data, and determine several image frames thereafter as P frames or B frames, thereby obtaining a video frame group. Figure 4 In the example, video frame group 1 includes 5 image frames, and video frame group 2 includes 4 image frames. That is, the first device can dynamically determine the number of image frames included in each video frame group according to the needs of the actual scene, which is not limited here.

[0089] It can be understood that video frame group 1 and video frame group 2 are continuous video frame groups, and the video frame groups before video frame group 1 and after video frame group 2 are all continuous video frame groups, that is, video frame group 1 and video frame group 2 are a fragment of multiple continuous video frame groups.

[0090] S230, cropping non-key frames in each video frame group based on the object detection frame to obtain cropped non-key frames, and performing background filling on the cropped non-key frames in the same video frame group based on the key frames to obtain a processed video frame group.

[0091] Specifically, the first device includes a cropping region generation module, and using the cropping region generation module, the first device can determine a connected area based on the object detection frame included in the non-key frame in each video frame group. The connected area can also be called a cropping area or a cropping area. The connected area can be determined based on the number, position and size of the object detection frame in each image frame. When the number, position and size of the object detection frame are different, different strategies can be used to determine the connected area.

[0092] Optionally, when determining the connected area, the first device may also learn the optimal connectivity rule by training the cropping model through supervised learning, so that the determined connected area can completely cover all object detection frames in the image frame and make the area of ​​the connected area as small as possible. Among them, the cropping model can be a machine learning model. In this way, the recognition accuracy of the first device can be improved, and the amount of data of subsequent video encoding data can be reduced. In order to realize the adaptive selection of connected areas for detection objects of different sizes, a neural network can be used to learn the rules of the optimal connected areas in different scenarios, so as to generate the most suitable connected area for each image frame in real time.

[0093] Specifically, Figure 5 As shown, the video data collected by the first device and the labels of the connected areas determined by the first device can be input into the cropping model for training, and the optimal connected area rule can be derived, thereby providing more accurate connected area information for subsequent cropping of image frames according to the connected areas.

[0094] Optionally, since the first device needs to crop the image frame based on the connected area, in order to reduce the negative impact of subsequent video cropping on video encoding, a neural network can be used to learn the part of the video frame that plays a key role in the encoding bit rate, and this part is also regarded as being in the connected area. That is, in addition to retaining the key connected area containing pedestrian information, the area in the image frame that plays a key role in the bit rate can also be used as a connected area, so as to avoid the loss of redundant information required for subsequent video encoding. Among them, the area that plays a key role in the bit rate is also used as a connected area and the key connected area containing pedestrian information can be connected or disconnected. That is, an image frame can have multiple connected areas. However, in order to ensure that the connected area is as small as possible, those skilled in the art can set whether an image frame can contain multiple connected areas according to actual needs, and the embodiments of the present application are not limited. For the convenience of description, the embodiments of the present application are described by taking an image frame containing a connected area as an example, which is not limited.

[0095] Furthermore, the first device may crop the non-key frame based on the determined connected area, so that the cropped non-key frame only retains the data in the connected area and removes the area other than the connected area. In this way, the image data contained in the non-key frame can be reduced.

[0096] Optionally, when the first device crops the non-key frames in each video frame group based on the object detection frame, it may crop part of the non-key frames in each video frame group, that is, it is not necessary to crop all the non-key frames in a video frame group.

[0097] Optionally, the first device may crop all image frames included in a portion of the video frame group.

[0098] Optionally, the first device may crop a portion of the image frames in each video frame group in a portion of the video frame groups.

[0099] Furthermore, after the first device obtains the cropped non-key frames, it can fill the background of the cropped non-key frames in the same video frame group based on the key frames, thereby obtaining multiple filled non-key frames, thereby obtaining the processed video frame group corresponding to each video frame group.

[0100] It can be understood that the processed video frame group includes a key frame and a plurality of filled non-key frames.

[0101] Currently, when encoding video data, the encoder requires that the image sizes of all image frames in the same video data must be the same. Filling the cropped non-key frames with backgrounds can make the image sizes of the filled non-key frames the same as the image sizes of the key frames, and the encoder can then be used for encoding.

[0102] Accordingly, the first device may not crop the key frames in each video frame group, that is, the key frames retain complete image information, including information about the detected object and background information. In this way, during subsequent encoding, the information retained in the key frames can provide more reference information for non-key frames.

[0103] In an embodiment of the present application, the first device can execute the step of grouping the video data in S220 after determining the connected areas of each image frame in the video data. In this way, the obtained multiple continuous video frame groups are more reasonable, thereby improving the accuracy of subsequent video encoding.

[0104] S240, encoding is performed based on the processed video frame group to obtain video encoding data.

[0105] The number of the processed video frame groups is multiple, and they correspond one-to-one to the multiple video frame groups in the video data. A processed video frame group includes a complete key frame and multiple cropped non-key frames. The first device can encode each processed video frame group to obtain video encoding data.

[0106] Through this method, first, the cropping operation can remove the background information in the non-key frame and retain the image information in the object detection frame, thereby reducing the image data volume of the non-key frame. Since the non-key frame usually contains a lot of redundant background information, removing this redundant information can effectively reduce the data volume of the non-key frame.

[0107] Secondly, during the encoding process, by using key frames to fill the background of the cropped non-key frames, the background of the non-key frames can be made consistent with the background of the key frames, so that the size of each image frame in each video frame group is consistent, which meets the requirements of the encoding protocol for the image size in the video frame group.

[0108] Finally, during the encoding process, only the pixels in the connected area contained in the filled non-key frame are encoded, without encoding the background information, which reduces the amount of video encoding data. This is because the pixels in the connected area contain important information about the target object, and the background information can be filled in by the key frame without repeating the encoding in the non-key frame.

[0109] Therefore, this method reduces the amount of non-key frame encoding data by cropping non-key frames, filling the background, and only encoding pixels in connected areas, thereby reducing the overall data volume of video encoding data and improving encoding efficiency and compression performance.

[0110] In one embodiment of the present application, another video processing method is provided, which can be executed by a first device. Figure 6 As shown, the video processing method may include S210 to S220, S610 to S630 and S240. That is, S610 to S630 are Figure 2 The specific implementation method of S230 is shown.

[0111] S610 to S630 are described below:

[0112] S610, connecting the object detection frames in the same image frame to obtain connected areas contained in the corresponding image frame.

[0113] S610 may specifically include S611 to S612, or S613.

[0114] It should be noted that S611 to S612 are steps executed when the number of object detection frames is two, and S613 is a step executed when the number of object detection frames is multiple.

[0115] S611 to S612 included in S610 are described below:

[0116] S611, determine two corner points on each object detection box.

[0117] The corner points may be two vertices outside the object detection box, or two points on the edge of the object detection box.

[0118] For example, Figure 7a The figure shows an exemplary schematic diagram of cropping non-key frames based on connected areas. Figure 7a In the figure, for the two object detection boxes, their two corner points are selected at the two outermost vertices respectively.

[0119] S612: Generate a polygonal frame based on the determined corner points, and use the image area where the polygonal frame is located as a connected area.

[0120] Based on the determined four corner points, the first device may generate a polygonal frame, which may be a rectangle, a parallelogram, etc., which is not limited here. Figure 7a As shown, the polygonal box is a rectangle, which includes the object detection boxes of the two detection objects, forming a connected area.

[0121] S613 included in S610 is described below:

[0122] S613, detecting a minimum circle area including multiple object detection frames in the same image frame, and using the minimum circle area as a connected area.

[0123] The minimum circle area may be determined based on the vertices of the plurality of object detection frames. The circle may be a circle or an ellipse.

[0124] For example, Figure 7b FIG. 4 is another exemplary schematic diagram of cropping non-key frames based on connected areas. Figure 7b In , the object detection boxes of the three detected objects are surrounded by an ellipse, and the area covered by the ellipse can be regarded as a connected area.

[0125] In the embodiment of the present application, when the number of object detection frames is 1, the area covered by the object detection frame can be directly used as the connected area.

[0126] Optionally, after the first device is trained by the cropping model, it can crop out any connected area. As long as the connected area can cover all object detection frames in an image frame, it can meet the conditions of being a connected area. In other words, the connected area of ​​the embodiment of the present application includes but is not limited to the several shapes mentioned above. Those skilled in the art can change the shape of the connected area according to actual needs, such as an area composed of irregular curves or straight lines, etc., which is not limited by the embodiment of the present application. In this way, adaptive area cropping is achieved.

[0127] The cropping method in the embodiment of the present application can not only ensure that the field of view of all detected objects is covered, but also avoid incorporating irrelevant areas into the connected area, thereby improving the accuracy and efficiency of recognition. The design of the cropping area generation module has good practicality and universality, can adapt to people of different sizes, and can accurately detect the dynamic trajectory of pedestrians, ensuring the accuracy and efficiency of recognition.

[0128] S620: Crop the non-key frames in each video frame group based on the connected regions contained in the image frames to obtain cropped non-key frames.

[0129] That is, the first device retains the images within the connected area contained in the non-key frames in each video frame group, and deletes the images outside the connected area, so as to obtain the cropped non-key frames.

[0130] like Figure 7a and Figure 7b Each of them obtained their own cropped non-key frames.

[0131] S630, performing background filling on the cropped non-key frames in the same video frame group based on the key frames, and obtaining a processed video frame group accordingly.

[0132] Specifically, S630 may include S631 to S633.

[0133] S631 to S633 are described below:

[0134] S631, extracting background information from the key frame.

[0135] Specifically, the first device may retain the image area other than the connected area in the key frame as background information.

[0136] S632, performing background filling on the cropped non-key frames in the same video frame group according to the background information, and obtaining corresponding filled non-key frames.

[0137] The image size of the padded non-key frame is the same as that of the key frame.

[0138] Specifically, the first device includes a background supplement module, which can directly overlay the background information of the key frame on the cropped non-key frame in the same video frame group, and merge the two together to form a complete image frame, that is, the filled non-key frame.

[0139] For example, in Figure 7a In the non-keyframe background, the leftmost tree is upright; Figure 8a In the background of the key frame, the leftmost tree is slightly tilted. Also, in the connected area of ​​the key frame, the position of the little man on the right is different from that in the cropped non-key frame. Therefore, Figure 8a In the filled non-key frame shown, the connected area contains the image of the non-key frame, and the outside of the connected area contains the background information of the key frame.

[0140] Optionally, in the first device, a filling template may be generated based on the background information, and the image size of the filling template is the same as the image size of the background information. The filling template may adopt a preset style, such as pure white, pure green, or contain some textures. By using the filling template to fill the background of the cropped non-key frame, a filled non-key frame may be obtained accordingly. Figure 8b As shown, the filled non-key frame includes a cropped non-key frame image and a filling template, and the image size thereof is the same as that of the key frame.

[0141] In an embodiment of the present application, in the first device, since the encoder requires that the size of each image frame in the video data is the same, the method of background filling is crucial to maintaining the consistency of the image size. If background filling is not performed, the first device will have to stretch the cropped non-key frame to make it reach the same image size as the key frame, which will cause serious image distortion, and the resulting video encoding data will not be able to be used for subsequent video processing. Therefore, the method of the embodiment of the present application avoids stretching the cropped non-key frame, but retains the original information in the cropped non-key frame, and makes it consistent with the image size of the key frame by background filling. The advantage of doing this is to improve the reliability of video encoding and ensure that the encoder can effectively encode each video frame without losing image quality or using severely distorted image data.

[0142] S633: Generate a processed video frame group based on the key frames and the filled non-key frames in the same video frame group.

[0143] The first device may sort the key frames and the filled non-key frames in the same video frame group according to the original order of the key frames and the non-key frames in the video frame group, so as to obtain a generated and processed video frame group.

[0144] Through this method, the first device can fill the background of the cropped non-key frame based on the background information in the key frame, so that the image size of the filled non-key frame is the same as the image size of the key frame, so that the encoder can smoothly encode the filled non-key frame.

[0145] In one embodiment of the present application, another video processing method is provided, which can be executed by a first device. Fig. 9 As shown, the video processing method may include S210 to S230 and S910 to S930. That is, S910 to S930 are Figure 2 The specific implementation method of S240 is shown.

[0146] S910 to S930 are described below:

[0147] S910: extracting temporal redundant information from the connected regions included in the key frame.

[0148] Among them, the temporal redundant information refers to repeated or similar information that appears in the video. Therefore, the temporal redundant information for the filled non-key frame can be extracted in the key frame, and the temporal redundant information is extracted within the connected area. For example, for a pixel in the connected area of ​​the filled non-key frame (i.e., the cropped non-key frame), the temporal redundant information for the pixel can be extracted in the same pixel of the key frame. Therefore, the first device can extract temporal redundant information for each pixel in the connected area of ​​the filled non-key frame from the corresponding pixel in the connected area of ​​the key frame.

[0149] S920, encoding the padded non-key frames contained in the same video frame group based on the temporal redundancy information to obtain encoded non-key frame data.

[0150] Specifically, S920 may include S921 to S922.

[0151] S921 to S922 are described below:

[0152] S921, extracting spatial redundant information from the connected areas included in the filled non-key frames.

[0153] The first device can also extract spatial redundant information from the connected area included in the padded non-key frame. The spatial redundant information can reflect the information between adjacent pixels. Therefore, for each pixel in the connected area included in the padded non-key frame, the first device can extract information of pixels around the pixel as the spatial redundant information.

[0154] S922, encoding the corresponding filled non-key frame based on the temporal redundant information and the spatial redundant information to obtain encoded non-key frame data.

[0155] Since both the temporal redundant information and the spatial redundant information are extracted within the connected region, the data within the connected region of the corresponding filled non-key frame is encoded based on the temporal redundant information and the spatial redundant information. In this way, each element within the connected region of the filled non-key frame has corresponding temporal redundant information and spatial redundant information.

[0156] Specifically, the first device may extract pixel information of the connected area contained in the filled non-key frame, and encode the pixel information of the connected area based on the temporal redundant information and the spatial redundant information to obtain encoded non-key frame data.

[0157] Optionally, the first device may not encode the data in the area other than the connected area of ​​the padded non-key frame, which can reduce the data volume of the video encoding data.

[0158] S930: Generate video encoding data based on the encoded key frame data and the encoded non-key frame data.

[0159] The first device also needs to encode the key frame data through an encoder to obtain encoded key frame data.

[0160] The first device may perform the same sorting on the encoded key frame data and the encoded non-key frame data according to the original order of the key frames and the non-key frames, thereby generating the video encoding data.

[0161] Through this method, after extracting temporal redundant information and spatial redundant information, the first device only encodes the pixels within the connected area contained in the filled non-key frame, and does not encode the pixels outside the connected area, which greatly reduces the number of pixels involved in the encoding, thereby reducing the data amount of the video encoding data finally generated.

[0162] In one embodiment of the present application, another video processing method is provided, which can be performed by a first device and a second device. Fig.10 As shown, the video processing method may include S1001 to S1009.

[0163] S1001 to S1009 are described below:

[0164] S1001: A first device acquires video data.

[0165] The image sensor included in the first device can be used to collect video data in real time.

[0166] S1002: The first device detects a detection object contained in an image frame included in video data, and obtains an object detection frame containing the detection object.

[0167] The first device may use a lightweight pedestrian detection algorithm, such as the YOLOX algorithm, to detect the position and shape of each detection object, thereby determining an object detection frame corresponding to each detection object.

[0168] S1003: The first device connects the object detection frames in the same image frame to obtain a connected area contained in the corresponding image frame.

[0169] The cropping area generation module included in the first device can analyze the object detection frame in the same image frame and can be combined with the cropping algorithm to find the best connected area, which is not limited here.

[0170] S1004: The first device performs picture grouping processing on the video data to obtain a plurality of continuous video frame groups.

[0171] The first device may perform picture grouping processing based on the connected area determined for each image frame in the video data to obtain a plurality of continuous video frame groups.

[0172] S1005: The first device crops non-key frames in each video frame group based on the object detection frame, and obtains cropped non-key frames accordingly.

[0173] The first device retains the images within the connected area contained in the non-key frames in each video frame group, and deletes the images outside the connected area, so as to obtain the cropped non-key frames.

[0174] S1006: The first device performs background filling on the cropped non-key frames in the same video frame group based on the key frames, and obtains a processed video frame group accordingly.

[0175] Specifically, the first device includes a background supplement module, which can directly apply the background information of the key frame to the cropped non-key frame in the same video frame group, and merge the two together to form a complete image frame, that is, a filled non-key frame. Alternatively, a filling template can be generated based on the background information, and the cropped non-key frame is filled with the background through the filling template to obtain a filled non-key frame accordingly. In this way, the image size of the filled non-key frame is the same as the image size of the key frame.

[0176] S1007: The first device performs encoding based on the processed video frame group to obtain video encoding data.

[0177] Specifically, the first device encodes the corresponding filled non-key frame through an encoder based on the temporal redundant information and the spatial redundant information to obtain encoded non-key frame data, and encodes the key frame data through an encoder to obtain encoded key frame data.

[0178] S1008: The first device sends the video encoding data to the second device.

[0179] The first device may send the video encoding data to the second device through the transmission channel.

[0180] S1009: The second device receives the video encoding data, and decodes the video encoding data to obtain video decoding data.

[0181] After receiving the video encoding data, the second device will use a corresponding decoding algorithm to decode the data and restore it to the original video signal, that is, the video decoding data.

[0182] Specifically, the second device can parse the cropped non-key frames, key frames, and connected areas in the key frames contained in each video frame group from the received video encoding data, and perform background filling on the cropped non-key frames in the same video frame group based on the key frames and the connected areas. In this way, each cropped non-key frame can be filled with the background information of the key frame to form a complete image frame, and finally obtain video decoding data.

[0183] Optionally, the second device needs to perform a decoding operation after the received video encoding data reaches a certain data volume, which can avoid the second device frequently turning on and off the decoder.

[0184] After obtaining the decoded video data, the second device may perform action recognition on the detected object in the decoded video data for use by downstream applications.

[0185] In the present application embodiment:

[0186] The first device uses the object detection frame to crop the non-key frames in each video frame group to remove background information in the non-key frames and only retain image information in the connected area. In this way, the image data volume of the non-key frames can be effectively reduced, thereby improving the efficiency of video transmission.

[0187] At the same time, the first device performs background filling on the cropped non-key frames based on the key frames so that their image sizes are consistent with the key frames. This enables the encoder to use the background information of the key frames during the encoding process without encoding the complete image of the non-key frames. Since the key frames already contain the background information, only the image information in the connected area of ​​the non-key frames needs to be encoded, which greatly reduces the amount of data encoded for the non-key frames.

[0188] After receiving the video encoding data with a smaller data volume, the second device can restore the complete video image through the decoding process. This not only reduces the amount of encoded data and improves the efficiency of video transmission, but also ensures the stability of image quality during transmission and ensures that important target information is retained.

[0189] Therefore, the first device can reduce the amount of data during video encoding, improve the efficiency of video transmission, and maintain high image quality. This is of great significance for resource-constrained environments, such as transmitting video data collected by drones to the cloud for processing.

[0190] Fig.11 FIG. 1 is a block diagram of a video processing device according to an embodiment of the present application. Fig.11 As shown, the video processing device can be applied to unmanned aerial vehicle equipment, and the device includes:

[0191] The processing unit 1110 is used to detect a detection object contained in an image frame contained in the video data, and obtain an object detection frame containing the detection object;

[0192] The grouping unit 1120 is used to perform picture grouping processing on the video data to obtain a plurality of continuous video frame groups; wherein each video frame group includes a key frame and a plurality of non-key frames;

[0193] The cropping unit 1130 is used to crop the non-key frames in each video frame group based on the object detection frame to obtain cropped non-key frames, and to perform background filling on the cropped non-key frames in the same video frame group based on the key frames to obtain a processed video frame group;

[0194] The encoding unit 1140 is used to perform encoding based on the processed video frame group to obtain video encoding data.

[0195] In one embodiment of the present application, based on the aforementioned scheme, the processing unit 1110 is further used to extract background information from the key frames; fill the background of the cropped non-key frames in the same video frame group with the background information, and obtain the filled non-key frames accordingly; wherein the image size of the filled non-key frames is the same as the image size of the key frames; and generate a processed video frame group based on the key frames and the filled non-key frames in the same video frame group.

[0196] In one embodiment of the present application, based on the aforementioned scheme, the processing unit 1110 is also used to generate a filling template based on background information, and the image size of the filling template is the same as the image size of the background information; the background of the cropped non-key frame is filled by the filling template, and the filled non-key frame is obtained accordingly.

[0197] In one embodiment of the present application, based on the aforementioned scheme, the processing unit 1110 is further used to connect the object detection frames in the same image frame to obtain the connected areas contained in the corresponding image frame; the cropping unit 1130 is further used to crop the non-key frames in each video frame group based on the connected areas contained in the image frame to obtain cropped non-key frames.

[0198] In one embodiment of the present application, based on the aforementioned scheme, the processing unit 1110 is also used to extract temporal redundant information from the connected area contained in the key frame; the encoding unit 1140 is also used to encode the padded non-key frames contained in the same video frame group based on the temporal redundant information to obtain encoded non-key frame data; the processing unit 1110 is also used to generate video encoding data based on the encoded key frame data and the encoded non-key frame data.

[0199] In one embodiment of the present application, based on the aforementioned scheme, the processing unit 1110 is further used to extract spatial redundant information from the connected area contained in the filled non-key frame; the encoding unit 1140 is further used to encode the corresponding filled non-key frame based on the temporal redundant information and the spatial redundant information to obtain the encoded non-key frame data.

[0200] In one embodiment of the present application, based on the aforementioned scheme, the processing unit 1110 is also used to extract the pixel information of the connected area contained in the filled non-key frame; the encoding unit 1140 is also used to encode the pixel information of the connected area based on the temporal redundant information and the spatial redundant information to obtain the encoded non-key frame data.

[0201] In one embodiment of the present application, based on the aforementioned scheme, the number of object detection frames in the same image frame is two; the processing unit 1110 is further used to determine two corner points on each object detection frame respectively; a polygonal frame is generated based on the determined corner points, and the image area where the polygonal frame is located is used as a connected area.

[0202] In one embodiment of the present application, based on the aforementioned scheme, there are multiple object detection frames in the same image frame; the processing unit 1110 is also used to detect the smallest circle area containing multiple object detection frames in the same image frame, and use the smallest circle area as a connected area.

[0203] In one embodiment of the present application, based on the aforementioned scheme, the processing unit 1110 is further used to perform feature detection on the detection object contained in the image frame in the video data to obtain the position information and body shape information of the detection object; determine the position and size of the object detection frame corresponding to the detection object based on the position information and the body shape information; and generate the object detection frame based on the position and size.

[0204] A video processing system, the system comprising a first device and a second device, wherein:

[0205] The grouping unit 1110 in the first device is used to perform picture grouping processing on the video data to obtain a plurality of continuous video frame groups, each of which includes a key frame and a plurality of non-key frames, and the cropping unit 1130 is used to crop the non-key frames in each video frame group based on the object detection frame included in the image frame in the video data to obtain cropped non-key frames accordingly, and the processing unit 1110 is used to perform background filling on the cropped non-key frames in the same video frame group based on the key frames to obtain processed video frame groups accordingly;

[0206] The encoding unit 1140 in the first device is used to perform encoding based on the processed video frame group to obtain video encoding data, and send the video encoding data to the second device;

[0207] The second device receives the video encoding data, and the processing unit 1110 is further configured to decode the video encoding data to obtain video decoding data.

[0208] In one embodiment of the present application, based on the aforementioned scheme, the processing unit 1110 in the second device is also used to parse the cropped non-key frames, key frames and connected areas in each video frame group from the video encoding data, and perform background filling on the cropped non-key frames in the same video frame group based on the key frames and the connected areas to obtain video decoding data.

[0209] It should be noted that the apparatus provided in the aforementioned embodiment and the method provided in the aforementioned embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment.

[0210] An embodiment of the present application also provides a video processing device, including: one or more processors; a memory for storing one or more programs, when the one or more programs are executed by the one or more processors, the electronic device implements the above-mentioned video processing method.

[0211] Fig.12 It is a structural diagram of a computer system suitable for implementing the video processing device of the embodiment of the present application.

[0212] It should be noted that Fig.12 The computer system 1200 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0213] like Fig.12 As shown, the computer system 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1202 or the program loaded from the storage part 1208 to the random access memory (RAM) 1203, such as executing the method in the above embodiment. In the RAM 1203, various programs and data required for system operation are also stored. The CPU 1201, the ROM 1202 and the RAM 1203 are connected to each other through the bus 1204. The input / output (I / O) interface 1205 is also connected to the bus 1204.

[0214] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1210 as needed so that a computer program read therefrom is installed into the storage section 1208 as needed.

[0215] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1209, and / or installed from a removable medium 1211. When the computer program is executed by a central processing unit (CPU) 1201, various functions defined in the system of the present application are executed.

[0216] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. Computer-readable media may be, for example, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0217] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0218] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0219] Another aspect of the present application further provides a computer readable medium having a computer program stored thereon, which implements the above video processing method when the computer program is executed by a processor. The computer readable medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0220] Another aspect of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable medium. A processor of a computer device reads the computer instruction from the computer-readable medium, and the processor executes the computer instruction, so that the computer device executes the video processing method provided in each of the above embodiments.

[0221] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A video processing method, characterized in that: include: Detecting a detection object contained in an image frame included in the video data, and obtaining an object detection frame containing the detection object; Performing picture group processing on the video data to obtain a plurality of continuous video frame groups; wherein each video frame group includes a key frame and a plurality of non-key frames; Cropping non-key frames in each video frame group based on the object detection frame to obtain cropped non-key frames, and performing background filling on the cropped non-key frames in the same video frame group based on the key frames to obtain a processed video frame group; Encoding is performed based on the processed video frame group to obtain video encoding data.

2. The method according to claim 1, characterized in that The background filling of the cropped non-key frames in the same video frame group based on the key frames to obtain a processed video frame group accordingly includes: Extracting background information from the key frame; The background of the cropped non-key frame in the same video frame group is filled with the background information to obtain the filled non-key frame; wherein the image size of the filled non-key frame is the same as the image size of the key frame; A processed video frame group is generated based on the key frame and the filled non-key frames in the same video frame group.

3. The method according to claim 2, characterized in that The background filling of the cropped non-key frames in the same video frame group by using the background information to obtain the filled non-key frames accordingly includes: generating a filling template based on the background information, wherein the image size of the filling template is the same as the image size of the background information; The cropped non-key frame is background filled using the filling template to obtain a filled non-key frame.

4. The method according to claim 1, characterized in that: The step of cropping the non-key frames in each video frame group based on the object detection frame to obtain cropped non-key frames includes: Connect the object detection frames in the same image frame to obtain the connected areas contained in the corresponding image frame; The non-key frames in each video frame group are cropped based on the connected areas contained in the image frames to obtain cropped non-key frames.

5. The method according to claim 4, characterized in that The encoding based on the processed video frame group to obtain video encoding data includes: Extracting temporal redundant information from the connected regions included in the key frame; Encoding the padded non-key frames contained in the same video frame group based on the temporal redundant information to obtain encoded non-key frame data; The video encoding data is generated based on the encoded key frame data and the encoded non-key frame data.

6. The method according to claim 5, characterized in that The step of encoding the padded non-key frames contained in the same video frame group based on the temporal redundant information to obtain encoded non-key frame data includes: Extracting spatial redundant information from the connected areas contained in the padded non-key frames; The corresponding filled non-key frame is encoded based on the temporal redundant information and the spatial redundant information to obtain the encoded non-key frame data.

7. The method according to claim 6, characterized in that The encoding of the corresponding filled non-key frame based on the temporal redundant information and the spatial redundant information to obtain the encoded non-key frame data includes: Extract pixel information of connected areas contained in the filled non-key frames; The pixel information of the connected area is encoded based on the temporal redundant information and the spatial redundant information to obtain the encoded non-key frame data.

8. The method according to claim 4, characterized in that The number of object detection boxes in the same image frame is two; The connecting the object detection frames in the same image frame to obtain the connected regions contained in the corresponding image frame includes: Determine two corner points on each object detection box; A polygonal frame is generated based on the determined corner points, and the image area where the polygonal frame is located is used as the connected area.

9. The method according to claim 4, characterized in that There are multiple object detection frames in the same image frame; The connecting the object detection frames in the same image frame to obtain the connected regions contained in the corresponding image frame includes: A minimum circle area including a plurality of object detection frames is detected in the same image frame, and the minimum circle area is used as the connected area.

10. The method according to claim 1, characterized in that The detecting object contained in the image frame contained in the video data, and obtaining an object detection frame containing the detected object, comprises: Performing feature detection on a detection object contained in an image frame in the video data to obtain position information and shape information of the detection object; Determine the position and size of the object detection frame corresponding to the detection object based on the position information and the body shape information; The object detection frame is generated based on the position and the size.

11. A video processing system, characterized in that: The system comprises a first device and a second device, wherein: The first device performs picture group processing on the video data to obtain a plurality of continuous video frame groups, each of which includes a key frame and a plurality of non-key frames, and crops the non-key frames in each video frame group based on an object detection frame included in an image frame in the video data to obtain cropped non-key frames, and performs background filling on the cropped non-key frames in the same video frame group based on the key frames to obtain processed video frame groups; The first device performs encoding based on the processed video frame group to obtain video encoding data, and sends the video encoding data to the second device; The second device receives the video encoding data, and decodes the video encoding data to obtain video decoding data.

12. The system according to claim 11, characterized in that: The second device parses the video encoding data to obtain the cropped non-key frames, key frames and connected areas in each video frame group, and performs background filling on the cropped non-key frames in the same video frame group based on the key frames and the connected areas to obtain the video decoding data.

13. A video processing device, characterized in that: include: A processing unit, configured to detect a detection object contained in an image frame contained in the video data, and obtain an object detection frame containing the detection object; A grouping unit, used for performing picture grouping processing on the video data to obtain a plurality of continuous video frame groups; wherein each video frame group includes a key frame and a plurality of non-key frames; A cropping unit, configured to crop non-key frames in each video frame group based on the object detection frame to obtain cropped non-key frames, and to perform background filling on the cropped non-key frames in the same video frame group based on the key frames to obtain a processed video frame group; The encoding unit is used to perform encoding based on the processed video frame group to obtain video encoding data.

14. A video processing device, characterized in that: include: a memory storing computer-readable instructions; A processor reads the computer-readable instructions stored in the memory to execute the method according to any one of claims 1 to 10.

15. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the method according to any one of claims 1 to 10.

16. A computer program product, characterized in that The computer program product comprises a computer program, the computer program is stored in a computer-readable storage medium, a processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device performs the method according to any one of claims 1 to 10.

Images