Video processing method, device, electronic device and storage medium

Abstract

The embodiments of the present invention provide a video processing method, device, electronic device and storage medium. The method includes: determining the amount of video picture change generated by a video picture corresponding to a first video frame relative to a video picture corresponding to a second video frame, wherein the first video frame and the second video frame are different forward prediction coding frames belonging to the same image group; determining a target detection result corresponding to the first video frame based on the amount of video picture change and the target detection result corresponding to the second video frame, wherein the target detection result is used to indicate the detection result of the target object in the video picture corresponding to the video frame; blurring the video picture corresponding to the first video frame based on the target detection result corresponding to the first video frame. The technical solution of the present application can reduce the number of calls to the algorithm interface while maintaining the accuracy and real-time performance of mosaic processing of video frames.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of video processing technology, and in particular to a video processing method, device, electronic device and storage medium. Background Art

[0002] In today's era of rapid development of the Internet, while information dissemination and sharing are convenient, the problem of personal privacy leakage is becoming more and more serious. Video mosaic can effectively reduce the risk of privacy leakage. The relevant solution is generally to call the algorithm interface, use image recognition and other technologies to detect the target area in the video, and then apply a mosaic effect to the area to blur the content. However, with the development of video technology, the resolution has increased and the demand for real-time processing has increased. Although the execution speed of the algorithm interface has been optimized to the limit, the overall process is still too time-consuming, which seriously affects the efficiency of video processing. It cannot meet the actual needs in scenes with high real-time requirements such as live broadcast and real-time monitoring, which has become an important problem facing current video processing. Summary of the invention

[0003] The present invention provides a video processing method, device, electronic device and storage medium, so as to reduce the number of times of calling an algorithm interface while maintaining the accuracy and real-time performance of mosaic processing on video frames.

[0004] In a first aspect, an embodiment of the present invention provides a video processing method, the method comprising:

[0005] Determining a video picture change amount generated by a video picture corresponding to a first video frame relative to a video picture corresponding to a second video frame, wherein the first video frame and the second video frame are different forward prediction coded frames belonging to the same picture group, and the picture group to which the first video frame and the second video frame belong only includes independently coded key frames and forward prediction coded frames;

[0006] Determine the target detection result corresponding to the first video frame based on the video picture change amount and the target detection result corresponding to the second video frame, wherein the target detection result is used to indicate the detection result of detecting the target object in the video picture corresponding to the video frame;

[0007] Based on the target detection result corresponding to the first video frame, the video picture corresponding to the first video frame is blurred.

[0008] In a second aspect, an embodiment of the present invention further provides a video processing device, the device comprising:

[0009] A first determination module is used to determine a video picture change amount generated by a video picture corresponding to a first video frame relative to a video picture corresponding to a second video frame, wherein the first video frame and the second video frame are different forward prediction coding frames belonging to the same picture group, and the picture group to which the first video frame and the second video frame belong only includes independently encoded key frames and forward prediction coding frames;

[0010] A second determination module is used to determine the target detection result corresponding to the first video frame based on the video picture change amount and the target detection result corresponding to the second video frame, wherein the target detection result is used to indicate the detection result of detecting the target object in the video picture corresponding to the video frame;

[0011] A blur processing module is used to blur the video image corresponding to the first video frame based on the target detection result corresponding to the first video frame.

[0012] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor so that the at least one processor can perform the video processing method described in any one of the above embodiments.

[0016] In a fourth aspect, an embodiment of the present invention further provides a computer-readable medium, wherein the computer-readable medium stores computer instructions, and the computer instructions are used to enable a processor to implement the video processing method described in any one of the above embodiments when executed.

[0017] According to the technical solution of the embodiment of the present invention, during the video blur processing, the video picture change amount will be analyzed for the first video frame and the second video frame of different forward prediction coding frames belonging to the same image group. By accurately analyzing the video picture change amount and combining the target detection result of the second video frame that has completed target detection, the target detection result of the first video frame can be determined without the need to perform the target detection algorithm on the first video frame. This method avoids the tedious operation of re-performing a complex target detection algorithm on each video frame, reduces unnecessary calculations, and significantly reduces the frequency of calling the algorithm interface due to the significant reduction in repeated target object detection operations. In this way, when facing complex scenes where a large number of video frames need to be processed, the processing time can be significantly shortened, the efficiency of the entire video blur processing process can be improved, and the performance bottleneck problem of time-consuming algorithm interface calls can be effectively alleviated, so that when processing high-resolution videos or real-time processing requirements, good performance can still be maintained, avoiding problems such as freezes caused by slow processing speeds.

[0018] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other features, advantages and aspects of the embodiments of the present invention will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0020] Figure 1 is a flow chart of a video processing method provided by an embodiment of the present invention;

[0021] Figure 2 is a flow chart of another video processing method provided by an embodiment of the present invention;

[0022] Figure 3 is a specific schematic diagram of a video processing process provided by an embodiment of the present invention;

[0023] Figure 4 It is a block diagram of a video frame corresponding to a video screen provided by an embodiment of the present invention;

[0024] Figure 5 A schematic diagram of the structure of a video processing device provided by an embodiment of the present invention;

[0025] Figure 6 It is a structural schematic diagram of an electronic device for implementing a video processing method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0027] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0028] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0029] It should be noted that the concepts such as "first" and "second" mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0030] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0031] Figure 1 A flow chart of a video processing method provided in an embodiment of the present invention. The technical solution of the embodiment of the present invention is applicable to the situation where blurring is performed on a partial picture area in a video picture. The method can be executed by a video processing device. The video processing device can be implemented in the form of software and / or hardware and is generally integrated on any electronic device with network communication function, which can be a mobile terminal, a PC or a server, etc.

[0032] like Figure 1 As shown, the video processing method of the embodiment of the present invention may include the following process:

[0033] S110. Determine a video picture change amount caused by a video picture corresponding to the first video frame relative to a video picture corresponding to the second video frame, wherein the first video frame and the second video frame are different forward prediction coding frames belonging to the same picture group, and the picture group to which the first video frame and the second video frame belong only includes independently encoded key frames and forward prediction coding frames.

[0034] For videos such as H264 and H265 that use inter-frame compression technology for video encoding, their video sequences will be composed of several GOPs (Group of Pictures). Due to the high real-time requirements in the security field, bidirectional predictive coding frames are usually not used. Therefore, in this case, the first video frame in each picture group GOP is a key frame, and the video frames that follow the key frame in the same picture group GOP in the playback order are all forward predictive coding frames. On this basis, the picture group to which the first video frame and the second video frame belong only includes independently encoded key frames and forward predictive coding frames. The first video frame and the second video frame belong to the same picture group and are different forward predictive coding frames. The first video frame is played after the second video frame in the picture group. The video picture change is a measure of the degree of difference in the video picture between the first video frame and the second video frame.

[0035] By parsing two different forward prediction coding frames belonging to the same image group, the video picture corresponding to the first video frame and the video picture corresponding to the second video frame are obtained, and then the video picture change amount caused by the video picture corresponding to the first video frame relative to the video picture corresponding to the second video frame is calculated, and the picture change degree of the video picture corresponding to the first video frame relative to the video picture corresponding to the second video frame is obtained.

[0036] S120. Determine a target detection result corresponding to the first video frame based on the video picture change amount and the target detection result corresponding to the second video frame, where the target detection result is used to indicate a detection result of detecting a target object in the video picture corresponding to the video frame.

[0037] In order to mosaic the target object in the video, it is usually necessary to detect the target object and apply mosaic to the video screen corresponding to each frame of the video. In other words, the process of mosaicing the target object in the video involves calling the target detection algorithm interface to detect the target object, and then applying the mosaic effect to the target object. However, with the increase in video resolution and the increase in real-time processing requirements, the time consumption of calling the target detection algorithm interface has become a bottleneck of system performance. Although the execution speed of the target detection algorithm interface has been optimized to the limit, the overall process is still time-consuming, which will lead to low processing efficiency, especially when processing high-resolution or high-frame rate videos.

[0038] Based on the above situation, after calling the target detection algorithm interface to perform target object detection on the video screen corresponding to the second video frame to obtain the target detection result corresponding to the second video frame, considering that the video screen change can reflect the difference between the first video frame and the second video frame, if the video screen change is very small, it indicates that the target object in the video screen corresponding to the first video frame should be basically the same as the target object in the video screen corresponding to the second video frame and will not change much. Therefore, the intrinsic correlation between the video screens of the first video frame and the second video frame can be used to conditionally pass the target detection result of the second video frame to the first video frame for direct use, so as to achieve the effect of predicting and determining the target detection result corresponding to the first video frame based on the target detection result corresponding to the second video frame, so as to reduce the probability of calling the target detection algorithm interface to detect the target object on the first video frame. Among them, the target detection result is used to represent the detection of the target object in the video screen corresponding to the video frame, including information such as the position of the target object and the category of the target object.

[0039] By adopting the above scheme, it is possible to avoid re-calling the target detection algorithm interface to perform complex target object detection operations on the first video frame as much as possible, thereby reducing the large amount of computing resources and time consumed by re-calling the target detection algorithm interface for detection, and reducing unnecessary calculations by using the existing target detection result information. Especially when processing a large number of video frames, the link of independent detection of each video frame is reduced in the overall process of video processing, which speeds up the processing speed and significantly improves efficiency. In addition, by reasonably using the target detection results and picture changes of the second video frame, the target detection results of the first video frame can be determined more accurately. As long as the picture change evaluation is accurate and the association rules are reasonable, the reliability of target object detection can be guaranteed while reducing calculations.

[0040] As an optional but non-limiting implementation, determining the video picture change amount generated by the video picture corresponding to the first video frame relative to the video picture corresponding to the second video frame may include the following steps A1-A2:

[0041] Step A1, determine a first data amount associated with a first video frame, the first data amount is the data amount generated by accumulating data amounts associated with at least one first reference frame, the data amount associated with the first reference frame is the motion vector data and residual value data of the first reference frame compared to a previous video frame in the image group to which the first reference frame belongs, the first reference frame includes a first video frame or a video frame located between the first video frame and the second video frame and including the first video frame, the first video frame is a video frame that needs to rely on the second reference frame for predictive encoding in the video encoding process to restore the complete video picture, and the second reference frame is the second video frame or a video frame located between the first video frame and the second video frame and including the second video frame.

[0042] Step A2: Determine, based on the first data amount associated with the first video frame, a video picture change amount generated by the video picture corresponding to the first video frame relative to the video picture corresponding to the second video frame.

[0043] Forward predictive coding frame (P frame) is a frame type in video coding. In the process of video coding using inter-frame compression technology (such as H264, H265), P frame is mainly encoded by referring to the previous I frame (key frame) or other P frame. P frame does not record complete image information like I frame, but only records motion vector data and residual value data. The data volume of P frame is usually much smaller than that of key frame (I frame), and the smaller the video picture changes, the smaller the P frame data volume. This is because P frame is encoded based on the difference of the previous video frame. Small changes in the video picture mean small differences from the video frame, so less motion vector and residual information need to be recorded, and the data volume is naturally small.

[0044] According to the above situation, it is not difficult to find that for encoding methods such as H264 and H265 that use inter-frame compression technology, only the key frame (I frame) records complete image information, and the data volume is usually large, while the forward prediction coding frame (P frame) only records motion vector data and residual value data (among which, the security field has high real-time requirements, and usually does not use bidirectional prediction coding frames, which are not considered), and the data volume is usually much smaller than the key frame data volume, and the smaller the video picture changes, the smaller the data volume of the forward prediction coding frame. Using this feature, by detecting whether the data volume of the forward prediction coding frame is small enough, it can be determined whether the target detection result corresponding to the video frame before the forward prediction coding frame can be used, thereby achieving the purpose of reducing the number of calls to the target detection algorithm interface.

[0045] In video processing, in order to accurately evaluate the amount of change in the video picture, the concept of first data volume is introduced. The first data volume can reflect the degree of association and difference between the video picture corresponding to the second video frame and the video picture corresponding to the first video frame from a quantitative perspective. The first reference frame is one of the core elements of the entire calculation process. There are two specific cases. One is the first video frame itself, and the other is a number of video frames located between the first video frame and the second video frame and including the first video frame. The first data volume associated with the first video frame is the data volume generated by accumulating the data volume associated with at least one first reference frame.

[0046] For each first reference frame, it is necessary to calculate the amount of data associated with the first reference frame. The amount of data associated with the first reference frame is composed of the motion vector data and residual value data of the first reference frame compared to the previous video frame in the image group to which it belongs. In the video encoding process, the motion vector data and the residual value data contain dynamic change information between video frames. For example, the motion vector data can represent the direction and speed of movement of an object in the picture, and the residual value data reflects the difference in pixel value between the current frame and the previous frame. By obtaining these data, the change of each first reference frame relative to its previous frame can be understood. By accumulating the amount of data associated with all first reference frames, the first data amount is obtained. This accumulation process is to summarize the picture change information carried by each first reference frame to form a comprehensive indicator for measuring the overall degree of change related to the first video frame.

[0047] After obtaining the first data volume associated with the first video frame, the video picture change amount generated by the video picture corresponding to the first video frame relative to the video picture corresponding to the second video frame can be determined according to the first data volume associated with the first video frame. There is an inherent connection between the first data volume and the video picture change amount. The size of the first data volume can reflect the degree of picture change between the video pictures corresponding to the two video frames to a certain extent. This connection is established based on the inherent law of video coding and picture change. For example, when the first data volume is large, it may mean that there is a large change in the picture between the first video frame and the second video frame, which may be caused by violent object movement, scene switching or other factors; on the contrary, when the first data volume is small, it means that the change between the two video pictures is relatively small. On this basis, this feature can be used to determine whether the target detection result corresponding to the previous video frame can be used by detecting whether the data volume of the forward prediction coding frame is small enough, so as to reduce the number of calls to the target detection algorithm interface, thereby optimizing the overall performance consumption of the system and reducing the overall processing time of the system.

[0048] The above technical solution makes full use of the motion vector data and residual value data generated during the video encoding process, which originally exist in the video encoding structure. By tapping into the value of these data, there is no need to introduce additional complex image analysis algorithms to determine picture changes, thereby improving computing efficiency and reducing unnecessary computing resource consumption. By accumulating the amount of data in the first reference frame to determine the amount of video picture change, the picture change is quantified. This quantification method is more accurate and detailed than traditional methods such as those based on image similarity. It can not only determine whether the picture has changed, but also understand the degree of change from the amount of data. At the same time, it can achieve the goal of reducing the number of calls to the target detection algorithm interface, thereby optimizing the overall performance consumption of the system and reducing the overall processing time of the system.

[0049] S130: Based on the target detection result corresponding to the first video frame, blur the video image corresponding to the first video frame.

[0050] When the target detection result corresponding to the first video frame is determined, the target object position and target object type indicated in the target detection result corresponding to the first video frame can be used to blur the corresponding target object area in the video screen corresponding to the first video frame. For example, a mosaic or other form of occlusion can be added to the area where the target object is located through a specific algorithm or software tool to process the specific target object area in the video screen to protect privacy or sensitive information. In the above manner, by accurately blurring the target object in the first video frame, it is possible to effectively prevent the leakage of sensitive information such as privacy information and confidential content in the video.

[0051] According to the technical solution of the embodiment of the present invention, during the video blur processing, the video picture change amount will be analyzed for the first video frame and the second video frame of different forward prediction coding frames belonging to the same image group. By accurately analyzing the video picture change amount and combining the target detection result of the second video frame that has completed target detection, the target detection result of the first video frame can be determined without the need to perform the target detection algorithm on the first video frame. This method avoids the tedious operation of re-performing a complex target detection algorithm on each video frame, reduces unnecessary calculations, and significantly reduces the frequency of calling the algorithm interface due to the significant reduction in repeated target object detection operations. In this way, when facing complex scenes where a large number of video frames need to be processed, the processing time can be significantly shortened, the efficiency of the entire video blur processing process can be improved, and the performance bottleneck problem of time-consuming algorithm interface calls can be effectively alleviated, so that when processing high-resolution videos or real-time processing requirements, good performance can still be maintained, avoiding problems such as freezes caused by slow processing speeds.

[0052] Figure 2 A flow chart of another video processing method provided for an embodiment of the present invention. The technical scheme of this embodiment further optimizes the process of determining the target detection result corresponding to the first video frame based on the video picture change amount and the target detection result corresponding to the second video frame in the aforementioned embodiment on the basis of the technical scheme of the aforementioned embodiment. This embodiment can be combined with various optional schemes in one or more of the aforementioned embodiments.

[0053] like Figure 2 As shown, the video processing method of the embodiment of the present invention may include the following process:

[0054] S210, determining a video picture change amount generated by a video picture corresponding to a first video frame relative to a video picture corresponding to a second video frame, the first video frame and the second video frame being different forward prediction coding frames belonging to the same picture group.

[0055] S220. Determine a reference video picture change threshold to be used for the first video frame relative to the second video frame. The reference video picture change threshold is a threshold for the video picture corresponding to the first video frame and the video picture corresponding to the second video frame, which can make the similarity between the target detection result corresponding to the first video frame and the target detection result corresponding to the second video frame greater than an upper limit of the video picture change that needs to be satisfied for a preset similarity.

[0056] In video processing, in order to efficiently and accurately determine whether the target detection results of each video frame can be reused, this solution introduces the key indicator of the reference video picture change threshold, which will be used as the basis for judging whether to reuse the target detection results corresponding to the existing video frames. The reference video picture change threshold can be set for the video pictures corresponding to the first video frame and the second video frame. The reference video picture change threshold represents an upper limit value, which is closely related to the similarity of the target detection results.

[0057] Specifically, when the video picture change between the first video frame and the second video frame is within this threshold range, it can be ensured that the similarity between the target detection result corresponding to the first video frame and the target detection result corresponding to the second video frame is greater than the preset similarity. For example, if the preset similarity is set to 80%, then this threshold is the upper limit of the picture change that allows the similarity of the target detection results of the two video frames to reach this level. When calculating this threshold, it is necessary to comprehensively consider factors such as various characteristics of the video picture, the nature of the target object, and the characteristics of the target detection algorithm.

[0058] S230: If the video picture change amount is not greater than the reference video picture change amount threshold, directly multiplex the target detection result corresponding to the second video frame as the target detection result corresponding to the first video frame.

[0059] The target detection result is used to indicate the detection result of the target object in the video picture corresponding to the video frame.

[0060] When the calculated video picture change is not greater than the reference video picture change threshold, it means that the picture changes of the first video frame and the second video frame are small. In this case, from the perspective of video processing efficiency, the target detection result corresponding to the second video frame can be directly reused as the target detection result corresponding to the first video frame. Because the smaller picture change indicates that the position, features and other information of the target object in the two video frames do not change much, the previous target detection result of the second video frame still has a high reference value. In other words, when the video picture changes corresponding to the two video frames are small enough, the target detection results corresponding to the two video frames should also be similar enough, then the latter video frame can not perform the target object detection operation, and directly use the target detection result corresponding to the previous video frame, so as to realize the application of blurring processing (such as adding mosaic effect, etc.) in the target object area, so as to avoid the complex target object detection operation on the first video frame again, saving computing resources and time.

[0061] By setting the reference video picture change threshold to determine whether to reuse the target detection result, the repeated calls of the target detection algorithm are greatly reduced when the picture changes are small. This is particularly effective for processing a large number of video frames, and can significantly shorten the time spent on the target detection link in the entire video processing process, thereby improving processing efficiency. In addition, this processing method can avoid unnecessary target object detection operations and reduce the consumption of computing resources, such as CPU, memory and other resources, so that the video processing system can allocate resources more reasonably when processing videos, while ensuring processing quality, and improving the overall performance and stability of the system.

[0062] S240: If the video picture change amount is greater than the reference video picture change amount threshold, a target object detection operation is performed on the video picture corresponding to the first video frame to obtain a target detection result corresponding to the first video frame.

[0063] If it is found through calculation that the video picture change is greater than the reference video picture change threshold, this indicates that there is a significant change in the picture between the first video frame and the second video frame. This significant change may cause the position, form or quantity of the target objects in the video picture corresponding to the first video frame and the video picture corresponding to the second video frame to change significantly. At this time, in order to ensure the accuracy of the target detection result, it is necessary to re-call the target detection algorithm interface to perform the target object detection operation on the video picture corresponding to the first video frame, analyze each pixel and area in the first video frame, and obtain the target detection result corresponding to the first video frame.

[0064] When the picture changes greatly, the first video frame is re-detected in time to ensure that the target detection result can accurately reflect the actual situation of the target object in the video picture. This processing method will flexibly adjust the detection strategy according to the picture changes, avoiding target detection errors that may be caused by reusing old results, and ensuring the accuracy of target detection. The above scheme fully considers that the content of the video picture is often dynamically changing, and can adapt well to this dynamic nature. Whether it is a slight shake of the scene, the slow movement of the object, or a large picture change caused by scene switching, rapid movement of objects, etc., the quality of the target detection result can be guaranteed through a reasonable strategy, making the entire video processing process more robust and reliable.

[0065] As an optional but non-limiting implementation, determining a reference video picture change threshold to be used for the first video frame relative to the second video frame includes the following steps B1-B2:

[0066] Step B1, determine the similarity between the target detection result corresponding to the second video frame and the target detection result corresponding to the third video frame, the third video frame and the second video frame belong to the same image group, and the third video frame is a forward prediction coding frame or key frame that is located before and adjacent to the second video frame.

[0067] Step B2: If the similarity between the target detection result corresponding to the second video frame and the target detection result corresponding to the third video frame is not less than a preset similarity threshold, the second data amount associated with the second video frame is determined as the reference video picture change threshold to be adopted by the first video frame relative to the second video frame.

[0068] The second data amount associated with the second video frame is the amount of data generated by the motion vector data and the residual value data of the second video frame compared with the third video frame.

[0069] An important phenomenon was discovered during video processing: when the target detection results of two consecutive video frames show a sufficiently high similarity, this often means that the degree of change in the video picture between the two video frames may be relatively small. In other words, in this case, the amount of change in the video picture of a video frame relative to the previous video frame is within a specific range, and this range just meets the condition that the detection results of the two video frames are highly similar.

[0070] Especially when two video frames are encoded in a coding method such as H264 and H265 that uses inter-frame compression technology, there is a correlation between the degree of change in the video picture and the amount of data in the forward prediction coding frame. Specifically, the smaller the change in the video picture, the smaller the amount of data contained in the forward prediction coding frame. This is because in this coding method, the characteristics of the forward prediction coding frame determine that it mainly records the change information in the video picture. When the video picture changes less, naturally less content needs to be recorded, and the amount of data is correspondingly reduced.

[0071] Specifically, assuming that in these two consecutive video frames, the latter video frame happens to be a forward predictive coding frame, then this provides an important clue: the amount of video picture changes recorded by this forward predictive coding frame is small enough to make the detection results of the two video frames highly similar. From the perspective of data volume, this means that the amount of data carried in the forward predictive coding frame is small enough to meet the condition of high similarity.

[0072] If the data size of this specific forward prediction coding frame is set to threshold A, and this forward prediction coding frame is marked as reference frame A, and the target detection result of reference frame A is defined as reference result A. Then, several consecutive forward prediction coding frames starting from reference frame A form a new video sequence. As long as the total data size accumulated by these consecutive forward prediction coding frames does not exceed threshold A, it means that the accumulated video picture changes in this continuous process are small enough. This small video picture change can ensure that the target detection result of the current video frame to be processed maintains a highly similar relationship with reference result A.

[0073] In video processing, in order to accurately evaluate the impact of video picture changes between video frames on target detection results, it is necessary to first determine the second video frame and the third video frame. The third video frame has a specific relationship with the second video frame and belongs to the same image group. In addition, the third video frame is a forward prediction coding frame or key frame located before and adjacent to the second video frame. This adjacent and specific type of frame relationship makes the second video frame and the third video frame closely connected in time and coding logic. The similarity of their target detection results can reflect the impact of picture changes in a short period of time on target information. For example, in a surveillance video, two adjacent frames may only have a slight movement of an object or a slight change in illumination. The similarity between the target detection results corresponding to the two video frames is calculated.

[0074] When it is detected that the similarity between the target detection result corresponding to the second video frame and the target detection result corresponding to the third video frame is not less than the preset similarity threshold, this indicates that there is basically no significant change in the target detection result of the target object detection between the video images corresponding to the two video frames, which means that the position and state of the target object in the image have not changed significantly, or no new target object has appeared, etc. At this time, the second data amount associated with the second video frame can be directly determined as the reference video image change threshold to be adopted by the first video frame relative to the second video frame. Subsequently, as long as the image change between the two video frames does not exceed this reference video image change threshold, it can be considered that the target detection results of the video images corresponding to the two video frames are basically consistent. The second data amount here is the amount of data generated by the motion vector data and residual value data of the second video frame compared to the third video frame.

[0075] Among them, the motion vector data reflects the information such as the movement direction and speed of the object in the picture, and the residual value data reflects the difference in pixel values ​​between the two frames. The threshold determined in this way can comprehensively consider the dynamic changes of the picture and provide a quantitative standard for the subsequent judgment of the relationship between the first video frame and the second video frame. For example, if in a traffic monitoring video, a car passes by quickly, causing the similarity of the target detection results of the second video frame and the third video frame to decrease, then the reference video picture change threshold is determined based on the motion vector and residual value generated between the second video frame and the third video frame.

[0076] The above technical solution can automatically determine the reference video picture change threshold according to the similarity of the target detection results between the video frames. This adaptive method makes the threshold setting more in line with the actual changes in the video content, rather than using a fixed threshold, thereby improving the adaptability of the entire video processing system to different types of videos. By using the motion vector data and residual value data of the second video frame compared to the third video frame to determine the threshold, the existing information in the video encoding process is fully mined, and additional complex calculations are avoided to determine the threshold, thereby improving the calculation efficiency. Based on this reference video picture change threshold, it is possible to more accurately determine whether it is necessary to re-detect the target when the relationship between the first video frame and the second video frame is subsequently processed. If the picture change between the first video frame and the second video frame is within the threshold range, the previous detection results can be reasonably reused to reduce unnecessary target detection operations, thereby improving the efficiency of the entire video processing process.

[0077] For example, see Figure 3, define the video frames belonging to the same image group GOP as F0, F1, F2...Fn-1, Fn in the order of playback. Define Fi as the i-th video frame in GOP, and the target detection result obtained by calling the algorithm interface for Fi is Ri. The data size of the Fi frame (the Fi frame can be an I frame or a P frame in the image group) is Di. On this basis, it is assumed that there is a similarity detection scheme that can give a quantitative value for the similarity of the target detection results corresponding to the two groups of video frames, which is defined as the similarity L. A similarity threshold Lt can be defined. When the similarity of the target detection results corresponding to two consecutive video frames is not less than the similarity threshold Lt, the two groups of target detection results are considered to be highly similar. Define the sum of the data sizes of several consecutive forward reference frames as the cumulative difference. If the similarity between Ri and Ri+1 is lower than Lt, then perform target detection on Ri+2, and use Ri+1 as Ri and Ri+2 as Ri+1. If the similarity between Ri and Ri+1 is not less than Lt, that is, Ri and Ri+1 are highly similar, the data size of the Fi+1 frame (Fi+1 is the P frame in the image group) is set to the cumulative difference threshold Dt (that is, the reference video picture change threshold to be calculated above). Define the cumulative difference of several forward reference frames after the Ri+1 video frame as Dsum (recorded as the first data amount associated with the video frame after Ri+1):

[0078]

[0079] Based on the above-mentioned cumulative difference, if Dsum is not greater than Dt, the video frames Fi+2 to Fj do not need to call the target detection algorithm interface to perform target object detection operations, and can be directly mosaiced according to the target detection frame of the target object Ri+1; if Dsum+Dj+1 is greater than Dt, it is necessary to perform target object detection operations on Fj+1 and Fj+2, and use Rj+1 as Ri and Rj+2 as Ri+1.

[0080] As an optional but non-limiting implementation, determining the similarity between the target detection result corresponding to the second video frame and the target detection result corresponding to the third video frame includes the following steps C1-C4:

[0081] Step C1, determining the number of target objects in the target detection result corresponding to the second video frame and the number of target objects in the target detection result corresponding to the third video frame.

[0082] Step C2: If the numbers of the two target objects are different, determine that the similarity between the target detection result corresponding to the second video frame and the target detection result corresponding to the third video frame is a preset value, and the preset value is used to indicate that the similarity between the two target detection results is 0.

[0083] Step C3: If the number of two target objects is the same, determine the first size information and the second size information, the first size information is used to indicate the overlapping position area size of the target detection frame of the same target object in the target detection result corresponding to the second video frame and the target detection result corresponding to the third video frame, and the second size information is used to indicate the union area size of the position area range covered by the two target detection frames of the same target object in the target detection result corresponding to the second video frame and the target detection result corresponding to the third video frame.

[0084] Step C4: determining the similarity between the target detection result corresponding to the second video frame and the target detection result corresponding to the third video frame according to the size ratio of the first size information to the second size information.

[0085] In the target detection analysis phase of video processing, it is necessary to first determine the number of target objects in the target detection results corresponding to the second video frame and the third video frame. The number of target objects can intuitively reflect the basic situation of the target in the picture. For example, in a surveillance video, the target objects may be pedestrians, vehicles, etc. By determining the changes in their number, we can preliminarily understand the changes in the content of the picture. Optionally, by extracting data from the target detection results that have been completed in the second video frame and the third video frame, the number of target objects therein is counted respectively, which involves the analysis of the output results of the target detection algorithm. The target detection algorithm usually identifies the target in the picture and gives corresponding information, including the key data of the number of target objects.

[0086] When it is found that the number of target objects in the target detection results corresponding to the second video frame and the third video frame is different, this means that the picture between the two frames has changed significantly at the target level. It may be that a new target has entered the picture, or the original target has left the picture. In this case, the similarity between the target detection result corresponding to the second video frame and the target detection result corresponding to the third video frame can be further determined as a preset value. The preset value here is set to indicate that the similarity of the two target detection results is 0, that is, the two target detection results are considered different. This setting is based on the significant difference in the number of target objects, indicating that the content of the two frames has undergone essential changes in target composition.

[0087] When it is found that the number of target objects in the target detection results corresponding to the second video frame and the third video frame is the same, it means that the video images corresponding to the two video frames have certain similarities in the dimension of the number of target objects. However, this is not enough to completely determine the similarity between the two video frames, and further analysis is needed of information such as the position and size of the target object in the image. At this time, it is necessary to determine the first size information and the second size information, and further use the first size information and the second size information to determine the similarity between the target detection result corresponding to the second video frame and the target detection result corresponding to the third video frame.

[0088] Specifically, the first size information is used to indicate the size of the overlapping position area of ​​the target detection frame for the same target object in the target detection result corresponding to the second video frame and the target detection result corresponding to the third video frame. The target detection frame is a rectangular frame used to identify the position and range of the target object in the picture in the target detection algorithm. By analyzing the overlapping parts of the target detection frames of the same target object in two frames, the relative change of the position and size of the target between the two frames can be determined. To calculate the size of this overlapping area, it is necessary to use geometric calculation methods to analyze the coordinates and sizes of the two target detection frames.

[0089] Specifically, the second size information is used to indicate the size of the union area of ​​the position area covered by the two target detection frames of the same target object in the target detection result corresponding to the second video frame and the target detection result corresponding to the third video frame. Specifically, it is obtained by merging the ranges covered by the target detection frames of the same target object in the two video frames. That is to say, the difference between the sum of the position areas of the two target detection frames of the same target object in the target detection result corresponding to the second video frame and the target detection result corresponding to the third video frame and the overlapping area size indicated by the first size information reflects the overall occupation of the screen by the target object in the two frames. Similarly, the calculation of this second size information also needs to be calculated based on the coordinates and size information of the target detection frame.

[0090] After obtaining the first size information and the second size information, the size ratio of the first size information and the second size information is calculated by intersection and union calculation to determine the similarity between the target detection result corresponding to the second video frame and the target detection result corresponding to the third video frame. This ratio can more accurately measure the degree of change of the target object between the two frames. For example, if the size ratio is close to 1, it means that the position and size of the target object in the two frames have changed little, and the similarity of the two frames is high; conversely, if the size ratio is small, it means that the target object has a large position or size change between the two frames, and the similarity of the two frames is low. This similarity calculation method based on size ratio comprehensively considers the spatial position and size changes of the target object in the picture, and is more detailed and accurate than judgment based solely on the number of target objects.

[0091] Exemplarily, for the target detection results corresponding to two consecutive video frames, the following is a feasible similarity detection scheme: the prerequisite for calculating the similarity of the target detection results corresponding to two consecutive video frames is that the number of detected target objects indicated by the target detection results corresponding to the two consecutive video frames is the same. If the number of target objects is different, the similarity of the target detection results corresponding to the two consecutive video frames is considered to be 0. When the area of ​​the overlapping part of the target detection frame corresponding to the target object in the target detection results corresponding to the two consecutive video frames is larger, that is, the proportion of the overlapping part area of ​​the target detection frame corresponding to the two target objects to the union area size of the position area range covered by the two target detection frames is larger, it means that the target detection frames corresponding to the two target objects are more similar, and the value of the proportion can be used as the quantized value L of the similarity. When the number of targets corresponding to the target detection results of the two consecutive video frames is the same, the video frame with the earlier display order is defined as FA, the video frame with the later display order is defined as FB, the area of ​​FA is defined as SA, the area of ​​FB is defined as SB, and the area of ​​the overlapping part of FA and FB is defined as SAB, then the similarity of the target detection frames corresponding to the two target objects is:

[0092]

[0093] Assuming that the number of target objects in the target detection results corresponding to two consecutive video frames is n, the similarity of the target detection results corresponding to two consecutive video frames is:

[0094]

[0095] By comprehensively considering factors such as the number of target objects, the overlapping area of ​​the target detection frame, and the size of the union area indicated by the second size information to determine the similarity of the target detection results, it can more accurately reflect the similarity between the two frames of video at the target level. This multi-dimensional evaluation method avoids the limitations of single-factor judgment and makes the similarity evaluation results more in line with the actual situation. In complex video scenes, the number, position, and size of target objects may change frequently. The above method can effectively cope with this complexity. Whether it is the rapid movement of the target, short-term occlusion, or the appearance of a new target and the disappearance of the old target, the similarity between the two frames can be accurately evaluated through reasonable steps, thereby ensuring the stability and accuracy of the entire video processing system in various complex environments.

[0096] As an optional but non-limiting implementation, before determining the similarity between the target detection result corresponding to the second video frame and the target detection result corresponding to the third video frame, steps D1-D2 are also included:

[0097] Step D1: Use the second video frame and the third video frame as to-be-processed video frames respectively, and divide the to-be-processed video frames into blocks corresponding to the video pictures to obtain a plurality of to-be-processed picture blocks.

[0098] Step D2, performing target object detection operations on multiple to-be-processed picture blocks in parallel to obtain target detection results corresponding to each to-be-processed picture block, and merging the target detection results corresponding to each to-be-processed picture block to obtain the target detection results corresponding to the to-be-processed video frame.

[0099] In the process of video processing, in order to detect the target object more efficiently and accurately, the strategy of dividing the video screen into blocks is adopted because the target object detection is in a key position in the entire video analysis process and may be related to subsequent operations such as reuse of target detection results and evaluation of screen changes. Here, the second video frame and the third video frame can be selected as the video frames to be processed in turn for block processing. By dividing the video screen corresponding to the video frame to be processed into blocks, a complex large screen can be decomposed into multiple relatively simple small screen areas, that is, multiple screen blocks to be processed. In this way, dividing the video frame to be processed into blocks and performing parallel detection on the blocks can effectively reduce the detection time of a single video frame, thereby further reducing the overall time consumption.

[0100] The specific block operation can be determined according to different algorithms and application scenarios. For example, the picture can be evenly divided according to a fixed size, such as dividing the picture into several rectangular blocks of the same size; or adaptive block division can be performed according to the content features in the picture, such as dividing based on the outline, color and other features of the object, so that the content in each block has a certain similarity or independence. Regardless of the block division method used, the purpose is to better handle the problem of target object detection in the video picture.

[0101] The target object detection operation is performed in parallel on multiple to-be-processed picture blocks. Parallel processing means that multiple blocks are detected at the same time, and each to-be-processed picture block is detected independently using the target object detection algorithm. Among them, these algorithms can be based on machine learning models (such as convolutional neural networks) or traditional image processing methods (such as feature-based target detection algorithms). Through parallel processing, the detection of a large number of blocks can be completed in a short time. After completing the target object detection of each to-be-processed picture block, it is necessary to merge the target detection results corresponding to each to-be-processed picture block to obtain the target detection results corresponding to the to-be-processed video frame.

[0102] It is understandable that in the process of integrating the local detection results of each block into the global detection result of the entire video frame, the relationship and position information between the blocks need to be considered when merging. For example, if the uniform block method is adopted, the target detection results of each block are spliced ​​together according to the correct position relationship according to the coordinate order of the blocks; if the adaptive block method is adopted, the detection results need to be accurately merged according to the correlation between the blocks to form a complete target detection result of the video frame to be processed.

[0103] By dividing the video screen into blocks and performing target object detection operations in parallel, computing resources are fully utilized and the target detection time is greatly shortened. Compared with the traditional method of detecting the entire video frame in sequence, parallel processing can process multiple blocks at the same time, especially for high-resolution video frames, this efficiency improvement is more significant. Block processing can make the target object detection algorithm more focused on the target features in each small area, reducing the interference of other complex factors in the picture. The target objects in each block are relatively small and the features are more obvious, which is conducive to improving the accuracy of detection. It can be seen that dividing the video frame into blocks according to a specific block scheme and then detecting it in parallel can further reduce the overall time consumption without losing the target.

[0104] As an optional but non-limiting implementation method, there is a rectangular overlapping area between adjacent to-be-processed picture blocks in each to-be-processed picture block, the vertical height of the rectangular overlapping area is not less than a reference vertical height, and the horizontal width of the rectangular overlapping area is not less than a reference horizontal width, the reference vertical height is determined based on the maximum vertical height of the target detection box to which the target object belongs in the target detection results corresponding to the historical video frames collected before the to-be-processed video frame, and the reference horizontal width is determined based on the maximum horizontal width of the target detection box to which the target object belongs in the target detection results corresponding to the historical video frames collected before the video frame to be processed.

[0105] See also Figure 4, in order to avoid the target object on the dividing line of the detection area from being correctly identified when the image to be processed is divided into blocks, there must be enough overlapping areas between the image blocks to be processed. At the same time, in order to improve the performance as much as possible, the area of ​​the repeated detection area, that is, the overlapping area between the image blocks to be processed, should be minimized, and in order to ensure that the target object to be detected will not be unidentified because it is located at the edge of the detection area, the width and height of the overlapping area cannot be less than the width and height of any detected target object. The maximum width of the target detection frame to which all target objects belong in the target detection results corresponding to the historical video frame is recorded as the minimum overlapping area width, and the maximum height of all target detection frames in the target detection results corresponding to the historical video frame is recorded as the minimum overlapping area height. Subsequently, when the video frame to be processed is divided into blocks, it is necessary to ensure that the width of any overlapping area of ​​the detection area cannot be less than the minimum overlapping width, and the height cannot be less than the minimum overlapping height. Finally, each image block to be processed can be uniformly detected in parallel, and the detected results can be merged to obtain the detection result of the video frame.

[0106] For example, see Figure 4 , define the upper left corner of the video frame to be processed as point A0, the upper right corner as point B1, the lower left corner as point C2, and the lower right corner as point D3; make the perpendicular bisector of A0B1, intersecting with A0B1 at point AB, and intersecting with C2D3 at point CD; make the perpendicular bisector of A0C2, intersecting with A0C2 at point AC, and intersecting with B1D3 at point BD. Record the maximum width of all target detection frames in the historical detection results, defined as the minimum width W of the overlapping area, record the maximum height of all target detection frames in the historical detection results, defined as the minimum height H of the overlapping area. After the definition point AB moves W / 2 distance toward the point A0, point A1 is obtained; after the definition point AB moves W / 2 distance toward the point B1, point B0 is obtained; after the definition point AC moves H / 2 distance toward the point A0, point A2 is obtained; after the definition point AC moves H / 2 distance toward the point C2, point C0 is obtained; after the definition point BD moves H / 2 distance toward the point B1, point B3 is obtained; after the definition point BD moves H / 2 distance toward the point D3, point D1 is obtained; after the definition point CD moves W / 2 distance toward the point C2, point C3 is obtained; after the definition point CD moves W / 2 distance toward the point D3, point D2 is obtained. At this time, the 4 processed screen blocks to be detected are rectangle A0B0C0D0, rectangle A1B1C1D1, rectangle A2B2C2D2, and rectangle A3B3C3D3. The overlapping area is the cross area composed of rectangle A1B0C3D2 and rectangle A2B3C0D1.

[0107] See also Figure 4After completing the parallel detection of each block of the image to be processed, the detection results need to be merged, such as collecting and collating the detection results of each block of the image to be processed in each image to be processed obtained by the parallel detection. These detection results usually contain the location information of the target object (such as the coordinate range in the block, which may be expressed as the coordinates of the upper left corner and the lower right corner, etc.), the target object category (such as specific categories such as people, cars, objects, etc.); in the parallel detection process, different blocks may have repeated detections for target objects that are near the block boundary or have overlapping areas. Therefore, it is necessary to remove duplicate target detection results of the detection results of each block in the image to be processed, classify and integrate the detection results of each block of the image to be processed after deduplication and other processing according to the target object category, and summarize the integrated target detection results corresponding to the target objects of each category and position to form a complete final detection result applicable to the entire image.

[0108] Optionally, before merging the detection results of each to-be-processed screen block, the target detection frame coordinates of each to-be-processed screen block need to be converted into coordinates on the original video frame. Define the width of the original video frame as w, the height of the original video frame as h, the origin of the coordinate axis of the original video frame as the upper left corner of the video frame, and define the upper left corner coordinates of a single detected target object of any to-be-processed screen block as (x, y). Then, for rectangle A0B0C0D0, the coordinates converted into the original video frame are (x, y); for rectangle A1B1C1D1, the coordinates converted into the original video frame are (x+w / 2-W / 2, y); for rectangle A2B2C2D2, the coordinates converted into the original video frame are (x, y+h / 2-H / 2); for rectangle A3B3C3D3, the coordinates converted into the original video frame are (x+w / 2-W / 2, y+h / 2-H / 2). When merging detection results, if the coordinates of the upper left corners of multiple detection frames are the same, these target detection frames are merged, and the height of the merged detection frame takes the maximum value of the heights of these detection frames, and the width of the merged detection frame takes the maximum value of the widths of these detection frames.

[0109] S250: Based on the target detection result corresponding to the first video frame, blur the video image corresponding to the first video frame.

[0110] According to the technical solution of the embodiment of the present invention, during the video blurring process, the video picture change amount will be analyzed for the first video frame and the second video frame of different forward prediction coding frames belonging to the same image group. By accurately analyzing the video picture change amount and combining the target detection result of the second video frame that has completed target detection, the target detection result of the first video frame can be determined without the need to apply the target detection algorithm to the first video frame. This avoids the tedious operation of re-performing a complex target detection algorithm on each video frame, reduces unnecessary calculations, and significantly reduces the frequency of calling the algorithm interface due to the significant reduction in repeated target object detection operations. In this way, when faced with complex scenes where a large number of video frames need to be processed, the processing time can be significantly shortened, the efficiency of the entire video blurring process can be improved, and the performance bottleneck problem of time-consuming algorithm interface calls can be effectively alleviated, so that when processing high-resolution videos or real-time processing requirements, good performance can still be maintained, avoiding problems such as freezes caused by slow processing speeds.

[0111] Figure 5 This is a structural schematic diagram of a video processing device provided in an embodiment of the present invention. The technical solution of the embodiment of the present invention is applicable to the situation where blurring is performed on a partial picture area in a video picture. The video processing device can be implemented in the form of software and / or hardware and is generally integrated on any electronic device with network communication function, which can be a mobile terminal, a PC or a server, etc.

[0112] like Figure 5 As shown, the video processing device of the embodiment of the present invention may include the following:

[0113] A first determining module 510 is used to determine a video picture change amount generated by a video picture corresponding to a first video frame relative to a video picture corresponding to a second video frame, wherein the first video frame and the second video frame are different forward prediction coded frames belonging to the same picture group, and the picture group to which the first video frame and the second video frame belong only includes independently coded key frames and forward prediction coded frames;

[0114] A second determination module 520 is used to determine the target detection result corresponding to the first video frame based on the video picture change amount and the target detection result corresponding to the second video frame, wherein the target detection result is used to indicate the detection result of the target object in the video picture corresponding to the video frame;

[0115] The blur processing module 530 is used to blur the video image corresponding to the first video frame based on the target detection result corresponding to the first video frame.

[0116] Based on the above embodiment, optionally, determining the video picture change amount generated by the video picture corresponding to the first video frame relative to the video picture corresponding to the second video frame includes:

[0117] Determine a first data amount associated with a first video frame, where the first data amount is a data amount generated by accumulating data amounts associated with at least one first reference frame, where the data amount associated with the first reference frame is motion vector data and residual value data of the first reference frame compared to a previous video frame in an image group to which the first reference frame belongs, where the first reference frame includes the first video frame or a video frame located between the first video frame and the second video frame and including the first video frame, where the first video frame is a video frame that needs to rely on a second reference frame for predictive encoding in a video encoding process in order to restore a complete video picture, and where the second reference frame is the second video frame or a video frame located between the first video frame and the second video frame and including the second video frame;

[0118] According to the first data amount associated with the first video frame, a video picture change amount generated by the video picture corresponding to the first video frame relative to the video picture corresponding to the second video frame is determined.

[0119] Based on the above embodiment, optionally, before determining the video picture change amount generated by the video picture corresponding to the first video frame relative to the video picture corresponding to the second video frame, the method further includes:

[0120] Determine a reference video picture change threshold to be used for the first video frame relative to the second video frame, where the reference video picture change threshold is a video picture change upper limit value that needs to be satisfied for a video picture corresponding to the first video frame and a video picture corresponding to the second video frame so that a similarity between a target detection result corresponding to the first video frame and a target detection result corresponding to the second video frame is greater than a preset similarity;

[0121] Accordingly, determining the target detection result corresponding to the first video frame based on the video picture change amount and the target detection result corresponding to the second video frame includes:

[0122] If the video picture change amount is not greater than the reference video picture change amount threshold, directly multiplexing the target detection result corresponding to the second video frame as the target detection result corresponding to the first video frame;

[0123] If the video picture change amount is greater than the reference video picture change amount threshold, a target object detection operation is performed on the video picture corresponding to the first video frame.

[0124] Based on the above embodiment, optionally, determining a reference video picture change threshold to be used for the first video frame relative to the second video frame includes:

[0125] Determine a similarity between a target detection result corresponding to the second video frame and a target detection result corresponding to a third video frame, wherein the third video frame and the second video frame belong to the same picture group, and the third video frame is a forward predictive coding frame or a key frame that is located before and adjacent to the second video frame;

[0126] If the similarity between the target detection result corresponding to the second video frame and the target detection result corresponding to the third video frame is not less than a preset similarity threshold, determining the second data amount associated with the second video frame as a reference video picture change amount threshold to be adopted by the first video frame relative to the second video frame;

[0127] The second data amount associated with the second video frame is the amount of data generated by the motion vector data and the residual value data of the second video frame compared with the third video frame.

[0128] Based on the above embodiment, optionally, determining the similarity between the target detection result corresponding to the second video frame and the target detection result corresponding to the third video frame includes:

[0129] Determining the number of target objects in the target detection result corresponding to the second video frame and the number of target objects in the target detection result corresponding to the third video frame;

[0130] If the number of the two target objects is different, determining that the similarity between the target detection result corresponding to the second video frame and the target detection result corresponding to the third video frame is a preset value, where the preset value is used to indicate that the similarity between the two target detection results is 0;

[0131] If the number of the two target objects is the same, first size information and second size information are determined, wherein the first size information is used to indicate the size of the overlapping position area of ​​the target detection result corresponding to the second video frame and the target detection frame of the same target object in the target detection result corresponding to the second video frame, and the second size information is used to indicate the size of the union area of ​​the position area range covered by the target detection result corresponding to the second video frame and the two target detection frames of the same target object in the target detection result corresponding to the second video frame;

[0132] The similarity between the target detection result corresponding to the second video frame and the target detection result corresponding to the third video frame is determined according to the size ratio of the first size information to the second size information.

[0133] Based on the above embodiment, optionally, before determining the similarity between the target detection result corresponding to the second video frame and the target detection result corresponding to the third video frame, the method further includes:

[0134] The second video frame and the third video frame are respectively used as to-be-processed video frames, and the to-be-processed video frames are divided into blocks corresponding to video pictures to obtain a plurality of to-be-processed picture blocks;

[0135] The target object detection operation is performed on multiple to-be-processed picture blocks in parallel to obtain the target detection results corresponding to each to-be-processed picture block, and the target detection results corresponding to each to-be-processed picture block are merged to obtain the target detection results corresponding to the to-be-processed video frame.

[0136] On the basis of the above embodiment, optionally, there is a rectangular overlapping area between adjacent to-be-processed picture blocks in each to-be-processed picture block, the vertical height of the rectangular overlapping area is not less than the reference vertical height, the horizontal width of the rectangular overlapping area is not less than the reference horizontal width, the reference vertical height is determined based on the maximum vertical height of the target detection box to which the target object belongs in the target detection results corresponding to the historical video frames collected before the to-be-processed video frame, and the reference horizontal width is determined based on the maximum horizontal width of the target detection box to which the target object belongs in the target detection results corresponding to the historical video frames collected before the to-be-processed video frame.

[0137] According to the technical solution of the embodiment of the present invention, during the video blurring process, the video picture change amount will be analyzed for the first video frame and the second video frame of different forward prediction coding frames belonging to the same image group. By accurately analyzing the video picture change amount and combining the target detection result of the second video frame that has completed target detection, the target detection result of the first video frame can be determined without the need to apply the target detection algorithm to the first video frame. This avoids the tedious operation of re-performing a complex target detection algorithm on each video frame, reduces unnecessary calculations, and significantly reduces the frequency of calling the algorithm interface due to the significant reduction in repeated target object detection operations. In this way, when faced with complex scenes where a large number of video frames need to be processed, the processing time can be significantly shortened, the efficiency of the entire video blurring process can be improved, and the performance bottleneck problem of time-consuming algorithm interface calls can be effectively alleviated, so that when processing high-resolution videos or real-time processing requirements, good performance can still be maintained, avoiding problems such as freezes caused by slow processing speeds.

[0138] The video processing device provided in the embodiment of the present invention can execute the video processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the video processing method.

[0139] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present invention.

[0140] Figure 6 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 6 , which shows an electronic device (eg Figure 6 The terminal device in the embodiment of the present invention may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0141] like Figure 6 As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An edit / output (I / O) interface 505 is also connected to the bus 504.

[0142] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0143] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the video processing method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the video processing method of the embodiment of the present invention are executed.

[0144] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

[0145] The electronic device provided in the embodiment of the present invention and the video processing method provided in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0146] An embodiment of the present invention provides a computer storage medium on which a computer program is stored. When the program is executed by a processor, the video processing method provided by the above embodiment is implemented.

[0147] It should be noted that the computer-readable medium of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0148] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0149] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0150] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: determines the video picture change amount generated by the video picture corresponding to the first video frame relative to the video picture corresponding to the second video frame, the first video frame and the second video frame are different forward prediction coding frames belonging to the same image group, and the image group to which the first video frame and the second video frame belong only includes independently encoded key frames and forward prediction coding frames; based on the video picture change amount and the target detection result corresponding to the second video frame, determines the target detection result corresponding to the first video frame, the target detection result is used to indicate the detection result of the target object in the video picture corresponding to the video frame; based on the target detection result corresponding to the first video frame, blurs the video picture corresponding to the first video frame.

[0151] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0152] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0153] The units involved in the embodiments of the present invention may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0154] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0155] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0156] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) to form a technical solution.

[0157] In addition, although each operation is described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present invention. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0158] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A video processing method, characterized in that: The method comprises: Determining a video picture change amount generated by a video picture corresponding to a first video frame relative to a video picture corresponding to a second video frame, wherein the first video frame and the second video frame are different forward prediction coded frames belonging to the same picture group, and the picture group to which the first video frame and the second video frame belong only includes independently coded key frames and forward prediction coded frames; Based on the video picture change amount and the target detection result corresponding to the second video frame, determine the target detection result corresponding to the first video frame, and the target detection result is used to indicate the detection result of the target object in the video picture corresponding to the video frame; based on the video picture change amount and the target detection result corresponding to the second video frame, determine the target detection result corresponding to the first video frame, including: if the video picture change amount is not greater than the reference video picture change amount threshold, directly reuse the target detection result corresponding to the second video frame as the target detection result corresponding to the first video frame; the reference video picture change amount threshold is generated by the following method: Determine the target detection result corresponding to the second video frame similarity between the target detection result corresponding to the third video frame, the third video frame and the second video frame belong to the same image group, and the third video frame is a forward prediction coding frame or key frame located before and adjacent to the second video frame; if the similarity between the target detection result corresponding to the second video frame and the target detection result corresponding to the third video frame is not less than a preset similarity threshold, then determining the second data amount associated with the second video frame as a reference video picture change threshold to be adopted by the first video frame relative to the second video frame; wherein the second data amount associated with the second video frame is the amount of data generated by the motion vector data and residual value data of the second video frame compared to the third video frame; Based on the target detection result corresponding to the first video frame, the video picture corresponding to the first video frame is blurred.

2. The method according to claim 1, characterized in that Determining a video picture change amount generated by a video picture corresponding to the first video frame relative to a video picture corresponding to the second video frame includes: Determine a first data amount associated with a first video frame, where the first data amount is a data amount generated by accumulating data amounts associated with at least one first reference frame, where the data amount associated with the first reference frame is motion vector data and residual value data of the first reference frame compared to a previous video frame in an image group to which the first reference frame belongs, where the first reference frame includes the first video frame or a video frame located between the first video frame and the second video frame and including the first video frame, where the first video frame is a video frame that needs to rely on a second reference frame for predictive encoding in a video encoding process in order to restore a complete video picture, and where the second reference frame is the second video frame or a video frame located between the first video frame and the second video frame and including the second video frame; According to the first data amount associated with the first video frame, a video picture change amount generated by the video picture corresponding to the first video frame relative to the video picture corresponding to the second video frame is determined.

3. The method according to claim 1 or 2, characterized in that: Before determining the video picture change amount generated by the video picture corresponding to the first video frame relative to the video picture corresponding to the second video frame, the method further includes: Determine a reference video picture change threshold to be used for the first video frame relative to the second video frame, where the reference video picture change threshold is a video picture change upper limit value that needs to be satisfied for a video picture corresponding to the first video frame and a video picture corresponding to the second video frame so that a similarity between a target detection result corresponding to the first video frame and a target detection result corresponding to the second video frame is greater than a preset similarity; Accordingly, determining the target detection result corresponding to the first video frame based on the video picture change amount and the target detection result corresponding to the second video frame includes: If the video picture change amount is greater than the reference video picture change amount threshold, a target object detection operation is performed on the video picture corresponding to the first video frame.

4. The method according to claim 1, characterized in that: Determining a similarity between a target detection result corresponding to the second video frame and a target detection result corresponding to the third video frame includes: Determining the number of target objects in the target detection result corresponding to the second video frame and the number of target objects in the target detection result corresponding to the third video frame; If the number of the two target objects is different, determining that the similarity between the target detection result corresponding to the second video frame and the target detection result corresponding to the third video frame is a preset value, where the preset value is used to indicate that the similarity between the two target detection results is 0; If the number of the two target objects is the same, first size information and second size information are determined, wherein the first size information is used to indicate the size of the overlapping position area of ​​the target detection frame of the same target object in the target detection result corresponding to the second video frame and the target detection result corresponding to the third video frame, and the second size information is used to indicate the size of the union area of ​​the position area range covered by the two target detection frames of the same target object in the target detection result corresponding to the second video frame and the target detection result corresponding to the third video frame; The similarity between the target detection result corresponding to the second video frame and the target detection result corresponding to the third video frame is determined according to the size ratio of the first size information to the second size information.

5. The method according to claim 1, characterized in that Before determining the similarity between the target detection result corresponding to the second video frame and the target detection result corresponding to the third video frame, the method further includes: The second video frame and the third video frame are respectively used as to-be-processed video frames, and the to-be-processed video frames are divided into blocks corresponding to video pictures to obtain a plurality of to-be-processed picture blocks; The target object detection operation is performed on multiple to-be-processed picture blocks in parallel to obtain the target detection results corresponding to each to-be-processed picture block, and the target detection results corresponding to each to-be-processed picture block are merged to obtain the target detection results corresponding to the to-be-processed video frame.

6. The method according to claim 5, characterized in that There is a rectangular overlapping area between adjacent to-be-processed picture blocks in each to-be-processed picture block, the vertical height of the rectangular overlapping area is not less than the reference vertical height, the horizontal width of the rectangular overlapping area is not less than the reference horizontal width, the reference vertical height is determined based on the maximum vertical height of the target detection box to which the target object belongs in the target detection results corresponding to the historical video frames collected before the to-be-processed video frame, and the reference horizontal width is determined based on the maximum horizontal width of the target detection box to which the target object belongs in the target detection results corresponding to the historical video frames collected before the to-be-processed video frame.

7. A video processing device, characterized in that: The device comprises: A first determination module is used to determine a video picture change amount generated by a video picture corresponding to a first video frame relative to a video picture corresponding to a second video frame, wherein the first video frame and the second video frame are different forward prediction coding frames belonging to the same picture group, and the picture group to which the first video frame and the second video frame belong only includes independently encoded key frames and forward prediction coding frames; A second determination module is used to determine the target detection result corresponding to the first video frame based on the target detection result corresponding to the second video frame and the video picture change amount, wherein the target detection result is used to indicate the detection result of the target object in the video picture corresponding to the video frame; based on the target detection result corresponding to the second video frame and the video picture change amount, determine the target detection result corresponding to the first video frame, including: if the video picture change amount is not greater than a reference video picture change amount threshold, directly multiplexing the target detection result corresponding to the second video frame as the target detection result corresponding to the first video frame; the reference video picture change amount threshold is generated by the following method: determining the target object corresponding to the second video frame similarity between the target detection result and the target detection result corresponding to the third video frame, the third video frame and the second video frame belong to the same image group, and the third video frame is a forward prediction coding frame or key frame located before and adjacent to the second video frame; if the similarity between the target detection result corresponding to the second video frame and the target detection result corresponding to the third video frame is not less than a preset similarity threshold, then determining the second data amount associated with the second video frame as a reference video picture change amount threshold to be adopted by the first video frame relative to the second video frame; wherein the second data amount associated with the second video frame is the amount of data generated by the motion vector data and the residual value data of the second video frame compared to the third video frame; A blur processing module is used to blur the video image corresponding to the first video frame based on the target detection result corresponding to the first video frame.

8. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the video processing method described in any one of claims 1-6.

9. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to perform the video processing method described in any one of claims 1-6 when executed by a computer processor.

Images