Method and device for assisting target detection using video coding information

Abstract

The present invention discloses a method and device for assisting target detection using video coding information, including: determining a target reference frame for a video frame to be detected, the target reference frame including several sub-window blocks; for each sub-window block: determining a reference position of the sub-window block in the video frame to be detected; determining a reference value of the reference position; determining an overall reference value of the video frame to be detected based on the reference value of each sub-window block; searching for a target window in the video frame to be detected based on the overall reference value within the constraint range of a target deformation function; and performing target detection based on the target window. The method disclosed in the present invention can effectively reduce the amount of calculation, improve the efficiency of target detection, and save hardware resources and reduce power consumption and delay as a whole.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for assisting target detection using video coding information. Background Art

[0002] Object detection and recognition are common problems in computer vision, and they are widely used in many fields of life. The purpose of object detection is to distinguish objects in an image or video from parts that are not of interest, determine whether an object exists, and if so, determine the location of the object. Currently, high-precision object detection usually uses methods based on deep learning. The location and type of objects are discovered through convolutional neural networks, and their accuracy can reach or even exceed that of the human eye.

[0003] The biggest disadvantage of deep learning-based object detection is its high computational complexity. Using object detection on each frame of a video may require a lot of computing resources (usually using a GPU), and may also increase latency. Therefore, object detection is usually used in conjunction with object tracking. Even so, the computational complexity of object tracking is also very high. This greatly limits the application areas of object detection and also causes a lot of computing and energy consumption.

[0004] Another common method is video object detection using motion information. Currently, the most commonly used motion information is optical flow information. It measures the instantaneous speed of pixel motion of a moving object in space on the observation imaging plane. It uses the change of pixels in the time domain in the image sequence and the correlation between adjacent frames to find the corresponding relationship between the previous frame and the current frame, thereby calculating the motion information of the object between adjacent frames. Generally speaking, optical flow is generated by the movement of the foreground object itself in the scene, the movement of the camera, or the joint movement of the two. However, the amount of calculation required to extract and use the optical flow itself is very large.

[0005] Since the computational complexity of existing target detection is very high, almost all protocols require motion estimation during the video encoding and decoding process. Effectively using motion estimation to track targets can improve the efficiency and performance of target detection. Summary of the invention

[0006] The embodiment of the present invention provides a method and device for assisting target detection using video coding information, so as to realize assisting target detection and improve the efficiency of target detection.

[0007] In a first aspect, an embodiment of the present invention provides a method for assisting target detection using video coding information, comprising: determining a target reference frame for a video frame to be detected, the target reference frame comprising a plurality of sub-window blocks; for each sub-window block: determining a reference position of the sub-window block in the video frame to be detected; determining a reference value of the reference position; determining an overall reference value based on the reference values ​​of each sub-window block; searching for a target window in the video frame to be detected based on the overall reference value within the constraint range of a target deformation function; and performing target detection based on the target window.

[0008] In some embodiments, determining a target reference frame for a video frame to be detected includes:

[0009] Based on a preset time range and the to-be-detected video frame, a target reference frame is selected before the to-be-detected video frame.

[0010] In some embodiments, the preset time range is determined by a specified time interval, or is determined based on a time correlation function, wherein the closer the target reference frame is to the video frame to be detected, the higher the credibility of the target reference frame.

[0011] In some embodiments, determining the reference value of the reference position includes:

[0012] The reference value of the reference position is determined based on the size of the block corresponding to the reference position and the credibility of the reference position, wherein the reference value of the reference position increases as the size of the block corresponding to the reference position increases and / or the credibility of the reference position increases.

[0013] In some embodiments, searching for a target window in the to-be-detected video frame includes:

[0014] Within the constraint range of the target deformation function, a target window is searched in the video frame to be detected so that the overall reference value of the target window and the overall reference value of the video frame to be detected exceed a first threshold, and the sum of the reference values ​​of the reference positions contained in the target window and the size of the target window exceed a second threshold.

[0015] In some embodiments, it also includes: pre-configuring a target detection count value, when a target window is found, the target detection count value is reduced by 1, and when the target detection count value is 0, performing target detection based on each target window.

[0016] In some embodiments, it also includes: directly performing target detection when the target window cannot be found.

[0017] In some embodiments, the first threshold and / or the second threshold are adjusted to stabilize the search target window.

[0018] In a second aspect, an embodiment of the present invention further provides a target detection device for video coding, comprising a processor configured to execute the steps of the auxiliary target detection method for video coding described in each embodiment of the present disclosure.

[0019] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the auxiliary target detection method for video encoding described in each embodiment of the present disclosure are implemented.

[0020] The embodiment of the present invention searches for a target window in the video frame to be detected based on a determined overall reference value within the constraint range of the target deformation function, and performs target detection based on the target window, thereby effectively reducing the amount of calculation and improving the efficiency of target detection. As a whole, hardware resources are saved and power consumption and delay are reduced.

[0021] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0023] Figure 1 A schematic diagram of moving target vector detection is shown.

[0024] Figure 2 A basic flow chart of the target detection method according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0025] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0026] In the process of video encoding and decoding, almost all protocols need to perform motion estimation. The basic idea is to divide each frame of the image sequence into many non-overlapping blocks, and assume that the displacement of all pixels in the block is the same. Then, for each macroblock, the block that is most similar to the current block is found according to a certain block matching criterion within a certain search range of the reference frame, i.e., the matching block. The relative displacement between the matching block and the current block is the motion vector.

[0027] Generally speaking, the displacement of the object between the two frames adjacent in time is usually not very large, so we can start from the current macroblock position and search in a small area around the macroblock at the same position in the historical frame to find the matching block. Obviously, the larger the search area, the higher the calculation cost, so some tricks are needed in actual products to speed up the search and reduce the amount of calculation. For example, the H.264 protocol allows searching for matching blocks in any reconstructed frame in the DPB cache to find the best matching block. Obviously, the cost of the search is proportional to the number of reference frames. For example, the time (chip area) and power consumption required to search for matching blocks in two reference frames are basically twice that of a single reference frame. Therefore, from the perspective of cost saving, the simplest solution is to only refer to the previous frame, which saves both the reference frame cache and the encoder computing power. The disadvantage is that some bit rate performance will be sacrificed.

[0028] After finding an optimal matching block, the motion estimation will output a motion vector (MV), which is the position coordinate of the reference block relative to the current block. If no suitable matching block is found, intra-frame prediction can be used for this macroblock. Figure 1 As shown, in H.264 and H.265 technologies, both P frames and B frames can have Intra-coded macroblocks.

[0029] If the frequency of target detection is reduced, it can be used in some application scenarios. However, it is not applicable to scenarios that require video frame rate. For example, in the application of area enhancement, the video encoder needs to improve the image quality by adjusting the QP for special objects, which requires target recognition for each frame. Even when performing license plate recognition in a parking lot, some high-speed vehicles may be missed due to the interval of the timed snapshot.

[0030] Traditional video object detection that uses motion information requires additional computing resources to estimate motion information, resulting in a waste of resources and increased hardware and energy costs.

[0031] In order to use motion vectors to determine and track target detection, an embodiment of the present invention provides a method for assisting target detection using video coding information, including:

[0032] Step S201, determine the target reference frame for the video frame to be detected, and the target reference frame includes several sub-window blocks. Specifically, in this embodiment, the sub-window block can be a target position discovered by deep learning or other methods. Since the video is continuous in many cases, the previous video frame in the time dimension can be used to select the target reference frame, such as the previous frame. In step S202, for each sub-window block: determine the reference position of the sub-window block in the video frame to be detected; determine the reference value of the reference position. That is, for each block in the target window of the target reference frame, the reference position of the sub-window block can be determined in the video frame to be detected, and the reference position can be the corresponding position of the sub-window block in the video frame to be detected. Therefore, the reference value can be calculated according to the reference position, and the reference value of the embodiment of the present disclosure can be used to describe the time credibility of the sub-window block. In step S203, the overall reference value is determined based on the reference value of each sub-window block. For example, the overall reference value can be determined by summing, averaging or weighted averaging, which is not specifically limited here.

[0033] Then in step S204, within the constraint range of the target deformation function, based on the overall reference value, the target window is searched in the video frame to be detected. Finally, in step S205, target detection is performed based on the target window. In some embodiments, initially, the target detection result of the previous frame can be obtained by deep learning to determine the position of each sub-window block containing the moving target. And through the motion vector, the position of the sub-window block in the target detection result area in the previous frame to be detected can be found. These positions may be very concentrated or very scattered. In the case of dispersion, if all positions need to be covered, the new target window generated will be very large. In order to control this, the present application proposes to limit the scope of deformation by a target deformation function, and the target window satisfies the requirement of being similar to the previous frame, and also allows certain changes.

[0034] The method disclosed in the present invention can find the new position of the next frame (video frame to be detected) through motion estimation without reusing deep learning. Motion estimation is a necessary step in the video encoding process, so it effectively improves the efficiency of target detection, and the target detection result of each frame comes from the motion vector and target detection results of other frames, effectively saving computing resources. The target window is found by the target deformation function, thereby reducing the amount of search calculations.

[0035] In some embodiments, determining a target reference frame for a video frame to be detected includes:

[0036] Based on the preset time range and the video frame to be detected, a target reference frame is selected before the video frame to be detected. In some embodiments, the preset time range is determined by a specified time interval, or is determined based on a time correlation function, wherein the closer to the video frame to be detected, the higher the credibility of the target reference frame. Exemplarily, a video frame before the specified time interval can be selected as the target reference frame, for example, a previous frame can be selected as the target reference frame, or a second previous frame can be selected as the target reference frame. A time credibility function F(t) can also be used: its input is the time interval with the current frame, and its output is credibility. The shorter the interval, the closer the target reference frame is to the current frame, and the higher its credibility. It can be implemented using a one-dimensional LUT. The maximum value of its output is 1 and the minimum value is 0. For example, if you only want to refer to the previous frame (the previous frame is used as the target reference frame), F(t=-1)=1, F(t!=-1)=0. If you want to refer to the previous two frames, F(t=-1)=1, F(t=-2)=0.8, F(t!=-1,-2)=0.

[0037] In some embodiments, determining the reference value of the reference position includes: determining the reference value of the reference position based on the size of the block corresponding to the reference position and the credibility of the reference position, wherein the reference value of the reference position increases as the size of the block corresponding to the reference position increases and / or the credibility of the reference position increases. As an example, the size of the block corresponding to the reference position × the credibility value can be used. Step S103, determine the overall reference value based on the reference values ​​of each sub-window block. For example, the overall reference value can be determined by summing.

[0038] In some embodiments, searching for a target window in the video frame to be detected includes: searching for a target window in the video frame to be detected within the constraint range of a target deformation function, so that the overall reference value of the target window and the overall reference value of the video frame to be detected exceed a first threshold, and the sum of the reference values ​​of the reference positions contained in the target window and the size of the target window exceed a second threshold. A specific search method may be to use a target deformation function Q(X, Y): the deformation range allowed for the target window in the current frame. The deformation allowed for the target window corresponding to the current frame compared to the size and shape of the target window in the previous frame. For example, Q(0.1, 0.1) may allow a 10% deformation (increase or decrease) in width and height of the target window. It can also be a target window that only allows a fixed size [1.1W, 1.1H], [1.1W, 1.0H], [1.1W, 0.9H], [1.0W, 1.1H], [1.0W, 1.0H], [1.0W, 0.9H], [0.9W, 1.1H], [0.9W, 1.0H], [0.9W, 0.9H]. Q(0,0) means that the target window size is consistent with the previous frame. In some embodiments, the target deformation function can be configured to tolerate small-scale scene deformation. The purpose of the target deformation function in this example is to limit the scope of the search and reduce the amount of calculation for the search. At the same time, the deformation of the target in the video can be tolerated to a certain extent. Considering that the time between frames is very short, in this example, the deformation tolerance is set within a smaller range in most scenes. The first threshold value Pc and the second threshold value Pd can be pre-set, so that within the allowable range of the target deformation function, a new target window can be found in the current frame (frame to be detected), which can contain the ratio of the sum of the credible reference values ​​of the reference position to the overall credible reference value, which needs to exceed Pc. If a matching target window is found, further detection of the corresponding reference position size and the ratio of the size of the target window needs to exceed Pd.

[0039] In some embodiments, the first threshold and / or the second threshold can be adjusted to stabilize the search target window. In this example, it is considered that a large target window is easier to obtain a larger reference position coverage, which is also easier to be greater than Pc. A small target window is easier to obtain a higher coverage density. By adjusting Pc and Pd, it is possible to control the final stable target window.

[0040] In some embodiments, a new target window may be found in the current frame within the allowable range of the target deformation function, and the ratio of the sum of the credible reference values ​​of the reference positions to the overall credible reference value may need to exceed Pc.

[0041] The new window must contain a certain proportion of the blocks in the old window (moved to the new window) → the new window expands. The ratio of the sum of the corresponding reference position sizes and the size of the target window of the new target window found needs to exceed Pd, but there cannot be too many unknown blocks in the new window (which do not come from the old window) → the new window shrinks. Follow this process to match the target window. Based on the two factors of Pd and Pc, the position of the target in the current frame and the size of the target window are controlled to make the target window fit the actual needs.

[0042] In some embodiments, it also includes: pre-configuring a target detection count value, when the target window is found, the target detection count value is reduced by 1, and when the target detection count value is 0, performing target detection based on each target window. In some embodiments, it also includes: when the target window cannot be found, directly performing target detection. For example, the target detection count value N=k can be pre-configured, and the target detection count value can be configured in a forced target detection counter, and the forced target detection counter is used to perform the count of forced target detection.

[0043] If a new frame appears, if it is an I frame (key frame) or forced intra-frame prediction caused by other mechanisms, the number of reference frames is 0, and the motion vector cannot be determined. Therefore, target detection can be performed on the frame by a preset method, such as a deep learning method. Assign N = k.

[0044] If a target window that meets Pc and Pd is found, the target window N=N-1 is output.

[0045] If there is no target window that meets both Pc and Pd (does not meet the target tracking conditions), directly run target detection (for example, directly complete the target positioning or target detection of the frame through deep learning). Assign N=k.

[0046] If N=0, run target detection and reset N. N=k.

[0047] In summary, the method of the present invention can use motion vectors to achieve target tracking, thereby reducing the use of computing resources. The target detection result of each frame comes from the motion vectors and target detection results of other frames, and errors may accumulate continuously in this process. In this example, a forced target detection counter is also set to prevent accumulated errors. Thereby ensuring that new targets will not be missed in the case of target tracking. The method disclosed in the present invention cannot discover new targets. In the specific implementation process, deep learning can be used to detect the targets to be tracked at the same time, and then the target is tracked by the method disclosed in the present invention. The algorithm complexity of tracking the same target is 1 / 30 or even lower than that of using deep learning to complete target positioning between frames. Therefore, the method disclosed in the present invention effectively saves computing resources. The target deformation function is used to limit the search range, which greatly reduces the amount of search calculations. By adjusting Pc and Pd, the stable target window can be controlled to be finally obtained. Using the time credibility function, the weight of the reference frame is adjusted to obtain more accurate results. In the decoding process. The decoder can also obtain information about the motion vector. Therefore, the method disclosed in the present invention can also be used to accelerate target detection and save computational complexity.

[0048] In a second aspect, an embodiment of the present invention further provides a target detection device for video coding, comprising a processor configured to execute the steps of the auxiliary target detection method for video coding described in each embodiment of the present disclosure.

[0049] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the auxiliary target detection method for video encoding described in each embodiment of the present disclosure are implemented.

[0050] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0051] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0052] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for a terminal (which can be a mobile phone, computer, server or network equipment, etc.) to execute the methods described in each embodiment of the present invention.

[0053] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A method for assisting target detection using video coding information, characterized in that: include: Determine a target reference frame for the video frame to be detected, wherein the target reference frame includes a plurality of sub-window blocks; For each subwindow block: Determining a reference position of the sub-window block in the video frame to be detected; determining a reference value of the reference position; Determine an overall reference value based on the reference values ​​of each sub-window block; Searching for a target window in the to-be-detected video frame based on the overall reference value within the constraint range of the target deformation function; Performing target detection based on the target window; Determining a target reference frame for a video frame to be detected includes: Based on a preset time range and the video frame to be detected, selecting a target reference frame before the video frame to be detected; The preset time range is determined by a specified time interval, or is determined based on a time correlation function, wherein the closer to the to-be-detected video frame, the higher the credibility of the target reference frame; The reference values ​​for determining the reference position include: The reference value of the reference position is determined based on the size of the block corresponding to the reference position and the credibility of the reference position, wherein the reference value of the reference position increases as the size of the block corresponding to the reference position increases and / or the credibility of the reference position increases.

2. The method for assisting target detection using video coding information as claimed in claim 1, characterized in that: Searching for a target window in the video frame to be detected includes: Within the constraint range of the target deformation function, a target window is searched in the video frame to be detected so that the overall reference value of the target window and the overall reference value of the video frame to be detected exceed a first threshold, and the sum of the reference values ​​of the reference positions contained in the target window and the size of the target window exceed a second threshold.

3. The method for assisting target detection using video coding information as claimed in claim 2, characterized in that: Also includes: The target detection count value is pre-configured. When the target window is found, the target detection count value is reduced by 1. When the target detection count value is 0, target detection is performed based on each target window.

4. The method for assisting target detection using video coding information as claimed in claim 1, characterized in that: Also includes: If the target window cannot be found, target detection is performed directly.

5. The method for assisting target detection using video coding information as claimed in claim 2, characterized in that: The first threshold and / or the second threshold are adjusted to stabilize the search target window.

6. A device for assisting target detection using video coding information, characterized in that: It comprises a processor configured to execute the steps of the method for assisting target detection using video coding information as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for assisting target detection using video coding information as described in any one of claims 1 to 5 are implemented.

Images