Image processing method, device and storage medium based on motion estimation

Abstract

The present application discloses an image processing method, device and storage medium based on motion estimation, which includes: determining the target video frame in which the target object is identified and the to-be-estimated video frame in which the target object is not identified in the video frame cache queue according to the target recognition results of each video frame image in the video frame cache queue; wherein the target recognition result is obtained by performing target object recognition processing on the video frame image; performing motion estimation processing on the target object according to the target recognition results of each target video frame and the timing information of each target video frame to obtain the estimated area of ​​the target object in the to-be-estimated video frame; performing image processing on the estimated area in the to-be-estimated video frame and the area where the target object is located in the target video frame to obtain the target image. The above scheme can effectively protect the privacy of the video image detection process.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device and storage medium based on motion estimation. Background Art

[0002] In video image detection application scenarios, special attention should be paid to the protection of privacy issues.

[0003] Existing privacy coding technology relies on target recognition algorithms, which require the target recognition algorithm to identify the target object in each frame and then encode the image according to the area where the target object is located.

[0004] However, if the target recognition algorithm is inaccurate or there are some target objects that the target recognition algorithm cannot normally identify, it may still not be possible to process them in a timely manner, resulting in privacy issues. Summary of the invention

[0005] The present application at least provides an image processing method, apparatus, device and computer-readable storage medium based on motion estimation.

[0006] The first aspect of the present application provides an image processing method based on motion estimation, comprising: determining, according to target recognition results of each video frame image in a video frame cache queue, target video frames in which a target object is recognized and video frames to be estimated in which the target object is not recognized in the video frame cache queue; wherein the target recognition results are obtained by performing target object recognition processing on the video frame images; performing motion estimation processing on the target object according to the target recognition results of each target video frame and the timing information of each target video frame to obtain an estimated area of ​​the target object in the video frame to be estimated; performing image processing on the estimated area in the video frame to be estimated and the area where the target object is located in the target video frame to obtain a target image.

[0007] In one embodiment, the step of performing motion estimation processing on the target object according to the target recognition results of each target video frame and the timing information of each target video frame to obtain the estimated area of ​​the target object in the video frame to be estimated includes: determining the motion information of the target object according to the frame timing of each target video frame and the target recognition results of each target video frame; performing motion estimation processing on the target object according to the motion information to obtain the estimated area.

[0008] In one embodiment, the motion information includes a motion vector, and the step of determining the motion information of the target object based on the frame timing of each target video frame and the target recognition result of each target video frame includes: determining the first motion vector of the target object in the first timing direction based on the first timing direction, the frame timing of each target video frame and the target recognition result of each target video frame; and / or determining the second motion vector of the target object in the second timing direction based on the second timing direction, the frame timing of each target video frame and the target recognition result of each target video frame; wherein the first timing direction and the second timing direction are opposite.

[0009] In one embodiment, the estimated area includes a first estimated area and / or a second estimated area, and the step of performing motion estimation processing on the target object according to the motion information to obtain the estimated area includes: performing motion estimation processing in the first timing direction according to the first motion vector and the position information of the target object to obtain the first estimated area; and / or performing motion estimation processing in the second timing direction according to the second motion vector and the position information of the target object to obtain the second estimated area.

[0010] In one embodiment, the step of determining, based on the target recognition results of each video frame image in the video frame cache queue, the target video frame in which the target object is recognized and the video frame to be estimated in which the target object is not recognized, includes: traversing the target recognition results of each video frame image in the video frame cache queue to obtain trajectory information of the target object; and determining the target video frame and the video frame to be estimated in the video frame cache queue based on the trajectory information.

[0011] In one embodiment, the trajectory information includes trajectory start information and / or trajectory end information, and the step of determining the target video frame in the video frame cache queue and the video frame to be estimated according to the trajectory information includes: determining the video frame image with the trajectory information in the video frame cache queue as the target video frame; obtaining the starting video frame corresponding to the trajectory start information and the ending video frame corresponding to the trajectory end information in the target video frame; determining the video frame image in the video frame cache queue that is earlier than the starting video frame in timing as the video frame to be estimated; and / or determining the video frame image in the video frame cache queue that is later than the ending video frame in timing as the video frame to be estimated.

[0012] In one embodiment, before the step of determining the target video frames in the video frame cache queue that recognize the target object and the video frames to be estimated that do not recognize the target object in the video frame cache queue based on the target recognition results of each video frame image in the video frame cache queue, the method further includes: performing target object recognition processing on the acquired video frame images to obtain target recognition results corresponding to the video frame images; and storing the video frame images and the target recognition results in the video frame cache queue correspondingly according to the frame timing of each video frame image.

[0013] In one embodiment, the step of performing image processing on the estimated area in the video frame to be estimated and the area where the target object is located in the target video frame to obtain the target image includes: performing image occlusion processing on the estimated area in the video frame to be estimated and the area where the target object is located in the target video frame to obtain the target image; after the step of performing image processing on the estimated area to obtain the target image, the method also includes: inputting the target image into an image encoder for encoding processing to obtain a target code stream.

[0014] The second aspect of the present application provides an image processing device based on motion estimation, including: a video frame determination module, which is used to determine the target video frame in which the target object is identified and the video frame to be estimated in which the target object is not identified in the video frame cache queue according to the target recognition results of each video frame image in the video frame cache queue; wherein the target recognition result is obtained by performing target object recognition processing on the video frame image; a motion estimation module, which is used to perform motion estimation processing on the target object according to the target recognition results of each target video frame and the timing information of each target video frame to obtain the estimated area of ​​the target object in the video frame to be estimated; an image processing module, which is used to perform image processing on the estimated area in the video frame to be estimated and the area where the target object is located in the target video frame to obtain the target image.

[0015] A third aspect of the present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned image processing method based on motion estimation.

[0016] A fourth aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, which implement the above-mentioned image processing method based on motion estimation when the program instructions are executed by a processor.

[0017] The above scheme determines the target video frames in the video frame cache queue that recognize the target object and the to-be-estimated video frames in which the target object is not recognized, based on the target recognition results of each video frame image in the video frame cache queue; wherein the target recognition results are obtained by performing target object recognition processing on the video frame images, thereby determining the target video frames in which the target object is successfully recognized in each video frame image in the video frame cache queue, and the to-be-estimated video frames in which the target object is not successfully recognized and need to be determined by motion estimation; motion estimation processing is performed on the target object according to the target recognition results of each target video frame and the timing information of each target video frame to obtain the estimated area of ​​the target object in the to-be-estimated video frame; image processing is performed on the estimated area in the to-be-estimated video frame and the area where the target object is located in the target video frame, so as to protect the privacy of the target object as much as possible, and the obtained target image is output, thereby realizing effective protection of privacy in the video image detection scenario.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0020] Figure 1 It is a schematic diagram of a scene where the problem solved by the image processing method based on motion estimation of the present application is solved;

[0021] Figure 2 is a flowchart of an exemplary embodiment of the image processing method based on motion estimation of the present application;

[0022] Figure 3 It is a schematic diagram of a scene of inverse motion estimation in the image processing method based on motion estimation of the present application;

[0023] Figure 4 is a block diagram of an image processing device based on motion estimation shown in an exemplary embodiment of the present application;

[0024] Figure 5 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0025] Figure 6 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0026] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0027] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0028] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0029] It should be noted that in the prior art, it is usually necessary to perform privacy masking (such as mosaic) on some sensitive image content during video encoding. Privacy coding technology mainly relies on target recognition algorithms, which identify the target object in each frame of the image, obtain the regional position of the target object in the image, and then process the image according to its regional position before encoding it into a bitstream for transmission.

[0030] For ease of understanding, an example of one of the applicable scenarios of this application is now described. Figure 1 , Figure 1 This is a schematic diagram of a scene in which the problem is solved by the image processing method based on motion estimation of the present application. Figure 1 The privacy masking process in the prior art is shown in the figure, which mainly processes the target objects identified in the video frame image based on the target recognition technology. The video frame image shows three target objects, but due to the limitations of the target recognition algorithm in model training and / or the recognition accuracy of the target recognition algorithm and / or the low image quality of the video frame images involved in target recognition, some target objects in the video frame image are not recognized (for example, they are blocked by walls), and the subsequent privacy smearing will only process the image of the area where the identified target objects are located, while the target objects not recognized by the target recognition algorithm are not processed. However, in fact, some privacy content of the target objects will still be leaked, which does not meet the requirements of privacy protection.

[0031] Exemplarily, the above problem is very common in implementation scenarios such as when the target object has just entered the acquisition field of view of the image acquisition device, when the target object has just left the acquisition field of view of the image acquisition device, etc.

[0032] See also Figure 2 , Figure 2This is a flowchart of an exemplary embodiment of the image processing method based on motion estimation of the present application. Specifically, it may include the following steps:

[0033] Step S110, according to the target recognition results of each video frame image in the video frame cache queue, determine the target video frames in the video frame cache queue that recognize the target object and the to-be-estimated video frames in the video frame cache queue that do not recognize the target object; wherein the target recognition results are obtained by performing target object recognition processing on the video frame images.

[0034] Among them, the target recognition result is obtained by performing target object recognition processing on the collected video frame image according to a preset target recognition algorithm. The target object of the present application may include but is not limited to one or more of people, limbs, clothing accessories, vehicles, license plates, pets, etc., and the type of target object can be determined by the implementation requirements of the specific application scenario. The target recognition algorithm of the present application can refer to the existing mature technical solutions, and can be directly analyzed and judged by image pixels and / or analyzed and judged by a neural network model, etc., which will not be elaborated here.

[0035] Exemplarily, the collected continuous video frame sequence can be input into a pre-trained target recognition model to obtain the target recognition result output by the target recognition model. The target recognition result of the present application may include, but is not limited to: the target object in each video frame image, the position information of the target object in each video frame image, etc., and the target object can also be marked in the video frame image in the form of a rectangular detection frame, and a unique identifier (target ID) can be set for each target object. Among them, the target ID can be unique in the video frame sequence, that is, in a video frame sequence, the same target ID represents the same target object. And Rect is a coordinate area representing a rectangle (rectangular detection frame), Rect = (x, y, w, h), x, y are the coordinates of the upper left vertex of the rectangular detection frame, w, h are the width and height of the rectangular detection frame, respectively, and Rect represents the position information of the target object in the video frame image. In addition, the present application can also use shapes such as elliptical detection frames and triangular detection frames for corresponding representation, which are not limited here.

[0036] It should be noted that one or more target objects may exist simultaneously in a video frame image of the present application, and different target objects can be distinguished by target ID, which is not limited here. For example, the target recognition result may include all target IDs and corresponding coordinate areas Rect in the current frame, recorded as (ID, Rect). When multiple target objects are recognized in a frame, the target recognition results obtained may form a set, recorded as:

[0037]

[0038] Where i is the frame number, and the index number of each frame can be unique; j=0~N, represents the target object number in a frame, and N+1 is the number of target objects in this frame.

[0039] Specifically, for each video frame image, after target recognition, it can be determined whether there is a recognized target object in each video frame image according to the target recognition result. The video frame image in which the target object is recognized is determined as the target video frame, and the video frame image in which the target object is not recognized is determined as the video frame to be estimated. It can be understood from the above example that the video frame image in which the target object is not recognized does not necessarily mean that there is no target object, which will not be elaborated here.

[0040] It should also be noted that the video frame cache queue of the present application can be a preset first-in-first-out queue for caching video frame images and corresponding target recognition results. The queue length can be set to L, indicating that the queue can cache a maximum of L frames of images and their recognition results. When the actual queue length = L, the queue is full; otherwise (the actual queue length < L), the queue is not full. Exemplarily, L of the present application can be set to 5 (or other values, not limited here), which means that a maximum of 5 frames of video frame images and their corresponding target recognition results can be cached in the video frame cache queue. Among them, if the video frame cache queue is not full, the acquired video frame images and the corresponding target recognition results are inserted into the cache queue in sequence; if the video frame cache queue is full, the video frame image Frame` and its target recognition result that are first added to the queue (at the head of the queue) in the current cache queue can be output to the queue according to the first-in-first-out principle of the queue, and the currently acquired video frame image Frame and its target recognition result are inserted into the tail of the queue.

[0041] Step S120 , performing motion estimation processing on the target object according to the target recognition results of each target video frame and the timing information of each target video frame, to obtain an estimated area of ​​the target object in the video frame to be estimated.

[0042] In conjunction with the above steps, after the video frame buffer queue is full, motion estimation can be performed based on the video frame images in the video frame buffer queue.

[0043] It is understandable that to perform motion estimation on the target object, it is necessary to perform the motion estimation based on the target video frame in which the target object is identified, and estimate the estimated area where the target object should be located in the to-be-estimated video frame in which the target object is not identified.

[0044] Specifically, in combination with the description of the aforementioned embodiment, it can be understood that in the video frame buffer queue, there may be a situation where the target object has just entered the acquisition field of view and / or the target object has just left the acquisition field of view. Therefore, the video frame to be estimated that has not identified the target object may be at the head and / or tail of the video frame buffer queue (it may also be in the middle of the queue). Therefore, in order to ensure the accuracy of the estimated area, forward motion estimation (estimation based on the same direction of the time flow) and / or reverse motion estimation (estimation based on the reverse direction of the time flow) can be performed according to the timing information of the video frame. That is, in the forward motion estimation, the estimated area where the target object is located in the next frame of the video frame image can be forward predicted based on the timing and the motion trajectory of the target object; and in the reverse motion estimation, the estimated area where the target object is located in the previous frame of the video frame image can be reversely predicted based on the timing and the motion trajectory of the target object. In addition, if the video frame to be estimated is in the middle of the video frame cache queue, it can also be estimated based on the two adjacent target video frames on the left and right of the video frame to be estimated (for example, taking the average of the sum of the coordinate information of the two adjacent target video frames on the left and right to estimate the position of the target object in the middle video frame to be estimated, which will not be elaborated here).

[0045] In summary, there may be one or more video frames to be estimated in the video frame buffer queue of the present application, and therefore, one or more estimated regions may be obtained.

[0046] Step S130 , performing image processing on the estimated area in the to-be-estimated video frame and the area where the target object is located in the target video frame to obtain a target image.

[0047] In conjunction with the above steps, in order to achieve accurate privacy protection, image processing is required for the area where the target object is identified in the target video frame, and image processing is also required for the estimated area in the video frame to be estimated, to obtain the target image. Of course, it is not limited to processing both the estimated area and the area where the target object is located, and one of them can be processed; or after filtering the two areas separately, other image areas in the video frame to be estimated except the estimated area, and other image areas in the target video frame except the area where the target object is located are processed. Only then can the target image be sent to the video image encoder for encoding processing to obtain a code stream, and then the code stream is transmitted to implement subsequent business processes, which will not be elaborated here.

[0048] The image processing method of the present application may include, but is not limited to, image occlusion processing, image smearing processing, image cropping processing, image deformation processing, etc., which are not limited here. For example, the pixel values ​​of the estimated area and the area where the target object is located are set to 0 or 255, or the estimated area and the area where the target object is located are cropped, or the estimated area and the area where the target object is located are distorted, etc.

[0049] It should also be noted that the estimated area may not exist in the video frame to be estimated in the present application. Specifically, it is necessary to determine whether the estimated area is in the image content of the video frame to be estimated based on the motion information of the target object. If the estimated area does not exist in the video frame to be estimated, image processing can be performed on the area where the target object is located in the target video frame, and the processed target video frame and the unprocessed video frame to be estimated are output as the target image.

[0050] It can be seen that the present application determines the target video frames in which the target objects are identified and the to-be-estimated video frames in which the target objects are not identified in the video frame cache queue according to the target recognition results of each video frame image in the video frame cache queue; wherein the target recognition results are obtained by performing target object recognition processing on the video frame images, thereby determining the target video frames in which the target objects are successfully identified in each video frame image in the video frame cache queue, and the to-be-estimated video frames in which the target objects are not successfully identified and which need to be determined by motion estimation; motion estimation processing is performed on the target objects according to the target recognition results of each target video frame and the timing information of each target video frame to obtain the estimated area of ​​the target object in the to-be-estimated video frame; image processing is performed on the estimated area in the to-be-estimated video frame and the area where the target object is located in the target video frame, so as to protect the privacy of the target object as much as possible, and the obtained target image is output, thereby realizing effective protection of privacy in the video image detection scenario.

[0051] Based on the above embodiments, the embodiment of the present application describes the steps of performing motion estimation processing on the target object according to the target recognition results of each target video frame and the timing information of each target video frame to obtain the estimated area of ​​the target object in the video frame to be estimated. Specifically, the method of this embodiment includes the following steps:

[0052] According to the frame timing of each target video frame and the target recognition result of each target video frame, the motion information of the target object is determined; according to the motion information, the motion estimation processing is performed on the target object to obtain the estimated area.

[0053] In combination with the above-mentioned embodiments, the motion estimation processing process of the present application may include forward motion estimation and / or reverse motion estimation, which needs to be performed based on the frame timing of each target video frame in the cache queue and the target recognition results of each target video frame.

[0054] Specifically, after determining the estimated direction, the motion information (such as forward motion information and / or reverse motion information) of the target object can be determined according to the frame timing of the target video frame. The motion information may include the position information, moving speed information, moving direction information, etc. of the target object, which is not limited here.

[0055] Then, by constructing a motion equation (motion model) and substituting the acquired motion information of the target object into the calculation, the estimated area of ​​the target object in the video frame to be estimated can be estimated. In addition, the motion estimation method of the present application can also predict the estimated area of ​​the target object in the video frame to be estimated by the optical flow method, etc., which will not be elaborated here.

[0056] Based on the above embodiment, the embodiment of the present application describes the steps of determining the motion information of the target object according to the frame timing of each target video frame and the target recognition result of each target video frame. The motion information includes a motion vector. Specifically, the method of this embodiment includes the following steps:

[0057] Determine a first motion vector of the target object in the first timing direction according to the first timing direction, the frame timing of each target video frame and the target recognition result of each target video frame; and / or determine a second motion vector of the target object in the second timing direction according to the second timing direction, the frame timing of each target video frame and the target recognition result of each target video frame; wherein the first timing direction and the second timing direction are opposite.

[0058] In conjunction with the above-mentioned embodiments, the motion estimation processing of the present application may include forward motion estimation and / or reverse motion estimation. Among them, forward and reverse refer to the frame timing (time flow) relative to the video frame image. Forward motion estimation and reverse motion estimation may be performed in a sequential order or synchronously, which is not limited here. In the present application, the estimation result obtained after the motion estimation processing may also be recorded in the cache queue like the target recognition result.

[0059] For example, in this embodiment, the first timing direction and the second timing direction are opposite, if the first timing direction is the time forward direction, then the second timing direction is the time reverse direction; if the first timing direction is the time reverse direction, then the second timing direction is the time forward direction. For ease of explanation, in the embodiment of this application, the first timing direction is the time reverse direction and the second timing direction is the time forward direction.

[0060] For example, assuming that the cache queue length L = 5, the cache queue contains 5 video frame images and the corresponding target recognition results when performing motion estimation processing. By traversing the cache queue from the tail to the head, the target recognition results can be obtained as follows: , , , , . 100, 99, 98, 97, and 96 are the frame numbers of the corresponding video frame images, that is, in the current cache queue, the 100th frame image is at the end of the cache queue, and the 96th frame image is at the head of the cache queue. The timing of the 96th frame image is earlier than the timing of the 100th frame image. For the convenience of subsequent description and understanding, the direction in which the frame number decreases (time reverse) can be defined as the right, and the direction in which the frame number increases (time forward) can be defined as the left.

[0061] Furthermore, we can assume that there are n different target IDs (i.e. n target objects) in total in the 5 video frames. By classifying the target ID values, we can obtain a Rect set of each ID value (here referred to as a trajectory set, in which the elements are called trajectory boxes, which are one of the target recognition results obtained through target recognition). For example: Trajectory = {Rect0, Rect1, ...}, where the elements are Match all track boxes found in the video frame images in the cache queue; Trajectory = {Rect0, Rect1, ...}, where the elements are Match all track boxes found in the video frame images in the cache queue; Trajectory = {Rect0, Rect1, ...}, where the elements are Match all trajectory boxes found in the video frame images in the cache queue; ...; Trajectory = {Rect0, Rect1, ...}, where the elements are All track boxes found in the video frame images of the matching cache queue. It should be noted that the track box information corresponding to the same ID above follows: , , , , The traversal order is arranged, that is, from left to right, thereby obtaining the order of reverse motion estimation.

[0062] For reference Figure 3 As shown, Figure 3 This is a schematic diagram of the scene of inverse motion estimation in the image processing method based on motion estimation of this application. If the target object enters the acquisition field of view at the time of acquisition of the 96th frame image, the target recognition algorithm may not recognize the target object in the 96th frame image. (i.e. It will not exist Instead of identifying the target object in the 97th frame and the target object in the 98th frame, the 96th frame is determined as the video frame to be estimated, and the 97th and 98th frames are determined as the target video frames. The estimation area in the 96th frame needs to be estimated through the 97th and 98th frames.

[0063] Specifically, in the reverse motion estimation of the above example, the current motion estimation direction of the target object should be — — , that is, the estimated area in the 96th frame image is inferred by the target recognition result of the 97th frame image and the target recognition result of the 98th frame image. Therefore, the first motion vector of the target object in the first time sequence direction is determined by the target recognition result of the 97th frame image and the target recognition result of the 98th frame image, and then the estimated area of ​​the target object in the 96th frame image can be determined based on the target recognition result of the 97th frame image and the first motion vector.

[0064] Need to target Figure 3 It is explained that Figure 3 Frame (index=96) in is the video frame to be estimated. The target object in this frame is partially blocked and is not successfully identified by the target recognition algorithm. Therefore, its target recognition result does not contain relevant information about the target object. Therefore, its trajectory frame ( ) can be obtained from the motion information and coordinate information of the same target object in two consecutive frames on the left ( and ) to perform motion estimation. After obtaining the estimated trajectory frame (estimated area) After that, Insert the target recognition result of the corresponding frame in the cache queue.

[0065] Among them, it is necessary to specially explain that: according to the actual time lapse order, the acquisition order of video frames from early to late is: Frame (index=96) - Frame (index=97) - Frame (index=98), that is, Frame (index=96) is actually the earliest video frame in the time sequence. In the real world, the motion trajectory is the process of the target object moving from the lower right corner of the acquisition field of view to the upper left corner. The reverse estimation is actually to use the subsequent target video frame to predict the estimated area in the previous video frame to be estimated. Its significance lies in that it can solve the situation where the target object such as Frame (index=96) has just entered the acquisition field of view, and the target recognition algorithm cannot accurately identify the target object at this time.

[0066] Similarly, the motion estimation direction of the forward motion estimation (second timing direction) in this embodiment is opposite to that in the aforementioned embodiment, but the implementation principle is the same, which will not be described in detail here.

[0067] Optionally, in addition to the method of determining the target video frame and the video frame to be estimated according to the target recognition result in the above example and performing motion estimation processing, this embodiment can also be implemented by other methods. According to the target recognition results in the cache queue, track = { , ,…, }, where m is the number of Rects (e.g., if the target object is recognized in 3 of the 5 video frames, , we can get 3 target recognition results Rect, m=3).

[0068] Then the video frame to be estimated for inverse motion estimation can be determined by the following three judgment conditions: 1. m>1; 2. , ,…, The video frames are continuous, that is, their frame numbers are continuous; 3. In the current cache queue, There is at least one video frame image to the right of the video frame image (that is, there is a video frame image in the current cache queue that is earlier than The target video frame is the video frame image of the target video frame), which is equivalent to the 96th frame image in the above example. The target object is not successfully identified in this image. If all three judgment conditions are met at the same time, The first video frame on the right of the video frame is determined as the video frame to be estimated, otherwise no processing is performed.

[0069] Now define the target object ( The direction of movement is — — (time reverse), where are the target trajectory frames of the first and second frames on the left side of the frame to be estimated, and are the known conditions in the target recognition results. The coordinate frame information of the estimated area in the video frame to be estimated. The estimation method can use conventional motion vector prediction, which is briefly described below: , is the position box of two consecutive frames on the left side of the video frame to be estimated (the area where the identified target object is located), and the target can be obtained from arrive The motion vector is V = ( , ),in , which is the distance that the target object moves on the x-axis between two adjacent video frames. Then it is the movement distance on the y-axis:

[0070]

[0071] Then we can find:

[0072]

[0073] That is, the target object in the video frame to be estimated ( The predicted location of , that is, the estimated area.

[0074] For forward motion estimation, the cache queue can be traversed in the same manner as in the above example to obtain a set of trajectories of the same target object. Take the following as an example to illustrate: For example, its trajectory set = { , ,…, }. Determine whether there is a video frame that needs to estimate the trajectory frame in the cache queue. If all of the following three conditions are met, it is determined that there is a frame that needs to estimate the trajectory frame (video frame to be estimated): Judgment condition 1: m>1; Judgment condition 2: , ,…, The frames are continuous, that is, their frame numbers are continuous; Judgment condition 3: In the current cache queue, There is at least one frame to the left of the frame where the estimated frame is located. If there is a video frame to be estimated, then take The first frame to the left of the frame is the video frame to be estimated, otherwise no processing is performed.

[0075] The movement direction of the target object is defined as — — (time forward, i.e. the real direction of movement of the target object in the real world), where The target position boxes of the first and second frames on the right side of the video frame to be estimated (the area where the identified target object is located), is the trajectory frame to be estimated (estimated area). The specific motion estimation method can refer to the principle of the above embodiment, using arrive The motion vector of , which will not be described here. After that, it is inserted into the target recognition result of the corresponding frame in the cache queue. It can be understood that the significance of reverse estimation is that the target object in the first frame image when the target object enters the acquisition field of view may be incomplete, resulting in the problem that the algorithm cannot accurately recognize the target object. The significance of forward estimation is that the target object in the last frame image when the target object walks out of the acquisition field of view may be incomplete, resulting in the problem that the algorithm cannot accurately recognize the target object.

[0076] Based on the above embodiment, the embodiment of the present application describes the steps of performing motion estimation processing on the target object according to the motion information to obtain the estimated area. The estimated area includes the first estimated area and / or the second estimated area. Specifically, the method of this embodiment includes the following steps:

[0077] According to the first motion vector and the position information of the target object, motion estimation processing is performed in a first timing direction to obtain a first estimated area; and / or, according to the second motion vector and the position information of the target object, motion estimation processing is performed in a second timing direction to obtain a second estimated area.

[0078] It should be noted that the estimated area of ​​the present application may include a first estimated area and / or a second estimated area, depending on whether forward motion estimation and / or reverse motion estimation are performed during the implementation process and whether the motion estimation result is obtained normally. The first estimated area and the second estimated area may be the same area in different video frame images (for example, the position where the target object enters the acquisition field of view and the position where it leaves the acquisition field of view are the same), or they may be different areas, which is not limited here.

[0079] The specific calculation method of the motion vector can be referred to the above embodiment, which will not be described in detail here.

[0080] Based on the above embodiments, the embodiment of the present application describes the steps of determining the target video frames in the video frame cache queue that recognize the target object and the to-be-estimated video frames in the video frame cache queue that do not recognize the target object according to the target recognition results of each video frame image in the video frame cache queue. Specifically, the method of this embodiment includes the following steps:

[0081] The target recognition results of each video frame image in the video frame buffer queue are traversed to obtain the trajectory information of the target object; and the target video frame and the video frame to be estimated in the video frame buffer queue are determined according to the trajectory information.

[0082] In conjunction with the above-mentioned embodiments, in the specific implementation process of the present application, the target object's trajectory information (trajectory frame) can be obtained by traversing the target recognition results of each video frame image in the video frame cache queue. Among them, the video frame image with corresponding trajectory information in the video frame cache queue can be determined as the target video frame, and the video frame image without corresponding trajectory information in the video frame cache queue can be determined as the video frame to be estimated.

[0083] Based on the above embodiment, the embodiment of the present application describes the steps of determining the target video frame in the video frame cache queue and the video frame to be estimated according to the trajectory information. The trajectory information includes the trajectory start information and / or the trajectory end information. Specifically, the method of this embodiment includes the following steps:

[0084] Determine a video frame image with trajectory information in a video frame cache queue as a target video frame; obtain a starting video frame corresponding to the trajectory starting information and an ending video frame corresponding to the trajectory ending information in the target video frame; determine a video frame image in the video frame cache queue that is earlier than the starting video frame in timing as a video frame to be estimated; and / or determine a video frame image in the video frame cache queue that is later than the ending video frame in timing as a video frame to be estimated.

[0085] In combination with the above-mentioned embodiment, the trajectory start information refers to the earliest trajectory information in the video frame buffer queue, and the start video frame refers to the video frame image corresponding to the trajectory start information, that is, the start video frame is the video frame image of the first frame in the video frame buffer queue that recognizes the target object according to the time sequence.

[0086] In the previous example, the trajectory start information is equivalent to In Rect97, the starting video frame is equivalent to the 97th video frame image. If there is a video frame image (the 96th frame image) that is earlier than the starting video frame in the video frame cache queue, it means that the target object is not recognized in the frame image, and it is determined as the video frame to be estimated. Therefore, motion estimation can be performed using the method provided in the above example to obtain the estimated area.

[0087] Similarly, the trajectory termination information refers to the latest trajectory information in the video frame buffer queue, and the termination video frame refers to the video frame image corresponding to the trajectory termination information, which means that the termination video frame is the video frame image in the video frame buffer queue that recognizes the target object in the last frame according to the timing. If there are video frame images in the video frame buffer queue that are later than the termination video frame, it can also indicate that the target object is not recognized in the image, so the frame image is also a video frame to be estimated. The specific understanding can be combined with the above description, and it will not be repeated here.

[0088] On the basis of the above-mentioned embodiment, the embodiment of the present application describes the steps before determining the target video frame in which the target object is identified and the to-be-estimated video frame in which the target object is not identified in the video frame cache queue according to the target recognition results of each video frame image in the video frame cache queue. Specifically, the method of this embodiment includes the following steps:

[0089] The acquired video frame images are processed for target object recognition to obtain target recognition results corresponding to the video frame images; according to the frame timing of each video frame image, the video frame images and the target recognition results are correspondingly stored in the video frame cache queue.

[0090] In conjunction with the above-mentioned embodiments, in the specific implementation process of the present application, before determining the target video frame and the video frame to be estimated in the video frame buffer queue, each newly captured video frame image (Frame) of the image acquisition device can be sent to the target recognition algorithm module to perform target object recognition processing on it, and obtain the target recognition result of each video frame image (which may include a successful recognition result or a failed recognition result). The target recognition result of each video frame image is recorded, and each video frame image and its corresponding target recognition result are stored in the video frame buffer queue according to the frame timing.

[0091] If the video frame buffer queue is not full, the video frame image and the corresponding target recognition result are continuously acquired for storage. Furthermore, if the video frame buffer queue is not full and the video frame image and the corresponding target recognition result are not continuously acquired, the subsequent steps can be performed directly based on the current video frame buffer queue. Specifically, a time threshold can also be set, and after receiving each video frame image and the corresponding target recognition result, a timing process is performed to obtain the interruption time. In response to the interruption time being greater than the time threshold, the subsequent steps can be performed directly based on the current video frame buffer queue without having to wait for the video frame buffer queue to be full.

[0092] Optionally, if the interval time is greater than the time threshold, the video frame images in the current video frame cache queue can be directly cleared after the judgment of the video frame to be estimated or the image processing is completed, and the video frame images and their target recognition results are stored again. This avoids the existence of video frame images with too long intervals in the video frame cache queue, which in turn affects the judgment accuracy of the present application.

[0093] If the video frame buffer queue is full, the video frame images in the video frame buffer queue are analyzed to determine the target video frame and the video frame to be estimated in the video frame buffer queue, so as to perform subsequent motion estimation processing.

[0094] Based on the above embodiments, the embodiment of the present application exemplarily illustrates the steps of performing image occlusion processing on the estimated area in the to-be-estimated video frame and the area where the target object is located in the target video frame to obtain the target image, and the steps after performing image processing on the estimated area to obtain the target image. Specifically, the method of this embodiment includes the following steps:

[0095] The estimated area in the estimated video frame and the area where the target object is located in the target video frame are subjected to image occlusion processing to obtain a target image; the target image is input into an image encoder for encoding processing to obtain a target bitstream.

[0096] With reference to the foregoing embodiment, after obtaining the estimated area of ​​the video frame to be estimated, you can choose to perform image occlusion processing on the estimated area in the video frame to be estimated and the area where the target object is located in the target video frame to obtain a target image; then input the target image into the image encoder for encoding processing to obtain a target code stream.

[0097] In addition, if it is determined after motion estimation that there is no estimation area in the video frame to be estimated, image occlusion processing can be performed only on the area where the target object is located in the target video frame, and the processed target video frame and the unprocessed video frame to be estimated are used as target images for encoding processing to obtain a target bitstream. The specific video image encoding and decoding method can refer to the existing encoding and decoding technology, which will not be described in detail here.

[0098] It should be further explained that the execution subject of the image processing method based on motion estimation may be an image processing device based on motion estimation, for example, the image processing method based on motion estimation may be executed by a terminal device or a server or other processing device, wherein the terminal device may be a user equipment (UE), a computer, a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the image processing method based on motion estimation may be implemented by a processor calling a computer-readable instruction stored in a memory.

[0099] Figure 4 FIG. 1 is a block diagram of an image processing device based on motion estimation, which is shown as an exemplary embodiment of the present application. Figure 4 As shown, the exemplary motion estimation-based image processing device 400 includes: a video frame determination module 410, a motion estimation module 420 and an image processing module 430. Specifically:

[0100] The video frame determination module 410 is used to determine the target video frames in which the target objects are identified and the to-be-estimated video frames in which the target objects are not identified in the video frame cache queue according to the target recognition results of each video frame image in the video frame cache queue; wherein the target recognition results are obtained by performing target object recognition processing on the video frame images.

[0101] The motion estimation module 420 is used to perform motion estimation processing on the target object according to the target recognition results of each target video frame and the timing information of each target video frame to obtain the estimated area of ​​the target object in the video frame to be estimated.

[0102] The image processing module 430 is used to perform image processing on the estimated area in the video frame to be estimated and the area where the target object is located in the target video frame to obtain a target image.

[0103] In this exemplary image processing device based on motion estimation, according to the target recognition results of each video frame image in the video frame cache queue, the target video frames in which the target object is recognized and the to-be-estimated video frames in which the target object is not recognized in the video frame cache queue are determined; wherein the target recognition results are obtained by performing target object recognition processing on the video frame images, thereby determining the target video frames in which the target object is successfully recognized in each video frame image in the video frame cache queue, and the to-be-estimated video frames in which the target object is not successfully recognized and need to be determined by motion estimation; according to the target recognition results of each target video frame and the timing information of each target video frame, motion estimation processing is performed on the target object to obtain an estimated area of ​​the target object in the to-be-estimated video frame; image processing is performed on the estimated area in the to-be-estimated video frame and the area where the target object is located in the target video frame, so as to protect the privacy of the target object as much as possible, and the obtained target image is output, thereby realizing effective protection of privacy in the video image detection scenario.

[0104] It should be noted that the device provided in the above embodiment and the method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In practical applications, the device provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0105] The functions of each module can be found in the embodiment of the image processing method based on motion estimation, which will not be described in detail here.

[0106] See also Figure 5 , Figure 51 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 100 includes a memory 101 and a processor 102, and the processor 102 is used to execute program instructions stored in the memory 101 to implement the steps in any of the above-mentioned image processing method embodiments based on motion estimation. In a specific implementation scenario, the electronic device 100 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 100 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0107] Specifically, the processor 102 is used to control itself and the memory 101 to implement the steps in any of the above-mentioned image processing method embodiments based on motion estimation. The processor 102 can also be called a CPU (Central Processing Unit). The processor 102 may be an integrated circuit chip with signal processing capabilities. The processor 102 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 102 can be implemented by an integrated circuit chip.

[0108] In this exemplary electronic device, target video frames in which target objects are identified and to-be-estimated video frames in which target objects are not identified are determined in the video frame cache queue based on target recognition results of each video frame image in the video frame cache queue; wherein the target recognition results are obtained by performing target object recognition processing on the video frame images, thereby determining target video frames in which target objects are successfully identified in each video frame image in the video frame cache queue, and to-be-estimated video frames in which target objects are not successfully identified and which require determination of the target objects through motion estimation; motion estimation processing is performed on the target object based on the target recognition results of each target video frame and the timing information of each target video frame to obtain an estimated area of ​​the target object in the to-be-estimated video frame; image processing is performed on the estimated area in the to-be-estimated video frame and the area where the target object is located in the target video frame, so as to protect the privacy of the target object as much as possible, and the obtained target image is output, thereby achieving effective protection of privacy in the video image detection scenario.

[0109] See also Figure 6 , Figure 6The computer-readable storage medium 110 stores program instructions 111 that can be executed by a processor, and the program instructions 111 are used to implement the steps in any of the above-mentioned image processing method embodiments based on motion estimation.

[0110] In this exemplary storage medium, by running the program instructions in the storage medium, according to the target recognition results of each video frame image in the video frame cache queue, the target video frames in which the target object is recognized and the to-be-estimated video frames in which the target object is not recognized in the video frame cache queue are determined; wherein the target recognition results are obtained by performing target object recognition processing on the video frame images, thereby determining the target video frames in which the target object is successfully recognized in each video frame image in the video frame cache queue, and the to-be-estimated video frames in which the target object is not successfully recognized and need to be determined by motion estimation; motion estimation processing is performed on the target object according to the target recognition results of each target video frame and the timing information of each target video frame to obtain an estimated area of ​​the target object in the to-be-estimated video frame; image processing is performed on the estimated area in the to-be-estimated video frame and the area where the target object is located in the target video frame, so as to protect the privacy of the target object as much as possible, and the obtained target image is output, thereby realizing effective protection of privacy in the video image detection scenario.

[0111] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0112] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0113] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0114] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. An image processing method based on motion estimation, characterized in that: The method comprises: According to the target recognition results of each video frame image in the video frame buffer queue, determine the target video frame in which the target object is recognized and the to-be-estimated video frame in which the target object is not recognized in the video frame buffer queue; wherein the target recognition result is obtained by performing target object recognition processing on the video frame image; Performing motion estimation processing on the target object according to the target recognition results of each target video frame and the timing information of each target video frame to obtain an estimated area of ​​the target object in the video frame to be estimated; wherein the motion estimation processing includes forward motion estimation and / or reverse motion estimation; Image processing is performed on the estimated area in the to-be-estimated video frame and the area where the target object is located in the target video frame to obtain a target image.

2. The method according to claim 1, characterized in that The step of performing motion estimation processing on the target object according to the target recognition results of each target video frame and the timing information of each target video frame to obtain the estimated area of ​​the target object in the video frame to be estimated includes: Determining motion information of the target object according to the frame timing of each target video frame and the target recognition result of each target video frame; Perform motion estimation processing on the target object according to the motion information to obtain the estimated area.

3. The method according to claim 2, characterized in that The motion information includes a motion vector, and the step of determining the motion information of the target object according to the frame timing of each target video frame and the target recognition result of each target video frame includes: Determine a first motion vector of the target object in the first temporal direction according to the first temporal direction, the frame timing of each target video frame, and the target recognition result of each target video frame; And / or, based on the second timing direction, the frame timing of each target video frame and the target recognition result of each target video frame, determine the second motion vector of the target object in the second timing direction; wherein the first timing direction is opposite to the second timing direction.

4. The method according to claim 3, characterized in that The estimated area includes a first estimated area and / or a second estimated area, and the step of performing motion estimation processing on the target object according to the motion information to obtain the estimated area includes: According to the first motion vector and the position information of the target object, performing motion estimation processing in the first time sequence direction to obtain the first estimation area; And / or, based on the second motion vector and the position information of the target object, motion estimation processing is performed in the second time sequence direction to obtain the second estimated area.

5. The method according to claim 1, characterized in that The step of determining, according to the target recognition results of each video frame image in the video frame cache queue, the target video frame in which the target object is recognized and the to-be-estimated video frame in which the target object is not recognized in the video frame cache queue comprises: Traversing the target recognition results of each video frame image in the video frame buffer queue to obtain the trajectory information of the target object; The target video frame and the to-be-estimated video frame in the video frame buffer queue are determined according to the trajectory information.

6. The method according to claim 5, characterized in that The trajectory information includes trajectory start information and / or trajectory end information, and the step of determining the target video frame and the to-be-estimated video frame in the video frame buffer queue according to the trajectory information includes: Determine the video frame image containing the trajectory information in the video frame buffer queue as the target video frame; Obtaining a starting video frame corresponding to the track starting information and an ending video frame corresponding to the track ending information in the target video frame; Determine a video frame image in the video frame buffer queue that is earlier than the start video frame in time sequence as the video frame to be estimated; And / or, a video frame image in the video frame buffer queue whose timing is later than the termination video frame is determined as the video frame to be estimated.

7. The method according to claim 1, characterized in that Before the step of determining, according to the target recognition results of each video frame image in the video frame buffer queue, the target video frame in which the target object is recognized and the to-be-estimated video frame in which the target object is not recognized in the video frame buffer queue, the method further includes: Performing target object recognition processing on the acquired video frame image to obtain a target recognition result corresponding to the video frame image; According to the frame timing of each video frame image, the video frame image and the target recognition result are correspondingly stored in the video frame buffer queue.

8. The method according to claim 1, characterized in that: The step of performing image processing on the estimated area in the video frame to be estimated and the area where the target object is located in the target video frame to obtain the target image comprises: Performing image occlusion processing on the estimated area in the to-be-estimated video frame and the area where the target object is located in the target video frame to obtain the target image; After the step of performing image processing on the estimated area to obtain a target image, the method further includes: The target image is input into an image encoder for encoding to obtain a target code stream.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images