Video processing method, video recognition method, video processing device and vehicle

Abstract

The present application relates to a video processing method, a video recognition method, a video processing device and a vehicle. The video processing method includes: obtaining the current position of a moving target in a current video frame; calculating the inter-frame displacement of the moving target from the position in a forward key frame to the current position; in response to the inter-frame displacement satisfying a preset screening condition, determining that the current video frame is a key frame that can be used for preset recognition processing. This method can be used to filter key frames for preset recognition processing from video frames, so as to reduce the recognition burden of preset recognition processing, thereby effectively reducing the delay of target recognition and improving the timeliness of target recognition. At the same time, video processing and video recognition can be realized locally, which is conducive to improving the security of information.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a video processing method, a video recognition method, a video processing device and a vehicle. Background Art

[0002] When using deep learning technologies such as DNN (Deep Neural Network) and RNN (Recurrent Neural Network) to perform video stream processing tasks, a large amount of storage resources and computing resources are required when the task is running, and there is a risk that mobile platforms such as vehicles and monitoring devices cannot afford the resource occupation.

[0003] Usually, neural networks deployed in the cloud and on the server are used to perform video stream processing tasks such as recognition, and then the task results are returned to the mobile platform. However, off-site data processing has the risk of increasing user privacy exposure, and data transmission such as video streams and task results affects the real-time performance of tasks. If video stream processing tasks are performed on mobile platforms, serious delays are likely to occur due to the occupation of storage resources and computing resources and task performance requirements. Summary of the invention

[0004] Based on this, it is necessary to provide a video processing method, video recognition method, video processing device and vehicle that can filter key frames from video frames for preset recognition processing to address the above-mentioned technical problems, so as to reduce the recognition burden of preset recognition processing, and further effectively reduce the delay in target recognition and improve the timeliness of target recognition. At the same time, video processing and video recognition can be realized locally, which is conducive to improving the security of information.

[0005] On the one hand, a video processing method is provided, which includes: obtaining the current position of a moving target in a current video frame; calculating the inter-frame displacement of the moving target from the position in a forward key frame to the current position; and in response to the inter-frame displacement satisfying a preset filtering condition, determining that the current video frame is a key frame that can be used for preset recognition processing.

[0006] In one embodiment of the present application, the preset screening condition includes: the inter-frame displacement is greater than a screening threshold; wherein the screening threshold is adjustable.

[0007] In one embodiment of the present application, obtaining the current position of a moving target in a current video frame includes: obtaining motion information of an area of ​​interest in the current video frame relative to a first video frame, superimposing the motion information to a first position of the moving target in the first video frame, and obtaining an observed position of the moving target; wherein the first video frame is a forward video frame of the current video frame; obtaining position change information of the moving target in a second video frame relative to the first video frame, superimposing the position change information to the first video frame, and obtaining a predicted position of the moving target; wherein the second video frame is a forward video frame of the first video frame; fusing the observed position and the predicted position to obtain a corrected predicted position, and obtaining the current position based on the corrected predicted position.

[0008] In one embodiment of the present application, obtaining motion information of a region of interest in a current video frame relative to a first video frame includes: dividing the region where the moving target is located into multiple macroblocks, and using the multiple macroblocks as the region of interest; wherein the number of macroblocks is adjustable; and using motion vectors of the multiple macroblocks as motion information.

[0009] In one embodiment of the present application, using the motion vectors of multiple macroblocks as motion information includes: dividing the motion vectors into multiple vector groups based on the similarity of the motion trajectories of the motion vectors; selecting a vector group including the most motion vectors, calculating the average value of the motion vectors in the selected vector group, and obtaining a balanced motion vector as the motion information.

[0010] In one embodiment of the present application, the observed position and the predicted position are fused to obtain a revised predicted position, including: obtaining the area between the observed position and the predicted position as the new information area; assigning weights to the new information area for weighting, and superimposing one of the observed position and the predicted position with the weighted new information area to obtain the revised predicted position.

[0011] In one embodiment of the present application, position change information of a moving target in a second video frame relative to a first video frame is obtained, and the position change information is superimposed on the first video frame to obtain a predicted position of the moving target, including: obtaining a second position of the moving target in the second video frame; calculating a moving speed and a moving direction of the moving target from the second position to the first position; taking the first position as a starting position, predicting a position reached by the moving target at the moving speed and the moving direction within an inter-frame duration as the predicted position; wherein the inter-frame duration is the interval between the first video frame and the current video frame.

[0012] Another method provides a video recognition method, which includes: obtaining key frames using a video processing method such as in any of the above embodiments; and inputting the key frames into a preset recognition model for preset recognition processing.

[0013] On the other hand, a video processing device is provided, including: a capture unit and a screening unit; the capture unit is used to obtain the current position of a moving target in a current video frame; the screening unit is connected to the capture unit, and is used to calculate the inter-frame displacement of the moving target from the position in a previous key frame to the current position; in response to the inter-frame displacement satisfying a preset screening condition, the current video frame is determined to be a key frame that can be used for preset recognition processing.

[0014] On the other hand, a vehicle is provided, comprising: an encoder, a video processing device and a preset recognition model; the encoder is used to extract motion information when compressing a video stream; the video processing device as in the above embodiment is connected to the encoder for capturing moving targets; the preset recognition model is connected to the video processing device for receiving key frames and performing preset recognition processing on the key frames.

[0015] The above-mentioned video processing method, video recognition method, video processing device and vehicle calculate the inter-frame displacement of the moving target from the position in the forward key frame to the current position, and screen whether the current video frame can be used as a key frame for preset recognition processing. When screening key frames, the inherent motion information in the video is used to improve the efficiency of screening key frames, and to achieve fast and complete selection of single-frame images for subsequent recognition from continuous video frames. Screening video frames based on the inter-frame displacement of the moving target is conducive to reducing the recognition burden of the preset recognition model compared to using a preset recognition model for each video frame to perform preset recognition processing, so as to reduce the computational cost, thereby helping to reduce the delay in target recognition, and thus helping to improve the timeliness of target recognition. At the same time, it can also reduce the occupation of storage resources and computing resources, so as to help balance the resource allocation of mobile platforms such as vehicles, thereby ensuring performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flowchart of an embodiment of the video processing method of the present application;

[0017] Figure 2 is a flowchart of another embodiment of the video processing method of the present application;

[0018] Figure 3 It is a structural diagram of an embodiment of a video frame of the present application;

[0019] Figure 4 It is a flowchart of an embodiment of the video recognition method of the present application;

[0020] Figure 5 is a structural diagram of an embodiment of a video processing device of the present application;

[0021] Figure 6 It is a structural schematic diagram of a vehicle embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0023] In order to solve the technical problems of information leakage risk and delay problem in the video recognition method in the prior art, the present application provides a video processing method, a video recognition method, a video processing device and a vehicle, which are described in detail below.

[0024] See also Figure 1 , Figure 1 It is a flow chart of an embodiment of the video processing method of the present application. It should be noted that the video processing method of the present application can be applied to monitoring devices, vehicles and other application environments, and is not limited here.

[0025] S101: Obtain the current position of the moving target in the current video frame.

[0026] In this embodiment, the current video frame is a video frame that is being screened to determine whether to be used as a key frame.

[0027] Key frames are video frames that can be used for subsequent preset recognition processing. In this way, it is not necessary to input all video frames into the preset recognition model for preset recognition processing, which is conducive to reducing the recognition burden of the preset recognition model, thereby reducing the computational cost, thereby helping to reduce the delay in target recognition, and further helping to improve the timeliness of target recognition.

[0028] The moving target may be a movable object, such as an animal or a person, etc. In this embodiment, the moving target is not identified yet.

[0029] The current position is the position of the moving target in the current video frame. The current position can be directly obtained based on the motion vector extracted by the encoder. Alternatively, the current position can be obtained by filtering the predicted position and the observed position, where the predicted position is predicted based on the motion trend at the historical moment, and the observed position is the position read in the current video frame.

[0030] The current position can be expressed as coordinates of a coordinate system established based on the current video frame, such as the coordinates of the center point of the moving target or the diagonal coordinates of the moving target rectangular frame; it can also be expressed as a connected area of ​​pixels where the moving target is located, which is not limited here.

[0031] S102: Calculate the inter-frame displacement of the moving target from the position in the previous key frame to the current position.

[0032] In this embodiment, the forward key frame is a video frame that is determined to be a key frame before.

[0033] In this embodiment, the forward key frame can be an adjacent forward key frame, that is, a forward key frame that is closest to the acquisition time of the current video frame. Alternatively, the selected forward key frame is a selected key frame that has a certain interval between it and the current video frame. Alternatively, the selected forward key frame is the video frame in which the current moving target is first recognized.

[0034] The representation form of the position of the moving target in the adjacent forward key frame matches the representation form of the current position, and may be represented in the same form or in forms that can be converted to each other.

[0035] The interval between the position of the moving target in the adjacent forward key frame and the current position is calculated, and the interval is used as the inter-frame displacement. The distance that the moving target moves between video frames is used to characterize the similarity between video frames, which is used as the condition for filtering key frames.

[0036] S103: In response to the inter-frame displacement satisfying a preset screening condition, determining that the current video frame is a key frame that can be used for a preset recognition process.

[0037] In this embodiment, the preset screening condition is used to screen / determine whether the current video frame is a key frame.

[0038] The preset screening condition may be that the inter-frame displacement is greater than a preset distance threshold, wherein the preset distance threshold may be a fixed value; or the preset distance threshold may be positively correlated with the time interval between the current video frame and the forward video frame, which is not limited here.

[0039] In response to the moving distance of the moving target between the forward key frame and the current key frame satisfying the preset filtering conditions, that is, the inter-frame displacement of the moving target satisfies the preset filtering conditions, it is considered that there is a large difference in the position of the moving target between the current video frame and the forward key frame, and the moving target can be subjected to preset recognition processing, then the current video frame is determined to be a key frame that can be used for preset recognition processing.

[0040] The preset recognition process may be to recognize the specific type of the moving target, such as whether it is a human being, etc. Alternatively, the moving target may be identified, etc., which is not limited here.

[0041] The movement information of the moving target between video frames is the information that the encoder needs to extract when compressing the video bitstream. Using this as a condition for filtering key frames is conducive to utilizing the inherent motion information in the video to improve the efficiency of filtering key frames and achieve fast and complete selection of single-frame images for subsequent recognition from continuous video frames.

[0042] It can be seen that in this embodiment, the inherent motion information in the video is used when screening key frames, which is conducive to improving the efficiency of screening key frames and realizing the rapid and complete selection of single-frame images for subsequent recognition from continuous video frames. Screening video frames based on the inter-frame displacement of the moving target is conducive to reducing the recognition burden of the preset recognition model compared to using a preset recognition model for each video frame to perform preset recognition processing, so as to reduce the computational cost, thereby helping to reduce the delay in target recognition, and then helping to improve the timeliness of target recognition. At the same time, it can also reduce the occupation of storage resources and computing resources, etc., so as to help balance the resource allocation of the mobile platform where the video processing device is located, thereby ensuring performance. In other words, the video processing method in this embodiment can be run locally on the mobile platform.

[0043] See also Figure 2 , Figure 2 FIG. 2 is a flow chart of another embodiment of the identification processing method of the present application. In this embodiment, the forward video frame of the key frame used for screening is an example of a forward key frame adjacent to the current video frame.

[0044] S201: Obtain the motion vector of the region of interest in the current video frame.

[0045] In this embodiment, the region of interest is an image region from the current video frame, and various operators and functions commonly used in machine vision software such as Halcon, OpenCV, and Matlab can be used to obtain ROI (region of interest), which is not limited here.

[0046] In this embodiment, the region of interest is an image region containing a moving object.

[0047] Specifically, Figure 3 As shown in the example, Figure 3 The structure diagram of the video frame of the present application is a schematic diagram of an embodiment of the present application. The current video frame including the region where the moving target is located can be divided into multiple macroblocks, and multiple macroblocks in the region where the moving target is located are selected as the region of interest.

[0048] Furthermore, the number of macroblocks can be adjusted, and the number of macroblocks in the region of interest can be limited, or the number of macroblocks around the region of interest can be limited. Limiting and adjusting the number of macroblocks is conducive to reducing the risk of excessive background image noise affecting the accuracy of current position prediction, and is also conducive to increasing the richness of image information and improving the accuracy of current position recognition.

[0049] Optionally, the number of macroblocks around the moving target may be preset to three, four, five, six, etc.

[0050] The motion vectors of multiple macroblocks may be used as the motion information. For example, all the motion vectors may be directly used as the motion information.

[0051] Optionally, the background image can be removed by a reasonable macroblock division method. For example, the motion vector obtained from the pixel points around the moving target that are close to the background image can be discarded to reduce the interference of the uncertain motion vector on the prediction of the current position.

[0052] Furthermore, the equalized motion vector serving as the motion information may also be obtained through the processing method in step S202 to step S203.

[0053] S202: Divide the motion vectors into a plurality of vector groups based on similarities of motion trajectories of the motion vectors.

[0054] In this embodiment, a plurality of motion vectors are obtained, so as to improve the richness and reliability of the motion trajectory expression compared with a single motion vector.

[0055] The motion vectors may be grouped based on similarities in their motion trajectories.

[0056] For example, the motion vectors may be clustered using a clustering algorithm to form a plurality of vector groups. The motion vectors may also be grouped based on the direction of the motion vector.

[0057] For example, four directional groups, namely, upper left, upper right, lower left, and lower right, may be preset, and the motion vectors may be divided into the corresponding groups based on the directional characteristics contained in the motion vectors.

[0058] S203: Select a vector group including the most motion vectors, calculate the average value of the motion vectors in the selected vector group, and obtain an equalized motion vector as motion information.

[0059] In this embodiment, the vector group including the most motion vectors is used as the vector group to be used, and it is considered that the motion vector of the current group can represent the motion trajectory of the moving target.

[0060] The average value of the motion vectors on the coordinate axis in the selected vector group is calculated, that is, the balanced motion vector, and the balanced motion vector is used as the motion information of the motion target.

[0061] S204: Superimpose the motion information to the first position where the moving target is located in the first video frame to obtain the observed position of the moving target.

[0062] In this embodiment, the first video frame is a forward video frame of the current video frame, that is, the first video frame acquired before the current video frame in the time dimension.

[0063] The motion vector obtained in step S201 is a motion vector of the current video frame relative to the first video frame.

[0064] Furthermore, if the first video frame is a forward video frame adjacent to the current video frame, the multiple motion vectors obtained in step S201 may be multiple motion vectors directly extracted by the encoder. The calculation formula of the observation position may be as follows:

[0065] P t =P t-1 +m t-1,t Formula 1-1

[0066] Among them, P t is the observed position of the moving target in the current video frame; P t-1 is the position of the moving target in the first video frame; m t-1,t is the equalized motion vector of the current video frame relative to the first video frame.

[0067] Alternatively, there may be an interval of video frames between the first video frame and the current video frame, and the motion vector from the first video frame to the current video frame may be obtained in step S201, and steps S202 to S203 may be executed.

[0068] S205: Obtaining position change information of the moving target in the second video frame relative to the first video frame.

[0069] In this embodiment, the second video frame is a forward video frame of the first video frame.

[0070] The second position of the moving target in the second video frame is obtained. The moving speed and moving direction of the moving target from the second position to the first position are calculated. The moving speed and moving direction are used as position change information.

[0071] In this embodiment, considering that the moving direction and speed of an object between consecutive frames are similar, the moving trend (ie, position change information) of consecutive video frames can be used to predict the current position of the moving object in the current video frame.

[0072] S206: Superimpose the position change information on the first video frame to obtain the predicted position of the moving target.

[0073] In this embodiment, the first position is used as the starting position, and the position reached by the moving target at the moving speed and moving direction within the inter-frame duration is predicted as the predicted position. The inter-frame duration is the interval between the first video frame and the current video frame.

[0074] Alternatively, the second video frame may be an adjacent forward video frame of the first video frame, and the first video frame may be an adjacent current video frame of the current video frame, so as to reduce the amount of computation required for current position prediction. The position change information may be a displacement distance of the moving target between the second video frame and the first video frame, such as a Euclidean distance. The displacement distance is superimposed on the first video frame to obtain the predicted position of the moving target.

[0075] The calculation formula for the predicted position can be as follows:

[0076] P pr t =P t-1 +P pr t-2，t-1 Formula 1-2

[0077] Among them, P pr t is the predicted position of the moving target in the current video frame; P t-1 is the position of the moving target in the first video frame; pr t-2，t-1 It is the position change information of the moving object in the second video frame relative to the first video frame.

[0078] S207: Fuse the observed position and the predicted position to obtain a corrected predicted position.

[0079] In this embodiment, it is considered that whether it is an observed position obtained by reading based on a hardware device such as an encoder, or a predicted position obtained by combining motion trend prediction, the obtained position information may still have errors compared to the actual position of the moving target. Therefore, in this embodiment, the observed position and the predicted position are processed twice to obtain a corrected predicted position that is as close to the actual position of the moving target as possible.

[0080] Specifically, the area between the observed position and the predicted position may be obtained as the new information area, a weight is assigned to the new information area for weighting, and one of the observed position and the predicted position is superimposed with the weighted new information area to obtain a modified predicted position.

[0081] The value range of the weight can be [0, 1].

[0082] The specific calculation formula for superimposing the selected predicted position with the weighted information area to obtain the corrected predicted position can be as follows:

[0083] P est t =P pr t +K t (P t -P pr t) Formula 1-3

[0084] Among them, P est t is the corrected predicted position of the moving target in the current video frame; pr t is the predicted position of the moving target in the current video frame; K t is the weight, the value range is [0,1]; P t is the observed position of the moving target in the current video frame.

[0085] Optionally, the weight may be a fixed value set by a preset setting. Alternatively, the weight may also correspond to the new information area. Alternatively, the weight may also be the Kalman gain of the Kalman filter, and the value of the Kalman gain is determined according to the relative sizes of the state covariance matrix, the measurement matrix, and the measurement noise covariance matrix of the prior estimate. Specifically, the larger the value of the Kalman gain, the greater the influence of the measurement value on the state estimation, and the more the system trusts the measurement value; and the smaller the value of the Kalman gain, the more the system trusts the state of the prior estimate and the less consideration of the measurement value.

[0086] Among them, Kalman filters mostly use minimizing mean square error as the optimization function to measure noise variables and system states, hoping to make a good fit to the input signal. In addition to being able to achieve state estimation, Kalman filters can also reduce noise, which helps to filter out the noise of motion vectors. Kalman filters continuously update the filter output through predicted values ​​and actual values, and the filter output changes as the observation is continuously iterated. Noise is also an uncertain interference factor, which will be controlled and reduced to a certain extent in the feedback process. The Kalman filter takes the statistical information of noise as known information into consideration in the design of the filter output.

[0087] In an alternative embodiment, weights may be assigned to the observed position and the predicted position respectively, and the two may be weightedly fused to obtain a modified predicted position, which is not limited here.

[0088] S208: Obtain the current position of the moving target in the current video frame based on the corrected predicted position.

[0089] In this embodiment, the corrected predicted position may be used as the current position of the moving target. Alternatively, the corrected predicted position may be further processed based on the time error, space error, precision error, etc. to obtain the current position.

[0090] S209: Calculate the inter-frame displacement of the moving target from the position in the adjacent forward key frame to the current position.

[0091] In this embodiment, as described in the above embodiments, the inter-frame position of the moving target from the adjacent forward key frame to the current position can be calculated based on the current position of the moving target.

[0092] For example, the inter-frame displacement can be represented by the Euclidean distance, which is the "ordinary" (i.e. straight-line) distance between two points in the Euclidean space. Using this distance, the Euclidean space becomes a metric space. The specific calculation formula for the inter-frame displacement can be as follows:

[0093] δ t =ED((P r t , P l t ), (P r t-1 , P l t-1 )) Formula 1-4

[0094] Among them, δ t is the inter-frame displacement; ED(x, y) represents the Euclidean distance between x and y; P r t is the coordinate of the upper left corner of the moving target in the current video frame; l t is the coordinate of the lower right corner of the moving target in the current video frame; P r t-1 is the coordinate of the upper left corner of the moving object in the first video frame; l t-1 is the lower right corner coordinate of the moving object in the first video frame.

[0095] S210: Determine whether the inter-frame displacement meets a preset condition.

[0096] In this embodiment, if the inter-frame displacement meets the preset condition, step S211 is executed. If the inter-frame displacement does not meet the preset condition, the process for the current video frame ends. When the next video frame is acquired, step S201 or step S101 is repeatedly executed.

[0097] Optionally, the preset screening condition includes: the inter-frame displacement is greater than the screening threshold. For example, the screening threshold may be 9, 10, 11, etc., so as to reduce the risk that the screening threshold is too large and too few key frames are screened, affecting the accuracy of the preset recognition processing, and can also reduce the risk that the screening threshold is too small and too many key frames are screened, affecting the timeliness of the preset recognition processing.

[0098] Furthermore, the screening threshold is adjustable. In this way, the number of the screened key frames can be adjusted according to the requirements of both accuracy and time efficiency by adjusting the screening threshold to meet the requirements of accuracy and timeliness.

[0099] S211: Determine whether the current video frame is a key frame that can be used for preset recognition processing.

[0100] In this embodiment, in response to the inter-frame displacement satisfying the preset screening condition, the current video frame is determined to be a key frame that can be used for the preset recognition process.

[0101] It can be seen that in this embodiment, the negative impact of presetting the recognition processing of the video frame frame by frame can be alleviated, which is conducive to alleviating the recognition burden of the preset recognition model to reduce the computational cost. As such, the video processing method in this embodiment can make the video recognition method run locally on the mobile platform. At the same time, when the key frame is screened, the inherent motion information in the video is used to select the single frame image for subsequent recognition from the continuous video frame quickly and completely. In addition, the method for performing secondary processing on the observation position and the predicted position in this embodiment is also conducive to improving the reliability of the current position obtained. In this embodiment, when the first key frame frame is screened out, the above-mentioned video processing method will be used to continue to capture the movement of the object, but the accuracy of the capturer based on the motion vector will decrease as the number of frames increases. In order to reduce the risk of large deviations in the recognition process, the key frame is screened to reduce the correction by using the preset recognition model frame by frame to maintain the accuracy and real-time requirements, that is, the appropriate frame will be continuously screened out before the new object appears.

[0102] It is easy to understand that the video stream methods in the related art mainly include background subtraction method and frame difference method.

[0103] Background subtraction is a method for taking differences in pixel values. The pixel difference between the background and the target is calculated on each frame of the image, and compared with the threshold to determine whether the point is the current background pixel. The algorithm of this method is simple, but when the background changes rapidly and the information is complicated, there is a risk that the background update will become a great obstacle and difficult to distinguish.

[0104] The inter-frame difference method calculates the pixel difference between consecutive frames to obtain the changed pixel points and thus obtain the target information for recognition. The inter-frame difference method is similar to the background subtraction method, but the background subtraction method models the constantly updated background, while the inter-frame method models the previous frame, so the background does not need to be considered. However, the disadvantage of the inter-frame difference method is that it is not suitable for objects that change too quickly, and it cannot fully recognize objects when the changing target is quickly subtracted.

[0105] However, the video processing method in this application can overcome the defects of the above-mentioned background subtraction method, frame difference method and other related technologies. The recognition of moving targets and the screening of key frames in this application mainly rely on motion information, rather than identifying moving targets frame by frame. After the key frames are screened out, the key frames are subjected to preset recognition processing. That is, compared with the related technology, the video processing method of this application is equivalent to pre-processing the video code stream in advance.

[0106] See also Figure 4 , Figure 4 It is a flowchart of an embodiment of the video recognition method of the present application.

[0107] S301: Obtain key frames using a video processing method.

[0108] In this embodiment, the video processing method may be as in the above embodiment. For specific limitations on the video processing device, reference may be made to the above limitations on the video processing method, which will not be described in detail here.

[0109] S302: Input the key frame into a preset recognition model for preset recognition processing.

[0110] It can be seen that, compared with pre-setting the recognition processing of the video frame by frame, this embodiment is conducive to reducing the delay of target recognition, thereby improving the timeliness of target recognition, and can also reduce the occupation of storage resources and computing resources. As such, the video recognition method in this embodiment can be applied to mobile platforms such as vehicles, smart phones, computers, and even monitoring equipment itself, which is conducive to reducing the risk of user privacy information being exposed during uploading to the server / cloud or returning the recognition results, and is conducive to improving the security of the video recognition method. In addition, since the number of video frames for preset recognition processing is reduced, it is conducive to balancing the resource allocation of the mobile platform where the video processing device is located, thereby ensuring performance.

[0111] It should be understood that although Figure 1-Figure 2 , Figure 4 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1-Figure 2 , Figure 4 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0112] See also Figure 5 , Figure 5 It is a structural diagram of an embodiment of a video processing device of the present application.

[0113] In this embodiment, the video processing device includes a capturing unit 41 and a screening unit 42 .

[0114] The capturing unit 41 is used to obtain the current position of the moving object in the current video frame.

[0115] The screening unit 42 is connected to the capture unit 41 and is used to calculate the inter-frame displacement of the moving target from the position in the forward key frame to the current position; in response to the inter-frame displacement satisfying the preset screening condition, the current video frame is determined to be a key frame that can be used for preset recognition processing.

[0116] For the specific definition of the video processing device, please refer to the definition of the video processing method above, which will not be repeated here. Each module in the above video processing device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0117] See also Figure 6 , Figure 6 It is a structural schematic diagram of a vehicle embodiment of the present application.

[0118] In this embodiment, the vehicle includes a video processing device 40 , an encoder 50 , and a preset recognition model 60 .

[0119] The encoder 50 is used to extract motion information when compressing the video code stream.

[0120] The video processing device 40 is as described in the above embodiment. The video processing device 40 is connected to the encoder 50 to capture the moving target.

[0121] The preset recognition model 60 is connected to the video processing device 40 and is used to receive key frames and perform preset recognition processing on the key frames.

[0122] It can be seen that the number of key frames processed by the video processing device 40 is less than the total number of video frames in the video code stream, which is beneficial to reducing the resource requirements of the preset recognition model 60 when performing the preset recognition processing. Therefore, the preset recognition model 60 can be set locally in the vehicle, which is beneficial to improving the security of user information.

[0123] Those skilled in the art will understand that Figure 6The structure shown in the figure is only a block diagram of a part of the structure related to the present application scheme, and does not constitute a limitation on the vehicle to which the present application scheme is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0124] In one embodiment, the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0125] S101: Obtain the current position of the moving target in the current video frame.

[0126] S102: Calculate the inter-frame displacement of the moving target from the position in the previous key frame to the current position.

[0127] S103: In response to the inter-frame displacement satisfying a preset screening condition, determining that the current video frame is a key frame that can be used for a preset recognition process.

[0128] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0129] S201: Obtain the motion vector of the region of interest in the current video frame.

[0130] In this embodiment, the region where the moving object is located is divided into a plurality of macroblocks, and the plurality of macroblocks are used as the region of interest, wherein the number of macroblocks is adjustable.

[0131] S202: Divide the motion vectors into a plurality of vector groups based on similarities of motion trajectories of the motion vectors.

[0132] In this embodiment, motion vectors of multiple macroblocks may be used as motion information.

[0133] S203: Select a vector group including the most motion vectors, calculate the average value of the motion vectors in the selected vector group, and obtain an equalized motion vector as motion information.

[0134] S204: Superimpose the motion information to the first position where the moving target is located in the first video frame to obtain the observed position of the moving target.

[0135] In this embodiment, the first video frame is a forward video frame of the current video frame.

[0136] S205: Obtaining position change information of the moving target in the second video frame relative to the first video frame.

[0137] In this embodiment, the second video frame is a forward video frame of the first video frame.

[0138] The second position of the moving target in the second video frame is obtained. The moving speed and moving direction of the moving target from the second position to the first position are calculated. The moving speed and moving direction are used as position change information.

[0139] S206: Superimpose the position change information on the first video frame to obtain the predicted position of the moving target.

[0140] In this embodiment, the first position is used as the starting position, and the position reached by the moving target at the moving speed and moving direction within the inter-frame duration is predicted as the predicted position. The inter-frame duration is the interval between the first video frame and the current video frame.

[0141] S207: Fuse the observed position and the predicted position to obtain a corrected predicted position.

[0142] In this embodiment, the area between the observed position and the predicted position is obtained as the new information area.

[0143] Assign weights to the new information area for weighting, take one of the observed position and the predicted position and superimpose it with the weighted new information area to obtain the corrected predicted position.

[0144] S208: Obtain the current position of the moving target in the current video frame based on the corrected predicted position.

[0145] S209: Calculate the inter-frame displacement of the moving target from the position in the adjacent forward key frame to the current position.

[0146] S210: Determine whether the inter-frame displacement meets a preset condition.

[0147] In this embodiment, if the inter-frame displacement meets the preset condition, step S211 is executed. If the inter-frame displacement does not meet the preset condition, the process for the current video frame ends.

[0148] Optionally, the preset screening condition includes: the inter-frame displacement is greater than a screening threshold, wherein the screening threshold is adjustable.

[0149] S211: Determine whether the current video frame is a key frame that can be used for preset recognition processing.

[0150] In this embodiment, in response to the inter-frame displacement satisfying the preset screening condition, the current video frame is determined to be a key frame that can be used for the preset recognition process.

[0151] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0152] S301: Obtain key frames using a video processing method.

[0153] In this embodiment, the video processing method may be as in the above embodiment. For specific limitations on the video processing device, reference may be made to the above limitations on the video processing method, which will not be described in detail here.

[0154] S302: Input the key frame into a preset recognition model for preset recognition processing.

[0155] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0156] S101: Obtain the current position of the moving target in the current video frame.

[0157] S102: Calculate the inter-frame displacement of the moving target from the position in the previous key frame to the current position.

[0158] S103: In response to the inter-frame displacement satisfying a preset screening condition, determining that the current video frame is a key frame that can be used for a preset recognition process.

[0159] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0160] S201: Obtain the motion vector of the region of interest in the current video frame.

[0161] In this embodiment, the region where the moving object is located is divided into a plurality of macroblocks, and the plurality of macroblocks are used as the region of interest, wherein the number of macroblocks is adjustable.

[0162] S202: Divide the motion vectors into a plurality of vector groups based on similarities of motion trajectories of the motion vectors.

[0163] In this embodiment, motion vectors of multiple macroblocks may be used as motion information.

[0164] S203: Select a vector group including the most motion vectors, calculate the average value of the motion vectors in the selected vector group, and obtain an equalized motion vector as motion information.

[0165] S204: Superimpose the motion information to the first position where the moving target is located in the first video frame to obtain the observed position of the moving target.

[0166] In this embodiment, the first video frame is a forward video frame of the current video frame.

[0167] S205: Obtaining position change information of the moving target in the second video frame relative to the first video frame.

[0168] In this embodiment, the second video frame is a forward video frame of the first video frame.

[0169] The second position of the moving target in the second video frame is obtained. The moving speed and moving direction of the moving target from the second position to the first position are calculated. The moving speed and moving direction are used as position change information.

[0170] S206: Superimpose the position change information on the first video frame to obtain the predicted position of the moving target.

[0171] In this embodiment, the first position is used as the starting position, and the position reached by the moving target at the moving speed and moving direction within the inter-frame duration is predicted as the predicted position. The inter-frame duration is the interval between the first video frame and the current video frame.

[0172] S207: Fuse the observed position and the predicted position to obtain a corrected predicted position.

[0173] In this embodiment, the area between the observed position and the predicted position is obtained as the new information area.

[0174] Assign weights to the new information area for weighting, take one of the observed position and the predicted position and superimpose it with the weighted new information area to obtain the corrected predicted position.

[0175] S208: Obtain the current position of the moving target in the current video frame based on the corrected predicted position.

[0176] S209: Calculate the inter-frame displacement of the moving target from the position in the adjacent forward key frame to the current position.

[0177] S210: Determine whether the inter-frame displacement meets a preset condition.

[0178] In this embodiment, if the inter-frame displacement meets the preset condition, step S211 is executed. If the inter-frame displacement does not meet the preset condition, the process for the current video frame ends.

[0179] Optionally, the preset screening condition includes: the inter-frame displacement is greater than a screening threshold, wherein the screening threshold is adjustable.

[0180] S211: Determine whether the current video frame is a key frame that can be used for preset recognition processing.

[0181] In this embodiment, in response to the inter-frame displacement satisfying the preset screening condition, the current video frame is determined to be a key frame that can be used for the preset recognition process.

[0182] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0183] S301: Obtain key frames using a video processing method.

[0184] In this embodiment, the video processing method may be as in the above embodiment. For specific limitations on the video processing device, reference may be made to the above limitations on the video processing method, which will not be described in detail here.

[0185] S302: Input the key frame into a preset recognition model for preset recognition processing.

[0186] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0187] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0188] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A video processing method, characterized in that: include: Get the current position of the moving target in the current video frame; Calculating the inter-frame displacement of the moving target from the position in the forward key frame to the current position; In response to the inter-frame displacement satisfying a preset screening condition, determining that the current video frame is a key frame that can be used for a preset recognition process; The obtaining of the current position of the moving target in the current video frame comprises: fusing the observed position and the predicted position to obtain a corrected predicted position, and obtaining the current position based on the corrected predicted position; The predicted position is predicted based on the movement trend at a historical moment, and the observed position is the position read in the current video frame.

2. The video processing method according to claim 1, characterized in that: The preset screening condition includes: the inter-frame displacement is greater than a screening threshold; wherein the screening threshold is adjustable.

3. The video processing method according to claim 1, characterized in that: The obtaining of the current position of the moving target in the current video frame comprises: Obtaining motion information of the region of interest in the current video frame relative to the first video frame, and superimposing the motion information to a first position where the moving target is located in the first video frame to obtain an observed position of the moving target; wherein the first video frame is a forward video frame of the current video frame; Obtain position change information of the moving target in the second video frame relative to the first video frame, and superimpose the position change information on the first video frame to obtain the predicted position of the moving target; wherein the second video frame is a forward video frame of the first video frame.

4. The video processing method according to claim 3, characterized in that: The step of obtaining motion information of the region of interest in the current video frame relative to the first video frame includes: Dividing the region where the moving target is located into a plurality of macroblocks, wherein the plurality of macroblocks serve as the region of interest; wherein the number of the macroblocks is adjustable; The motion vectors of the plurality of macroblocks are used as the motion information.

5. The video processing method according to claim 4, characterized in that: The using the motion vectors of the plurality of macroblocks as the motion information comprises: Based on the similarity of the motion trajectories of the motion vectors, the motion vectors are divided into a plurality of vector groups; A vector group including the most motion vectors is selected, and an average value of the motion vectors in the selected vector group is calculated to obtain an equilibrium motion vector as the motion information.

6. The video processing method according to claim 3, characterized in that: The fusing the observed position and the predicted position to obtain a corrected predicted position comprises: Acquire an area between the observed position and the predicted position as an information area; A weight is assigned to the new information region for weighting, and one of the observed position and the predicted position is superimposed with the weighted new information region to obtain the modified predicted position.

7. The video processing method according to claim 3, characterized in that: The step of obtaining the position change information of the moving target in the second video frame relative to the first video frame, and superimposing the position change information on the first video frame to obtain the predicted position of the moving target includes: Acquire a second position of the moving object in the second video frame; Calculating a moving speed and a moving direction of the moving target from the second position to the first position; Taking the first position as the starting position, predict the position reached by the moving target at the moving speed and in the moving direction within the inter-frame duration as the predicted position; wherein the inter-frame duration is the interval between the first video frame and the current video frame.

8. A video recognition method, characterized in that: The video recognition method comprises: Obtaining a key frame using the video processing method according to any one of claims 1 to 7; The key frame is input into a preset recognition model for preset recognition processing.

9. A video processing device, characterized in that: The video processing device comprises: A capture unit, used to obtain the current position of the moving target in the current video frame; A screening unit is connected to the capture unit and is used to calculate the inter-frame displacement of the moving target from the position in the forward key frame to the current position; in response to the inter-frame displacement satisfying a preset screening condition, the current video frame is determined to be a key frame that can be used for preset recognition processing; the current position of the moving target in the current video frame is obtained by fusing the observed position and the predicted position to obtain a corrected predicted position, and the current position is obtained based on the corrected predicted position; the predicted position is predicted based on the motion trend at a historical moment, and the observed position is the position read in the current video frame.

10. A vehicle, characterized in that: The vehicle comprises: Encoder, used to extract motion information when compressing the video stream; The video processing device according to claim 9, connected to the encoder, for capturing a moving target; The preset recognition model is connected to the video processing device and is used to receive key frames and perform preset recognition processing on the key frames.

Images