Dynamic engineering vehicle detection method based on optical flow and rotation positioning

Abstract

The invention discloses a dynamic engineering vehicle detection method based on optical flow and rotation positioning, and relates to the field of robot navigation, and the method comprises the following parts: an optical flow calculation module used for capturing the motion information of a target and realizing the detection and tracking of a dynamic target; the rotation positioning module is used for extracting the rotation angle and direction change information of the target and analyzing the dynamic behavior of the target in combination with the calculation result of the optical flow calculation module; the state judgment module is used for integrating the information, judging the final state of the target and distinguishing the static state, the driving state or the working state of the target; according to the invention, motion characteristics of a target are analyzed and captured through an optical flow field, and the working posture of the engineering vehicle is calculated by combining a three-point rotation positioning technology, so that the working state of the construction vehicle can be accurately judged, and meanwhile, the efficient detection performance of a static scene is kept; and dual optimization of engineering vehicle detection and state identification in dynamic and static scenes is realized.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision, image processing, autonomous driving and robot navigation, and in particular to a dynamic engineering vehicle detection method based on optical flow and rotational positioning. Background Art

[0002] UAV inspection systems based on target detection algorithms have been widely used in the field of engineering inspections. However, most current inspection systems can only achieve static detection of targets, which may lead to the following problems:

[0003] The existing system can only realize static identification of engineering vehicles and cannot accurately determine whether the target is in operation. The main technical bottlenecks are:

[0004] 1. Insufficient dynamic feature extraction: The system lacks the ability to extract dynamic features of the engineering vehicle's motion state (such as speed and direction) and operating posture (such as the robot arm's extension angle and bucket position);

[0005] 2. Insufficient utilization of timing information: An effective timing analysis model has not been established, making it impossible to judge operating behavior based on target state changes between consecutive frames;

[0006] 3. Lack of operating status identification standards: The system has not established a complete operating status identification indicator system and lacks the ability to model and identify the characteristics of typical operating modes (such as excavation, loading, transportation, etc.), resulting in the inability to accurately distinguish between the working state and the static state of the target.

[0007] Based on the above reasons, the present invention designs a dynamic engineering vehicle detection method based on optical flow and rotation positioning. Summary of the Invention

[0008] The purpose of this invention is to solve the problem in the existing technology that most inspection systems can only realize the static detection function of the target and cannot effectively identify the actual operating status of engineering vehicles in dynamic scenes. To address this technical bottleneck, this patent proposes an improved solution that integrates optical flow detection technology and three-point rotation positioning technology.

[0009] A dynamic engineering vehicle detection method based on optical flow and rotation positioning includes the following parts:

[0010] The optical flow calculation module is used to capture the target's motion information, including the optical flow vector and the intensity of the optical flow change, to detect and track dynamic targets;

[0011] A rotation positioning module is used to extract the target's rotation angle and direction change information, and analyze the target's dynamic behavior in combination with the calculation results of the optical flow calculation module;

[0012] The state discrimination module is used to integrate the optical flow calculation results and information such as the change in rotation direction to judge the final state of the target and distinguish the target's stationary, moving or operating state;

[0013] It also includes an image preprocessing module and a target detection module, which specifically includes the following steps:

[0014] S1: Process the input image through the image preprocessing module.

[0015] S2: Detect the target in the image through the target detection module. If the target is detected, proceed to the next step;

[0016] S3: Determine the target's optical flow field and rotation state information through the optical flow calculation module and the three-point rotation positioning module respectively:

[0017] S4: The state is judged by the state judgment module and expressed as a numerical value. If the numerical value exceeds the threshold, the target is in the working state. If it is lower than the threshold, the target is in the driving or stationary state.

[0018] In the above-mentioned dynamic engineering vehicle detection method based on optical flow and rotation positioning, the calculation process of the optical flow calculation module includes the following steps:

[0019] S1: Grayscale processing: The input video frame is grayscaled through the image preprocessing unit and Gaussian filtering is used to eliminate image noise;

[0020] Grayscale processing converts the RGB image into a single-channel grayscale image. The calculation formula is as follows:

[0021] I(x,y)=0.299·R(x,y)+0.587·G(x,y)+0.114·B(x,y)

[0022] Gaussian filtering is used to smooth the image, and the calculation formula is:

[0023]

[0024] Among them, G(i,j) is the Gaussian kernel function;

[0025] S2: Calculate the optical flow field: Based on the Lucas-Kanade optical flow algorithm, the motion vectors of pixels between adjacent frames are calculated to construct the optical flow field distribution map of the target area. The core formula of the Lucas-Kanade algorithm is:

[0026]

[0027] Among them, I x and I y are the gradients of the image in the x and y directions, I tis the temporal gradient, u and v are optical flow vectors;

[0028] S3: Pyramid layered calculation, which introduces a pyramid layered calculation method to improve the accuracy and robustness of optical flow calculation through multi-scale analysis. Pyramid layered calculation generates multi-resolution images through downsampling. The calculation formula is:

[0029]

[0030] Among them, k represents the pyramid level, and the optical flow field is optimized layer by layer from coarse to fine;

[0031] S4: GPU parallel acceleration: To optimize computing efficiency, the module uses GPU parallel acceleration technology to decompose the optical flow calculation task into multiple parallel threads, ensuring that optical flow field data can be stably output at a frame rate of 30fps in real-time drone inspection scenarios;

[0032] S5: Smoothing: The calculation results are smoothed by the optical flow field post-processing unit to eliminate outlier interference and provide reliable dynamic feature input for subsequent job status recognition. Smoothing uses median filtering, and the calculation formula is:

[0033] V sot (x,y)=median(V(x+i,y+j))

[0034] Among them, V is the optical flow vector field, and median is the median filtering operation.

[0035] In the above-mentioned dynamic engineering vehicle detection method based on optical flow and rotational positioning, the calculation of the rotational positioning module includes the following steps:

[0036] S1: Key point selection: In the target box output by the target detection algorithm, three representative key points (such as the upper left corner, upper right corner and center point of the target box) are selected as the reference points for rotation state analysis;

[0037] S2: Key point coordinate extraction: Get the coordinate positions of the three key points in the current frame and the previous frame, which are recorded as (x1, y1), (x2, y2), (x3, y3) (current frame) and (x′1, y′1), (x′2, y′2), (x′3, y′3) (previous frame).

[0038] S3: Rotation Angle Calculation: Calculate the target's rotation angle based on the coordinate changes of the three key points. This is achieved through the following steps:

[0039] Calculate the center point coordinates of the target frame (x c ,y c ):

[0040]

[0041] Calculate the vector of each keypoint relative to the center point:

[0042] v i =(x i -x c ,y i -y c ),i=1,2,3

[0043] Calculate the angle between the corresponding vectors in the previous frame and the current frame to get the rotation angle θ:

[0044]

[0045] Among them, v′ i is the vector in the previous frame;

[0046] S4: Rotation direction calculation: By calculating the rotation direction (clockwise or counterclockwise) of the three key points, the rotation state of the target is further described. The rotation direction can be determined by the vector cross product:

[0047] Direction = sign((v i ×v′ i )·z)

[0048] Where z is the unit vector perpendicular to the image plane, sign is the sign function;

[0049] S5: Result output: The rotation angle θ and the rotation direction information are output as input data of the state determination module for subsequent operation state determination.

[0050] In the above-mentioned dynamic engineering vehicle detection method based on optical flow and rotation positioning, the working process of the discrimination module includes the following steps:

[0051] S1: Input data, receiving the optical flow calculation results (including optical flow vector and optical flow change intensity) from the optical flow calculation module and the rotation angle θ and rotation direction information from the rotation positioning module;

[0052] S2: Optical flow feature extraction:

[0053] The optical flow vector in the target area is extracted, and the average intensity and directional consistency of the optical flow change are calculated.

[0054] S3: Rotation feature extraction:

[0055] Extract the rotation angle θ and rotation direction information to analyze the rotation state of the target;

[0056] S4: Status judgment is performed through status judgment rules;

[0057] S5: Output the final state judgment result of the target, which can be stationary, moving or working.

[0058] In the above-mentioned dynamic engineering vehicle detection method based on optical flow and rotation positioning, in step S2 of the operation of the discrimination module, the average intensity and directional consistency of the optical flow change are calculated;

[0059] If the intensity of optical flow variation is low and the directional consistency is high, the target is stationary;

[0060] If the optical flow variation intensity is high and the direction consistency is high, the target is in a straight-line driving state;

[0061] If the optical flow variation intensity is high and the directional consistency is low, the target is in an operating state.

[0062] In the above-mentioned dynamic engineering vehicle detection method based on optical flow and rotation positioning, in step S3 of the operation of the discrimination module, if the rotation angle θ is close to 0 and the rotation direction does not change significantly, the target is in a stationary or straight-line driving state;

[0063] If the rotation angle θ changes significantly and the rotation direction changes regularly, the target is in an operating state.

[0064] In the above-mentioned dynamic engineering vehicle detection method based on optical flow and rotation positioning, the state discrimination rules include:

[0065] The target is stationary: the optical flow change intensity is low, the rotation angle θ is close to 0, and the rotation direction does not change significantly;

[0066] The target is in a straight-line driving state: the optical flow change intensity is high, the direction consistency is high, the rotation angle θ is close to 0, and the rotation direction does not change significantly.

[0067] The beneficial effects of the present invention are: capturing the motion characteristics of the target through optical flow field analysis, and combining it with three-point rotation positioning technology to accurately solve the operating posture of the construction vehicle, it can not only accurately judge the operating status of the construction vehicle (such as excavation, loading, transportation, etc.), but also maintain efficient detection performance for static scenes, realizing the dual optimization of construction vehicle detection and status recognition in dynamic and static scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a flow chart of a dynamic engineering vehicle detection method based on optical flow and rotation positioning. DETAILED DESCRIPTION

[0069] Reference Figure 1 ,A dynamic engineering vehicle detection method based on optical flow and rotation positioning,

[0070] The dotted box shows the improvement of this patent over the prior art. It includes three modules, namely the optical flow calculation module, the three-point rotation positioning module, and the state discrimination module. Through the optical flow calculation module, the system can capture the target's motion information (such as the optical flow vector and the intensity of the optical flow change) in real time, thereby realizing the detection and tracking of dynamic targets. The optical flow calculation can reflect the target's motion direction and speed changes, providing a basis for dynamic target recognition. Through the rotation positioning module, the system can extract the target's rotation angle and direction change information, and combine the optical flow calculation results to further analyze the target's dynamic behavior. For example, when the rotation angle changes significantly and the consistency of the optical flow direction is low, it can be determined that the target is in an operating state. The optical flow calculation module can effectively separate the motion information of the target and the background, and the rotation positioning module further extracts the target's rotation features. The combination of the two can improve the robustness of the system in complex scenes.

[0071] Optical flow calculation module

[0072] (1) Module function

[0073] This module is responsible for calculating the optical flow field of the target object and is the core component of the system for dynamic feature extraction.

[0074] 2) Working principle and process

[0075] (1) Grayscale processing: The input video frame is grayscaled through the image preprocessing unit, and Gaussian filtering is used to eliminate image noise.

[0076] Grayscale processing converts the RGB image into a single-channel grayscale image. The calculation formula is as follows:

[0077] I(x,y)=0.299·R(x,y)+0.587·G(x,y)+0.114·B(x,y)

[0078] Gaussian filtering is used to smooth the image, and the calculation formula is:

[0079]

[0080] Among them, G(i,j) is the Gaussian kernel function.

[0081] (2) Calculate the optical flow field: Based on the Lucas-Kanade optical flow algorithm, the optical flow field distribution map of the target area is constructed by calculating the motion vectors of pixels between adjacent frames. The core formula of the Lucas-Kanade algorithm is:

[0082]

[0083] Among them, I x and I y are the gradients of the image in the x and y directions, It is the temporal gradient, u and v are the optical flow vectors.

[0084] (3) Pyramid layered calculation: The pyramid layered calculation method is introduced to improve the accuracy and robustness of optical flow calculation through multi-scale analysis. Pyramid layered calculation generates multi-resolution images through downsampling. The calculation formula is:

[0085]

[0086] Among them, k represents the pyramid level, and the optical flow field is optimized layer by layer from coarse to fine.

[0087] (4) GPU parallel acceleration: To optimize computing efficiency, the module uses GPU parallel acceleration technology to decompose the optical flow calculation task into multiple parallel threads, ensuring that the optical flow field data can be stably output at a frame rate of 30fps in the real-time drone inspection scenario.

[0088] (5) Smoothing: The calculation results are smoothed by the optical flow field post-processing unit to eliminate the interference of abnormal values ​​and provide reliable dynamic feature input for subsequent job status recognition. The smoothing process uses median filtering, and the calculation formula is:

[0089] V sot (x,y)=median(V(x+i,y+j))

[0090] Among them, V is the optical flow vector field, and median is the median filtering operation.

[0091] 2) Rotation positioning module

[0092] (1) Module function

[0093] The rotation localization module is used to further analyze the target's rotation state after the object detection algorithm detects the object. By extracting the rotation information of three key points within the target frame, the target's rotation angle and direction change are calculated, providing accurate rotation feature input for the subsequent state discrimination module.

[0094] (2) Working principle and process

[0095] 1) Key point selection: In the target box output by the target detection algorithm, three representative key points (such as the upper left corner, upper right corner and center point of the target box) are selected as the reference points for rotation state analysis.

[0096] 2) Key point coordinate extraction: Get the coordinate positions of the three key points in the current frame and the previous frame, which are recorded as (x1, y1), (x2, y2), (x3, y3) (current frame) and (x′1, y′1), (x′2, y′2), (x′3, y′3) (previous frame).

[0097] 3) Rotation Angle Calculation: Based on the coordinate changes of the three key points, the target's rotation angle is calculated. This is achieved through the following steps:

[0098] Calculate the center point coordinates of the target frame (x c ,y c ):

[0099]

[0100] Calculate the vector of each keypoint relative to the center point:

[0101] v i =(x i -x c ,y i -y c ),i=1,2,3

[0102] Calculate the angle between the corresponding vectors in the previous frame and the current frame to get the rotation angle θ:

[0103]

[0104] Among them, v′ i is the vector in the previous frame.

[0105] 4) Rotation direction calculation: By calculating the rotation direction (clockwise or counterclockwise) of the three key points, the rotation state of the target is further described. The rotation direction can be determined by the vector cross product:

[0106] Direction = sign((v i ×v′ i )·z)

[0107] Where z is the unit vector perpendicular to the image plane and sign is the sign function.

[0108] 5) Result output: The rotation angle θ and rotation direction information are output as input data of the state judgment module for subsequent operation state judgment.

[0109] State discrimination module

[0110] (1) Module function

[0111] The state discrimination module combines optical flow calculation results with information such as rotational direction changes to determine the target's final state. By analyzing the changing characteristics of optical flow and the patterns of rotational direction changes, it can distinguish whether the target is stationary, moving, or operating, providing a basis for system decision-making.

[0112] (2) Working principle and process

[0113] 1) Input data: Receive the optical flow calculation results (including optical flow vector and optical flow change intensity) from the optical flow calculation module and the rotation angle θ and rotation direction information from the rotation positioning module.

[0114] 2) Optical flow feature extraction:

[0115] The optical flow vector in the target area is extracted, and the average intensity and directional consistency of the optical flow change are calculated.

[0116] If the intensity of optical flow variation is low and the direction consistency is high, the target may be stationary;

[0117] If the optical flow variation intensity is high and the direction consistency is high, the target may be traveling in a straight line;

[0118] If the optical flow variation intensity is high and the directional consistency is low, the target may be in an operational state (such as rotating or turning).

[0119] 3) Rotation feature extraction:

[0120] Extract the rotation angle θ and rotation direction information to analyze the rotation state of the target.

[0121] If the rotation angle θ is close to 0 and the rotation direction does not change significantly, the target may be stationary or moving in a straight line;

[0122] If the rotation angle θ changes significantly and the rotation direction changes regularly, the target may be in an operating state.

[0123] 4) Status determination rules:

[0124] Static state: The intensity of optical flow change is low, the rotation angle θ is close to 0, and the rotation direction does not change significantly.

[0125] Straight-line driving state: The optical flow change intensity is high, the direction consistency is high, the rotation angle θ is close to 0, and the rotation direction does not change significantly.

[0126] Working status: The intensity of optical flow change is high, the direction consistency is low, the rotation angle θ changes significantly, and the rotation direction changes regularly.

[0127] Result output: Based on the above analysis, the final state judgment result of the target (stationary, moving or working) is output.

[0128] It is understood from common technical knowledge that the present invention may be implemented by other embodiments that do not depart from its spirit or essential features. Therefore, the embodiments disclosed above are, in all respects, merely illustrative and not exclusive. All modifications within the scope of the present invention or equivalent to the scope of the present invention are intended to be encompassed by the present invention.

Claims

1. A dynamic engineering vehicle detection method based on optical flow and rotation positioning, characterized in that: Includes the following sections: The optical flow calculation module is used to capture the target's motion information, including the optical flow vector and the intensity of the optical flow change, to detect and track dynamic targets; A rotation positioning module is used to extract the target's rotation angle and direction change information, and analyze the target's dynamic behavior in combination with the calculation results of the optical flow calculation module; The state discrimination module is used to integrate the optical flow calculation results and information such as the change in rotation direction to judge the final state of the target and distinguish the target's stationary, moving or operating state; It also includes an image preprocessing module and a target detection module, which specifically includes the following steps: S1: Process the input image through the image preprocessing module; S2: Detect the target in the image through the target detection module. If the target is detected, proceed to the next step; S3: Determine the target's optical flow field and rotation state information through the optical flow calculation module and the three-point rotation positioning module respectively: S4: The state is judged by the state judgment module and expressed as a numerical value. If the numerical value exceeds the threshold, the target is in the working state. If it is lower than the threshold, the target is in the driving or stationary state.

2. The dynamic engineering vehicle detection method based on optical flow and rotation positioning according to claim 1 is characterized in that: The calculation process of the optical flow calculation module includes the following steps: S1: Grayscale processing: The input video frame is grayscaled through the image preprocessing unit and Gaussian filtering is used to eliminate image noise; Grayscale processing converts the RGB image into a single-channel grayscale image. The calculation formula is as follows: I(x,y)=0.299·R(x,y)+0.587·G(x,y)+0.114·B(x,y) Gaussian filtering is used to smooth the image, and the calculation formula is: Among them, G(i,j) is the Gaussian kernel function; S2: Calculate the optical flow field: Based on the Lucas-Kanade optical flow algorithm, the motion vectors of pixels between adjacent frames are calculated to construct the optical flow field distribution map of the target area. The core formula of the Lucas-Kanade algorithm is: Among them, I x and I y are the gradients of the image in the x and y directions, I t is the temporal gradient, u and v are the optical flow vectors; S3: Pyramid layered calculation, which introduces a pyramid layered calculation method to improve the accuracy and robustness of optical flow calculation through multi-scale analysis. Pyramid layered calculation generates multi-resolution images through downsampling. The calculation formula is: Among them, k represents the pyramid level, and the optical flow field is optimized layer by layer from coarse to fine; S4: GPU parallel acceleration: To optimize computing efficiency, the module uses GPU parallel acceleration technology to decompose the optical flow calculation task into multiple parallel threads, ensuring that optical flow field data can be stably output at a frame rate of 30fps in real-time drone inspection scenarios; S5: Smoothing: The calculation results are smoothed by the optical flow field post-processing unit to eliminate outlier interference and provide reliable dynamic feature input for subsequent job status recognition. The smoothing process uses median filtering. The calculation formula is: V sot (x,y)=median(V(x+i,y+j)) Among them, V is the optical flow vector field, and median is the median filtering operation.

3. The dynamic engineering vehicle detection method based on optical flow and rotation positioning according to claim 1 is characterized in that: The calculation of the rotation positioning module includes the following steps: S1: Key point selection: In the target box output by the target detection algorithm, three representative key points (such as the upper left corner, upper right corner and center point of the target box) are selected as the reference points for rotation state analysis; S2: Key point coordinate extraction: Get the coordinate positions of the three key points in the current frame and the previous frame, which are recorded as (x1, y1), (x2, y2), (x3, y3) (current frame) and (x1′, y1′), (x2′, y2′), (x3′, y3′) (previous frame). S3: Rotation Angle Calculation: Based on the coordinate changes of the three key points, the target's rotation angle is calculated through the following steps: Calculate the center point coordinates of the target frame (x c ,y c ): Calculate the vector of each keypoint relative to the center point: v i =(x i -x c ,y i -y c ),i=1,2,3 Calculate the angle between the corresponding vectors in the previous frame and the current frame to get the rotation angle θ: Among them, v′ i is the vector in the previous frame; S4: Rotation direction calculation: By calculating the rotation direction (clockwise or counterclockwise) of the three key points, the rotation state of the target is further described. The rotation direction can be determined by the vector cross product: Direction = sign((v i ×v i ′)·z) Where z is the unit vector perpendicular to the image plane, sign is the sign function; S5: Result output: The rotation angle θ and the rotation direction information are output as input data of the state determination module for subsequent operation state determination.

4. The dynamic engineering vehicle detection method based on optical flow and rotation positioning according to claim 1 is characterized in that: The working process of the discrimination module includes the following steps: S1: Input data, receiving the optical flow calculation results (including optical flow vector and optical flow change intensity) from the optical flow calculation module and the rotation angle θ and rotation direction information from the rotation positioning module; S2: Optical flow feature extraction: Extract the optical flow vector within the target area and calculate the average intensity and directional consistency of the optical flow change; S3: Rotation feature extraction: Extract the rotation angle θ and rotation direction information to analyze the rotation state of the target; S4: Status judgment is performed through status judgment rules; S5: Output the final state judgment result of the target, which can be stationary, moving or working.

5. The dynamic engineering vehicle detection method based on optical flow and rotation positioning according to claim 1 is characterized in that: In step S2 of the discrimination module, the average intensity and directional consistency of the optical flow change are calculated; If the intensity of optical flow variation is low and the directional consistency is high, the target is stationary; If the optical flow variation intensity is high and the direction consistency is high, the target is in a straight-line driving state; If the optical flow variation intensity is high and the directional consistency is low, the target is in an operating state.

6. The dynamic engineering vehicle detection method based on optical flow and rotation positioning according to claim 1 is characterized in that: In step S3 of the discrimination module, if the rotation angle θ is close to 0 and the rotation direction does not change significantly, the target is in a stationary or straight-line driving state; If the rotation angle θ changes significantly and the rotation direction changes regularly, the target is in an operating state.

7. The dynamic engineering vehicle detection method based on optical flow and rotation positioning according to claim 2 is characterized in that: The state determination rules include: The target is stationary: the optical flow change intensity is low, the rotation angle θ is close to 0, and the rotation direction does not change significantly; The target is in a straight-line state: the optical flow change intensity is high, the direction consistency is high, the rotation angle θ is close to 0, and the rotation direction does not change significantly; The target is in an operating state: the optical flow variation intensity is high, the directional consistency is low, the rotation angle θ changes significantly, and the rotation direction changes regularly.

Images