An anti-shake method and device based on vehicle-mounted road surface disease identification system

Abstract

This application provides an anti-shake method and apparatus based on an on-board road surface defect recognition system, solving the technical problem that existing technologies cannot eliminate shake with high quality. It includes: acquiring preprocessed road image data; converting the coordinates of the preprocessed road image data in a local coordinate system to coordinates in a camera coordinate system; extracting each feature point in the camera coordinate system; tracking each feature point using optical flow estimation to obtain the motion trajectory of the feature point; performing smoothing processing to obtain the de-shake road image data. This application can be widely applied in the field of anti-shake technology for on-board platforms.

Description

Technical Field

[0001] The present application belongs to the technical field of anti-shake of vehicle-mounted platforms, and more specifically, to an anti-shake method and device based on a vehicle-mounted road surface disease identification system. Background Art

[0002] After a long period of rain and snow impact and vehicle load, the road surface will have different degrees of road damage. The vehicle-mounted road damage detection system can collect road surface information and analyze the degree and type of road damage, and this application is designed for this detection system.

[0003] An existing pod drone video stabilization method and system (application number: 202410870416.2) uses multiple sensors to detect jitter and compensate for the jitter data, but its attention mechanism can only focus on key feature points. In the case of large noise, it cannot correctly identify feature points, and even eliminates some effective feature points, resulting in image quality being affected and unable to eliminate jitter with high quality. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an anti-jitter method and device based on a vehicle-mounted road surface defect identification system to solve the technical problem that jitter cannot be eliminated with high quality in the prior art.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application provides an anti-shake method based on a vehicle-mounted road surface disease identification system, comprising the following steps: obtaining pre-processed road image data;

[0006] Convert the coordinates of the preprocessed road image data in the local coordinate system into the coordinates of the camera coordinate system;

[0007] Extract each feature point in the camera coordinate system;

[0008] Use the optical flow estimation method to track each feature point, obtain the motion trajectory of the feature point, perform smoothing, and obtain the road image data after de-jittering;

[0009] The process of tracking each feature point using the optical flow estimation method includes: reverse tracking and screening the feature points to obtain positive and negative samples, performing self-training to obtain the predicted target of the sample, outputting the training data for the predicted target, classifying the training data according to the confidence level, removing the low-confidence targets, and using the nearest neighbor algorithm to find the nearest neighbor samples of the remaining targets as positive samples for self-training. Finally, with the same feature point as the center, the normalization coefficient in the surrounding area is calculated as compensation data to compensate for the reversely tracked feature points.

[0010] Preferably, screening means taking the average value of the difference as a threshold, dividing the feature points obtained by reverse tracing, and obtaining positive and negative samples;

[0011] The difference refers to the difference between the position of all feature points obtained by reverse tracking and the actual position of the feature points.

[0012] Preferably, the smoothing formula is as follows:

[0013] ;

[0014] In the formula, , is the coordinate of the feature point after smoothing correction, is the total number of selected frames. For the The tracking path direction of the frame feature points along the x-axis, For the The tracking path direction of the frame feature points along the y-axis.

[0015] Preferably, the process of obtaining the pre-processed road image data includes: obtaining the road image data, establishing a road surface flatness model to obtain a power spectrum density curve, performing anti-shake correction, and obtaining the pre-processed road image data;

[0016] Anti-shake correction includes optical image stabilization and mechanical image stabilization;

[0017] Mechanical anti-shake refers to the use of the buffer provided by the suspension spring to prevent shaking;

[0018] Optical image stabilization refers to rotating the camera to the corresponding angles based on the power spectral density curve and the acquired roll and yaw angles to prevent shaking.

[0019] Preferably, the formulas for the roll angle and the yaw angle are obtained:

[0020] ;

[0021] In the formula, and is the coordinate of the original coordinate system, and is the coordinate of the coordinate system after the original coordinate system is tilted 45°, is the distance from the camera sensor to the road surface, is the angle between the vibration plane and the driving plane of the vehicle-mounted device, is the roll angle, is the yaw angle.

[0022] Preferably, the formula for the power spectral density curve is obtained:

[0023] ;

[0024] ;

[0025] ;

[0026] In the formula, is the time frequency, is the number of waves contained in a unit distance, is the vehicle speed, is the spatial road surface power spectrum, is the reference spatial frequency The road power spectrum under is the reference spatial frequency, is the frequency coefficient, is the power spectral density.

[0027] Preferably, the transformation includes a translation transformation and a rotation transformation.

[0028] Preferably, the conversion formula is as follows:

[0029] ;

[0030] ;

[0031] ;

[0032] ;

[0033] In the formula, , , is the coordinate after translation transformation; , , is the original coordinate, that is, the camera coordinate system coordinate, , , is the translation amount, , , is the coordinate after the translation transformation and then the rotation transformation, Point is Plane and A common point on a plane, for and The angle of is the angle of rotation around the z-axis, is the rotation matrix, is the length of the line segment connecting the origin of the coordinate system and point P.

[0034] Preferably, each feature point in the camera coordinate system is extracted using the Hessian matrix and maximum suppression.

[0035] A second aspect of the embodiment of the present application provides an anti-shake device based on a vehicle-mounted road surface disease identification system, comprising: an encoder, a control terminal, a posture sensor arranged on a vehicle roof, a differential positioning device, and a camera stabilization device; the camera stabilization devices are arranged in pairs and face opposite directions, and are distributed at both ends of the vehicle roof; the camera stabilization device is sequentially provided with a suspension spring, a pan / tilt bracket, and a camera sensor from bottom to top; the posture sensor, the camera sensor, and the encoder are all electrically connected to the control terminal;

[0036] The control terminal controls the camera stabilization device, the attitude sensor, the encoder and the differential positioning device to obtain the pre-processed road image data;

[0037] The control terminal converts the coordinates of the pre-processed road image data in the local coordinate system into the coordinates in the camera coordinate system; extracts each feature point in the camera coordinate system; tracks each feature point using the optical flow estimation method to obtain the motion trajectory of the feature point, and performs smoothing processing to obtain the road image data after de-jittering.

[0038] The beneficial effects of the present application are as follows: the present application proposes an anti-shake method based on a vehicle-mounted road surface disease recognition system. First, the improved optical flow estimation method is used to track the feature points in reverse, which effectively realizes the tracking of unknown targets and meets the needs of the vehicle-mounted device when facing complex road scenes; secondly, by introducing a self-training semi-supervised learning form to train the samples, the system is allowed to continuously update parameters during operation, has better generalization ability, can make more accurate predictions, better remove noise, and improves the accuracy and reliability of recognition; finally, separate compensation data is set to compensate for the reversely tracked feature points, which reduces the system coupling, reduces the missed operations caused by the control terminal switching tasks, and significantly improves the safety and stability of the system.

[0039] The present application also proposes an anti-shake device based on a vehicle-mounted road surface defect recognition system. The device controls the camera stabilization device to change the angle through a posture sensor and can adapt to various road surface defects. The above method is used in the control terminal to eliminate the shaking and blur of the picture to achieve the effect of eliminating the shaking.

[0040] In summary, the anti-shake method and device based on the vehicle-mounted road disease identification system proposed in this application can eliminate the shaking blur of the picture to make the picture clearer. At the same time, reverse tracking can track unknown targets, meeting the needs of the vehicle-mounted device for complex road scenes and eliminating the shaking of the picture with high quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0042] Figure 1 A schematic diagram of the overall process of an anti-shake method based on a vehicle-mounted road surface disease identification system provided in one embodiment of the present application;

[0043] Figure 2 A schematic diagram of two coordinate systems established by a gimbal bracket in an optical anti-shake method provided in an embodiment of the present application;

[0044] Figure 3 A schematic diagram of an improved optical flow estimation method provided by an embodiment of the present application;

[0045] Figure 4 A three-dimensional diagram of an anti-shake device based on a vehicle-mounted road surface disease identification system provided in one embodiment of the present application;

[0046] Figure 5 A workflow diagram for the fusion of differential positioning device and encoder data provided in one embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] A first aspect of an embodiment of the present application provides an anti-shake method based on a vehicle-mounted road surface defect identification system.

[0049] See also Figure 1 , an anti-shake method based on a vehicle-mounted road surface disease identification system provided in an embodiment of the present application, comprising:

[0050] S1: Obtain pre-processed road image data.

[0051] The camera sensor is used to obtain road image data and transmit it to the control terminal. Then, a mathematical model of road surface smoothness is established to obtain a power spectrum density curve, which is used to guide the camera stabilization device to perform anti-shake correction to obtain pre-processed road image data.

[0052] Specifically, after the control terminal receives the road image transmitted by the camera sensor, it first establishes a mathematical model of road surface smoothness. In the actual operation of the vehicle-mounted device, since the undulating curves of the road surface are random, the road surface smoothness can be regarded as a random signal. By introducing white noise interference, the spatial road surface power spectrum is obtained. , the power spectrum can be used to represent the road roughness coefficient, and its formula is shown as follows:

[0053] ;

[0054] In the formula, is the spatial road surface power spectrum, is the reference spatial frequency The road power spectrum under is the number of waves contained in a unit distance, is the reference spatial frequency, Take the constant 2 as the frequency coefficient, is the vehicle speed, is the temporal frequency, and q is the height relative to the road surface reference.

[0055] Among them, the time frequency It is motivated by the road and can be expressed as:

[0056] ;

[0057] In the formula, is the time frequency, is the number of waves contained in a unit distance, For vehicle speed.

[0058] Spatial road power spectral density and power spectral density The relationship between them is shown as follows:

[0059] ;

[0060] In the formula, is the power spectral density, is the spatial road surface power spectrum density, is the reference spatial frequency The road power spectrum under is the vehicle speed, is the time frequency, is the number of waves contained in a unit distance, is the reference spatial frequency, and q is the height relative to the road surface reference.

[0061] When the vehicle-mounted device is traveling at different speeds, the spatial frequency and the temporal frequency will also change. The vehicle is driven and the power spectral density curve in the spatial domain is obtained using the mathematical model of road surface smoothness established above, which is used to guide the movement of the suspension spring and the gimbal bracket in the camera stabilization device.

[0062] Specifically, several sets of PID parameters (proportional, integral, and differential) are used in conjunction with attitude sensors to control the motor in the camera stabilization device. Different softness and hardness (damping size) are used to control the movement of the gimbal bracket according to different road conditions to adapt to various undulating road surfaces, thereby controlling the movement of the suspension spring and gimbal bracket in the camera stabilization device.

[0063] Furthermore, anti-shake correction refers to mechanical anti-shake and optical anti-shake, integrating the advantages of both to adapt to the acquisition of different scenes. Anti-shake correction mainly uses the suspension spring and gimbal bracket in the camera stabilization device for anti-shake. Among them, the suspension spring plays the role of mechanical anti-shake. When the road surface is uneven and the vehicle-mounted device encounters slight bumps, it can effectively attenuate vibrations, absorb energy, and redistribute the load of the camera stabilization device, so that it is buffered and some shakes are eliminated.

[0064] The gimbal supports optical image stabilization, which reduces the deviation in all directions, obtains the acceleration, angular velocity and other data of the vehicle-mounted device in real time, and calculates the roll angle through the following formula: and yaw angle Attitude angles in two directions:

[0065] ;

[0066] In the formula, and is the coordinate of the original coordinate system, and Tilt to original coordinate system °Coordinates of the latter coordinate system.

[0067] The camera sensor is tilted downward at 45° relative to the vehicle body of the vehicle system, and the following is obtained:

[0068] ;

[0069] In the formula, and is the coordinate of the original coordinate system, and is the coordinate of the coordinate system after the original coordinate system is tilted 45°, is the distance from the camera sensor to the road surface, It is the angle between the plane where the shaking occurs and the plane on which the vehicle-mounted device travels.

[0070] Combining the above formulas, the roll angle can be obtained and yaw angle , respectively drive the motors on the gimbal bracket to rotate corresponding angles to achieve an anti-shake effect and obtain the pre-processed road image data.

[0071] S2: Convert the coordinates of the preprocessed road image data in the local coordinate system into coordinates in the camera coordinate system.

[0072] After receiving the pre-processed road image data, the control terminal converts the coordinates of the pre-processed road image data in the local coordinate system into coordinates in the camera coordinate system, wherein the conversion refers to rotation transformation and translation transformation.

[0073] Please refer to Figure 2 Specifically, since the orientation of the attitude sensor is based on the body of the vehicle-mounted device, it is necessary to convert the coordinates of the pre-processed road image data in the local coordinate system into the coordinates of the camera coordinate system. At this time, the relationship between the two coordinate systems can be regarded as a translation transformation and a rotation transformation, where the translation transformation formula can be expressed as:

[0074] ;

[0075] In the formula, , , is the coordinate after translation transformation; , , is the original coordinate, , , is the translation amount.

[0076] In an optional embodiment, taking rotation around the z-axis as an example, the rotation transformation can be expressed as: ;

[0077] In the formula, , is the coordinate after the translation transformation and then the rotation transformation, Point is Plane and A common point on a plane, for and The angle of is the angle of rotation around the z-axis, is the length of the line segment connecting the origin of the coordinate system and point P.

[0078] Bundle , Bring in , At the same time, the z'-axis coordinate of point P in the coordinate system before the rotation transformation is the same as that in the coordinate system after the rotation transformation. The axis coordinates are equal, so we have:

[0079] ;

[0080] The matrix multiplication is shown in the following formula:

[0081] ;

[0082] In the formula, is the rotation matrix, , , The coordinates of the translation transformation followed by the rotation transformation.

[0083] S3: Extract each feature point in the camera coordinate system.

[0084] The Hessian matrix and maximum suppression are used to extract stable and trackable feature points.

[0085] Specifically, in the camera coordinate system, the preprocessed road image data is segmented by frame, and the feature points of each frame are extracted. These feature points are scale-invariant and rotation-invariant, and are very stable to the ambient lighting, so they can be used to track motion trajectories. In order to obtain feature points with scale invariance, it is necessary to calculate the Hessian matrix of each pixel in the entire image and suppress its maximum value. This application uses a second-order Gaussian function to construct the Hessian matrix, and the Hessian matrix can be expressed as:

[0086] ;

[0087] In the formula, is the two-dimensional Hessian matrix, , , , are the second-order partial derivatives of Gaussian functions, is the x-coordinate value of the independent variable in the camera coordinate system, is the standard deviation of the Gaussian function.

[0088] Furthermore, each feature point obtained is screened, the following determinant of maximum value suppression is calculated, and it is determined whether it is an extreme point:

[0089] ;

[0090] ;

[0091] In the formula, is the Hessian matrix determinant, is the x-axis coordinate of the camera coordinate system, is the y-axis coordinate of the camera coordinate system, is a two-dimensional Gaussian function, is the standard deviation of the Gaussian function.

[0092] In this way, all extreme points in a frame are obtained, among which the extreme points are used as candidate feature points.

[0093] When the above algorithm is used in the control terminal, the second-order Gaussian function in the Hessian matrix is ​​split into the product of two first-order Gaussian functions, which reduces the use of multiplication in the control terminal and ensures the real-time performance of the overall system. The specific description is:

[0094] ;

[0095] In the formula, is the Gaussian function of the independent variables x and y, is the Gaussian function of the independent variable x, is the Gaussian function of the independent variable y, is the x-axis coordinate of the camera coordinate system, is the y-axis coordinate of the camera coordinate system, is a two-dimensional Gaussian function, is the standard deviation of the Gaussian function.

[0096] Assuming that the length of the convolution kernel convolved with each frame is r, this application can reduce r each time. 2 -2r multiplications are used. At the same time, under the condition that the feature point extraction effect remains almost unchanged, this application uses 5 video frame images as a group, and a total of 4 groups of images are used to constrain the scale invariance, which occupies less DSP resources of the control terminal. When describing the obtained feature points, this application avoids using Euclidean distance and instead uses binary Hamming distance, which is more convenient for hardware to perform logical calculations including the most commonly used XOR.

[0097] The present application imposes constraints on rotation invariance. To determine the rotation invariance of feature points extracted from the captured image, it is necessary to determine a main direction for the feature points, so as to ensure that the overall image is not affected when the camera sensor's viewing angle is rotated and shaken.

[0098] The present application calculates the gradient feature description in the nearest neighborhood of each feature point, and the division of the gradient direction takes into account the data storage of the control terminal, and selects 32-bit data storage, that is, one division every 11.25°. The main direction of the feature point is determined by summing the statistical amplitude according to the gradient direction and the weight of the feature point, where the weight is The calculation formula is:

[0099] ;

[0100] In the formula, is the weight, is the x-coordinate value of the independent variable in the camera coordinate system, is the y-axis coordinate of the camera coordinate system.

[0101] S4: Use the optical flow estimation method to track each feature point and obtain the motion trajectory of the feature point.

[0102] In the continuous multi-frame images, the improved optical flow estimation method is used to track the obtained feature points to obtain the motion trajectory of the feature points.

[0103] Specifically, to track the trajectory of the feature points obtained in the above steps, it is necessary to predict and estimate each feature point to achieve the purpose of de-jittering. The gray value of each feature point is regarded as brightness, and the movement of each object in the picture is abstracted into the movement of light points. At this time, the improved optical flow estimation method is used to connect the positions of the light points of each feature point in each continuous frame to estimate the motion trajectory. First, there are:

[0104] ;

[0105] In the formula, for The feature points at the reference frame The brightness at the moment, and it is related to the next frame at the feature point The same feature point at The brightness at the moment remains unchanged, x is the x-axis coordinate of the camera coordinate system, and y is the y-axis coordinate of the camera coordinate system.

[0106] right Taylor expansion is performed, because the interval between the two frames is , so we can get:

[0107] ;

[0108] in, , , can be calculated from the image data, so this method can calculate the movement direction of the light spot by the above formula , that is, the movement direction of the feature points, and obtain the preliminary predicted movement trajectory of the feature points in the first frame.

[0109] Based on the traditional optical flow estimation algorithm, this application introduces a tracking algorithm with a training mechanism (TBT, Training-Based-Tracking) to achieve continuous tracking of each feature point. Since the target composed of feature points in the picture collected by the camera sensor is to be detected, that is, the feature point object is unknown before running the traditional optical flow estimation algorithm, so the TBT algorithm is introduced to improve the continuous tracking accuracy of the feature points, and finally obtain the motion trajectory of each feature point.

[0110] For details, please refer to Figure 3 , track the position of the feature point in the second frame in reverse order to obtain the position of the feature point in the first frame, calculate the distance difference between the position of the feature point in the first frame and the actual position of the feature point in the first frame, calculate the average value of all the differences in this frame as the threshold, if the difference of the feature point is greater than the threshold, the feature point is considered a negative sample, otherwise it is a positive sample.

[0111] A semi-supervised FlowNet algorithm is used to self-train the obtained positive and negative samples, obtain the predicted target of the sample, and output the training data for the predicted target.

[0112] The training data for the predicted target is classified, and the confidence threshold is set to 0.65. Samples with low confidence, that is, unreliable targets, are eliminated, and the nearest neighbor algorithm is used to find the nearest neighbor samples of the remaining targets, which are used as positive samples to continue self-training.

[0113] Finally, in the first frame and the second frame, the normalized correlation coefficient (NCC) is calculated for two square areas with a side length of 6 pixels centered on the same feature point, and this is used as compensation data to compensate for the reversely tracked feature points.

[0114] Traditional optical flow methods are generally used to determine targets composed of feature points, which does not meet the needs of vehicle-mounted devices for complex road scenes. The improved optical flow estimation algorithm of this application can track unknown targets; it is worth noting that the traditional optical flow estimation algorithm does not have separate compensation data when performing reverse tracking, while this algorithm has dedicated compensation data to compensate for the feature points of reverse tracking, which reduces the system coupling and is less likely to cause missed operations due to switching tasks of the control terminal; semi-supervision is introduced on the basis of the traditional optical flow estimation method, and parameters are continuously updated during system operation, which has better generalization ability.

[0115] S5: Smoothing the image according to the motion trajectory of the feature points to obtain a de-jittered image.

[0116] The bumps encountered by the vehicle body cause noise and jitter in the motion trajectory of the feature points extracted from the image collected by the camera sensor. Therefore, the motion trajectory of the obtained feature points needs to be smoothed, specifically:

[0117] ;

[0118] In the formula, , is the coordinate of the feature point after smoothing correction, It is the abbreviation of the first letter of fixed. is the total number of selected frames. , For the The tracking path direction of the frame feature points, is the x-component of the tracking direction, The y component of the tracking direction.

[0119] See also Figure 4 The second aspect of the embodiment of the present application provides an anti-shake device based on a vehicle-mounted road surface disease identification system, including: a control terminal 3 and an encoder 4, a posture sensor 11, a differential positioning device 12 and a camera stabilization device 2 are provided on the roof 1.

[0120] The camera stabilizing devices 2 are arranged in pairs and face opposite directions, and are distributed at both ends of the roof 1 .

[0121] The camera stabilizing device 2 is provided with a suspension spring 21, a gimbal bracket 22 and a camera sensor 23 in sequence from bottom to top.

[0122] The posture sensor 11 , the camera sensor 23 , and the encoder 4 are all electrically connected to the control terminal 3 .

[0123] The camera sensor 23 is used to obtain pre-processed road image data.

[0124] The attitude sensor 11 is used to obtain real-time attitude data and guide the gimbal bracket 22 to adjust its position.

[0125] The suspension spring 21 is used to cooperate with the pan / tilt bracket 22 to reduce vibration and adapt to different degrees of bumps.

[0126] The differential positioning device 12 is RTK, and is used in conjunction with the encoder 4 to obtain the longitude and latitude position information of the bumpy road surface and the distance traveled by the vehicle.

[0127] For details, please refer to Figure 5, the image collected by the camera sensor 23 of the vehicle-mounted device is severely shaken due to the bumping of the vehicle body. This situation is highly correlated with encountering serious road damage, and the location information of the vehicle-mounted system at this time needs to be recorded. The present application realizes positioning by combining the differential positioning device 12 and the encoder 4. The differential positioning device 12 provides a large range of geographical locations, while the encoder 4 can accurately record the movement state of the vehicle during bumps and the distance the wheels rotate. Through data fusion, the position changes before and after the bumps are analyzed, and finally the specific location information of the vehicle can be accurately identified.

[0128] The specific process is to first synchronize the position data obtained by the differential positioning device 12 and the encoder 4 to ensure that the data at the same time point is accurate; then use the attitude sensor 11 to monitor the vehicle body bumps. When no bumps are detected, the differential positioning device 12 and the encoder 4 continue to obtain position data records; when bumps are detected, the current position and encoder 4 data are recorded; then the data of the two sensors are fused to obtain accurate position information. The use of data fusion eliminates the applicable limitations of each of the two and improves the overall positioning accuracy.

[0129] The control terminal 3 controls the camera stabilization device 2, the attitude sensor 11, the encoder 4 and the differential positioning device 12 to obtain the pre-processed road picture data.

[0130] The control terminal 3 converts the coordinates of the preprocessed road image data in the local coordinate system into coordinates in the camera coordinate system, extracts each feature point in the camera coordinate system, tracks each feature point using the optical flow estimation method, obtains the motion trajectory of the feature point, and performs smoothing processing to obtain the road image data after de-jittering.

[0131] Specifically, the control terminal 3 has relatively rich CPU (central processing unit) resources and DSP (digital signal processing) resources, and the power meets the limitations of the vehicle-mounted device. The posture sensor 11 of the roof 1 is connected to the control terminal 3 through the CAN bus. The rotation angle of the motor of the pan-tilt bracket 22 is calculated by the control terminal 3, and each motor is driven by an SPWM pulse (sine wave pulse width modulation pulse); the acquired pre-processed road picture data is connected to the control terminal 3 through the MIPI interface (Mobile Industry Processor Interface), and the control terminal 3 performs coordinate conversion, extracts tracking feature points, and smoothes the pre-processed road picture data, and records the position information recorded by the differential positioning device 12 and the encoder 4.

[0132] The present application proposes an anti-shake method based on a vehicle-mounted road surface disease recognition system. First, an improved optical flow estimation method is used to track feature points in reverse, effectively realizing the tracking of unknown targets and meeting the needs of vehicle-mounted equipment when facing complex road scenes; secondly, samples are trained by introducing a self-training semi-supervised learning form, allowing the system to continuously update parameters during operation, with better generalization ability, capable of making more accurate predictions, better removing noise, and improving the accuracy and reliability of recognition; finally, separate compensation data is set to compensate for the reversely tracked feature points, thereby reducing the system coupling, reducing the missed operations caused by the control terminal switching tasks, and significantly improving the safety and stability of the system.

[0133] The present application also proposes an anti-shake device based on a vehicle-mounted road surface defect recognition system. The device controls the camera stabilization device to change the angle through a posture sensor and can adapt to various road surface defects. The above method is used in the control terminal to eliminate the shaking and blur of the picture to achieve the effect of eliminating the shaking.

[0134] In summary, the anti-shake method and device based on the vehicle-mounted road disease identification system proposed in this application can eliminate the shaking blur of the picture to make the picture clearer. At the same time, reverse tracking can track unknown targets, meeting the needs of the vehicle-mounted device for complex road scenes and eliminating the shaking of the picture with high quality.

[0135] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0136] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An anti-shake method based on a vehicle-mounted road surface disease identification system, characterized in that: The following steps are involved: Acquire pre-processed road image data; Convert the coordinates of the preprocessed road image data in the local coordinate system into the coordinates of the camera coordinate system; Extract each feature point in the camera coordinate system; Use the optical flow estimation method to track each feature point, obtain the motion trajectory of the feature point, perform smoothing, and obtain the road image data after de-jittering; The process of tracking each feature point using the optical flow estimation method includes: reverse tracking and screening the feature point to obtain positive and negative samples, self-training to obtain the predicted target of the sample, outputting training data for the predicted target, classifying the training data according to the confidence level, removing the low-confidence target, using the nearest neighbor algorithm to find the nearest neighbor sample of the remaining target as the positive sample for self-training, and finally, in two adjacent frames, taking the same feature point as the center, calculating the normalized correlation coefficient of two surrounding areas as compensation data to compensate the reversely tracked feature point; The screening refers to taking the average value of the difference as a threshold, dividing the feature points obtained by the reverse tracking, and obtaining positive and negative samples; The difference refers to the difference between the positions of all feature points obtained by reverse tracking and the actual positions of the feature points.

2. The anti-shake method based on the vehicle-mounted road surface disease identification system according to claim 1 is characterized in that: The process of obtaining the pre-processed road image data includes: obtaining the road image data, establishing a road surface flatness model to obtain a power spectrum density curve, performing anti-shake correction, and obtaining the pre-processed road image data; The anti-shake correction includes optical anti-shake and mechanical anti-shake; The mechanical anti-shake refers to the use of the buffer provided by the suspension spring to prevent shaking; The optical image stabilization refers to rotating the corresponding angle according to the power spectrum density curve and the obtained roll angle and yaw angle to prevent shaking; The formula for obtaining the power spectral density curve is: ； ； ； In the formula, is the time frequency, is the number of waves contained in a unit distance, is the vehicle speed, is the spatial road power spectrum, is the reference spatial frequency The road power spectrum under is the reference spatial frequency, is the frequency coefficient, is the power spectral density.

3. The anti-shake method based on the vehicle-mounted road surface disease identification system according to claim 2 is characterized in that: The formula for obtaining the roll angle and yaw angle is: ； In the formula, and is the coordinate of the original coordinate system, and is the coordinate of the coordinate system after the original coordinate system is tilted 45°, is the distance from the camera sensor to the road surface, is the angle between the vibration plane and the driving plane of the vehicle-mounted device, is the roll angle, is the yaw angle.

4. The anti-shake method based on the vehicle-mounted road surface disease identification system according to claim 1 is characterized in that: The transformation includes a translation transformation and a rotation transformation.

5. The anti-shake method based on the vehicle-mounted road surface disease identification system according to claim 1 or 4, characterized in that: The formula for the conversion is as follows: ； ； ； ； In the formula, , , is the coordinate after translation transformation; , , is the original coordinate, , , is the translation amount, , , is the coordinate after the translation transformation and then the rotation transformation, Point is Plane and A common point on a plane, for and The angle of is the angle of rotation around the z-axis, is the rotation matrix, is the length of the line segment connecting the origin of the coordinate system and point P.

6. The anti-shake method based on the vehicle-mounted road surface disease identification system according to claim 1 is characterized in that: Each feature point in the camera coordinate system is extracted using the Hessian matrix and maximum suppression.

7. An anti-shake device based on a vehicle-mounted road surface disease identification system, characterized in that: The anti-shake method based on the vehicle-mounted road surface disease identification system described in claim 1 above is used, comprising: an encoder, a control terminal, a posture sensor arranged on the roof, a differential positioning device and a camera stabilizing device; the camera stabilizing devices are arranged in pairs and face opposite directions, distributed at both ends of the roof, the camera stabilizing device is provided with a suspension spring, a pan / tilt bracket and a camera sensor in sequence from bottom to top, and the posture sensor, the camera sensor and the encoder are all electrically connected to the control terminal; The control terminal controls the camera stabilization device and the attitude sensor to obtain pre-processed road image data; Positioning is achieved through encoders and differential positioning devices; The control terminal converts the coordinates of the preprocessed road image data in the local coordinate system into the coordinates in the camera coordinate system, extracts each feature point in the camera coordinate system, tracks each feature point using the optical flow estimation method, obtains the motion trajectory of the feature point, and performs smoothing processing to obtain the road image data after de-jittering.

Images