Vehicle-mounted camera jitter recognition method, device and computer-readable storage medium
Through semantic segmentation model and motion matrix calculation, combined with vehicle motion parameters, the difficulty of identifying the jitter recognition of vehicle cameras under weak texture background is solved, and accurate jitter recognition is achieved under dynamic background of vehicle cameras.
Patent Information
- Application Number
- CN202111424049.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-26
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-11-26
AI Technical Summary
The existing camera jitter recognition method is not effective in vehicle camera applications, especially in weak texture backgrounds, which cannot accurately detect jitter of vehicle cameras, resulting in failure of recognition.
The pre-trained semantic segmentation model is used to detect the video images captured by the on-board camera in real time. The top-view projection and ground motion normalization generate a top-view projection map, and the motion matrix is calculated by combining the motor vehicle wheel speed pulse and steering angle to perform motion compensation and frames. The difference information entropy is calculated, and the jitter state is judged using the adaptive threshold.
Accurate jitter recognition under the dynamic multi-image background of the car camera is achieved, overcomes the influence of weak texture background, and improves the accuracy and robustness of jitter detection of the car camera.
Smart Images

Figure CN114245102B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the technical field of vehicle-mounted image processing, and in particular to a vehicle-mounted camera jitter recognition method, device and computer-readable storage medium. Background Art
[0002] At present, when a motor vehicle encounters bumps during driving, it often causes the on-board camera to shake. When the jitter amplitude of the on-board camera image is too large, the image captured by it will have poor robustness and reduced accuracy, and may even cause errors in the navigation system and positioning system that rely on the video images captured by the on-board camera. Therefore, it is particularly important to detect and identify the jitter of the on-board camera during the driving of the motor vehicle.
[0003] An existing camera shake recognition method mainly calculates the motion vector of the video scene captured by the camera based on a feature point matching model or an optical flow algorithm model, and further determines the camera shake amplitude by evaluating the amplitude and phase of the motion vector.
[0004] However, the above-mentioned camera shake recognition method is usually only applicable to application scenarios where the camera is relatively fixed (for example: security monitoring scenarios). When the above-mentioned method is applied to the shake detection of a vehicle-mounted camera, the detection effect of the weak texture background of the image during the detection process is poor, and the motion vector of the motion video scene captured by the vehicle-mounted camera cannot be accurately calculated, which ultimately leads to the failure of the shake recognition of the vehicle-mounted camera. Summary of the invention
[0005] The technical problem to be solved by the embodiments of the present invention is to provide a vehicle-mounted camera shaking recognition method, which can accurately recognize the shaking of the vehicle-mounted camera.
[0006] A further technical problem to be solved by the embodiments of the present invention is to provide a vehicle-mounted camera shake recognition device that can accurately recognize the shake of the vehicle-mounted camera.
[0007] A further technical problem to be solved by an embodiment of the present invention is to provide a computer-readable storage medium for storing a computer program that can accurately identify the shaking of a vehicle-mounted camera.
[0008] In order to solve the above technical problems, the embodiment of the present invention provides the following technical solution: a vehicle camera jitter recognition method, comprising the following steps:
[0009] Acquire real-time image frames from the original video images captured by the vehicle-mounted camera, use a pre-trained semantic segmentation model to detect the ground around the motor vehicle on the image frames, and then perform a bird's-eye view projection on the semantic segmentation results and normalize the scale of ground motion to generate a bird's-eye view projection map;
[0010] The initial value of the motion matrix of the current image frame is calculated based on the wheel speed pulse and the steering angle of the motor vehicle, and then the accurate motion matrix of the current image frame is calculated based on the minimization optimization of the reprojection error of the ground;
[0011] Generate a motion compensated image frame of the current image frame based on the precise motion matrix;
[0012] Calculating frame difference information entropy between the current image frame and the motion compensated image frame; and
[0013] The current shaking state of the vehicle-mounted camera is determined by combining the frame difference information entropy and a preset frame difference information entropy adaptive threshold.
[0014] Furthermore, after performing a bird's-eye view projection on the semantic segmentation results and normalizing the scale of the ground motion to generate a bird's-eye view projection map, a ground region of interest of a predetermined size is captured from the bird's-eye view projection map. The motion compensated image frame is a compensation region corresponding to the ground region of interest of the current image frame, and the frame difference information entropy is the frame difference information entropy between the compensation region of the motion compensated image frame and the ground region of interest of the current image frame.
[0015] Furthermore, the calculating of the frame difference information entropy between the current image frame and the motion compensated image frame specifically includes:
[0016] Calculating the frame difference between the ground region of interest of the current image frame and the compensation region; and
[0017] The frame difference information entropy H1 between the ground region of interest of the current image frame and the compensation region is calculated based on the frame difference.
[0018] Furthermore, the combining the frame difference information entropy and the preset frame difference information entropy adaptive threshold to determine the current shaking state of the vehicle-mounted camera specifically includes:
[0019] Calculate the information entropy H2 of the ground area of interest of the current image frame;
[0020] Calculate the current jitter amplitude of the vehicle camera H0 = |H1-H2| / H2; and
[0021] The jitter amplitude H0 is compared with the frame difference information entropy adaptive threshold, and when the jitter amplitude H0 is greater than the frame difference information entropy adaptive threshold, it is determined that the vehicle-mounted camera currently has jitter.
[0022] Furthermore, the calculation of the initial value of the motion matrix of the current image frame based on the wheel speed pulse of the motor vehicle and the steering angle of the motor vehicle specifically includes:
[0023] Calculating the three-dimensional motion parameters of the vehicle within the frame difference time length based on the vehicle wheel speed pulse and the vehicle steering angle; and
[0024] The initial value of the motion matrix of the current image frame is calculated by combining the three-dimensional motion parameters and the calibration parameters of the vehicle-mounted camera.
[0025] Furthermore, the ground-based reprojection error minimization optimization and calculation of the precise motion matrix of the current image frame specifically include:
[0026] Calculate the projection error of the ground region of interest of the previous frame of the current image frame in the current image frame based on the initial value of the motion matrix;
[0027] Calculate the best inter-frame motion vector of the current image frame based on the projection error and nonlinear optimization principle;
[0028] Predicting the precise motion vector of the current image frame based on the best inter-frame motion vector of the latest predetermined number of image frames in the original video image based on the smoothness assumption of motor vehicle motion and the polynomial fitting principle; and
[0029] An accurate motion matrix of the current image frame is generated based on the accurate motion vector of the current image frame.
[0030] Further, the predicting the accurate motion matrix of the current image frame based on the best inter-frame motion vector of the image frame with the latest predetermined number of frames in the original video image based on the smoothness assumption of motor vehicle motion and the polynomial fitting principle specifically includes: comparing the total number of frames N of the image frames in the original video image with the predetermined number of frames T;
[0031] When the total number of frames N is less than the predetermined number of frames T, the best inter-frame motion vector of each image frame in the original video image is used and the accurate motion vector of the current image frame is predicted based on the smoothness assumption of the motor vehicle motion and the polynomial fitting principle; and
[0032] When the total number of frames N is greater than or equal to the predetermined number of frames T, the best inter-frame motion vector of the latest T frames in the original video image is used and the precise motion vector of the current image frame is predicted based on the smoothness assumption of motor vehicle motion and the polynomial fitting principle.
[0033] Further, the generating of the motion compensated image frame of the current image frame based on the precise motion matrix specifically comprises: calculating the coordinates of compensated pixel points in the compensation area corresponding to the coordinates of pixel points in the ground area of interest of the current image frame based on the precise motion matrix; and
[0034] The compensation area is obtained by calculating according to the coordinates of the compensation pixel points and the pixel size of the ground area of interest of the current image frame.
[0035] On the other hand, in order to further solve the above-mentioned technical problems, an embodiment of the present invention provides the following technical solutions: a vehicle-mounted camera jitter recognition device, which is respectively connected to a vehicle-mounted camera for shooting video images around a motor vehicle and providing original video images, a motor vehicle motion sensor for detecting the motion state of the motor vehicle and providing motor vehicle wheel speed pulses and motor vehicle steering angles, and a jitter processing device for processing jitter when jitter exists in the vehicle-mounted camera. The vehicle-mounted camera jitter recognition device includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the vehicle-mounted camera jitter recognition method as described above is implemented.
[0036] On the other hand, in order to further solve the above-mentioned technical problems, an embodiment of the present invention provides the following technical solution: a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the vehicle-mounted camera jitter recognition method as described above.
[0037] After adopting the above technical scheme, the embodiment of the present invention has at least the following beneficial effects: after extracting image frames frame by frame from the original video images collected and provided by the vehicle-mounted camera, the embodiment of the present invention performs semantic segmentation and overhead projection on the image frames in sequence to finally obtain an overhead projection map, further calculates the initial value of the motion matrix of the current image frame based on the wheel speed pulse of the motor vehicle and the steering angle of the motor vehicle, and calculates the precise motion matrix of the current image frame in combination with the ground-constrained reprojection error minimization optimization, thereby realizing fast and robust vehicle body motion estimation; by calculating the frame difference information entropy between the current image frame and the motion-compensated image frame, and finally combining the frame difference information entropy and the preset frame difference information entropy adaptive threshold to judge the current jitter state of the vehicle-mounted camera, there is no need to calculate the feature corners of the image, thereby overcoming the influence of weak texture without feature-significant corners, and using the adaptive threshold of information entropy to fully adapt to the dynamic and changeable imaging background of the vehicle-mounted camera, thereby accurately identifying the jitter of the vehicle-mounted camera. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 The present invention is a flowchart of an optional embodiment of the vehicle camera jitter recognition method of the present invention.
[0039] Figure 2 This is a practical schematic diagram of the semantic segmentation result of an optional embodiment of the vehicle camera jitter recognition method of the present invention.
[0040] Figure 3 This is a top view projection diagram of an optional embodiment of the vehicle-mounted camera jitter recognition method of the present invention.
[0041] Figure 4This is a specific flow chart of step S4 of an optional embodiment of the vehicle-mounted camera jitter recognition method of the present invention.
[0042] Figure 5 This is a specific flow chart of step S5 of an optional embodiment of the vehicle-mounted camera jitter recognition method of the present invention.
[0043] Figure 6 The present invention is a flow chart showing an optional embodiment of the vehicle camera jitter recognition method for calculating the initial value of the motion matrix of the current image frame.
[0044] Figure 7 The present invention is a flow chart of calculating the precise motion matrix of the current image frame in an optional embodiment of the vehicle camera jitter recognition method.
[0045] Figure 8 This is a specific flow chart of step S25 of an optional embodiment of the vehicle-mounted camera jitter recognition method of the present invention.
[0046] Fig. 9 This is a specific flow chart of step S3 of an optional embodiment of the vehicle-mounted camera jitter recognition method of the present invention.
[0047] Fig.10 The present invention is a schematic diagram of an optional embodiment of the vehicle camera shake recognition device of the present invention.
[0048] Fig.11 This is a functional module diagram of an optional embodiment of the vehicle-mounted camera jitter recognition device of the present invention. DETAILED DESCRIPTION
[0049] The present application is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following illustrative embodiments and descriptions are only used to explain the present invention and are not intended to limit the present invention. In addition, the embodiments and features in the embodiments of the present application may be combined with each other without conflict.
[0050] like Figure 1 As shown, an optional embodiment of the present invention provides a vehicle-mounted camera jitter recognition method, comprising the following steps:
[0051] S1: Obtain real-time image frames from the original video images captured by the vehicle-mounted camera 1, use the pre-trained semantic segmentation model to detect the ground around the vehicle, and then perform semantic segmentation on the image frames (such as Figure 2 The top-down projection is performed by normalizing the scale of ground motion to generate a top-down projection map (as shown in Figure 3 shown);
[0052] S2: Calculate the initial value of the motion matrix of the current image frame based on the wheel speed pulse and the steering angle of the motor vehicle, and then calculate the precise motion matrix of the current image frame based on the minimization optimization of the reprojection error of the ground;
[0053] S3: generating a motion compensated image frame of the current image frame based on the precise motion matrix;
[0054] S4: Calculating the frame difference information entropy between the current image frame and the motion compensated image frame; and
[0055] S5: Determine the current shaking state of the vehicle-mounted camera 1 by combining the frame difference information entropy and a preset frame difference information entropy adaptive threshold.
[0056] The embodiment of the present invention extracts image frames frame by frame from the original video images collected and provided by the vehicle-mounted camera 1, and then performs semantic segmentation and overhead projection on the image frames in sequence to finally obtain an overhead projection image, further calculates the initial value of the motion matrix of the current image frame based on the wheel speed pulse and the steering angle of the vehicle, and calculates the precise motion matrix of the current image frame in combination with the ground-constrained reprojection error minimization optimization, thereby realizing fast and robust vehicle body motion estimation; by calculating the frame difference information entropy between the current image frame and the motion-compensated image frame, and finally combining the frame difference information entropy and a preset frame difference information entropy adaptive threshold to judge the current jitter state of the vehicle-mounted camera, there is no need to calculate the feature corner points of the image, thus overcoming the influence of weak texture without feature-significant corner points, and using the adaptive threshold of information entropy to fully adapt to the dynamic and changeable imaging background of the vehicle-mounted camera 1, the jitter of the vehicle-mounted camera 1 can be accurately identified.
[0057] In an optional embodiment of the present invention, after performing a top-down projection on the semantic segmentation result and normalizing the scale of the ground motion to generate a top-down projection map, a ground region of interest of a predetermined size is also intercepted from the top-down projection map, the motion compensation image frame is a compensation region corresponding to the ground region of interest of the current image frame, and the frame difference information entropy is the frame difference information entropy between the compensation region of the motion compensation image frame and the ground region of interest of the current image frame. In this embodiment, by intercepting a ground region of interest of a predetermined size from the top-down projection map, there is no need to perform subsequent image calculation and processing on the entire top-down projection map, which can effectively improve data processing efficiency. In specific implementation, since the motor vehicle is constantly moving, the top-down projection map of the ground corresponding to the image frame in the original video image changes at any time, so the ground region of interest therein can be actually identified and adjusted according to the ground region in the image frame.
[0058] In another optional embodiment of the present invention, Figure 4 As shown, the step S4 specifically includes:
[0059] S41: Calculating the frame difference between the ground region of interest of the current image frame and the compensation region; and
[0060] S42: Calculate the frame difference information entropy H1 between the ground region of interest of the current image frame and the compensation region based on the frame difference.
[0061] In this embodiment, by first calculating the frame difference between the ground area of interest of the current image frame and the compensation area, and then calculating the frame difference information entropy H1 based on the frame difference, the calculation process is relatively simple and the data processing efficiency is effectively improved.
[0062] In another optional embodiment of the present invention, Figure 5 As shown, the step S5 specifically includes:
[0063] S51: Calculate the information entropy H2 of the ground area of interest of the current image frame;
[0064] S52: Calculate the current shaking amplitude of the vehicle-mounted camera 1 H0 = |H1-H2| / H2; and
[0065] S53: Compare the jitter amplitude H0 with the frame difference information entropy adaptive threshold, and determine that the vehicle-mounted camera 1 currently has jitter when the jitter amplitude H0 is greater than the frame difference information entropy adaptive threshold.
[0066] In this embodiment, by recalculating the information entropy H2 of the ground area of interest of the current image frame, and by taking the difference between the frame difference information entropy H1 of the ground area of interest of the current image frame and the compensation area and the information entropy H2 of the ground area of interest of the current image frame as the current jitter amplitude H0 of the vehicle camera 1, and comparing it with the frame difference information entropy adaptive threshold, it is possible to accurately determine whether the vehicle camera 1 is currently jittering.
[0067] In another optional embodiment of the present invention, Figure 6 As shown, the calculation of the initial value of the motion matrix of the current image frame based on the wheel speed pulse of the motor vehicle and the steering angle of the motor vehicle specifically includes:
[0068] S21: Calculating three-dimensional motion parameters of the vehicle within the frame difference time length based on the vehicle wheel speed pulse and the vehicle steering angle; and
[0069] S22: Calculate and obtain the initial value of the motion matrix of the current image frame by combining the three-dimensional motion parameters and the calibration parameters of the vehicle-mounted camera 1.
[0070] In this embodiment, the three-dimensional motion parameters of the motor vehicle within the frame difference time are first calculated using the motor vehicle wheel speed pulse and the motor vehicle steering angle, and then the initial value of the motion matrix of the current image frame is calculated based on the three-dimensional motion parameters and the calibration parameters of the vehicle-mounted camera. The calculation process is simple and the data processing efficiency is high.
[0071] In another optional embodiment of the present invention, Figure 7 As shown, the ground-based reprojection error minimization optimization and calculation of the precise motion matrix of the current image frame specifically include:
[0072] S23: calculating the projection error of the ground region of interest of the previous frame of the current image frame in the current image frame based on the initial value of the motion matrix;
[0073] S24: Calculating the best inter-frame motion vector of the current image frame based on the projection error and the nonlinear optimization principle;
[0074] S25: predicting the precise motion vector of the current image frame based on the best inter-frame motion vector of the image frame of the latest predetermined number of frames in the original video image and the smoothing assumption of the motor vehicle motion and the polynomial fitting principle; and
[0075] S26: Generate an accurate motion matrix of the current image frame based on the accurate motion vector of the current image frame.
[0076] In this embodiment, the projection error of the ground area of interest of the previous frame of the current image frame is first calculated in the current image frame according to the initial value of the motion matrix, and then the optimal inter-frame motion vector of the current image frame can be calculated by combining the projection error and the nonlinear optimization principle. Since the error of the motion vector calculated by only two image frames is relatively large, the best inter-frame motion vector of the image frame with the latest predetermined number of frames is used to predict the precise motion vector of the current image frame based on the smoothing assumption of motor vehicle motion and the polynomial fitting principle, thereby reducing the error and improving the accuracy of the motion vector. Finally, the precise motion matrix of the current image frame is generated according to the precise motion vector, which can effectively improve the calculation accuracy of the motion matrix.
[0077] In another optional embodiment of the present invention, Figure 8 As shown, the step S25 specifically includes:
[0078] S251: Compare the total number of frames N in the original video image with the predetermined number of frames T;
[0079] S252: when the total number of frames N is less than the predetermined number of frames T, the best inter-frame motion vector of each image frame in the original video image is used and the accurate motion vector of the current image frame is predicted based on the smoothness assumption of the motor vehicle motion and the polynomial fitting principle; and
[0080] S253: When the total number of frames N is greater than or equal to the predetermined number of frames T, the best inter-frame motion vector of the latest T frames in the original video image is used and the precise motion vector of the current image frame is predicted based on the smoothness assumption of motor vehicle motion and the polynomial fitting principle.
[0081] In this embodiment, since the original video image is the corresponding video stream data, when the total number of frames N of the image frames in the original video image is relatively small and does not meet the predetermined number of frames T, all the best inter-frame motion vectors of the image frames in the original video image are used as the predicted original data set to predict the target inter-frame motion vector of the current image frame; and when the total number of frames N of the original video image is greater than or equal to the predetermined number of frames T during the gradual accumulation process, the best inter-frame motion vectors of the latest T frames of the image frames in the original video image are used as the original data set to predict the target inter-frame motion vector of the current image frame, which can improve data processing efficiency and ensure prediction accuracy; in addition, the prediction is performed using the smoothing assumption based on vehicle body motion and the polynomial fitting principle. After specific experimental measurements, it is found that the prediction efficiency is relatively high and the accuracy is also higher. For example: when the total number of image frames in the original video image is N=3, and the predetermined number of frames is T=4, N is less than T, and the best inter-frame motion vector of each frame (the 2nd to 3rd frames) in the original video image will be adopted. It can be understood that since the 1st frame in the original video image is the initial frame, there is no best inter-frame motion vector, so it is not adopted by default; when the total number of image frames in the original video image is N=7, N is greater than T, and the best inter-frame motion vectors of the latest 4 frames (the 4th to 7th frames) in the original video image will be adopted.
[0082] In another optional embodiment of the present invention, Fig. 9 As shown, the step S3 specifically includes:
[0083] S31: Calculating the coordinates of compensated pixel points in the compensation area corresponding to the coordinates of pixel points in the ground region of interest of the current image frame based on the precise motion matrix; and
[0084] S32: Calculate the compensation area according to the coordinates of the compensation pixel points and the pixel size of the ground region of interest of the current image frame.
[0085] In this embodiment, the coordinates of the compensation pixel points in the compensation area corresponding to the pixel point coordinates of the ground area of interest of the current image frame are first calculated based on the precise motion matrix to achieve a one-to-one correspondence between the coordinate points, and then point-to-point one-to-one motion compensation is performed based on the compensation pixel point coordinates and the pixel size of the ground area of interest of the current image frame, so that the compensation area can be quickly obtained.
[0086] In the specific implementation, the semantic segmentation model is a semantic segmentation model based on deep learning. In this embodiment, the semantic segmentation model is generated through learning and training based on a large number of template images. When performing actual semantic segmentation, the semantic segmentation effect is good, which is conducive to improving the accuracy of jitter judgment. In the specific implementation, a semantic segmentation neural network based on the U-net network as the core can be used, and its specific learning and training process is not described in detail here.
[0087] In specific implementation, in step S21, it can be understood that the horizontal displacement u0, vertical displacement v0 and rotation angle θ0 of the motor vehicle can be obtained according to the wheel speed pulse of the motor vehicle and the steering angle of the motor vehicle.
[0088] In step S22, according to the three-dimensional motion parameters of the motor vehicle and based on the pre-calibrated internal and external parameters of the vehicle-mounted camera 1, a conversion relationship T between the space coordinates and the image coordinates is obtained. According to the conversion relationship T, the horizontal displacement u0, vertical displacement v0 and rotation angle θ0 of the motor vehicle can be converted from the physical space to the image space, and the following conversion formula can be obtained:
[0089] (u0 camera ,v0 camera ,θ0 camera )=T(u0,v0,θ0) (Formula 1);
[0090] The initial value M of the motion matrix of the current image frame can be obtained from the above formula (1) and is expressed as:
[0091]
[0092] In step S23, the projection error S of the ground region of interest of the previous frame of the current image frame in the current image frame is calculated according to formula (2): dif It is expressed as:
[0093]
[0094] in, i and j represent the coordinates of each pixel point of the ground region of interest in the previous frame of the current image frame, i′ and j′ represent the coordinates of each pixel point of the area corresponding to the ground region of interest in the previous frame calculated according to the initial value M of the motion matrix in the image frame, m and n represent the length and width of the ground region of interest in the previous frame of the current image frame, Represents the ground region of interest in the previous frame of the current image frame.
[0095] In step S24, the initial value M of the motion matrix and the projection error S are combined. dif Based on the nonlinear optimization principle, the optimal inter-frame motion vector of the current image frame is estimated and calculated, which can be specifically expressed as:
[0096] min(S dif (u camera ,v camera ,θ camera ))=S dif (u obj ,v obj ,θ obj ) formula (4).
[0097] In step S25, according to the relationship between the total number of frames N in the original video image and the predetermined number of frames T, the data set used to determine the precise motion vector of the current image frame is determined. Based on the smooth assumption of motor vehicle motion and the principle of polynomial fitting, the precise motion vector of the current image frame can be expressed as: (u_i obj ,v_i obj ,θ_i obj ), where the value of i is the number of frames used.
[0098] In step S26, the precise motion matrix M_cur of the current image frame is generated based on the precise motion vector of the current image frame, which is specifically expressed as:
[0099]
[0100] In step S3, after calculating the precise motion matrix M_cur of the current image frame, the compensation area I_roi_obj corresponding to the ground region of interest I_roi_pre (whose size is m0*n0) in the current image frame can be solved by the precise motion matrix M_cur. The specific solution process includes: using the pixel reference coordinates of the ground region of interest in the current image frame as (i, j), solving the pixel coordinates (i′, j′) of the compensation area I_roi_obj, and the solution formula is:
[0101]
[0102] I _roi_obj (i,j)=I _semantic_brid_cur (i′,j′) (Formula 7);
[0103] Among them, I _semantic_brid_cur Represents the top-down projection image corresponding to the current image frame; finally, let i be equal to 1~m0 and j be equal to 1~n0 in sequence, and double loop can be used to solve and obtain the compensation area I_roi_obj.
[0104] In step S41, the frame difference I between the ground region of interest I_roi_pre of the current image frame and the compensation region I_roi_obj is dif It is expressed as:
[0105] Idif =|I _roi_obj -I _roi_pre | (Formula 8).
[0106] In step S42, the frame difference information entropy H1 between the ground region of interest I_roi_pre of the current image frame and the compensation region I_roi_obj is expressed as:
[0107]
[0108] Among them, p (i,j)_ dif is the grayscale occurrence probability of the target frame difference I_dif at the pixel coordinate (i, j).
[0109] In step S51, the information entropy H2 of the ground region of interest I_roi_pre of the current image frame can be expressed as:
[0110]
[0111] Among them, p (i,j) It is the grayscale occurrence probability of the ground region of interest I_roi_pre at the pixel coordinate (i, j) in the frame before the current image frame.
[0112] In step S52 , the current shaking amplitude H0 of the vehicle-mounted camera 1 can be expressed as H0=|H2−H1| / H1.
[0113] Finally, in step S53, by comparing the jitter amplitude H0 with the frame difference information entropy adaptive threshold, it can be determined whether the vehicle camera 1 has jitter, wherein the frame difference information entropy adaptive threshold is a normalized threshold whose value range is between 0 and 1.
[0114] On the other hand, Fig. 9 As shown, an embodiment of the present invention provides a vehicle-mounted camera jitter recognition device 3, which is respectively connected to a vehicle-mounted camera 1 for shooting video images around a vehicle and providing original video images, a vehicle motion sensor 5 for detecting the motion state of a vehicle and providing a vehicle wheel speed pulse and a vehicle steering angle, and a jitter processing device 7 for processing the jitter when the vehicle-mounted camera 1 is jittered. The vehicle-mounted camera jitter recognition device 3 includes a processor 30, a memory 32, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the vehicle-mounted camera jitter recognition method described in the above embodiment is implemented. In specific implementation, it can be understood that the vehicle motion sensor 5 can be composed of a vehicle wheel speed pulse sensor and a steering sensor of a vehicle electronic power steering system.
[0115] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 32 and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the vehicle-mounted camera shake recognition device 3. For example, the computer program may be divided into Fig.10 The functional modules in the vehicle-mounted camera jitter recognition device 3, wherein the image extraction module 41, the motion matrix calculation module 42, the compensation frame generation module 43, the information entropy calculation module 44 and the jitter judgment module 45 respectively execute the above steps S1 to S5.
[0116] The vehicle-mounted camera shake recognition device 3 can be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The vehicle-mounted camera shake recognition device 3 can include, but is not limited to, a processor 30 and a memory 32. Those skilled in the art can understand that the schematic diagram is only an example of the vehicle-mounted camera shake recognition device 1, and does not constitute a limitation on the vehicle-mounted camera shake recognition device 3, and can include more or fewer components than shown in the figure, or combine certain components, or different components. For example, the vehicle-mounted camera shake recognition device 1 can also include input and output devices, network access devices, buses, etc.
[0117] The processor 30 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 30 is the control center of the vehicle-mounted camera shake recognition device 3, and uses various interfaces and lines to connect various parts of the vehicle-mounted camera shake recognition device 3.
[0118] The memory 32 can be used to store the computer program and / or module. The processor 30 realizes various functions of the vehicle-mounted camera shake recognition device 3 by running or executing the computer program and / or module stored in the memory 32 and calling the data stored in the memory 32. The memory 32 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a graphic recognition function, a graphic overlay function, etc.), etc.; the data storage area can store data created according to the use of the recognition device (such as graphic data, etc.), etc. In addition, the memory 32 can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0119] If the functions described in the embodiments of the present invention are implemented in the form of software function modules or units and sold or used as independent products, they can be stored in a storage medium readable by a computing device. Based on this understanding, the embodiments of the present invention implement all or part of the processes in the above-mentioned embodiments, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor 30, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0120] On the other hand, an embodiment of the present invention provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the vehicle camera jitter recognition method as described above.
[0121] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0122] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, which all belong to the protection scope of the present invention.
Claims
1. A vehicle camera jitter recognition method, characterized in that: The method comprises the following steps: Acquire real-time image frames from the original video images captured by the vehicle-mounted camera, use a pre-trained semantic segmentation model to detect the ground around the motor vehicle, and then perform a bird's-eye view projection on the semantic segmentation results and normalize the scale of ground motion to generate a bird's-eye view projection map; Cut out a ground area of interest of a predetermined size from the bird's-eye view projection image; Calculating the three-dimensional motion parameters of the motor vehicle within the frame difference time length based on the motor vehicle wheel speed pulse and the motor vehicle steering angle; Calculate and obtain the initial value of the motion matrix of the current image frame by combining the three-dimensional motion parameters and the calibration parameters of the vehicle-mounted camera; Calculate the projection error of the ground region of interest of the previous frame of the current image frame in the current image frame based on the initial value of the motion matrix; Calculate the best inter-frame motion vector of the current image frame based on the projection error and nonlinear optimization principle; Predicting the precise motion vector of the current image frame based on the best inter-frame motion vector of the image frame with the latest predetermined number of frames in the original video image based on the smoothness assumption of the motor vehicle motion and the polynomial fitting principle; Generate an accurate motion matrix of the current image frame based on the accurate motion vector of the current image frame; Generate a motion compensated image frame of the current image frame based on the precise motion matrix, wherein the motion compensated image frame is a compensation area corresponding to the ground area of interest of the current image frame; Calculating the frame difference information entropy between the current image frame and the motion compensated image frame, the frame difference information entropy being the frame difference information entropy between the compensation area of the motion compensated image frame and the ground area of interest of the current image frame; and The current shaking state of the vehicle-mounted camera is determined by combining the frame difference information entropy and a preset frame difference information entropy adaptive threshold.
2. The vehicle-mounted camera jitter recognition method according to claim 1, characterized in that: The calculating of the frame difference information entropy between the current image frame and the motion compensated image frame specifically includes: Calculating the frame difference between the ground region of interest of the current image frame and the compensation region; and The frame difference information entropy H1 between the ground region of interest of the current image frame and the compensation region is calculated based on the frame difference.
3. The vehicle-mounted camera jitter recognition method according to claim 2, characterized in that: The combining the frame difference information entropy and the preset frame difference information entropy adaptive threshold to determine the current shaking state of the vehicle camera specifically includes: Calculate the information entropy H2 of the ground area of interest of the current image frame; Calculate the current jitter amplitude of the vehicle camera H0=|H1-H2| / H2; and The jitter amplitude H0 is compared with the frame difference information entropy adaptive threshold, and when the jitter amplitude H0 is greater than the frame difference information entropy adaptive threshold, it is determined that the vehicle-mounted camera currently has jitter.
4. The vehicle-mounted camera jitter recognition method according to claim 1, characterized in that: The method of predicting the precise motion matrix of the current image frame based on the best inter-frame motion vector of the latest predetermined number of image frames in the original video image based on the smoothness assumption of motor vehicle motion and the polynomial fitting principle specifically includes: Comparing the total number of frames N of the image frames in the original video image with the predetermined number of frames T; When the total number of frames N is less than the predetermined number of frames T, the best inter-frame motion vector of each image frame in the original video image is used and the accurate motion vector of the current image frame is predicted based on the smoothness assumption of the motor vehicle motion and the polynomial fitting principle; and When the total number of frames N is greater than or equal to the predetermined number of frames T, the best inter-frame motion vector of the latest T frames in the original video image is used and the precise motion vector of the current image frame is predicted based on the smoothness assumption of motor vehicle motion and the polynomial fitting principle.
5. The vehicle-mounted camera jitter recognition method according to claim 1, characterized in that: The step of generating a motion compensated image frame of the current image frame based on the precise motion matrix specifically includes: Calculating the coordinates of compensated pixel points in the compensation area corresponding to the coordinates of pixel points in the ground region of interest of the current image frame based on the precise motion matrix; and The compensation area is obtained by calculating according to the coordinates of the compensation pixel points and the pixel size of the ground area of interest of the current image frame.
6. A vehicle camera jitter recognition device, connected to a vehicle camera for shooting video images around a vehicle and providing original video images, a vehicle motion sensor for detecting the motion state of the vehicle and providing vehicle wheel speed pulses and vehicle steering angles, and a jitter processing device for processing the jitter when the vehicle camera jitters, characterized in that: The vehicle-mounted camera jitter recognition device includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the vehicle-mounted camera jitter recognition method according to any one of claims 1 to 5 is implemented.
7. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the vehicle-mounted camera jitter recognition method according to any one of claims 1 to 5.
Citation Information
Patent Citations
Image processing apparatus and method, imaging apparatus, and computer readable storage medium
CN108737726A
Vehicle measurement method based on map image
CN111477013A
Image processing device, image processing method, program and imaging system
CN113364983A
On-vehicle environment recognition device
JP2014026608A
