Image processing method, device and system for measuring vehicle wheelbase

Abstract

The present invention relates to the field of visual image processing technology, and discloses an image processing method, device, and system for measuring vehicle wheelbase, the method comprising: step 1, collecting image data of the front and rear wheels of a vehicle parked in a buffer parking space through a camera, performing distortion correction on the image data, and obtaining a corrected image; step 2, identifying the ground contact point data of the front and rear wheels of the vehicle in the corrected image through a wheel ground contact point detection model, and obtaining the vehicle wheelbase; step 2 comprises: step 2-1, calling a self-trained wheel ground contact point detection model to calculate the pixel positions of the ground contact points of the front and rear wheels of the vehicle in the corrected image; step 2-2, according to the buffer parking space boundary line calibrated in the corrected image, calculating the distance between the pixel position of each wheel ground contact point and the buffer parking space boundary line, and calculating the vehicle wheelbase according to the vector method and the Pythagorean theorem. The present invention ensures the accuracy of high-precision non-contact measurement of vehicle wheelbase, and also improves the convenience and efficiency of measurement.

Description

Technical Field

[0001] The present invention relates to the technical field of visual image processing, and in particular to an image processing method, device and system for measuring vehicle wheelbase. Background Art

[0002] Traditionally, the wheelbase of a vehicle is measured manually, but the measurement process is not only time-consuming, but the tools or equipment used for manual measurement may also damage the vehicle and cause losses. At present, the wheelbase of a vehicle is calculated through deep learning or machine learning after collecting vehicle images. However, due to the size of buffer parking spaces and the large number of vehicle models, the amount of data required for model training is very large, and it is easily affected by camera parameters when collecting vehicle images, making the model output prediction inaccurate. Therefore, how to accurately and quickly measure the wheelbase of a vehicle through visual image methods is an urgent problem to be solved. Summary of the invention

[0003] The purpose of the present invention is to accurately and quickly measure the wheelbase of a vehicle by correcting the distortion of a visual image, and to provide an image processing method, device and system for measuring the wheelbase of a vehicle.

[0004] In order to achieve the above-mentioned object of the invention, the present invention provides the following technical solutions:

[0005] The image processing method for measuring vehicle wheelbase comprises the following steps:

[0006] Step 1, collecting image data of the front and rear wheels of the vehicle parked in the buffer parking space through a camera, and performing distortion correction on the image data to obtain a corrected image;

[0007] In step 1, the step of performing distortion correction on the image data to obtain a corrected image includes:

[0008] Step 1-1, convert the image from the world coordinate system to the pixel coordinate system;

[0009] Step 1-2, calibrate the camera through the calibration plate image taken by the camera, and calculate the product of the camera's intrinsic parameter matrix and extrinsic parameter matrix;

[0010] Step 1-3, calculating the internal parameter matrix and the external parameter matrix based on the product of the internal parameter matrix and the external parameter matrix;

[0011] Step 1-4, solve the radial distortion formula, calibrate the distortion correction parameters, and obtain the corrected image;

[0012] Step 2, identifying the contact point data of the front and rear wheels of the vehicle in the rectified image through a wheel contact point detection model, thereby obtaining the vehicle wheelbase;

[0013] The step 2 specifically includes the following steps:

[0014] Step 2-1, calling the self-trained wheel contact point detection model to calculate the pixel positions of the contact points of the front and rear wheels of the vehicle in the rectified image;

[0015] Step 2-2, according to the buffer parking space boundary line calibrated in the corrected image, calculate the distance between the pixel position of each wheel contact point and the buffer parking space boundary line, and calculate the wheelbase of the vehicle according to the vector method and the Pythagorean theorem.

[0016] An image processing device for measuring vehicle wheelbase comprises:

[0017] An image acquisition module, used to acquire image data of the front and rear wheels of the vehicle in the buffer parking space;

[0018] A distortion correction module, used for performing distortion correction on image data to obtain a corrected image;

[0019] The wheel contact point detection module is used to identify the contact point data of the front and rear wheels of the vehicle in the corrected image, so as to calculate the vehicle wheelbase.

[0020] Compared with the prior art, the present invention has the following beneficial effects: the present invention measures the vehicle wheelbase through a visual image processing method without contacting the vehicle, and performs distortion correction on the camera at the same time. By deeply optimizing the combination of image distortion correction algorithm and intelligent recognition technology, it not only ensures the accuracy of high-precision non-contact measurement of the vehicle wheelbase, but also significantly improves the convenience and efficiency of measurement. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a flow chart of the method of the present invention;

[0023] Figure 2 Detailed flow chart of the method of the present invention;

[0024] Figure 3 This is a schematic diagram of the camera installation position according to an embodiment of the present invention;

[0025] Figure 4 A schematic diagram of the distance between a wheel and a buffer parking space boundary line according to an embodiment of the present invention;

[0026] Figure 5 A schematic diagram of the network structure of a wheel contact point detection model according to an embodiment of the present invention;

[0027] Figure 6 This is a block diagram of a device module according to an embodiment of the present invention;

[0028] Figure 7 This is a block diagram of a distortion correction module according to an embodiment of the present invention;

[0029] Figure 8 This is a block diagram of a wheel contact point detection module according to an embodiment of the present invention;

[0030] Fig. 9 The figure is a block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0032] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance, or implying any such actual relationship or order between these entities or operations.

[0033] The present invention is achieved through the following technical solutions: Figure 1 , Figure 2 As shown, an image processing method for measuring vehicle wheelbase comprises the following steps:

[0034] Step 1: collect image data of the front and rear wheels of the vehicle parked in the buffer parking space through a camera, perform distortion correction on the image data, and obtain a corrected image.

[0035] Drive the vehicle into the buffer parking space. Figure 3 As shown in the figure, the buffer parking space consists of four conveyor belts stretched left and right (left and right are relative to the left and right directions of the vehicle). When the vehicle is parked, each wheel corresponds to the area of ​​a conveyor belt. A camera is installed above the center of the side of each conveyor belt in the buffer parking space. Figure 3The circle in the middle is 2m to 3m above the conveyor belt, so that the four cameras can capture the image data of the four wheels of the vehicle in all directions, as well as the boundary lines of the two buffer parking spaces closest to the wheels. The layout of the cameras not only ensures that the collected image data is comprehensive and accurate, but also facilitates the precise processing of subsequent algorithms. After the vehicle is parked, the warning light on the top of the buffer parking space changes from blue to white, which not only indicates the presence of a vehicle, but also enhances the lighting environment for the camera to capture. At this time, the camera collects image data.

[0036] Before installing the camera, the camera's intrinsic parameter matrix is ​​obtained by matching the calibration plate. The camera is then installed above the buffer parking space. The camera's extrinsic parameter matrix is ​​obtained by collecting on-site image data. The product of the intrinsic parameter matrix and the extrinsic parameter matrix is ​​then solved using the camera imaging principle to perform distortion correction on the image data.

[0037] In step 1, the step of performing distortion correction on the image data to obtain a corrected image further includes:

[0038] Step 1-1, transform the image from the world coordinate system to the pixel coordinate system.

[0039] When obtaining the intrinsic parameter matrix, let P = (X, Y, Z) be a point in the shooting scene. In the pinhole camera model, it must undergo the following transformations and eventually become an image point on the two-dimensional image. , where X, Y, and Z are the coordinates of point P on the x-axis, y-axis, and z-axis in the world coordinate system:

[0040] ① Convert point P from the world coordinate system to the camera coordinate system through rigid body transformation (i.e. rotation and translation). The conversion process uses the relative pose between cameras, that is, the external parameters of the camera.

[0041] ②Transform the image point p`=(x,y) from the camera coordinate system to the camera's imaging coordinate system through perspective projection.

[0042] ③ Transform the image point p' from the imaging coordinate system to the image point on the pixel coordinate system by scaling and translating ,in , are the horizontal and vertical coordinates of point p in the pixel coordinate system respectively.

[0043] Transforming the camera's 3D point in the scene into a 2D point in the image is a combination of various coordinate system transformations. The above transformation process can be organized into the form of matrix multiplication:

[0044]

[0045] Define K as the intrinsic parameter of the camera:

[0046]

[0047] Where s represents the first scale factor; Indicates the number of pixels within a horizontal unit distance on the image. Indicates the number of pixels within a unit distance in the vertical direction of the image; Indicates the transformation of the camera's focal length fw into pixel measurements in the x direction. Indicates the transformation of the camera's focal length fw into pixel measurements in the y direction; c x 、c y Respectively represent the horizontal and vertical coordinates of the center point of the image; R represents the external parameter matrix; Represents the translation vector.

[0048] Step 1-2: Calibrate the camera using the calibration plate image captured by the camera, and calculate the product of the camera's intrinsic parameter matrix and extrinsic parameter matrix.

[0049] Solve the product of the intrinsic parameter matrix and the extrinsic parameter matrix, and fix the world coordinate system on the calibration plate. This embodiment assumes that there are 8 corner points in the horizontal direction and 11 corner points in the vertical direction on a calibration plate, with a total of 8×11 corner points. Select the physical coordinate W=0 of any point on the calibration plate, and the original single-point undistorted imaging model is expressed as follows:

[0050]

[0051] Wherein, Z represents the second scale factor, u and v represent the horizontal and vertical coordinates of any corner point in the calibration plate in the pixel coordinate system, respectively; A represents the intrinsic parameter matrix; (R1 R2 T) represents the extrinsic parameter matrix, R1 represents the first column of the extrinsic parameter matrix, R2 represents the second column of the extrinsic parameter matrix, and T represents the translation vector; U and V represent the horizontal and vertical coordinates of (u, v) in the world coordinate system; f represents the image distance; dX and dY represent the physical length of a pixel on the camera photosensitive plate in the X and Y directions, respectively (the camera photosensitive plate refers to the component inside the camera used to capture light and convert it into electrical signals, also known as the image sensor; the physical length is the length of a pixel on the camera photosensitive plate, in millimeters). In the process of camera calibration, knowing the accurate values ​​of dX and dY can more accurately calculate the focal length fw of the camera; Indicates the angle between the horizontal and vertical sides of the camera's photosensitive plate; u 0 、v 0 They respectively represent the horizontal and vertical coordinates of the center of the camera's photosensitive plate in the pixel coordinate system.

[0052] Let H be the product of the internal parameter matrix and the external parameter matrix, H is a 3×3 matrix, and we have:

[0053]

[0054] And there are:

[0055]

[0056]

[0057]

[0058]

[0059] The size of each grid of the calibration plate is fixed and known, and the world coordinate system is also predefined, so after obtaining the pixel coordinates (u, v), the world coordinates (U, V) can be calculated. Where H is a homogeneous matrix with 8 independent unknown elements (H 11 ,H 12 ,H 13 ,H 21 ,H 22 ,H 23 ,H 31 ,H 32 ). Each calibration plate corner point can provide two constraint equations (i.e., the corresponding relationship between u, U, and V, and the corresponding relationship between v, U, and V). Therefore, when the number of calibration plate corner points on an image data is equal to 4, the H corresponding to the image data can be obtained. When the number of calibration plate corner points on an image data is greater than 4, the least squares regression method is used to obtain the optimal H.

[0060] Step 1-3, based on the product of the internal parameter matrix and the external parameter matrix, calculate the internal parameter matrix and the external parameter matrix.

[0061] Given the matrix H=A(R1 R2 T), solve the camera's intrinsic parameter matrix A. Using R1 and R2 as the first two columns of the extrinsic matrix, there is a unit orthogonal relationship:

[0062]

[0063]

[0064] From the relationship between H and R1, R2, we can know that:

[0065]

[0066]

[0067] Substituting in, we get:

[0068]

[0069]

[0070] Both of the above constraint equations exist in the matrix A -T A-1 , remember A -T A -1 =B, then matrix B is a symmetric matrix. The camera's intrinsic parameter matrix A is solved by matrix B, and the constraint equation obtained by the unit orthogonality of R1 and R2 is:

[0071]

[0072]

[0073] To solve the matrix B, we need to calculate ,but:

[0074]

[0075] Among them, i=1,2,3, j=1,2,3. Simplify the above equation and let:

[0076]

[0077]

[0078] Then there is: Hi T BHj=v ij b, where the matrix .

[0079] How to get the matrix v:

[0080] ① Calculate Hi T The transposed product of BHj yields a 6×9 matrix;

[0081] ② Divide this matrix into blocks by column, each block is a 6×1 vector v ij ;

[0082] ③ All v ij Put together to form a 6×6 matrix v;

[0083] ④Finally, subtract the last column from each column of the matrix v to get the matrix v The subscripts i and j in the part are not equal to 3.

[0084] The purpose of this is to eliminate redundancy, make the equations more concise, reduce the amount of calculation, and improve the efficiency of solving. In this way, a 6×6 equation system can be obtained instead of a larger equation system.

[0085] At this time, we only need to solve the vector b to get the matrix B. Each calibration plate can provide a constraint relationship of vb=0, which contains two constraint equations, namely the homography matrix constraint equation and the polar line constraint equation.

[0086] The homography matrix C describes the projective transformation from one plane to another. For each image, the homography matrix from the calibration plane to the image plane can be calculated. If the calibration plane is parallel to the camera optical axis, then the homography matrix C will satisfy a specific constraint, that is, the third row of the homography matrix C should be zero (except the last element c33, which represents the scaling factor). This is because in this case, the projection of the calibration plane on the image will remain unchanged, without perspective effects.

[0087] Mathematically, using homogeneous coordinates, the homography matrix C can be written as a 3×3 matrix:

[0088]

[0089] Then the constraint relationship of vb=0 can be expressed as:

[0090]

[0091] The epipolar constraint is derived from epipolar geometry and describes the relationship between the projection points of a point in the scene on the images taken by two cameras when two cameras capture the same scene. If the projection of a point on the first image is known, then the projection of the point on the second image must fall on the epipolar line defined by the point in the first image. When the calibration plane is parallel to the camera optical axis, the epipolar line will pass through the center of the image.

[0092] Mathematically, the epipolar line G can be calculated from the fundamental matrix N and the coordinates Q of any point in the image:

[0093]

[0094] in, is the homogeneous coordinate of the polar line, G 1 Represents the first element in G, G 2 Represents the second element in G, G 3 represents the third element in G; is the homogeneous coordinate of any point in the image, q1 represents the horizontal coordinate of the point, and q2 represents the vertical coordinate of the point. When vb=0, the epipolar line G must satisfy the following constraints:

[0095]

[0096] This is because the epipolar lines should be perpendicular to the camera optical axis, while in this case, the epipolar lines are stationary in the image vertical direction, that is, the epipolar lines have no component in the image vertical direction.

[0097] Combining the homography matrix constraint equation and the polar line constraint equation, we get the two constraint equations in the vb=0 constraint relationship. However, vector b has 6 unknown elements:

[0098]

[0099] Matrix B will be converted into a vector b. To facilitate the solution, the elements of matrix B are rearranged into a vector. When converting matrix B to vector b, the elements of matrix B are stacked in column-major or row-major order. Regardless of the stacking method, element 0 in matrix B corresponds to the 3rd and 6th items in vector b, and element 1 corresponds to the 9th item:

[0100]

[0101] These values ​​of 0 and 1 are considered fixed during the calibration process because they do not change and do not carry unknown information about the intrinsic parameters of the camera.

[0102]

[0103] But in fact, considering homogeneous coordinates and mathematical redundancy, we don't really care about the last item in vector b, so we can set it to any non-zero constant, usually 1. In addition, since there are two zero items in the first and second rows of matrix B, they will not bring new information in the solution process. Therefore, we only care about the first six non-zero or non-redundant items in vector b:

[0104]

[0105] These six elements are the six unknown elements of vector b, and the two constraint equations provided by a single calibration plate are not enough to solve vector b. Therefore, three calibration plates are needed to obtain three constraint relationships of vb=0, that is, six equations, to solve vector b. When the number of calibration plates is greater than 3 (in fact, 15 to 20 calibration plates are generally required), the least squares method can be used to fit the best vector b, and then the matrix B can be obtained.

[0106]

[0107] in, , Represents the square of the focal length fw (or equivalent focal length) of the camera in the x-axis and y-axis directions respectively; The skew coefficient in the camera's intrinsic parameter matrix K, that is, the element at (1,2) or (2,1) in K, can be used to obtain the camera's intrinsic parameter matrix A (3×4 matrix):

[0108]

[0109] It should be noted that if the last column of a 3-row 4-column matrix is ​​all 0, it can theoretically be regarded as a 3×3 matrix plus a 3×1 column vector of all 0s. When a matrix is ​​used to represent a linear transformation or a system of equations, if a row or a column of the matrix is ​​all 0, the row or column often does not provide any additional information or affect the result during the calculation process.

[0110] For the same camera, the camera's intrinsic parameter matrix depends on the camera's internal parameters. Regardless of the positional relationship between the calibration plate and the camera, the camera's intrinsic parameter matrix remains unchanged. -1 H, the external parameter matrix (R1 R2 T) corresponding to each image data can be obtained.

[0111] Step 1-4, solve the radial distortion formula, calibrate the distortion correction parameters, and obtain the corrected image.

[0112] Solve the camera distortion coefficient. The radial distortion has the greatest impact on the camera distortion. The formula for radial distortion is:

[0113]

[0114]

[0115] in, is the horizontal coordinate of the ideal distortion-free normalized image, is the ordinate of the ideal distortion-free normalized image; represents the horizontal coordinate of the normalized image after distortion correction, represents the ordinate of the normalized image after distortion correction; r is the distance from the pixel to the center of the image, that is, ;k 1 represents the first-order radial distortion coefficient, k 2 represents the secondary radial distortion coefficient, which is used to quantify the degree of distortion. 1 The effect on distortion is greater than k 2 If k 1 and k 2 If the value is positive, it means that the image has pincushion distortion, that is, the image edge is compressed; if the value is negative, it means that there is barrel distortion, that is, the image edge is stretched.

[0116] The conversion relationship between image coordinates and pixel coordinates is:

[0117]

[0118] Substituting into the radial distortion formula, we can simplify it to get:

[0119]

[0120] in, , are the ideal undistorted pixel horizontal and vertical coordinates respectively; , Respectively represent the horizontal and vertical coordinates of the pixel after distortion correction.

[0121] In the above formula , , , It can be obtained by identifying the corner points of the calibration plate. For each corner point, two of the above equations can be constructed. If there are m images, and each image has n calibration plate corner points, then all the equations can be combined to obtain m×n unknowns k=[k 1 ,k 2 ] T The constraint equation is Denoted as D, the non-homogeneous term Let it be d, which can be written as:

[0122]

[0123] Using the least squares method we get:

[0124]

[0125] There is an error between the actual position of the current camera with distortion correction in the pixel coordinate system and the fitted distortion curve. The smaller the reprojection error, the more consistent the fitted curve is with the actual distortion of the current camera, and the better the correction effect. At this point, the camera's distortion correction parameters have been calibrated.

[0126] In order to improve the accuracy and robustness of calibration, the Levenberg-Marquardt (LM) algorithm is optimized and improved to obtain the optimal intrinsic parameter matrix, extrinsic parameter matrix and distortion parameters of the camera. The specific optimization logic is as follows:

[0127] ① Initialization:

[0128] Set the initial parameters, use the camera's intrinsic parameter matrix, extrinsic parameter matrix and distortion parameters as the initial parameters, and set the initial camera parameter vector , Initialization parameters , Trust region radius .

[0129] ②Construct a quadratic approximation model:

[0130] Use the current parameter vector Calculate the Jacobian matrix J and the residual vector r to construct a quadratic approximation model of the objective function:

[0131]

[0132] in, Represents an approximate model of a quadratic function; represents the Jacobian matrix; represents the function value at the kth iteration; represents the camera parameter vector at the kth iteration.

[0133] ③ Solve the update step size in the trust region , solve the following constrained optimization problem:

[0134]

[0135] in, represents the trust region radius at the kth iteration, that is, the maximum allowed value of the parameter update step; I represents the identity matrix.

[0136] ④Accept or reject the update:

[0137] Calculate the actual reduction:

[0138]

[0139] in, Indicates the actual reduction. If it is greater than 0.25, accept the update, otherwise reject it and shrink the trust region.

[0140] ⑤Update and the trust region:

[0141] If you accept the update, Sizing and the trust region: Decrease or remain unchanged; if the update is rejected, increase And keep the trust region unchanged.

[0142] ⑥ Repeated iteration:

[0143] Use the updated parameters and adjusted and trust region, repeat steps ② to ⑤ until the termination condition is met.

[0144] ⑦Output results:

[0145] Final parameter estimation: Output the optimized camera intrinsic parameter matrix, extrinsic parameter matrix and distortion parameters.

[0146] After distortion correction, ideally, the points captured by the camera in the pixel coordinate system should fall exactly and unbiased on the predicted positions after correction by the distortion model. The metric that measures the degree of this match is called "reprojection error". This error quantifies the deviation between the actual captured image feature points and the corresponding points predicted based on the correction model theory. The smaller the reprojection error, the more accurately the distortion correction model can reflect the actual distortion characteristics of the camera, thus ensuring a better correction effect.

[0147] When the reprojection error reaches a satisfactory low level, it can be considered that the camera's distortion correction parameters have been successfully calibrated and can effectively compensate for lens distortion and improve image quality. At this point, the entire camera distortion correction calibration process is completed.

[0148] This step is crucial to improving image quality and reducing the difficulty of algorithm processing. Through distortion correction, the distortion and deformation in the image can be eliminated to obtain a corrected image, so that the subsequent algorithm can more accurately identify the characteristics of the vehicle.

[0149] Step 2: The contact point data of the four wheels of the vehicle in the rectified image are identified by the wheel contact point detection model to obtain the vehicle wheelbase.

[0150] The step 2 specifically includes the following steps:

[0151] Step 2-1, call the self-trained wheel contact point detection model to calculate the pixel positions of the contact points of the front and rear wheels of the vehicle in the corrected image.

[0152] First, the wheel contact point data set is collected. When training the wheel contact point detection model, a large number of wheel contact point data from a bird's-eye view of different vehicle models on the buffer parking space are collected to construct the wheel contact point data set. The process is as follows:

[0153] ① The vehicle drives forward into the buffer parking area. After the vehicle stops, the tires are straightened, the lights above the vehicle are turned on, and the camera is controlled to collect images. In order to ensure the accuracy of the wheel contact point detection, data is collected twice for each wheel. The first pass is the original data L 1 , and the second pass is the physical mark L of the wheel contact point in the real world 2 . The physical mark L 2 The pixel coordinates (x 2 ,y 2 ) is mapped to the original data collected in the first pass, then the physical identifier L of the wheel contact point in the original data is 1 =L 2 This method can not only eliminate the interference of the physical mark of the grounding point on the algorithm, but also ensure the true accuracy of the pixel coordinates of the grounding point; then the vehicle drives into the buffer parking space from different angles, and the image is collected cyclically according to the above steps.

[0154] ② The vehicle drives into the buffer parking area in reverse. After the vehicle stops, the tires are straightened, the lights above the vehicle are turned on, and the camera is controlled to collect images in the same way as in step ①. Then the vehicle drives into the buffer parking area in reverse from different angles and collects images in a loop.

[0155] ③ According to the above steps, vehicles of different brands and models are driven into the buffer parking space in turn, and several wheel contact point data sets are collected and constructed, for example, 3,000.

[0156] Then the wheel contact point detection data is preprocessed, and the Mosic data enhancement method is used to randomly combine the four image data into a new image data, thereby increasing the diversity of the data set and improving the generalization ability of the model. Passing such a new image data into the neural network for learning is equivalent to passing in the learning ability of four image data, which greatly enriches the background of the detected object, and the information of the four image data will be directly calculated during the standardized BN calculation.

[0157] Based on the Mosic data enhancement method, the TMosic data enhancement method is designed to improve it: the image data collected by the four cameras are randomly cropped, randomly combined, and randomly scaled. In order to ensure that the key annotation information of the wheel contact point will not be lost with random cropping, the wheel is first detected before random cropping to obtain the wheel's BoundingBOX. During random cropping, the cropped ROI area must include the wheel's BoundingBOX area, that is:

[0158]

[0159] The network structure of the wheel contact point detection model is as follows: Figure 5 As shown in the figure, it includes a feature extraction network (Backbone), a neck network (Neck), and a head network (Head). The feature extraction network of the wheel contact point detection model uses a multi-branch stacking module (ELAN). The multi-branch stacking module is an efficient network structure that enables the network to learn more features and has stronger robustness by controlling the shortest and longest gradient paths. The feature extraction network introduces a residual structure, which is composed of multiple convolutional layers + batch normalization layers + activation functions + pooling layers. The output size of any convolutional layer is calculated as follows:

[0160]

[0161] Among them, W is the input of the convolutional layer, O is the output of the convolutional layer, K is the filter size, E is the padding, and S is the stride.

[0162] The batch normalization layer calculates the mean and variance of a batch of samples and processes the batch samples using the following formula:

[0163]

[0164] in, Represents the data after batch normalization; Represents the data before batch normalization; Represents the mean of the batch samples; Represents the variance of the batch samples; represents the stretching parameter; It can be seen that the batch normalization layer stretches and shifts the data on the traditional basis (i.e. and ).

[0165] The activation function uses the Silu activation function, the formula is f(x)=x*sigmoid(x), where x represents the sum of the weighted inputs of a neuron in the neural network plus the bias term, sigmoid represents the sigmoid function, and sigmoid(x)=1 / (1+exp(-x)). When x is greater than 0, the Silu activation function will amplify x; when x is less than 0, the Silu activation function will reduce x, which can suppress overfitting.

[0166] The pooling layer uses the maximum pooling method, and the formula is:

[0167]

[0168] Among them, output[m,n] represents a pixel value in the output feature map after pooling, m represents the horizontal coordinate of the pixel, and n represents the vertical coordinate of the pixel; input represents the input feature map; pool_size represents the window size of the pooling operation.

[0169] The loss function of the wheel contact point detection model consists of the target box regression loss function, the target box confidence loss function, the target box category loss function, and the key point regression loss function. Among them, the target box regression loss function is represented by c_iouloss, the target box confidence loss function and the target box category loss function share the bce loss representation, and the key point regression loss function is represented by dynamics-wing-loss.

[0170] Among them, dynamics-wing-loss is the loss function for regression problems improved in this solution, which is particularly suitable for key point detection, such as facial key point positioning, hand joint positioning, etc. The design purpose is to solve the problem that in regression problems, when the predicted value is greatly different from the true value, the L1 or L2 loss function will cause the gradient to be too large, and when the predicted value is close to the true value, the gradient disappears. Therefore, by defining a nonlinear loss function, the loss function behaves like L1 loss when the error is small, and the loss function becomes smooth when the error is large, which can effectively avoid gradient explosion or disappearance (L1 and L2 are regularized loss functions). And for the regression problem of a single key point, the hyperparameters are dynamically adjusted to meet the different needs of model training.

[0171] In the process of designing the wheel contact point detection algorithm, considering that there is only one key point detection for the target wheel, in order to improve the accuracy of wheel key point detection, the improved key point regression loss function dynamics-wing-loss is:

[0172]

[0173] in, represents the key point regression loss function; y is the true value, is the predicted value; w(t) and is a hyperparameter, w(t) is used to control the smoothness of the loss function, Used to control the turning point from L1 loss to smooth loss.

[0174] In the improved key point regression loss function dynamics-wing-loss, for the regression problem of a single key point, for w and No default value is set. At different stages of training, w and The value of can be adjusted to meet different needs of model training. For example, a larger w and a smaller w may be required in the early stage of training. To promote fast convergence, while in the later stages of training, a smaller w and a larger To optimize the details and improve the accuracy of key point prediction.

[0175] The dynamic adjustment process is expressed in mathematical formulas:

[0176]

[0177]

[0178] Among them, w 0 and is the initial value, w(t) represents the hyperparameter of the tth iteration, represents the hyperparameter of the tth iteration, and t represents the number of iterations; and is the adjustment rate. The hyperparameter settings of the wheel contact point detection model are shown in Table 1.

[0179] Table 1 Hyperparameter settings of wheel contact point detection model

[0180]

[0181] Table 2 Wheel contact point detection model server configuration table

[0182]

[0183] According to the category classification probability of the candidate frames output by the wheel contact point detection model, they are sorted from high to low. For example, the wheel contact point detection model outputs six candidate frames A, B, C, D, E, and F, which are sorted according to the category classification probability as A>B>C>D>E>F. First, mark the candidate frame A with the largest category classification probability; calculate the overlap rate IoU (intersection over union ratio of two candidate frames) of B~E with A respectively. Assuming that the IoU of B, D with A is greater than the set threshold, and the IoU of C with A is not greater than the set threshold, then B and D are considered to be duplicated and removed, and the candidate frame C is retained; then continue to calculate the IoU of E, F with C, and remove the candidate frames with large overlap rates; repeat the calculation until all candidate frames are calculated and all candidate frames to be retained are obtained.

[0184] Next, use the monocular camera ranging principle to calculate the following formula:

[0185]

[0186]

[0187] Among them, distance_machine_direction is the longitudinal distance, distance_transverse_direction is the transverse distance; f_x is the camera intrinsic reference transverse focal length, f_y is the camera intrinsic reference longitudinal focal length; h is the camera height from the ground; imageW is the width of the image, imageH is the height of the image; pixe_x is the horizontal coordinate of the wheel contact point, and pixe_y is the vertical coordinate of the wheel contact point. After experimental testing, the wheel contact point detection model trained as above can achieve an accuracy rate of 99.8% for wheel contact point detection.

[0188] Step 2-2, according to the buffer parking space boundary line calibrated in the corrected image, calculate the distance between the pixel position of each wheel contact point and the buffer parking space boundary line, and calculate the wheelbase of the vehicle according to the vector method and the Pythagorean theorem.

[0189] like Figure 4 As shown, the distance between the wheel and the two closest boundary lines of the buffer parking space is calculated. Since the four boundary lines of the buffer parking space are also fixed relative to the camera after the camera position is fixed, the position of the boundary line can be calibrated in advance, and the wheel contact point detection model only needs to output the contact point position of the wheel. Taking the left front wheel as an example, it is known that the lengths of the left boundary line and the right boundary line of the buffer parking space are both Length, the contact point coordinates of the left front wheel are P1(x1,y1), and the coordinates of the two vertices of the left boundary line of the buffer parking space are A(x2,y2) and B(x3,y3). The distance P1L between the left front wheel and the left boundary line is calculated according to the vector method. d Similarly, the distance P1F between the left front wheel and the front boundary line of the buffer parking space can be obtained d The distances between the remaining three wheels and the two nearest boundary lines of the buffer parking space can be calculated in the same way.

[0190] Calculate the difference between the left and right directions and the front and rear directions of the same side wheel. Take the left wheel as an example. The distance between the left front wheel and the left boundary line is P1L. d , the distance between the left front wheel and the front boundary line is P1F d , the distance between the left rear wheel and the left boundary line is P2L d , the distance between the left rear wheel and the rear boundary line is P2B d , we can calculate the difference between the distance between the left front wheel and the left boundary line and the distance between the left rear wheel and the left boundary line, and get the difference Dx=|P1L between the left wheel and the left and right directions of the vehicle d -P2L d |, and calculate the difference between the left front wheel and the left rear wheel in the direction of the left boundary line, and get the difference Dy of the left wheel in the front and rear direction of the vehicle = |Length-P1F d -P2B d |.

[0191] Finally, use the Pythagorean theorem to square the sum of the squares of the differences between the left and right directions and the front and rear directions of the same side of the wheel to get the wheelbase values ​​on both sides. Take the left wheelbase measurement as an example:

[0192]

[0193] Among them, C L is the wheelbase on the left side, and similarly, the wheelbase on the right side can be obtained as C R Then for C L and C R Find the average value and finally get the vehicle wheelbase .

[0194] In actual application, the accuracy error of manual measurement is removed. The error of the vehicle wheelbase measurement method of this scheme is within the range of ±15mm, and the response time is about 50ms. The measurement accuracy and processing speed are at a high level.

[0195] like Figure 6 As shown, the present solution also proposes an image processing device for measuring vehicle wheelbase, comprising:

[0196] Image acquisition module 1, used to collect image data of the front and rear wheels of the vehicle in the buffer parking space;

[0197] The distortion correction module 2 is used to perform distortion correction on the image data to obtain a corrected image;

[0198] The wheel contact point detection module 3 is used to identify the contact point data of the front and rear wheels of the vehicle in the rectified image, so as to calculate the vehicle wheelbase.

[0199] like Figure 7 As shown, the distortion correction module 2 further includes: a coordinate conversion unit 21, used to convert the image from the world coordinate system to the pixel coordinate system; a matrix calculation unit 22, used to calculate the product of the camera's intrinsic parameter matrix and the extrinsic parameter matrix, and solve the radial distortion formula to perform distortion correction.

[0200] like Figure 8 As shown, the wheel contact point detection module 3 further includes:

[0201] The wheel position calculation unit 31 is used to call the self-trained wheel contact point detection model to calculate the pixel positions of the contact points of the front and rear wheels of the vehicle in the corrected image; the vehicle wheelbase calculation unit 32 is used to calculate the distance between the pixel position of each wheel contact point and the boundary line of the buffer parking space, and calculate the wheelbase of the vehicle.

[0202] The wheel contact point detection module has a self-trained wheel contact point detection model, and the loss function of the wheel contact point detection model includes a target frame regression loss function, a target frame confidence loss function, a target frame category loss function and a key point regression loss function.

[0203] The scheme also proposes an image processing system for measuring vehicle wheelbase, including:

[0204] At least one camera, an image processing device and a display device for displaying the measurement results, wherein the camera is used to collect image data of the front and rear wheels of the vehicle; the image processing device is used to process the image data and calculate the vehicle wheelbase.

[0205] The present solution also proposes a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the image processing method for measuring the vehicle wheelbase as described in this embodiment is implemented.

[0206] like Fig. 9 As shown, the present solution also proposes an electronic device, which may include a processor 41 and a memory 42, wherein the memory 42 is coupled to the processor 41. It is worth noting that Fig. 9 This is exemplary, and other types of structures may be used to supplement or replace this structure. In other embodiments, the electronic device may also include: an input unit 43, a display unit 44, and a power supply 45. It is worth noting that the electronic device does not necessarily have to include Fig. 9 In addition, electronic devices may also include Fig. 9 For components not shown, reference may be made to the prior art.

[0207] The processor 41 is sometimes also called a controller or an operation control, and may include a microprocessor or other processor devices and / or logic devices. The processor 41 receives inputs and controls the operations of various components of the electronic device.

[0208] The memory 42 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices, and may store the configuration information of the processor 41, instructions executed by the processor 41, recorded table data and other information. The processor 41 may execute the program stored in the memory 42 to implement information storage or processing, etc. In one embodiment, the memory 42 also includes a buffer memory, i.e., a buffer, to store intermediate information.

[0209] The input unit 43 is used, for example, to provide the processor 41 with data of the entity body or data owned by the data holder. The display unit 44 is used to display various results of the processing process, such as the acquired layers, the divided primitive layers, the strategy table, etc. The display unit can be, for example, an LCD display, but the present invention is not limited thereto. The power supply 45 is used to provide power to the electronic device.

[0210] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An image processing method for measuring vehicle wheelbase, characterized in that: The following steps are involved: Step 1, collecting image data of the front and rear wheels of the vehicle parked in the buffer parking space through a camera, and performing distortion correction on the image data to obtain a corrected image; In step 1, the step of performing distortion correction on the image data to obtain a corrected image includes: Step 1-1, convert the image from the world coordinate system to the pixel coordinate system; Step 1-2, calibrate the camera through the calibration plate image taken by the camera, and calculate the product of the camera's intrinsic parameter matrix and extrinsic parameter matrix; Step 1-3, based on the product of the internal parameter matrix and the external parameter matrix, calculate the internal parameter matrix and the external parameter matrix; Step 1-4, solve the radial distortion formula, calibrate the distortion correction parameters, and obtain the corrected image; Step 2, identifying the contact point data of the front and rear wheels of the vehicle in the rectified image through a wheel contact point detection model, thereby obtaining the vehicle wheelbase; The step 2 specifically includes the following steps: Step 2-1, calling the self-trained wheel contact point detection model to calculate the pixel positions of the contact points of the front and rear wheels of the vehicle in the rectified image; Step 2-2, according to the buffer parking space boundary line calibrated in the corrected image, calculate the distance between the pixel position of each wheel contact point and the buffer parking space boundary line, and calculate the wheelbase of the vehicle according to the vector method and the Pythagorean theorem.

2. The image processing method for measuring vehicle wheelbase according to claim 1, characterized in that: In step 1, the step of collecting image data of the front and rear wheels of the vehicle parked in the buffer parking space by means of a camera comprises: Four cameras are respectively arranged above the buffer parking spaces, and each camera is controlled to respectively collect images of the wheels at positions corresponding to the vehicles parked in the buffer parking spaces, and the boundary lines of the two buffer parking spaces closest to the wheels.

3. The image processing method for measuring vehicle wheelbase according to claim 1, characterized in that: The step 1-1 specifically includes the following steps: Let P = (X, Y, Z) be a point in the shooting scene, and the image point on the two-dimensional image is obtained by the following transformation: , where X, Y, and Z are the coordinates of point P on the x-axis, y-axis, and z-axis in the world coordinate system: ①Convert point P from the world coordinate system to the camera coordinate system through rigid body transformation; ② Transform the image point p`=(x,y) from the camera coordinate system to the camera imaging coordinate system through perspective projection; ③ Transform the image point p' from the imaging coordinate system to the image point on the pixel coordinate system by scaling and translating ,in , are the horizontal and vertical coordinates of point p in the pixel coordinate system respectively; The above transformation process is expressed as: Where s represents the first scale factor; Indicates the number of pixels within a horizontal unit distance on the image. Indicates the number of pixels within a unit distance in the vertical direction of the image; Indicates the transformation of the camera's focal length fw into pixel measurements in the x direction. Indicates the transformation of the camera's focal length fw into pixel measurements in the y direction; c x 、c y Respectively represent the horizontal and vertical coordinates of the center point of the image; R represents the external parameter matrix; Represents the translation vector.

4. The image processing method for measuring vehicle wheelbase according to claim 3, characterized in that: The steps 1-2 specifically include the following steps: Fix the world coordinate system on the calibration plate, select any point on the calibration plate with the physical coordinate W=0, and the original single-point undistorted imaging model is expressed as follows: Wherein, Z represents the second scale factor, u and v represent the horizontal and vertical coordinates of any corner point in the calibration plate in the pixel coordinate system, respectively; A represents the intrinsic parameter matrix; (R1 R2 T) represents the extrinsic parameter matrix, R1 represents the first column of the extrinsic parameter matrix, R2 represents the second column of the extrinsic parameter matrix, and T represents the translation vector; U and V represent the horizontal and vertical coordinates of (u, v) in the world coordinate system; f represents the image distance; dX and dY represent the physical length of a pixel in the X and Y directions on the camera photosensitive plate, respectively; Indicates the angle between the horizontal and vertical sides of the camera's photosensitive plate; u0 and v0 respectively represent the horizontal and vertical coordinates of the center of the camera's photosensitive plate in the pixel coordinate system; Let H be the product of the internal parameter matrix and the external parameter matrix, H is a 3×3 matrix, and we have: And there are: Among them, H is a homogeneous matrix, H1, H2, and H3 are the three-column matrices of H, and H ij Represents the element in the i-th row and j-th column of H, i=1,2,3, j=1,2,3.

5. The image processing method for measuring vehicle wheelbase according to claim 4, characterized in that: The steps 1-3 specifically include the following steps: Given a matrix H=A(R1 R2 T), using R1 and R2 as the first two columns of the external parameter matrix, there is a unit orthogonal relationship: Right now: Note A -T A -1 =B, and then solve the camera's intrinsic parameter matrix A through matrix B: in, , Respectively represent the square of the focal length fw of the camera in the x-axis and y-axis directions; Represents the skew coefficient in the camera's intrinsic parameter matrix K; obtain the camera's intrinsic parameter matrix A: By the formula (R1 R2 T) = A -1 H, obtain the external parameter matrix (R1 R2 T).

6. The image processing method for measuring vehicle wheelbase according to claim 5, characterized in that: The steps 1-4 specifically include the following steps: The formula for radial distortion is: in, is the horizontal coordinate of the ideal distortion-free normalized image, is the ordinate of the ideal distortion-free normalized image; represents the horizontal coordinate of the normalized image after distortion correction, represents the ordinate of the normalized image after distortion correction; r is the distance from the pixel to the center of the image; k1 represents the first-order radial distortion coefficient, and k2 represents the second-order radial distortion coefficient; The error between the actual position of the current camera with distortion correction in the pixel coordinate system and the fitted distortion curve. The smaller the reprojection error, the more consistent the fitted curve is with the actual distortion of the current camera, and the better the correction effect. At this point, the camera's distortion correction parameter calibration is completed and the corrected image is obtained.

7. The image processing method for measuring vehicle wheelbase according to claim 1, characterized in that: The step 2-1 specifically includes the following steps: The feature extraction network of the wheel contact point detection model consists of multiple convolutional layers, batch normalization layers, activation functions, and pooling layers. The output size of any convolutional layer is calculated as follows: Among them, W is the input of the convolutional layer, O is the output of the convolutional layer, K is the filter size, E is the padding, and S is the stride; The batch normalization layer processes batches of samples using the following formula: in, Represents the data after batch normalization; Represents the data before batch normalization; Represents the mean of the batch samples; Represents the variance of the batch samples; represents the stretching parameter; Indicates the offset parameter; The activation function uses the Silu activation function, the formula is f(x)=x*sigmoid(x), where x represents the sum of the weighted inputs of a neuron in the neural network plus the result of the bias term, sigmoid represents the sigmoid function, and sigmoid(x)=1 / (1+exp(-x)). When x is greater than 0, the Silu activation function will amplify x; when x is less than 0, the Silu activation function will reduce x. The pooling layer uses the maximum pooling method, and the formula is: Among them, output[m,n] represents a pixel value in the output feature map after pooling, m represents the horizontal coordinate of the pixel, and n represents the vertical coordinate of the pixel; input represents the input feature map; pool_size represents the window size of the pooling operation.

8. The image processing method for measuring vehicle wheelbase according to claim 1, characterized in that: The step 2-2 specifically includes the following steps: It is known that the length of the left and right boundary lines of the buffer parking space is Length, the coordinates of the ground contact point of the left front wheel are P1(x1,y1), and the coordinates of the two vertices of the left boundary line of the buffer parking space are A(x2,y2) and B(x3,y3). The distance P1L between the left front wheel and the left boundary line is calculated by the vector method. d Similarly, the distance P1F between the left front wheel and the front boundary line of the buffer parking space can be obtained d ; Similarly, the distance between the left rear wheel and the left boundary line is P2L d , the distance between the left rear wheel and the rear boundary line is P2B d ; Calculate the difference between the distance between the left front wheel and the left boundary line and the distance between the left rear wheel and the left boundary line, and get the difference Dx=|P1L of the left wheel in the left and right directions of the vehicle d -P2L d |, and calculate the difference between the left front wheel and the left rear wheel in the direction of the left boundary line, and get the difference Dy of the left wheel in the front and rear direction of the vehicle = |Length-P1F d -P2B d |; Take the square root of the sum of the squares of the differences Dx and Dy of the wheels on the same side to get the left wheelbase: Among them, C L is the left wheelbase, and the right wheelbase C is calculated in the same way. R ; Then for C L and C R Find the average value and finally get the vehicle wheelbase .

9. An image processing device for measuring vehicle wheelbase, used to implement the image processing method for measuring vehicle wheelbase according to any one of claims 1 to 8, characterized in that: include: An image acquisition module, used to acquire image data of the front and rear wheels of the vehicle in the buffer parking space; A distortion correction module, used for performing distortion correction on image data to obtain a corrected image; The wheel contact point detection module is used to identify the contact point data of the front and rear wheels of the vehicle in the corrected image, so as to calculate the vehicle wheelbase.

10. An image processing system for measuring vehicle wheelbase, characterized in that: include: at least one camera for collecting image data of the front and rear wheels of the vehicle; The image processing device for measuring vehicle wheelbase according to claim 9, used for processing image data and calculating the vehicle wheelbase; Display device, used to display measurement results.

Images