Vehicle matching method and system based on image recognition

Abstract

The present invention relates to the field of image processing, and particularly to a vehicle matching method and system based on image recognition. The method includes: acquiring continuous m frames of images, identifying target vehicles in each frame of image, performing corner detection on the target vehicles in the images, and marking the corner points of the target vehicles in each frame of image to obtain preprocessed images; performing super-resolution reconstruction on each frame of preprocessed image to obtain super-resolution reconstructed images; calculating corner key degrees according to the super-resolution reconstructed images, and the corner points with corner key degrees greater than a threshold are key motion points; acquiring the (m + 1)-th frame of image, predicting the range of the target vehicle in the (m + 1)-th frame of image based on the key motion points in the m-th frame of image, drawing a circle with the center of the predicted target vehicle range as the center and a radius of r to obtain a circular area, and matching all the vehicles within the circular area in the (m + 1)-th frame with the target vehicle in the m-th frame of image to obtain the target vehicle in the (m + 1)-th frame of image. The present invention effectively solves the problem of unable to accurately match vehicles.

Description

Technical Field

[0001] The present invention relates to the field of image processing. More specifically, the present invention relates to a vehicle matching method and system based on image recognition. Background Art

[0002] In an urban environment, due to the high similarity of vehicle types and appearances of the same model and color, when a vehicle passes through different locations at different times, phenomena such as license plate occlusion and vehicle appearance change may occur due to the influence of the external environment, resulting in the loss of the matching object and the inability to track the target vehicle. The prior art performs vehicle matching through motion trajectory tracking such as the optical flow method, inter-frame difference method, and background subtraction method.

[0003] However, since the traditional optical flow method relies on brightness consistency and small motion, and the dense optical flow of multiple vehicles may overlap, when facing a traffic road in a dynamic lighting environment, phenomena such as optical flow coverage, optical flow blank, or optical flow loss will occur, resulting in inaccurate vehicle matching.

[0004] Currently, the patent application document with the publication number CN103996292A and the name "A Moving Vehicle Tracking Method Based on Corner Matching" discloses a method for a video image obtained by a surveillance camera with a fixed field of view. The Harris algorithm is used to extract the corner points of the detection area, and the scale-invariant rotation feature descriptor of the corresponding corner points is calculated; then the corner points are matched according to the scale-invariant rotation feature descriptor, and then a trajectory is formed, and information such as the speed and direction of the trajectory is calculated; then the trajectory is dynamically grouped, and the rationality of the initial grouping result is judged to confirm the vehicle.

[0005] The above method uses corner points to track vehicles, but some corner points may not form a coherent trajectory after being occluded, and key motion points cannot be screened, so the vehicle position cannot be accurately predicted based on these corner points, and accurate vehicle matching cannot be performed. Summary of the Invention

[0006] To solve the above problem of inaccurate vehicle matching, the present invention provides solutions in the following aspects.

[0007] In a first aspect, the present invention provides a vehicle matching method based on image recognition, including: obtaining continuous m frames of images, recognizing target vehicles in each frame of image, performing corner detection on the target vehicles in each frame of image, and marking the corner points of the target vehicles in each frame of image to obtain preprocessed images; performing super-resolution reconstruction on each frame of preprocessed image to obtain at least one super-resolution reconstructed image; calculating the corner key degree according to the super-resolution reconstructed images, and taking the corner points with the corner key degree greater than the threshold as key motion points, where the corner key degree is positively correlated with the pixel gradient amplitude of the corner point in each super-resolution reconstructed image of each frame of preprocessed image and the color gradient amplitude of the corner point in each super-resolution reconstructed image of each frame of preprocessed image, and is negatively correlated with the absolute difference between the brightness value of the corner point in each frame of preprocessed image and the average brightness value of the corner point in all preprocessed images; collecting the (m + 1)-th frame of image, predicting the range of the target vehicle in the (m + 1)-th frame of image based on the key motion points of the target vehicle in the m-th frame of image, drawing a circle with the center of the predicted target vehicle range as the center and a radius of r to obtain a circular area, and matching all the vehicles within the circular area in the (m + 1)-th frame of image with the target vehicle in the m-th frame of image to obtain the target vehicle in the (m + 1)-th frame of image.

[0008] Super-resolution technology can enhance image details, making corner detection and vehicle boundary prediction more accurate, especially in low-resolution or complex scenarios. The prediction method based on key motion points can effectively capture the motion trend of the vehicle, so that the prediction of the target vehicle range is more in line with the actual motion trajectory. By combining multiple technologies such as refined image processing, super-resolution reconstruction, corner detection, and key point analysis, it provides strong support for the accurate range prediction of vehicles in images. Through accurate key motion point extraction and target detection correction, high-precision matching of target vehicles in complex environments can be achieved.

[0009] Preferably, the predicting the range of the target vehicle in the (m + 1)-th frame of image based on the key motion points of the target vehicle in the m-th frame of image includes:

[0010] Using the optical flow method to obtain the predicted positions of the key motion points of the target vehicle in the m-th frame of image in the (m + 1)-th frame of image, and the largest area covered by all the predicted positions is the predicted range of the target vehicle in the (m + 1)-th frame of image.

[0011] The optical flow method can capture the accurate displacement of each key point during the vehicle movement and provide fine motion estimation. Through the predicted positions of these key points, the actual position of the vehicle in the (m + 1)-th frame can be accurately inferred, and then the range of the vehicle can be predicted.

[0012] Preferably, the step of matching all vehicles within the circular region in the (m + 1)-th frame image with the target vehicle in the m-th frame image to obtain the target vehicle in the (m + 1)-th frame image includes: extracting the appearance features of each vehicle within the circular region in the (m + 1)-th frame image, and matching them with the appearance features of the target vehicle in the m-th frame image. The vehicle with the highest matching degree is the target vehicle in the (m + 1)-th frame image.

[0013] The method for predicting key motion points and matching appearance features of the target vehicle can efficiently and accurately track the target vehicle in consecutive frame images. By predicting the approximate position of the target vehicle and extracting possible candidate vehicles in the (m + 1)-th frame image based on this, and further performing precise identification through appearance feature matching, the accuracy, robustness, and real-time performance of target vehicle tracking can be improved.

[0014] Preferably, the step of matching all vehicles within the circular region in the (m + 1)-th frame image with the target vehicle in the m-th frame image to obtain the target vehicle in the (m + 1)-th frame image includes: extracting the image of each vehicle within the circular region in the (m + 1)-th frame image, inputting it and the target vehicle in the m-th frame image into a Siamese network to obtain a similarity score. The vehicle with the highest similarity score is the target vehicle in the (m + 1)-th frame image.

[0015] The method combining key motion point prediction of the target vehicle and Siamese network for appearance matching can provide high-precision and stable target vehicle recognition and tracking in dynamic and complex scenarios. By accurately predicting the range of the target vehicle and using the Siamese network for appearance feature matching, false detections, missed detections, and false matches can be effectively reduced, and the accuracy and robustness of target vehicle tracking can be improved.

[0016] Preferably, the step of performing super-resolution reconstruction on each preprocessed image to obtain at least one super-resolution reconstructed image includes: performing super-resolution reconstruction by upsampling each preprocessed image and then performing Gaussian filtering. After obtaining one super-resolution reconstructed image each time, calculate the reconstruction confidence of this super-resolution reconstructed image. When the reconstruction confidence is greater than the threshold, stop performing super-resolution reconstruction.

[0017] By performing super-resolution reconstruction on each preprocessed image through upsampling and Gaussian filtering, not only can the image resolution be effectively improved and more details be retained, but also the reconstruction process can be dynamically controlled through the reconstruction confidence mechanism, enhancing the computational efficiency and stability of the algorithm.

[0018] Preferably, the calculation of the reconstruction confidence of this super-resolution reconstructed image is specifically: , where represents the reconstruction confidence of this super-resolution reconstructed image, represents the standard deviation of the pixel values of the noise pixel points in the super-resolution reconstructed image, represents the standard deviation of the pixel values of the non-noise pixel points in the super-resolution reconstructed image, represents the edge density of the super-resolution reconstructed image.

[0019] By calculating the reconstruction confidence and combining the standard deviation of the noise pixels, the standard deviation of the non-noise pixels, and the edge density, the quality of the super-resolution reconstructed image can be effectively evaluated. This method provides a comprehensive evaluation mechanism for image quality, making the super-resolution process more efficient and adaptable. It can not only improve the clarity of the image but also save computing resources.

[0020] Preferably, the upsampling of each preprocessed image includes: performing upsampling on each preprocessed image using the Lanczos algorithm.

[0021] In a second aspect, the present invention also provides a vehicle matching system based on image recognition, including: a memory and a processor, where a computer program is stored on the memory, and the processor executes the computer program to implement the vehicle matching method based on image recognition described in any one of the above.

[0022] The beneficial effects of the present invention are as follows: The super-resolution technology can enhance image details, making corner detection and vehicle boundary prediction more accurate, especially in low-resolution or complex scenarios. The prediction method based on key motion points can effectively capture the motion trend of the vehicle, so that the prediction of the target vehicle range is more in line with the actual motion trajectory. By combining multiple technologies such as refined image processing, super-resolution reconstruction, corner detection, and key point analysis, strong support is provided for accurately predicting the range of the vehicle in the image. Through accurate extraction of key motion points and correction of target detection, high-precision matching of the target vehicle in a complex environment can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] By referring to the following detailed description with reference to the accompanying drawings, the above and other objects, features, and advantages of the exemplary embodiments of the present invention will become readily understandable. In the drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0024] Figure 1 is a flowchart of a vehicle matching method based on image recognition provided by an embodiment of the present invention;

[0025] Figure 2 is a block diagram of a vehicle matching system based on image recognition provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present invention.

[0027] The following will describe in detail the specific implementation manners of the present invention in conjunction with the accompanying drawings.

[0028] Figure 1 It is a flowchart of a vehicle matching method based on image recognition provided by an embodiment of the present invention, including the following steps:

[0029] S101. Obtain m consecutive frames of images, identify the target vehicle in each frame of image, and perform corner detection on the target vehicle in each frame of image, and mark the corners of the target vehicle in each frame of image to obtain a preprocessed image.

[0030] The obtaining of m consecutive frames of images can use a camera to shoot a video, and then extract each frame of image in the video as an image, or directly shoot m consecutive frames of images. Then, the vehicle feature vectors in different frames of images are extracted through the yolov5 algorithm, and the minimum cost matching is found by using the Kuhn-Munkras algorithm to obtain the target vehicle in each frame of image. In addition, the target vehicle in each frame of image can also be directly recognized by the human eye.

[0031] Corners are local feature points in an image. By detecting the corners on the target vehicle, the tracking of the target vehicle can be realized. Common corner detection methods include corner detection and corner detection methods. After detecting and marking the corners of the target vehicle in each frame of image, a preprocessed image is obtained. It should be noted that corner detection and corner detection methods both belong to well-known technologies commonly used in the field of image processing and will not be elaborated here.

[0032] S102. Perform super-resolution reconstruction on each frame of the preprocessed image to obtain at least one super-resolution reconstructed image.

[0033] Super-resolution reconstructed image is a technology for generating high-resolution images from low-resolution images. The super-resolution technology aims to restore the details and textures in the image to improve the image quality and make it clearer and more delicate. Generally, the method of interpolation can be used to perform multi-scale super-resolution reconstruction on the image, and a super-resolution reconstructed image can be obtained for each scale. For example, by different magnification factors ( etc.), multiple scales of super-resolution reconstructed images are generated, and these images will vary in resolution and detail restoration.

[0034] In some embodiments, performing super-resolution reconstruction on each frame of the preprocessed image to obtain at least one super-resolution reconstructed image specifically includes: performing super-resolution reconstruction by upsampling each frame of the preprocessed image and then performing Gaussian filtering. After obtaining one super-resolution reconstructed image each time, calculate the reconstruction confidence of the super-resolution reconstructed image. When the reconstruction confidence is greater than the threshold, stop performing super-resolution reconstruction. Since interpolation is used to perform super-resolution reconstruction on the image, noise is inevitably introduced or amplified. The more interpolation is performed, the greater the noise. That is, the higher the magnification factor of the super-resolution reconstructed image, the greater the noise. When the noise reaches a certain level, the corresponding super-resolution reconstructed image will be quite different from the image, and thus it will no longer be credible. Therefore, the reconstruction confidence of the corresponding super-resolution reconstructed image needs to be calculated after each super-resolution reconstruction. In addition, the present invention generally uses an algorithm to upsample each frame of the preprocessed image, and other upsampling algorithms can also be used, which are not limited herein.

[0035] In some embodiments, calculating the reconstruction confidence of the super-resolution reconstructed image specifically includes: , where represents the reconstruction confidence of the super-resolution reconstructed image, represents the standard deviation of the pixel values of the noise pixel points in the super-resolution reconstructed image, represents the standard deviation of the pixel values of the non-noise pixel points in the super-resolution reconstructed image, represents the edge density of the super-resolution reconstructed image.

[0036] S103. Calculate the corner key degree according to the super-resolution reconstructed image, and use the corners with the corner key degree greater than the threshold as the key motion points.

[0037] In the actual road conditions, there may be situations of occlusion or light change, resulting in some corner recognition errors or omissions. Therefore, it is necessary to find the key corners on the target vehicle as the key motion points. The present invention determines whether a corner is a key motion point by calculating the key degree of the corner. The corner key degree is positively correlated with the pixel gradient of the corner in each super-resolution reconstructed image of each frame of the preprocessed image and the color gradient of the corner in each super-resolution reconstructed image of each frame of the preprocessed image, and is inversely correlated with the absolute difference between the brightness value of the corner in each frame of the preprocessed image and the average brightness value of the corner in all preprocessed images. Specifically, the present invention provides a key degree calculation formula, and the key degree , where is the key degree of the i-th corner, is normalization, m is the total number of frames of the preprocessed image, is the k-th frame of the preprocessed image, is the average brightness of the i-th corner point in all preprocessed images, represents the brightness value of the i-th corner point in the k-th preprocessed image, represents the j-th super-resolution reconstructed image of the k-th preprocessed image, represents the number of super-resolution reconstructed images of the k-th preprocessed image. For example, when the k-th preprocessed image undergoes the third super-resolution reconstruction and its reconstruction confidence is greater than the preset threshold, its first two super-resolution reconstructed images are reliable. Therefore, , represents the pixel gradient magnitude of the i-th corner point in the j-th super-resolution reconstructed image of the preprocessed image of the k-th frame, represents the color gradient magnitude of the i-th corner point in the j-th super-resolution reconstructed image of the preprocessed image of the k-th frame. In the above method, it is also necessary to identify whether the corner points in each frame of the preprocessed image are the same corner point and number different corner points. For example, in the first frame of the preprocessed image, corner points a, b, c, and d are detected, and in the second frame, corner points x, y, z, and w are detected. By using feature matching methods such as SIFT, SURF, or ORB, it is identified that a and x, b and y, c and z are the same corner points. Then, a and x, b and y, c and z can be respectively marked as corner points 1, 2, and 3, d is marked as corner point 4, and w is marked as corner point 5. Since there is no corner point 5 in the first frame of the preprocessed image, the brightness value, pixel gradient magnitude, and color gradient magnitude of corner point 5 in this frame of the preprocessed image and all its super-resolution reconstructed images are set to the lowest value. Exemplarily, the pixel gradient magnitude , where, is the pixel gradient of corner point i in the x direction in this super-resolution reconstructed image, is the pixel gradient of corner point i in the y direction, and the color gradient magnitude , where, and are respectively the gradient magnitudes of corner point i in color channels and color channels in this super-resolution reconstructed image.

[0038] S104. Collect the (m + 1)-th frame image, predict the range of the target vehicle in the (m + 1)-th frame image based on the key motion points of the target vehicle in the m-th frame image. Take the center of the predicted target vehicle range as the center of a circle, draw a circle with radius r to obtain a circular area, and match all the vehicles within this circular area in the (m + 1)-th frame image with the target vehicle in the m-th frame image to obtain the target vehicle in the (m + 1)-th frame image.

[0039] In some embodiments, predicting the target vehicle range in the m-th frame image based on key motion points includes: obtaining the predicted positions of the key motion points of the target vehicle in the m-th frame image in the (m + 1)-th frame image by using the optical flow method, and the maximum area covered by all the predicted positions is the predicted target vehicle range in the (m + 1)-th frame image. By calculating the pixel changes between the m-th and (m + 1)-th frame images, the optical flow method can calculate the motion components of the key motion points in the horizontal and vertical directions, and then multiply them by the time difference between the two frames to obtain the position changes of the key motion points in the (m + 1)-th frame image relative to the m-th frame. The specific optical flow method is: constructing an adaptive optical flow equation and an optical flow cost function for the m-th frame image based on the brightness consistency and key degree of the key motion points in time: , where is the key degree of the a-th key motion point, , , are the partial derivatives of the brightness of this key motion point in the m-th frame with respect to the spatial coordinates , and time respectively, and are the optical flow components of this key motion point in and directions respectively. is the average error between the optical flow component solved by the optical flow equation for all frames before the m-th frame and the actual optical flow component of this key motion point, is the error between the optical flow component solved by the optical flow equation for the (m - 1)-th frame and the actual optical flow component of this key motion point, is the mean square of the gradient of the brightness of this key motion point in time before the m-th frame, and are the derivatives of the optical flow in and directions respectively, and are the second-order derivatives of this key motion point in and directions respectively. From the above equations, the optical flow components and of this key motion point in and directions can be calculated, and then using to calculate the displacements of this key motion point in the x and y directions in the (m + 1)-th frame relative to the m-th frame, is the time difference between collecting the (m + 1)-th and m-th frame images.

[0040] After obtaining the target vehicle range in the (m + 1)-th frame image, since there may be an error between the predicted target vehicle range and the actual vehicle position, it is also necessary to combine the vehicles within the target vehicle range in the predicted (m + 1)-th frame image and the target vehicle in the m-th frame image to obtain the target vehicle in the (m + 1)-th frame image. Specifically, it includes: taking the center of the predicted target vehicle range as the center of a circle, drawing a circle with a radius r to obtain a circular area, extracting the appearance features of each vehicle within this circular area in the (m + 1)-th frame image, and matching them with the appearance features of the target vehicle in the m-th frame image. The vehicle with the highest matching degree is the target vehicle in the (m + 1)-th frame image. In another embodiment, it is also possible to take the center of the target vehicle range as the center of a circle, draw a circle with a radius r, extract the images of each vehicle within this circular area in the (m + 1)-th frame image, input them together with the target vehicle in the m-th frame image into a Siamese network, obtain a similarity score, and the vehicle with the highest similarity score is the target vehicle in the (m + 1)-th frame image.

[0041] In some embodiments, taking the center of the predicted target vehicle range as the center of a circle and drawing a circle with a radius r, where r = , where v represents the driving speed of the target vehicle, l represents the optical flow prediction error, is the time difference between collecting the (m + 1)-th and m-th frame images. The driving speed of the target vehicle can be calculated based on the positions of the target vehicle in the previous two frame images and the time difference of collecting the images. The optical flow prediction error can be replaced by the mean value of the predicted optical flow vectors of multiple pixel points and the true optical flow vectors of the corresponding pixel points in the (m + 1)-th frame.

[0042] The present invention calibrates the target vehicle features through corner points, then screens out possible individual corner point recognition errors or occlusion situations by calculating the corner point criticality, predicts the approximate range of the target vehicle in the (m + 1)-th frame based on the key motion points, and finally accurately matches the target vehicle by comparing the vehicle features within this range in the (m + 1)-th frame image with the target vehicle features in the previous frame.

[0043] The present invention also provides a vehicle matching system based on image recognition. As Figure 2 shown, the system includes a processor and a memory. The memory stores computer program instructions, and when the computer program instructions are executed by the processor, the vehicle matching method based on image recognition according to the present invention is implemented.

[0044] In the description of this specification, the meanings of "a plurality of" and "several" are at least two, such as two, three or more, etc., unless otherwise specifically defined.

[0045] Although this specification has shown and described multiple embodiments of the present invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many variations, modifications, and alternative means will occur to those skilled in the art without departing from the spirit and scope of the present invention. It should be understood that various alternatives to the embodiments of the present invention described herein may be employed in practicing the present invention.

Claims

1. A vehicle matching method based on image recognition, characterized in that: include: Obtaining m consecutive frames of images, identifying the target vehicle in each frame, performing corner point detection on the target vehicle in each frame, marking the corner points of the target vehicle in each frame to obtain a preprocessed image; Performing super-resolution reconstruction on each frame of the pre-processed image to obtain at least one super-resolution reconstructed image, including: performing super-resolution reconstruction by upsampling each frame of the pre-processed image and then performing Gaussian filtering, and calculating a reconstruction confidence of the super-resolution reconstructed image after obtaining a super-resolution reconstructed image each time, and stopping the super-resolution reconstruction when the reconstruction confidence is greater than a threshold; Calculate the corner criticality based on the super-resolution reconstructed image and satisfy the relationship: , is the criticality of the i-th corner point, is normalization, m is the total number of preprocessed image frames, is the k-th frame preprocessed image, is the average brightness of the i-th corner point in all preprocessed images, represents the brightness value of the i-th corner point in the k-th frame preprocessed image, represents the jth super-resolution reconstructed image of the kth pre-processed image, represents the number of super-resolution reconstructed images of the k-th pre-processed image, represents the pixel gradient amplitude of the i-th corner point in the j-th super-resolution reconstructed image of the preprocessed image of the k-th frame image, represents the color gradient amplitude of the i-th corner point in the j-th super-resolution reconstructed image of the preprocessed image of the k-th frame image; The corner points whose keyness is greater than the threshold are regarded as key motion points; The m+1th frame image is acquired, and the target vehicle range in the m+1th frame image is predicted based on the key motion points of the target vehicle in the m-th frame image. A circular area is obtained by drawing a circle with a radius r with the center of the predicted target vehicle range as the center of the circle. All vehicles in the circular area in the m+1th frame image are matched with the target vehicle in the m-th frame image to obtain the target vehicle in the m+1th frame image.

2. The vehicle matching method based on image recognition according to claim 1, characterized in that: The method of predicting the range of the target vehicle in the m+1th frame image based on the key motion points of the target vehicle in the mth frame image includes: The optical flow method is used to obtain the predicted position of the key motion points of the target vehicle in the mth frame image in the m+1th frame image. The maximum area covered by all predicted positions is the predicted range of the target vehicle in the m+1th frame image.

3. The vehicle matching method based on image recognition according to claim 1, characterized in that: The step of matching all vehicles in the circular area in the m+1th frame image with the target vehicle in the mth frame image to obtain the target vehicle in the m+1th frame image comprises: The appearance features of each vehicle in the circular area in the m+1th frame image are extracted and matched with the appearance features of the target vehicle in the mth frame image. The vehicle with the highest matching degree is the target vehicle in the m+1th frame image.

4. The vehicle matching method based on image recognition according to claim 1, characterized in that: The step of matching all vehicles in the circular area in the m+1th frame image with the target vehicle in the mth frame image to obtain the target vehicle in the m+1th frame image comprises: The image of each vehicle in the circular area in the m+1th frame image is extracted, and the image and the target vehicle in the mth frame image are input into the Siamese network to obtain a similarity score. The vehicle with the highest similarity score is the target vehicle in the m+1th frame image.

5. The vehicle matching method based on image recognition according to claim 1, characterized in that: The calculation of the reconstruction confidence of the super-resolution reconstructed image is specifically as follows: ,in, Represents the reconstruction confidence of the super-resolution reconstructed image, It represents the standard deviation of the pixel values ​​of the noise pixels in the super-resolution reconstructed image. It represents the standard deviation of the pixel values ​​of non-noise pixels in the super-resolution reconstructed image. Represents the edge density of the super-resolution reconstructed image.

6. The vehicle matching method based on image recognition according to claim 1, characterized in that: The up-sampling of each frame of the pre-processed image includes: up-sampling each frame of the pre-processed image using a Lanczos algorithm.

7. A vehicle matching system based on image recognition, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the vehicle matching method based on image recognition as described in any one of claims 1 to 6.

Images