Vehicle matching method and system based on image recognition

Abstract

The invention relates to the field of image processing, in particular to a vehicle matching method and system based on image recognition. The method comprises the following steps: acquiring m continuous frames of images, identifying a target vehicle in each frame of image, performing corner detection on the target vehicle in the image, and marking the corner of the target vehicle in each frame of image to obtain a preprocessed image; performing super-resolution reconstruction on each frame of preprocessed image to obtain a super-resolution reconstructed image; calculating the corner criticality according to the super-resolution reconstructed image, wherein the corner with the corner criticality greater than a threshold value is a key motion point; and acquiring an (m + 1) th frame of image, predicting a target vehicle range in the (m + 1) th frame of image based on the key motion points in the mth frame of image, taking the center of the predicted target vehicle range as a circle center, drawing a circle with a radius r to obtain a circular region, and matching all vehicles in the circular region in the (m + 1) th frame with the target vehicle in the mth frame of image to obtain a target vehicle in the (m + 1) th frame of image. The problem that the vehicle cannot be accurately matched is effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and more specifically, to a vehicle matching method and system based on image recognition. Background Art

[0002] In an urban environment, since vehicles of the same model and color are highly similar in type and appearance, when a vehicle passes through different locations at different times, the license plate may be blocked or the vehicle appearance may change due to the influence of the external environment, resulting in the loss of the matching object and the inability to track the target vehicle. Existing technologies perform vehicle matching through motion trajectory tracking such as optical flow method, inter-frame difference method, and background subtraction method.

[0003] However, since traditional optical flow methods rely on brightness consistency and small motion, and the dense optical flows of multiple vehicles may overlap, optical flow coverage, optical flow gaps or optical flow loss may occur on traffic roads in dynamic lighting environments, resulting in inaccurate vehicle matching.

[0004] At present, the patent application document with publication number CN103996292A and name “A Moving Vehicle Tracking Method Based on Corner Point Matching” discloses a video image acquired by a surveillance camera with a fixed field of view, using the Harris algorithm to extract the corner points of the detection area, and calculating the scale-invariant rotation feature descriptor of the corresponding corner points; then the corner points are matched according to the scale-invariant rotation feature descriptor to form a trajectory, and the speed, direction and other information of the trajectory are calculated; then the trajectory is dynamically grouped, and the rationality of the initial grouping results is judged to confirm that it is a vehicle.

[0005] The above method uses corner points to track vehicles, but some corner points may not form a coherent trajectory after being blocked, and key motion points cannot be screened, so the vehicle position cannot be accurately predicted based on these corner points, and the vehicle cannot be accurately matched. Summary of the invention

[0006] In order to solve the above-mentioned problem of being unable to accurately match the vehicle, the present invention provides solutions in the following aspects.

[0007] In a first aspect, the present invention provides a vehicle matching method based on image recognition, comprising: acquiring m consecutive frames of images, identifying a target vehicle in each frame of the image, performing corner point detection on the target vehicle in each frame of the image, marking the corner points of the target vehicle in each frame of the image to obtain a preprocessed image; performing super-resolution reconstruction on each preprocessed image to obtain at least one super-resolution reconstructed image; calculating the corner point criticality based on the super-resolution reconstructed image, taking the corner point whose corner point criticality is greater than a threshold as a key motion point, and comparing the corner point criticality with the pixel gradient amplitude of the corner point in each super-resolution reconstructed image of each preprocessed image. , the color gradient amplitude of the corner point in each super-resolution reconstructed image of each frame of the preprocessed image is positively correlated, and is negatively correlated with the absolute difference between the brightness value of the corner point in each frame of the preprocessed image and the average brightness value of the corner point in all preprocessed images; collect the m+1th frame image, predict the target vehicle range in the m+1th frame image based on the key motion points of the target vehicle in the mth frame image, use the center of the predicted target vehicle range as the center of the circle, draw a circle with a radius r to obtain a circular area, match all vehicles in the circular area in the m+1th frame image with the target vehicle in the mth frame image to obtain the target vehicle in the m+1th frame image.

[0008] Super-resolution technology can enhance image details, making corner detection and vehicle boundary prediction more accurate, especially in low-resolution or complex scenes. The prediction method based on key motion points can effectively capture the movement trend of the vehicle, so that the prediction of the target vehicle range is more consistent with the actual motion trajectory. Through the combination of refined image processing, super-resolution reconstruction, corner detection and key point analysis, it provides strong support for the accurate range prediction of vehicles in images. Through accurate key motion point extraction and target detection correction, high-precision matching of target vehicles in complex environments can be achieved.

[0009] Preferably, the predicting of the target vehicle range in the m+1th frame image based on the key motion points of the target vehicle in the mth frame image includes: The optical flow method is used to obtain the predicted position of the key motion points of the target vehicle in the mth frame image in the m+1th frame image. The maximum area covered by all predicted positions is the predicted range of the target vehicle in the m+1th frame image.

[0010] The optical flow method can capture the accurate displacement of each key point during the vehicle's motion and provide a fine motion estimate. Through the predicted positions of these key points, the actual position of the vehicle in the m+1th frame can be inferred with high accuracy, and then the range of the vehicle can be predicted.

[0011] Preferably, the method of matching all vehicles in the circular area in the m+1th frame image with the target vehicle in the m-th frame image to obtain the target vehicle in the m+1th frame image includes: extracting the appearance features of each vehicle in the circular area in the m+1th frame image, and matching them with the appearance features of the target vehicle in the m-th frame image, and the vehicle with the highest matching degree is the target vehicle in the m+1th frame image.

[0012] The method of predicting the key motion points of the target vehicle and matching the appearance features can efficiently and accurately track the target vehicle in continuous frame images. By predicting the approximate position of the target vehicle and extracting the possible candidate vehicles in the m+1th frame image on this basis, and further accurately identifying them through appearance feature matching, the accuracy, robustness and real-time performance of target vehicle tracking can be improved.

[0013] Preferably, the step of matching all vehicles in the circular area in the m+1th frame image with the target vehicle in the m-th frame image to obtain the target vehicle in the m+1th frame image comprises: extracting the image of each vehicle in the circular area in the m+1th frame image, inputting the image and the target vehicle in the m-th frame image into a Siamese network to obtain a similarity score, wherein the vehicle with the highest similarity score is the target vehicle in the m+1th frame image.

[0014] The method combining the prediction of key motion points of the target vehicle and the appearance matching of the Siamese network can provide high-precision and stable target vehicle recognition and tracking in dynamic and complex scenes. By accurately predicting the range of the target vehicle and using the Siamese network for appearance feature matching, it can effectively reduce false detections, missed detections and false matches, and improve the accuracy and robustness of target vehicle tracking.

[0015] Preferably, the super-resolution reconstruction is performed on each frame of the pre-processed image to obtain at least one super-resolution reconstructed image, including: super-resolution reconstruction is performed by upsampling each frame of the pre-processed image and then Gaussian filtering, and each time a super-resolution reconstructed image is obtained, a reconstruction confidence of the super-resolution reconstructed image is calculated, and when the reconstruction confidence is greater than a threshold, the super-resolution reconstruction is stopped.

[0016] By upsampling and Gaussian filtering each frame of the pre-processed image for super-resolution reconstruction, not only can the image resolution be effectively improved and more details be retained, but the reconstruction process can also be dynamically controlled through the reconstruction confidence mechanism, thereby improving the computational efficiency and stability of the algorithm.

[0017] Preferably, the calculation of the reconstruction confidence of the super-resolution reconstructed image is specifically as follows: ,in, Represents the reconstruction confidence of the super-resolution reconstructed image, It represents the standard deviation of the pixel values ​​of the noise pixels in the super-resolution reconstructed image. It represents the standard deviation of the pixel values ​​of non-noise pixels in the super-resolution reconstructed image. Represents the edge density of the super-resolution reconstructed image.

[0018] By calculating the reconstruction confidence, combined with the standard deviation of noise pixels, the standard deviation of non-noise pixels and edge density, the quality of super-resolution reconstructed images can be effectively evaluated. This method provides a comprehensive evaluation mechanism for image quality, making the super-resolution process more efficient and adaptive. It not only improves the clarity of the image, but also saves computing resources.

[0019] Preferably, the upsampling of each frame of the pre-processed image comprises: up-sampling each frame of the pre-processed image using a Lanczos algorithm.

[0020] In a second aspect, the present invention further provides a vehicle matching system based on image recognition, comprising: a memory and a processor, wherein a computer program is stored in the memory, and the processor executes the computer program to implement any of the above-mentioned vehicle matching methods based on image recognition.

[0021] The beneficial effects of the present invention are as follows: super-resolution technology can enhance image details, making corner point detection and vehicle boundary prediction more accurate, especially in low-resolution or complex scenes. The prediction method based on key motion points can effectively capture the movement trend of the vehicle, so that the prediction of the target vehicle range is more consistent with the actual motion trajectory. By combining multiple technologies such as refined image processing, super-resolution reconstruction, corner point detection and key point analysis, strong support is provided for the accurate range prediction of the vehicle in the image. Through accurate key motion point extraction and target detection correction, high-precision matching of the target vehicle in a complex environment can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] By reading the following detailed description with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein: Figure 1 is a flow chart of a vehicle matching method based on image recognition provided by an embodiment of the present invention; Figure 2 The present invention provides a vehicle matching system based on image recognition. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0024] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0025] Figure 1 The flowchart of a vehicle matching method based on image recognition provided by an embodiment of the present invention includes the following steps: S101, acquiring m consecutive image frames, identifying a target vehicle in each image frame, performing corner point detection on the target vehicle in each image frame, marking the corner points of the target vehicle in each image frame to obtain a preprocessed image.

[0026] The acquisition of the continuous m-frame images may be performed by shooting a video with a camera, and then extracting each frame of the video as an image, or directly shooting the continuous m-frame images. Then, the vehicle feature vectors in different frames are extracted by the Yolov5 algorithm, and then the minimum cost matching is found by the Kuhn-Munkras algorithm to obtain the target vehicle in each frame. In addition, the target vehicle in each frame may be directly identified by human eyes.

[0027] Corner points are local feature points in an image. By detecting the corner points on a target vehicle, the target vehicle can be tracked. Common corner point detection methods include: Corner detection and The corner point detection method detects the corner points of the target vehicle in each frame and marks them to obtain a preprocessed image. Corner detection and Corner point detection methods are all well-known technologies commonly used in the field of image processing and will not be described in detail here.

[0028] S102 , performing super-resolution reconstruction on each frame of the pre-processed image to obtain at least one super-resolution reconstructed image.

[0029] Super-resolution image reconstruction is a technique for generating high-resolution images from low-resolution images. Super-resolution technology aims to restore the details and textures in the image to improve the image quality and make it clearer and more delicate. Generally, the interpolation method can be used to perform multi-scale super-resolution reconstruction of the image, and a super-resolution reconstructed image can be obtained at each scale. For example, by using different magnifications ( etc.), generate super-resolution reconstructed images of various scales, which will differ in resolution and detail restoration.

[0030] In some embodiments, the super-resolution reconstruction is performed on each frame of the pre-processed image to obtain at least one super-resolution reconstructed image, specifically: the super-resolution reconstruction is performed by upsampling each frame of the pre-processed image and then Gaussian filtering. After each super-resolution reconstructed image is obtained, the reconstruction confidence of the super-resolution reconstructed image is calculated. When the reconstruction confidence is greater than a threshold, the super-resolution reconstruction is stopped. Since the interpolation method is used to perform super-resolution reconstruction on an image, noise will inevitably be introduced or amplified. The more interpolations are performed, the greater the noise. That is, the higher the magnification of the super-resolution reconstructed image, the greater the noise. When the noise reaches a certain level, the corresponding super-resolution reconstructed image will be greatly different from the image and will no longer be credible. Therefore, the reconstruction confidence of the corresponding super-resolution reconstructed image must be calculated after each super-resolution reconstruction. In addition, the present invention generally adopts The algorithm upsamples each frame of the pre-processed image, and other upsampling algorithms may also be used, which are not limited here.

[0031] In some embodiments, the calculation of the reconstruction confidence of the super-resolution reconstructed image is specifically as follows: ,in, Represents the reconstruction confidence of the super-resolution reconstructed image, It represents the standard deviation of the pixel values ​​of the noise pixels in the super-resolution reconstructed image. It represents the standard deviation of the pixel values ​​of non-noise pixels in the super-resolution reconstructed image. Represents the edge density of the super-resolution reconstructed image.

[0032] S103, calculating the corner point keyness according to the super-resolution reconstructed image, and taking the corner points whose corner point keyness is greater than a threshold as key motion points.

[0033] In actual road conditions, there may be occlusions or changes in illumination, which may cause some corner points to be misidentified or missing, so it is necessary to find key corner points on the target vehicle as key motion points. The present invention determines whether a corner point is a key motion point by calculating the criticality of the corner point, and the corner point criticality is positively correlated with the pixel gradient of the corner point in each super-resolution reconstructed image of each frame of preprocessed image and the color gradient of the corner point in each super-resolution reconstructed image of each frame of preprocessed image, and is negatively correlated with the absolute difference between the brightness value of the corner point in each frame of preprocessed image and the average brightness value of the corner point in all preprocessed images. Specifically, the present invention provides a criticality calculation formula, and the criticality ,in, is the criticality of the i-th corner point, is normalization, m is the total number of preprocessed image frames, is the k-th preprocessed image, is the average brightness of the i-th corner point in all preprocessed images, represents the brightness value of the i-th corner point in the k-th frame preprocessed image, represents the jth super-resolution reconstructed image of the kth pre-processed image, Indicates the number of super-resolution reconstructed images of the k-th frame pre-processed image. For example, when the k-th frame pre-processed image is reconstructed for the third time with super-resolution, its reconstruction confidence is greater than the preset threshold, and its first two super-resolution reconstructed images are credible. , represents the pixel gradient amplitude of the i-th corner point in the j-th super-resolution reconstructed image of the preprocessed image of the k-th frame image, Represents the color gradient amplitude of the ith corner point in the jth super-resolution reconstructed image of the preprocessed image of the kth frame image. In the above method, it is also necessary to identify whether the corner points in each frame of the preprocessed image are the same corner points, and to number different corner points. For example, corner points a, b, c, and d are detected in the first frame of the preprocessed image, and corner points x, y, z, and w are detected in the second frame. Through feature matching methods such as SIFT, SURF, or ORB, it is identified that a and x, b and y, c and z are the same corner points. Then, a and x, b and y, c and z can be marked as corner points 1, 2, and 3 respectively, d is marked as corner point 4, and w is marked as corner point 5. Since there is no corner point 5 in the first frame of the preprocessed image, the brightness value, pixel gradient amplitude, and color gradient amplitude of corner point 5 in the preprocessed image of this frame and all its super-resolution reconstructed images are set to the lowest value. Exemplarily, the pixel gradient amplitude ,in, is the pixel gradient of corner point i in the x direction in the super-resolution reconstructed image, is the pixel gradient of corner point i in the y direction, the color gradient amplitude ,in, and They are respectively the color channels of the corner point i in the super-resolution reconstructed image Gradient magnitude and color channels The gradient amplitude.

[0034] S104, capturing the m+1th frame image, predicting the target vehicle range in the m+1th frame image based on the key motion points of the target vehicle in the m-th frame image, taking the center of the predicted target vehicle range as the center of a circle, drawing a circle with a radius r to obtain a circular area, matching all vehicles in the circular area in the m+1th frame image with the target vehicle in the m-th frame image to obtain the target vehicle in the m+1th frame image.

[0035] In some embodiments, the prediction of the target vehicle range in the m-th frame image based on the key motion points includes: using the optical flow method to obtain the predicted position of the key motion points of the target vehicle in the m-th frame image in the m+1-th frame image, and the maximum area covered by all predicted positions is the predicted target vehicle range in the m+1-th frame image. The optical flow method can calculate the motion components of the key motion points in the horizontal and vertical directions by calculating the pixel changes between the m-th frame and the m+1-th frame image, and then multiply it by the time difference between the two frames to obtain the position change of the key motion points in the m+1-th frame image relative to the m-th frame. The specific optical flow method is: constructing an adaptive optical flow equation and optical flow cost function for the m-th frame image based on the brightness consistency and criticality of the key motion points in time: ,in, is the criticality of the ath key motion point, , , are the brightness of the key motion point in the mth frame versus the spatial coordinates , and time The partial derivative of and The key motion points are and The optical flow component in the direction. is the average error between the optical flow component solved by the optical flow equation of the key motion point in all frames before the mth frame and the actual optical flow component, is the error between the optical flow component solved by the optical flow equation of the key motion point in the m-1th frame and the actual optical flow component, is the square mean of the brightness of the key motion point before the mth frame, and The optical flow is and The derivative in the direction, and The key motion points are and The second-order derivative in the direction. From the above equations, we can calculate the key motion point and Optical flow component in the direction and , reuse Calculate the displacement of the key motion point in the m+1th frame relative to the mth frame in the x and y directions. is the time difference between capturing the m+1th frame and the mth frame image.

[0036] After obtaining the target vehicle range in the m+1th frame image, since there may be an error between the predicted target vehicle range and the actual vehicle position, it is also necessary to combine the predicted vehicles within the target vehicle range in the m+1th frame image and the target vehicle in the mth frame image to obtain the target vehicle in the m+1th frame image, specifically including: taking the center of the predicted target vehicle range as the center of a circle, drawing a circle with a radius of r to obtain a circular area, extracting the appearance features of each vehicle in the circular area in the m+1th frame image, and matching them with the appearance features of the target vehicle in the mth frame image, and the vehicle with the highest matching degree is the target vehicle in the m+1th frame image. In another embodiment, the center of the target vehicle range can also be taken as the center of a circle, drawing a circle with a radius of r, extracting the image of each vehicle in the circular area in the m+1th frame image, and inputting the image and the target vehicle in the mth frame image into a Siamese network to obtain a similarity score, and the vehicle with the highest similarity score is the target vehicle in the m+1th frame image.

[0037] In some embodiments, the center of the predicted target vehicle range is used as the center of the circle and the radius is r, where r= , where v represents the target vehicle speed, l represents the optical flow prediction error, In order to collect the time difference between the m+1th frame and the mth frame image, the target vehicle's speed can be calculated based on the position of the target vehicle in the previous two frames and the time difference between the collected images. The optical flow prediction error can be replaced by the average of the predicted optical flow vectors of multiple pixel points and the actual optical flow vectors of the corresponding pixel points in the m+1 frame.

[0038] The present invention calibrates the target vehicle features through corner points, and then screens out possible individual corner point recognition errors or occlusions by calculating the corner point criticality, predicts the approximate range of the target vehicle in the m+1th frame according to the key motion points, and finally accurately matches the target vehicle by comparing the vehicle features in this range in the m+1th frame image with the target vehicle features in the previous frame.

[0039] The present invention also provides a vehicle matching system based on image recognition. Figure 2 As shown, the system includes a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, a vehicle matching method based on image recognition described in the present invention is implemented.

[0040] In the description of this specification, "plurality" or "several" means at least two, such as two, three or more, etc., unless otherwise clearly and specifically defined.

[0041] Although this specification has shown and described a number of embodiments of the present invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art will conceive of many modifications, changes and alternatives without departing from the ideas and spirit of the present invention. It should be understood that in the practice of the present invention, various alternatives to the embodiments of the present invention described herein may be employed.

Claims

1. A vehicle matching method based on image recognition, characterized in that: include: Obtaining m consecutive frames of images, identifying the target vehicle in each frame, performing corner point detection on the target vehicle in each frame, marking the corner points of the target vehicle in each frame to obtain a preprocessed image; Performing super-resolution reconstruction on each frame of the pre-processed image to obtain at least one super-resolution reconstructed image; Calculating the key of the corner point according to the super-resolution reconstructed image, taking the corner point whose key of the corner point is greater than a threshold as a key motion point, wherein the key of the corner point is positively correlated with the pixel gradient amplitude of the corner point in each super-resolution reconstructed image of each frame of the preprocessed image and the color gradient amplitude of the corner point in each super-resolution reconstructed image of each frame of the preprocessed image, and is negatively correlated with the absolute difference between the brightness value of the corner point in each frame of the preprocessed image and the average brightness value of the corner point in all preprocessed images; The m+1th frame image is acquired, and the target vehicle range in the m+1th frame image is predicted based on the key motion points of the target vehicle in the m-th frame image. A circular area is obtained by drawing a circle with a radius r with the center of the predicted target vehicle range as the center of the circle. All vehicles in the circular area in the m+1th frame image are matched with the target vehicle in the m-th frame image to obtain the target vehicle in the m+1th frame image.

2. The vehicle matching method based on image recognition according to claim 1, characterized in that: The method of predicting the range of the target vehicle in the m+1th frame image based on the key motion points of the target vehicle in the mth frame image includes: The optical flow method is used to obtain the predicted position of the key motion points of the target vehicle in the mth frame image in the m+1th frame image. The maximum area covered by all predicted positions is the predicted range of the target vehicle in the m+1th frame image.

3. The vehicle matching method based on image recognition according to claim 1, characterized in that: The step of matching all vehicles in the circular area in the m+1th frame image with the target vehicle in the mth frame image to obtain the target vehicle in the m+1th frame image comprises: The appearance features of each vehicle in the circular area in the m+1th frame image are extracted and matched with the appearance features of the target vehicle in the mth frame image. The vehicle with the highest matching degree is the target vehicle in the m+1th frame image.

4. The vehicle matching method based on image recognition according to claim 1, characterized in that: The step of matching all vehicles in the circular area in the m+1th frame image with the target vehicle in the mth frame image to obtain the target vehicle in the m+1th frame image comprises: The image of each vehicle in the circular area in the m+1th frame image is extracted, and the image and the target vehicle in the mth frame image are input into the Siamese network to obtain a similarity score. The vehicle with the highest similarity score is the target vehicle in the m+1th frame image.

5. The vehicle matching method based on image recognition according to claim 1, characterized in that: The super-resolution reconstruction is performed on each frame of the pre-processed image to obtain at least one super-resolution reconstructed image, including: Super-resolution reconstruction is performed by upsampling each frame of the pre-processed image and then performing Gaussian filtering. After each super-resolution reconstructed image is obtained, the reconstruction confidence of the super-resolution reconstructed image is calculated. When the reconstruction confidence is greater than a threshold, the super-resolution reconstruction is stopped.

6. The vehicle matching method based on image recognition according to claim 5, characterized in that: The calculation of the reconstruction confidence of the super-resolution reconstructed image is specifically as follows: ,in, Represents the reconstruction confidence of the super-resolution reconstructed image, It represents the standard deviation of the pixel values ​​of the noise pixels in the super-resolution reconstructed image. It represents the standard deviation of the pixel values ​​of non-noise pixels in the super-resolution reconstructed image. Represents the edge density of the super-resolution reconstructed image.

7. The vehicle matching method based on image recognition according to claim 5, characterized in that: The up-sampling of each frame of the pre-processed image includes: up-sampling each frame of the pre-processed image using a Lanczos algorithm.

8. A vehicle matching system based on image recognition, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the vehicle matching method based on image recognition as described in any one of claims 1 to 7.

Images