A traffic scene image matching method and system based on deep learning
By incorporating a triplet loss function and an image segmentation method that incorporate foreground proportion information during the model training phase, the problem of inaccurate matching caused by different foreground proportions in images is solved, thereby improving the accuracy and robustness of traffic scene image matching.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ISA TECH CO LTD
- Filing Date
- 2023-12-22
- Publication Date
- 2026-07-21
AI Technical Summary
Existing image matching methods fail to effectively consider the impact of different foreground proportions on matching accuracy, resulting in insufficient accuracy in matching traffic scene images.
During the model training phase, a triplet loss function with foreground proportion information is introduced to improve the model's ability to recognize foreground features. During the image matching phase, image segmentation is performed for cases with low similarity, followed by secondary feature extraction and similarity calculation.
It improves the accuracy and robustness of image matching, reduces the impact of different foreground proportions on matching results, and enhances the accuracy of image matching in traffic scenes.
Smart Images
Figure CN117671619B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a method and system for matching traffic scene images based on deep learning. Background Technology
[0002] In the field of image processing, image matching refers to a method for searching for images that are identical or similar to a query image. This method is widely used in fields such as object tracking, facial recognition, autonomous driving, and quality inspection, bringing great convenience to users' lives and work. However, the inventors discovered that current image matching methods do not consider the impact of different foreground proportions on the accuracy of image matching.
[0003] The patent application number 202111058412.7, entitled "A Deep Learning-Based Open Set Image Scene Matching Method," avoids the defect of deep learning methods being unable to identify untrained scenes by utilizing similarity; however, when training the matching model, it directly imports training datasets covering different image scenes into the deep learning model without considering the impact of different foreground proportions on the final matching results within the same scene.
[0004] The patent application number is 201711129595.0, entitled "An Image Matching Method Based on SURF". It improves the feature point detection range by introducing corner detection. However, in the image matching stage, the traditional method is used to perform corner detection on the reference image and the image to be matched in the traffic scene. However, only calculating the similarity of the whole image will result in the inability to capture detailed information. The features of the extracted corner points and other key points are relatively similar, thus failing to guarantee the matching accuracy.
[0005] Image matching in traffic scenes is primarily used to identify people, non-motorized vehicles, and motorized vehicles, requiring high matching accuracy. Therefore, improving the accuracy of image matching in traffic scenes is a pressing issue that needs to be addressed. Summary of the Invention
[0006] To address the aforementioned issues, this invention proposes a deep learning-based image matching method and system for traffic scenes. During model training, the method fully considers the foreground features of images with a large foreground area, thus using these features as prior knowledge to improve the model's matching performance and robustness. During image matching, when the confidence level of the original image similarity calculation is not high, the original image is split to minimize the influence of the foreground on image matching, thereby improving the accuracy of image matching.
[0007] To achieve the above objectives, the present invention adopts the following technical solution:
[0008] In a first aspect, the present invention provides a traffic scene image matching method based on deep learning, comprising:
[0009] The original reference image and the original image to be matched of the traffic scene are acquired, and preprocessed to obtain the preprocessed reference image and the image to be matched.
[0010] The preprocessed reference image and the image to be matched are input into the trained deep learning model, which outputs a first reference feature and a first feature to be matched. The similarity between the first reference feature and the first feature to be matched is calculated. If the similarity is greater than a set threshold, the image is successfully matched.
[0011] Otherwise, the reference image and the image to be matched are segmented to obtain a subset of the reference image and a subset of the image to be matched; each subset of the images contains four sub-images;
[0012] The sub-images in each image subset are input into the trained deep learning model, and the model outputs a second reference feature subset and a second feature subset to be matched. Each feature subset contains four sub-features. The similarity between the sub-features in the second reference feature subset and the second feature subset to be matched is calculated. If the similarity between the three sets of sub-features is greater than a set threshold, the image matching is successful; otherwise, it is unsuccessful.
[0013] Preferably, the preprocessing includes: using random horizontal image flipping, random vertical image flipping, and random rotation data augmentation methods to unify the size of the original reference image and the original image to be matched.
[0014] Preferably, the training process of the trained deep learning model is as follows:
[0015] Obtain a sample image set of traffic scenes, the sample image set including example images, positive sample images, and negative sample images;
[0016] Preprocess the positive and negative sample images to obtain the foreground area ratio information for each image;
[0017] The sample image set and the foreground area ratio of each image are input into the deep learning model, and the triplet loss is calculated. When the triplet loss is minimized, the trained deep learning model is obtained.
[0018] Preferably, the specific process of preprocessing the positive and negative sample images to obtain the foreground area ratio information of each image is as follows:
[0019] An object detection network is used to obtain the coordinates of the foreground in each of the positive and negative sample images; the foreground includes people, non-motorized vehicles, and motorized vehicles.
[0020] Based on the acquired coordinates, the area of the foreground in each image is calculated and summed. The ratio of the foreground area to the total area of the image is the foreground area proportion information.
[0021] Preferably, the triplet loss is:
[0022] L=max(d(a,p)*e -x -d(a,n)+margin,0)
[0023] Where d represents the distance between the two images, a represents the example image, p represents the positive sample image (i.e., the image in the same scene as a), n represents the negative sample image (i.e., the image in a different scene than a), margin represents a constant greater than 0, and x represents the foreground area ratio information.
[0024] Preferably, the trained deep learning model is a Siamese neural network.
[0025] Preferably, the similarity threshold is 0.9.
[0026] Secondly, the present invention provides a traffic scene image matching system based on deep learning, comprising:
[0027] Preprocessing module: Acquires the original reference image and the original image to be matched of the traffic scene, performs preprocessing, and obtains the preprocessed reference image and the image to be matched;
[0028] First calculation module: Input the preprocessed reference image and the image to be matched into the trained deep learning model respectively, and output the first reference feature and the first feature to be matched; calculate the similarity between the first reference feature and the first feature to be matched, and if the similarity is greater than a set threshold, the image is successfully matched;
[0029] Segmentation module: Otherwise, the reference image and the image to be matched are segmented to obtain a subset of the reference image and a subset of the image to be matched; each subset of the images contains four sub-images;
[0030] Secondary calculation module: Input the sub-images of each image subset into the trained deep learning model, and output the second reference feature subset and the second feature subset to be matched; each feature subset contains four sub-features; calculate the similarity of the sub-features in the second reference feature subset and the second feature subset to be matched. If the similarity of the three sets of sub-features is greater than a set threshold, the image matching is successful; otherwise, it is unsuccessful.
[0031] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the deep learning-based traffic scene image matching method described above.
[0032] Fourthly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the deep learning-based traffic scene image matching method described above.
[0033] Compared with the prior art, the beneficial effects of this disclosure are as follows:
[0034] (1) In order to enable the model to fully capture foreground information and improve model performance when training the matching model, this invention adds foreground proportion information to the triplet loss, thereby considering the contribution of images with large foreground proportion to the loss function; the foreground proportion information of the images is fed to the network as prior knowledge, and by dynamically adjusting the contribution of images with different foreground proportions to the loss, the trained deep learning model can output more reliable results, effectively solving the problem of inaccurate results when calculating cosine similarity in the final feature extraction due to different foreground proportions of images, and improving the robustness of the model.
[0035] (2) In the image matching stage, when the confidence level of the similarity calculation results between two images is not very high, the present invention performs a secondary matching by splitting the original image, which is beneficial for capturing detailed information. Among them, each sub-image of the reference image subset corresponds one-to-one with the positional relationship of each sub-image of the image subset to be matched. The features of each sub-image are extracted and the similarity is calculated based on the positional relationship, which improves the accuracy of similarity calculation, minimizes the influence of different foreground proportions on matching, and improves the accuracy of image matching. Attached Figure Description
[0036] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute a limitation thereof.
[0037] Figure 1 A flowchart illustrating a traffic scene image matching method based on deep learning, provided in Embodiment 1 of this disclosure;
[0038] Figure 2 This is a flowchart of traffic scene image matching provided in Embodiment 1 of this disclosure. Detailed Implementation
[0039] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0040] During model training, because the sample set contains diverse image samples, the foreground proportion of each image varies relative to the overall image. When training with images that have a small foreground proportion, the model cannot fully learn foreground features, resulting in inaccurate matching when using the trained model for image matching. Furthermore, during image matching, inputting images with different foreground proportions into the model and calculating similarity based solely on the entire image while ignoring the impact of different foreground proportions also leads to inaccurate matching results. To address these issues, this disclosure proposes the following solution:
[0041] Example 1
[0042] like Figure 1 As shown, this embodiment discloses a traffic scene image matching method based on deep learning, such as... Figure 2 The image matching flowchart shown below illustrates the method, which includes the following steps:
[0043] S1: Obtain the original reference image and the original image to be matched for the traffic scene, perform preprocessing, and obtain the preprocessed reference image and the image to be matched.
[0044] In one specific embodiment, the preprocessing includes: using data augmentation methods such as randomly horizontally flipping the image, randomly vertically flipping the image, and randomly rotating the image to unify the size of the original reference image and the original image to be matched.
[0045] S2: Input the preprocessed reference image and the image to be matched into the trained deep learning model, and output the first reference feature and the first feature to be matched; calculate the similarity between the first reference feature and the first feature to be matched, and if the similarity is greater than the set threshold, the image is successfully matched.
[0046] Specifically, the deep learning model is a feature extraction network used to extract features. The backbone of the feature extraction network is a Siamese network constructed from repVGG. The Siamese network consists of two identical sub-networks, each of which accepts an input and learns to map the input to a feature vector in the feature space. These two sub-networks are structurally identical and share parameters.
[0047] Before training the deep learning model, the traffic scene sample image set is first preprocessed. The traffic scene sample image set includes example images, positive sample images, and negative sample images.
[0048] Using an object detection network, the coordinates of the foreground in each of the positive and negative sample images are detected. The area of the foreground in each image is calculated based on the obtained coordinates, summed, and finally divided by the area of the entire image to obtain the foreground area ratio information. The result is then saved. It should be understood that this embodiment does not specifically limit the type of object detection network; those skilled in the art can choose according to actual needs. In this example, image matching in traffic scenes is mainly used to identify people, non-motorized vehicles, and motorized vehicles; therefore, the foreground in the images participating in the matching includes people, non-motorized vehicles, and motorized vehicles.
[0049] The sample image set and the foreground area ratio of each image are input into the deep learning model. Gradient backpropagation is performed using the improved triplet loss function. The trained deep learning model is obtained when the triplet loss is minimized. The triplet loss function is expressed as follows:
[0050] L=max(d(a,p)-d(a,n)+margin,0) (1)
[0051] L=max(d(a,p)*e -x -d(a,n)+margin,0) (2)
[0052] Equation (1) is the traditional triple loss function, where d represents the distance between two images, a represents the example image, p represents the positive sample image (i.e., the image in the same scene as a), n represents the negative sample image (i.e., the image in a different scene than a), and margin represents a constant greater than 0.
[0053] Equation (2) is the improved triplet loss function proposed in this embodiment, which considers the influence of foreground area during model training. Here, x represents the foreground area ratio, and x is the ratio of the foreground to the entire image during image preprocessing, with a value between 0 and 1. Thus, when the foreground area ratio of p is large, d(a,p)*e -x The smaller the value, the higher the similarity between the two images, effectively avoiding matching failures caused by large foreground variations in traffic scenarios. Specifically, the foreground area ratio of all images is calculated, but the loss is calculated only when considering the ratio of images corresponding to positive samples. This is because the negative sample image corresponding to this example image is the positive sample image of the other example images, meaning all images in the dataset are used.
[0054] In this specific embodiment, the similarity calculation is a cosine similarity calculation, and the threshold involved is 0.9. If the similarity is greater than 0.9, the output shows that the two images are successfully matched; otherwise, proceed to S3.
[0055] S3: Otherwise, the reference image and the image to be matched are segmented to obtain a subset of the reference image and a subset of the image to be matched; each subset of the images contains four sub-images.
[0056] In one specific embodiment, the reference image and the image to be matched are each divided into four equal sub-images according to a uniform segmentation method, forming a reference image subset and a image to be matched subset. The positional relationship between each sub-image of the reference image subset and each sub-image of the image to be matched subset is one-to-one.
[0057] S4: Input the sub-images of each image subset into the trained deep learning model, and output the second reference feature subset and the second feature subset to be matched; each feature subset contains four sub-features; calculate the similarity of the sub-features in the second reference feature subset and the second feature subset to be matched. If the similarity of the three sets of sub-features is greater than a set threshold, the image matching is successful; otherwise, it is unsuccessful.
[0058] In one specific embodiment, the computational reference image subset and the image subset to be matched are respectively input into the trained deep learning model described in S2, and secondary feature extraction is performed to output a second reference feature subset and a second feature subset to be matched.
[0059] Since there is a one-to-one correspondence between the positions of each sub-image in the reference image subset and the positions of each sub-image in the image subset to be matched, there is also a one-to-one correspondence between the positions of each sub-feature in the second reference feature subset and the positions of each sub-feature in the second feature subset to be matched. The cosine similarity of the corresponding sub-features in the second reference feature subset and the second feature subset to be matched is calculated. If there are three sets of sub-feature similarities greater than a set threshold of 0.9, the two scene images are considered successfully matched; otherwise, the scene images are considered unsuccessfully matched.
[0060] In the training phase of the matching model, this invention considers foreground area proportion information in the triplet loss function calculation to improve the model's ability to recognize foreground features. This increases the contribution of images with larger foreground proportions to the loss, enabling the trained deep learning model to output more reliable results. It effectively reduces the influence of foreground on matching and improves the robustness of the image matching model. During the image matching phase, when the confidence level of the similarity calculation results between two images is not high, a secondary matching is performed. This involves splitting the original image to minimize the impact of different foreground proportions on matching, thereby improving the accuracy of image matching.
[0061] Example 2
[0062] This embodiment provides a traffic scene image matching system based on deep learning, including:
[0063] Preprocessing module: Acquires the original reference image and the original image to be matched of the traffic scene, performs preprocessing, and obtains the preprocessed reference image and the image to be matched;
[0064] First calculation module: Input the preprocessed reference image and the image to be matched into the trained deep learning model respectively, and output the first reference feature and the first feature to be matched; calculate the similarity between the first reference feature and the first feature to be matched, and if the similarity is greater than a set threshold, the image is successfully matched;
[0065] Segmentation module: Otherwise, the reference image and the image to be matched are segmented to obtain a subset of the reference image and a subset of the image to be matched; each subset of the images contains four sub-images;
[0066] Secondary calculation module: Input the sub-images of each image subset into the trained deep learning model, and output the second reference feature subset and the second feature subset to be matched; each feature subset contains four sub-features; calculate the similarity of the sub-features in the second reference feature subset and the second feature subset to be matched. If the similarity of the three sets of sub-features is greater than a set threshold, the image matching is successful; otherwise, it is unsuccessful.
[0067] Example 3
[0068] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the deep learning-based traffic scene image matching method described in Embodiment 1 above.
[0069] In the training phase of the matching model, this invention considers foreground area proportion information in the triplet loss function calculation to improve the model's ability to recognize foreground features. This increases the contribution of images with larger foreground proportions to the loss, enabling the trained deep learning model to output more reliable results. It effectively reduces the influence of foreground on matching and improves the robustness of the image matching model. During the image matching phase, when the confidence level of the similarity calculation results between two images is not high, a secondary matching is performed. This involves splitting the original image to minimize the impact of different foreground proportions on matching, thereby improving the accuracy of image matching.
[0070] Example 4
[0071] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the deep learning-based traffic scene image matching method described in Embodiment 1 above.
[0072] In the training phase of the matching model, this invention considers foreground area proportion information in the triplet loss function calculation to improve the model's ability to recognize foreground features. This increases the contribution of images with larger foreground proportions to the loss, enabling the trained deep learning model to output more reliable results. It effectively reduces the influence of foreground on matching and improves the robustness of the image matching model. During the image matching phase, when the confidence level of the similarity calculation results between two images is not high, a secondary matching is performed. This involves splitting the original image to minimize the impact of different foreground proportions on matching, thereby improving the accuracy of image matching.
[0073] The steps or modules involved in Embodiments 2 to 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.
[0074] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A traffic scene image matching method based on deep learning, characterized in that, include: The original reference image and the original image to be matched of the traffic scene are acquired, and preprocessed to obtain the preprocessed reference image and the image to be matched. The preprocessed reference image and the image to be matched are respectively input into the trained deep learning model, and the first reference feature and the first matching feature are output. Calculate the similarity between the first reference feature and the first feature to be matched. If the similarity is greater than a set threshold, the image is successfully matched. Otherwise, the reference image and the image to be matched are segmented to obtain a subset of the reference image and a subset of the image to be matched; each subset of the images contains four sub-images; The sub-images in each image subset are input into the trained deep learning model, and the model outputs a second reference feature subset and a second feature subset to be matched. Each feature subset contains four sub-features. The similarity between the sub-features in the second reference feature subset and the second feature subset to be matched is calculated. If the similarity between the three sets of sub-features is greater than a set threshold, the image matching is successful; otherwise, it is unsuccessful. During the training process of the deep learning model, the sample image set and the foreground area ratio information of each image are input into the deep learning model to calculate the triplet loss; the triplet loss is: L=max(d(a,p)* -d(a,n)+margin,0) ； in, d Represents the distance between two images. a Representative example diagram, p The image representing the positive sample is and a Images of the same scene n The negative sample image is represented by the image and a Images from different scenes margin Represents a constant greater than 0. x This refers to the proportion of the foreground area.
2. The traffic scene image matching method based on deep learning as described in claim 1, characterized in that, The preprocessing includes: Data augmentation techniques, including random horizontal image flipping, random vertical image flipping, and random rotation, are used to unify the dimensions of the original reference image and the original image to be matched.
3. The traffic scene image matching method based on deep learning as described in claim 1, characterized in that, The training process of the trained deep learning model is as follows: Obtain a sample image set of traffic scenes, the sample image set including example images, positive sample images, and negative sample images; Preprocess the positive and negative sample images to obtain the foreground area ratio information for each image; The sample image set and the foreground area ratio of each image are input into the deep learning model, and the triplet loss is calculated. When the triplet loss is minimized, the trained deep learning model is obtained.
4. The traffic scene image matching method based on deep learning as described in claim 3, characterized in that, The specific process of preprocessing the positive and negative sample images to obtain the foreground area ratio information of each image is as follows: An object detection network is used to obtain the coordinates of the foreground in each of the positive and negative sample images; the foreground includes people, non-motorized vehicles, and motorized vehicles. Based on the acquired coordinates, the area of the foreground in each image is calculated and summed. The ratio of the foreground area to the total area of the image is the foreground area proportion information.
5. The traffic scene image matching method based on deep learning as described in claim 1, characterized in that, The trained deep learning model is a Siamese neural network.
6. The traffic scene image matching method based on deep learning as described in claim 1, characterized in that, The similarity threshold is 0.
9.
7. A traffic scene image matching system based on deep learning, based on the traffic scene image matching method based on deep learning as described in claim 1, characterized in that, include: Preprocessing module: Acquires the original reference image and the original image to be matched of the traffic scene, performs preprocessing, and obtains the preprocessed reference image and the image to be matched; First calculation module: Input the preprocessed reference image and the image to be matched into the trained deep learning model, and output the first reference feature and the first feature to be matched; Calculate the similarity between the first reference feature and the first feature to be matched. If the similarity is greater than a set threshold, the image is successfully matched. Segmentation module: Otherwise, the reference image and the image to be matched are segmented to obtain a subset of the reference image and a subset of the image to be matched; each subset of the images contains four sub-images; Secondary calculation module: Input the sub-images of each image subset into the trained deep learning model, and output the second reference feature subset and the second feature subset to be matched; each feature subset contains four sub-features; calculate the similarity of the sub-features in the second reference feature subset and the second feature subset to be matched. If the similarity of the three sets of sub-features is greater than a set threshold, the image matching is successful; otherwise, it is unsuccessful.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the steps in the deep learning-based traffic scene image matching method as described in claims 1-6.
9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the deep learning-based traffic scene image matching method as described in claims 1-6.