A zero-training vehicle re-identification method based on a visual language large model

Abstract

The present application relates to the field of computer vision and intelligent transportation, and provides a zero-training vehicle re-identification method based on a visual language large model, which comprises: dynamic multi-granularity text generation, visual feature analysis of a vehicle is performed by using the visual language large model to generate a structured hierarchical description, a hierarchical generation framework is constructed to generate basic semantic labels of a vehicle type, color and perspective and local semantic guided key detail area description in layers, and a description level is dynamically adjusted according to a confidence; adaptive feature fusion, adaptive fusion of visual-text features is realized for coarse sorting of vehicle similarity; combined comparison reasoning, TopN images in a visual-text coarse sorting list are divided into N / 2 comparison groups, and multi-image joint analysis is performed by using the visual language large model to realize fine sorting. The method generates a multi-level fine-granularity text description of a vehicle image, and realizes vehicle re-identification in an open scene without training.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and intelligent transportation, and in particular to a vehicle re-identification method that does not require training. Background Technology

[0002] Vehicle re-identification refers to determining the identity of a vehicle based on its appearance features, and it is an important means to overcome the limitations of license plate recognition verification. Existing vehicle re-identification methods mainly suffer from the following limitations: they rely on supervised training, requiring a large amount of labeled data for model training, making them difficult to adapt to new scenarios; they also have unimodal limitations, relying solely on visual features and being susceptible to interference from factors such as lighting changes and perspective differences. Existing large-scale visual language models can effectively generate image descriptions; therefore, there is an urgent need to explore zero-training vehicle re-identification methods based on large-scale visual language models. Summary of the Invention

[0003] To address the aforementioned problems in existing technologies, this invention proposes a zero-training vehicle re-identification method based on a large visual language model, which effectively improves the robustness and generalization ability of vehicle re-identification.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows.

[0005] This invention provides a zero-training vehicle re-identification method based on a large visual language model, comprising the following steps:

[0006] (1) Dynamic multi-granularity text generation: using a large visual language model to analyze the visual features of vehicles and generate structured hierarchical descriptions. By constructing a hierarchical generation framework, basic semantic labels for vehicle type, color, and perspective, as well as key detail descriptions of local semantic guidance are generated in layers, and the description level is dynamically adjusted according to the confidence level.

[0007] (2) Adaptive feature fusion to achieve adaptive fusion of visual and textual features for coarse ranking of vehicle similarity, including: image quality-guided visual feature weight calculation, semantic confidence-driven textual feature weight allocation, and visual-text fusion coarse ranking.

[0008] (3) Combinatorial comparative reasoning divides the TopN images of the visual-text coarse sorting list into N / 2 comparison groups, and uses the visual language big model to perform multi-image joint analysis to achieve fine sorting, including: automatic image stitching, group comparison based on prompting engineering, and sorting integration based on probabilistic graphical model.

[0009] In the above technical solution, the specific implementation steps of the dynamic multi-granularity text generation are as follows:

[0010] (1-1) Construct a hierarchical generation framework. The first level generates basic descriptions of vehicle models and colors, as well as visible perspectives (front, rear, side, front-side, rear-side). The second level generates detailed descriptions of the roof area. The third level guides the visual language model to focus on specific areas based on the perspective to generate detailed descriptions, such as focusing on the descriptions of windows, grilles, headlights, and logos for the front perspective.

[0011] (1-2) Confidence threshold control: Use a multimodal model such as CLIP model as a feature detector. When the confidence of local feature detection C_d < 0.6, the corresponding detailed description is masked. When the global feature matching degree C_t < 0.4, the re-detection process is triggered.

[0012] In the above technical solution, the adaptive feature fusion is specifically implemented through the following steps:

[0013] (2-1) Image quality-guided visual feature weight calculation: Based on the Tenengrad gradient function and entropy function, calculate the image sharpness Q_v, where Q_v∈[0,1], and calculate the visual feature weights based on the image sharpness.

[0014] (2-2) Semantic confidence-driven text feature weight allocation: text feature weights are calculated based on local feature detection confidence C_d and global feature matching degree C_t.

[0015] (2-3) Visual-text fusion coarse ranking: Image weights and text weights are normalized. Using a multimodal model supporting long text, such as LongCLIP, image features of all query and test images are extracted, along with multi-level text features generated using a multi-level generation framework. Visual similarity S_v and text similarity (S_t, S_r, S_d) of the query image and all images are calculated separately. The final similarity is calculated by weighted summation of visual and text similarity based on the visual feature weights and the text feature weights at different levels. Based on the final similarity, a coarsely ranked list of all query vehicle images is obtained.

[0016] In the above technical solution, the specific implementation steps of the combined comparative reasoning are as follows:

[0017] (3-1) Automatic image stitching: Construct a four-image stitching template "[query image][candidate image A][candidate image B][candidate image C]", scale the input image to a uniform size of 224×224 and then perform vertical stitching, and add learnable spatial location codes at the stitching boundary;

[0018] (3-2) Based on the group comparison of the prompting engineering, a structured prompt template is constructed: "As a vehicle recognition expert, please compare similarity from the following dimensions: 1. Vehicle outline matching degree (weight 40%) 2. Color spectrum consistency (weight 30%) 3. Detail feature consistency, such as sunroof, car logo, grid, windows, etc. (weight 30%). Output format: best matching ID > second best matching ID > least matching ID", and an abnormal response filtering mechanism is established to automatically discard the result when the model output does not conform to the preset format;

[0019] (3-3) Based on the probabilistic graphical model, a Markov chain transition matrix is ​​established between the first N (generally between 50 and 100) images in the coarse sorting list. The steady-state distribution probability is calculated by the random walk algorithm, and then the confidence weighted output is performed to form the final sorting list.

[0020] Compared with existing technologies, the advantages of the zero-training vehicle re-identification method based on a large visual language model in this invention are as follows:

[0021] 1. Breaking away from the dependence of traditional vehicle re-identification on closed scene data, it can be directly applied to open environments across cities and cameras without fine-tuning, significantly reducing model optimization costs;

[0022] 2. A dynamic multi-granularity text generation module was constructed to generate multi-granularity semantic descriptions that include global vehicle attributes (vehicle model, color) and local details (vehicle logo, decoration, damage), and can simultaneously support bidirectional queries of "text-to-image search" and "image-to-text search". Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating the implementation of an embodiment of the present invention. Detailed Implementation

[0024] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0025] like Figure 1 As shown in the figure, the zero-training vehicle re-identification method based on a large visual language model provided by this invention includes the following steps:

[0026] Step 1: Dynamic multi-granular text generation. The visual language big model is used to analyze the visual features of the vehicle and generate a structured hierarchical description. By constructing a hierarchical generation framework, basic semantic labels for vehicle type, color, and viewpoint are generated in layers, as well as key detail descriptions of local semantic guidance areas. The description level is dynamically adjusted according to the confidence level.

[0027] Step 2, Adaptive Feature Fusion, realizes adaptive fusion of visual and textual features for coarse ranking of vehicle similarity, including: image quality-guided visual feature weight calculation, semantic confidence-driven textual feature weight allocation, and visual-text fusion coarse ranking.

[0028] Step 3, combined comparative reasoning, divides the Top N images in the visual-text coarse ranking list into N / 2 comparison groups, and uses a large visual language model to perform multi-image joint analysis to achieve fine ranking, including: automatic image stitching, group comparison based on cue engineering, and ranking integration based on probabilistic graphical model.

[0029] The method will now be explained in detail.

[0030] Step 1 above specifically includes the following steps:

[0031] (1-1) Construct a hierarchical generation framework. The first level generates basic descriptions of vehicle models and colors, as well as visible perspectives (front, rear, side, front-side, rear-side). The second level generates detailed descriptions of the roof area. The third level guides the visual language model to focus on specific areas based on the perspective to generate detailed descriptions, such as focusing on the descriptions of windows, grilles, headlights, and logos for the front perspective.

[0032] (1-2) Confidence threshold control: Use a multimodal model such as CLIP model as a feature detector. When the confidence of local feature detection C_d < 0.6, the corresponding detailed description is masked. When the global feature matching degree C_t < 0.4, the re-detection process is triggered.

[0033] Step 2 above specifically includes the following steps:

[0034] (2-1) Image quality-guided visual feature weight calculation: Based on the Tenengrad gradient function and entropy function, the image sharpness Q_v is calculated, where Q_v∈[0,1]. The formula for calculating the visual feature weight is as follows:

[0035] α = sigmoid(Q_v × 0.5) × 0.7 (Formula 1)

[0036] (2-2) Semantic confidence-driven text feature weight allocation, the text feature weight calculation formula is as follows:

[0037] β = 0.25 × C_t + 1.0 × C_d (Formula 2)

[0038] (2-3) Visual-text fusion coarse sorting: The image weights and text weights are normalized and calculated using the following formula:

[0039]

[0040] Using multimodal models that support long text, such as LongCLIP, image features of all query and test images are extracted, along with multi-level text features generated using a multi-level generation framework. The visual similarity S_v between the query image and all images, and the text similarity (S_t, S_r, S_d) at different levels are calculated. The final similarity calculation formula is as follows:

[0041]

[0042] Based on the final similarity, a coarsely sorted list of all queried vehicle images is obtained.

[0043] Step 3 above specifically includes the following steps:

[0044] (3-1) Automatic image stitching: Construct a four-image stitching template “[query image][candidate image A][candidate image B][candidate image C]”, scale the input images to a uniform size of 224×224 and then stitch them vertically, and add learnable spatial location codes at the stitching boundaries.

[0045] (3-2) Based on the group comparison of the prompting engineering, a structured prompt template is constructed: "As a vehicle recognition expert, please compare similarity from the following dimensions: 1. Vehicle outline matching degree (weight 40%) 2. Color spectrum consistency (weight 30%) 3. Detail feature consistency, such as sunroof, car logo, grid, car window, etc. (weight 30%). Output format: best matching ID > second best matching ID > least matching ID", and an abnormal response filtering mechanism is established to automatically discard the result when the model output does not conform to the preset format.

[0046] (3-3) Based on the probabilistic graphical model, a Markov chain transition matrix is ​​established between the first N (generally between 50 and 100) images in the coarse sorting list. The steady-state distribution probability is calculated by the random walk algorithm, and then the confidence weighted output is performed to form the final sorting list.

[0047] The above step (3-3) specifically includes the following steps:

[0048] (3-3-1) Constructing the Markov transition matrix: First, define the state space, and then take the first N images of the coarsely sorted list as G = {g1, g2, ..., g...}. N} serves as a state node in a Markov chain. Each state s i The corresponding image g iBased on the output of (3-2), similarity scores are converted. Each set of comparison inference results (e.g., "g1>g2>g3") is numerically processed, with the image ranked 1st assigned a score1 = 1.0, the image ranked 2nd assigned a score2 = 0.7, and the image ranked 3rd assigned a score3 = 0.4. Then, each score is standardized using the following formula:

[0049]

[0050] Here, j and k represent other images in the comparison group. Then, the transition probabilities are calculated, and the state transition matrix T∈R is constructed. N ×N Among them, element T ij Indicates from state s i Transfer to s j The probability of is calculated using the following formula:

[0051]

[0052] Among them, the diagonal element T ii Set the self-loop probability ∈ = 0.05 to ensure the traversal of the Markov chain.

[0053] (3-3-2) Random walk calculation of steady-state distribution, initialization of probability distribution Then, a damped random walk iteration is performed, with the damping coefficient d = 0.85, and iterative updates are performed. The calculation formula is as follows:

[0054] π (t+1) =d·T T π (t) +(1-d)·π 0 (Formula 7)

[0055] The final steady-state probability π is obtained. * This indicates the similarity between the images of the various test vehicles.

[0056] (3-3-3) Confidence-weighted output, based on the number of times the image appears in the comparison group, C. i The weighted calculation is performed using the following formula:

[0057] S i =π i ×log(1+C i ) (Formula 8)

[0058] The weighted similarity is used as the final similarity to obtain the final sorted list.

[0059] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

[0060] Although the present invention has been described in detail above and embodiments have been provided, the present invention and its applicable embodiments are not limited thereto. Those skilled in the art can make various modifications based on the principles of the present invention, and some methods of the present invention can be applied to other systems. Therefore, all modifications made in accordance with the principles of the present invention should be understood to fall within the protection scope of the present invention.

Claims

1. A zero-training vehicle re-identification method based on a large visual language model, characterized in that, The method includes the following steps: (1) Dynamic multi-granularity text generation: A large visual language model is used to analyze the visual features of vehicles and generate structured hierarchical descriptions. A hierarchical generation framework is constructed to generate basic semantic labels for vehicle type, color, and viewing angle, as well as key detail descriptions guided by local semantics. The description hierarchy is dynamically adjusted based on confidence level. Specifically: (1-1) Construct a hierarchical generation framework. The first level generates basic descriptions of vehicle type and color, as well as visible viewpoints. Visible viewpoints include front, rear, side, front-side, and rear-side. The second level generates detailed descriptions of the roof area. The third level guides the visual language model to focus on specific areas based on the viewpoints to generate detailed descriptions. (1-2) Confidence threshold control: using a multimodal model as a feature detector, when the confidence of local feature detection C_d < 0.6, the corresponding detailed description is masked, and when the global feature matching degree C_t < 0.4, the re-detection process is triggered. (2) Adaptive feature fusion to achieve adaptive fusion of visual and textual features for coarse ranking of vehicle similarity, including: image quality-guided visual feature weight calculation, semantic confidence-driven textual feature weight allocation, and visual-textual fusion coarse ranking; specifically: (2-1) Image quality-guided visual feature weight calculation: Based on the Tenengrad gradient function and entropy function, calculate the image sharpness Q_v, where Q_v∈[0,1], and calculate the visual feature weights based on the image sharpness; (2-2) Semantic confidence-driven text feature weight allocation: text feature weights are calculated based on local feature detection confidence C_d and global feature matching degree C_t. (2-3) Visual-text fusion coarse sorting: normalize the image weights and text weights, use a multimodal model that supports long text to extract the image features of all query images and test images and the multi-level text features generated by the multi-level generation framework, calculate the visual similarity S_v of the query images and all images and the text similarity (S_t, S_r, S_d) of different levels. Calculate the final similarity by weighted sum of the visual similarity and text similarity according to the visual feature weights and the text feature weights of different levels. Based on the final similarity, obtain a coarse sorted list of all query vehicle images. (3) Combinatorial comparative reasoning: The Top N images in the visual-text coarse ranking list are divided into N / 2 comparison groups. A large visual language model is used for multi-image joint analysis to achieve fine ranking, including: automatic image stitching, group comparison based on cue engineering, and ranking integration based on a probabilistic graphical model; specifically: (3-1) Automatic image stitching: Construct a four-image stitching template "[query image] [candidate image A] [candidate image B] [candidate image C]", scale the input images to a uniform size of 224×224 and then stitch them vertically. Add learnable spatial location codes at the stitching boundaries. (3-2) Based on the group comparison of the prompting engineering, construct a structured prompt template: "As a vehicle recognition expert, please compare similarity from the following dimensions:

1. Vehicle outline matching degree, weight 40%; 2. Color spectrum consistency, weight 30%; 3. Detail feature consistency, weight 30%; Output format: best matching ID > second best matching ID > least matching ID", and establish an abnormal response filtering mechanism to automatically discard the results when the model output does not conform to the preset format; (3-3) Based on the probabilistic graphical model, a Markov chain transition matrix is ​​established among the first N images in the coarsely sorted list. The steady-state probability is calculated using a random walk algorithm, and the output is weighted by confidence level to form the final sorted list. N is between 50 and 100. Specifically, this includes: (3-3-1) Construct the Markov transition matrix. First, define the state space, and set the first N images of the coarsely sorted list into a set G = {g1, g2, ..., g...}. N As a state node in a Markov chain, each state s i The corresponding image g i Based on the output of step (3-2), similarity scores are converted, and the results of each comparison reasoning are quantified. Then, the scores of each group are standardized, and the transition probabilities are calculated based on the standardized scores to construct the state transition matrix T∈R. N×N , where the diagonal element T ii Set as self-loop probability This ensures the traversal property of the Markov chain; (3-3-2) Calculate the steady-state distribution using random walk and initialize the probability distribution. A damped random walk iteration was performed, with the damping coefficient d=0.85, and the results were updated iteratively to obtain the final steady-state probability. This indicates the similarity between the images of the various test vehicles; (3-3-3) Confidence-weighted output, based on the number of times the image appears in the comparison group C. i Calculate the weights, and use the weighted similarity as the final similarity to obtain the final ranked list.

Images