Traffic survey vehicle classification method based on multi-modal description and trajectory voting

Abstract

The application discloses a traffic volume survey vehicle classification method based on multi-modal description and trajectory voting. The method carries out target detection and multi-target tracking on a traffic monitoring video, selects a representative frame from an effective vehicle trajectory, and cuts a target image; a multi-modal artificial intelligence model is used to generate a structured description including an open vehicle type name and at least one auxiliary attribute dimension; a plurality of frame descriptions of the same trajectory are statistically aggregated according to fields, and the vehicle type name or an uncertain trajectory is corrected or marked according to a consistency checking result after aggregation of the attributes; and then the processed vehicle type name is mapped to a limited number of fine-grained target detection category labels to construct a training data set and train a fine-grained vehicle target detection model. The application does not need to assign fine-grained artificial category labels frame by frame, and can reduce the influence of single-frame description uncertainty on training labels.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation and computer vision technology, specifically relating to a traffic volume survey vehicle classification method based on multimodal description and trajectory voting. Background Technology

[0002] Traffic volume surveys are fundamental to traffic planning, road design, traffic organization optimization, and traffic safety assessments, with vehicle classification and counting being a crucial component. In some traffic volume survey operations, coarse-grained categories such as passenger cars, buses, trucks, and articulated vehicles are commonly used for statistical analysis. However, different vehicle types may vary in size, road traffic impact, and load characteristics. Therefore, obtaining more granular vehicle type information while maintaining the convertibility of existing statistical standards has practical value.

[0003] With the development of deep learning and computer vision technologies, vehicle detection models can automatically locate vehicle targets from traffic monitoring videos. However, existing solutions still have the following problems: 1. The category limitation of traditional target detection methods. Existing deep learning-based vehicle detection methods (such as YOLO, Faster R-CNN, etc.) usually use a predefined fixed category system for training and inference. This coarse-grained classification method cannot meet the needs of refined traffic surveys because it cannot effectively distinguish vehicle types with significant differences in axle load, traffic capacity, and safety risks. For example, dump trucks and box trucks, tank trucks and container trucks. These four types of trucks have significant differences in axle load, traffic capacity, and safety risks due to their different structures and uses: In terms of axle load, dump trucks have reinforced chassis and are used for transporting bulk heavy goods, so they have the strongest load-bearing capacity; box trucks mostly transport packaged goods and have a lighter load; tank trucks and container trucks are determined by the medium and standard container, and are subject to strict regulatory axle load and total mass limits. In terms of traffic capacity, dump trucks often face strict urban restrictions and transportation permits due to their tendency to generate dust and spillage; box trucks (especially light and medium-sized ones) have the fewest traffic restrictions; tank trucks (carrying hazardous chemicals) are subject to extremely strict route and time-of-day controls; and container trucks enjoy greater traffic convenience on main lines and at ports. Regarding safety risks, dump trucks are prone to tipping over or hydraulic bursts due to a shift in the center of gravity during lifting, and have numerous blind spots when turning; box trucks mainly face the risk of spillage due to unsecured cargo and the risk of fatigue driving on long journeys; tank trucks are prone to tipping over due to liquid sloshing, and are also accompanied by the risk of leakage and explosion; container trucks require vigilance against the risk of the container falling off due to a malfunctioning locking mechanism, and the risk of rear-end collisions caused by the large inertia of a fully loaded truck.

[0004] 2. The high cost of fine-grained annotation: To train a detection model that can recognize more fine-grained categories, a large amount of high-quality manually annotated data is required. Traffic monitoring video data is huge (usually 25-30 frames per second). Annotating vehicles in the video frame by frame with fine-grained data requires extremely high manpower and time costs, and it is difficult to ensure consistency among different annotators.

[0005] 3. Shortcomings of existing automatic annotation methods: The pre-trained models on general datasets have limited capabilities. Detection models pre-trained on general datasets such as COCO can typically only distinguish very coarse categories such as car, bus, and truck, failing to provide fine-grained annotations that meet the needs of traffic surveys. Single-frame uncertainty of visual language models (VLM): Although existing visual language models have open-ended image understanding capabilities, directly applying them to classify single-frame images results in unstable outcomes. The same vehicle may be described as different types by the model in different frames, under different lighting conditions, or at different angles, leading to significant label noise. Lack of effective utilization of temporal information: Currently, there is no effective method to systematically utilize the temporal continuity generated by target tracking technology to aggregate multi-frame descriptive information of the same vehicle in a video sequence, thereby eliminating the randomness and uncertainty brought about by single-frame inference.

[0006] In summary, the existing technology lacks a technical solution that can automatically discover and identify fine-grained target detection categories without requiring a large amount of manual annotation, while using temporal information to ensure classification stability. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention proposes a traffic volume survey vehicle classification method based on multimodal description and trajectory voting, comprising the following steps: Step S1: Perform target detection and multi-target tracking on the input traffic monitoring video, output vehicle detection boxes, detection confidence and original coarse category, and assign a unique trajectory number to the vehicle target to form a continuous vehicle trajectory across frames; Step S2: Filter valid vehicle trajectories and select representative frames from the multiple video frames contained in each valid vehicle trajectory, and crop the target image according to the detection box corresponding to the vehicle target; Step S3: Input the target image into the multimodal artificial intelligence model to generate an open multi-dimensional structured description. The structured description includes at least the fields generated by the multimodal artificial intelligence model based on the visible features of the vehicle, including: vehicle type name and at least one auxiliary attribute dimension, including: body structure, estimated vehicle length, cargo form, main color, and free description. Step S4: For the structured description corresponding to the target image of each representative frame of the same vehicle trajectory, perform mode voting for each field. Aggregate the winning values ​​selected for each field to obtain the field aggregation result of the vehicle trajectory. If there is no winning value, the vehicle trajectory is marked as an uncertain vehicle trajectory. Step S5: Perform attribute consistency verification using the field aggregation results, correct, retain, or mark uncertain vehicle type names according to preset consistency processing rules, and retain the mark of uncertain vehicle trajectories; Step S6: For vehicle trajectories that are not marked as uncertain, map the vehicle type names corrected in step S5 to a finite number of fine-grained target detection category labels according to the preset mapping rules; Step S7: Construct a training dataset based on the fine-grained object detection category labels, excluding vehicle trajectories labeled as uncertain from the training dataset; for valid vehicle trajectories that have not entered the multimodal description processing, use a preset fallback mapping rule to search step by step according to priority, stopping when a match is found and generating expanded training labels; if no match is found above, assign them to the default object detection category labels and record the label source. The priority order is as follows: vehicle body structure, freight type, and the original coarse category output from step S1; Step S8: Train a fine-grained vehicle target detection model using the training dataset constructed in step S7.

[0008] Further, in step S2, the screening conditions for effective vehicle trajectories include a trajectory duration frame threshold and a maximum detection box area threshold; from each screened effective vehicle trajectory, the frame with the highest detection confidence, the frame with the largest detection box area, and the middle frame of the trajectory detection record are selected as representative frames. When at least two representative frames are the same video frame, the remaining one or two unsampled representative frames are supplemented by the first and / or last frames of the trajectory.

[0009] Furthermore, the multimodal artificial intelligence model in step S3 is a visual language model with image understanding and structured output capabilities; the vehicle type name is an open natural language field; the auxiliary attribute dimension includes an "uncertain" option; the structured description also includes the main color and free description; the vehicle type name example is only used to illustrate the granularity of description and does not constitute a closed set of candidate categories.

[0010] Furthermore, before step S4, the steps include: performing a validity check on the structured description, normalizing the vehicle type name into a string, and performing a unique mode check on the vehicle type name, body structure type, estimated vehicle length range, and freight mode respectively. If there is no unique highest vote, the field is marked as uncertain. In step S4, the value that appears most frequently and has the highest number of unique votes in each field is taken as the aggregation result of the corresponding field; the main color and free description adopt the output value corresponding to the representative frame with the highest detection confidence; when the number of valid descriptions in the vehicle type name field is less than the preset number, there is no unique highest number of votes after normalization, or at least three valid descriptions contain three different names, the corresponding vehicle trajectory is marked as an uncertain vehicle trajectory, and the vehicle type name is marked as uncertain.

[0011] Furthermore, the consistency processing rules in step S5 include at least one of the following: Rule R1: If the vehicle type corresponding to the vehicle body structure is inconsistent with the vehicle model in the vehicle type name, and the original coarse category and at least one other independent auxiliary attribute of the sampled vehicle trajectory are consistent with the vehicle type corresponding to the vehicle body structure, then the vehicle type corresponding to the vehicle body structure is used to forcibly correct the vehicle model in the vehicle type name; otherwise, the original aggregation result is retained or marked as uncertain. Rule R2: If the vehicle model length corresponding to the estimated vehicle length is greater than the vehicle model length in the vehicle type name, and the original coarse category and at least one other independent auxiliary attribute of the sampled vehicle trajectory are consistent with the vehicle type corresponding to the vehicle body structure, then the vehicle model in the vehicle type name is corrected using the vehicle type corresponding to the estimated vehicle length; otherwise, the original aggregation result is retained or marked as uncertain. Rule R3: If the vehicle model length corresponding to the estimated vehicle length is less than the vehicle model length in the vehicle type name, and the original coarse category and at least one other independent auxiliary attribute of the sampled vehicle trajectory are consistent with the vehicle type corresponding to the vehicle body structure, then the vehicle model in the vehicle type name is corrected using the vehicle type corresponding to the estimated vehicle length; otherwise, the original aggregation result is retained or marked as uncertain. Rule R4: Retain the labels marked in step S4 for uncertain vehicle trajectories, but do not generate fine-grained object detection category labels for training on the uncertain vehicle trajectories.

[0012] Furthermore, the preset mapping rules in step S6 include: direct mapping, where when the aggregated vehicle type name contains a vehicle model name in the direct mapping table, it is mapped to the corresponding fine-grained target detection category; fallback mapping, where when the vehicle type name does not contain any vehicle model name in the direct mapping table, the corresponding vehicle model name is retrieved from the direct mapping table according to the vehicle body structure type, freight form, and the original coarse category output in step S1 to determine the fine-grained target detection category. If none of the above searches are successful, it is assigned to the default target detection category label; and backtracking mapping, which maps the fine-grained target detection category to the existing original coarse category. The fine-grained target detection categories include passenger cars, medium-sized trucks, heavy-duty box trucks, dump trucks, tank trucks, semi-trailers, and large buses.

[0013] Further, in an optional implementation, step S7 further includes: for valid vehicle trajectories that did not enter step S3, generating expanded training labels based on the original coarse category output in step S1 and the preset fallback mapping rule, and recording the label source; and performing category balancing processing, wherein the category balancing processing includes at least one of prioritizing the selection of representative frames containing rare category targets, performing whole-frame downsampling on representative frames containing only the dominant category targets, and performing whole-frame downsampling on representative frames of the dominant target detection category according to the set frame-level sampling weight, without deleting the bounding boxes of some vehicle targets in the same training image.

[0014] Further, in step S8, the target detection model weights trained with coarse-grained vehicle data in the same domain are used as initial weights, the category output dimension of the detection head is adjusted according to the number of fine-grained target detection categories, and at least some feature extraction modules, feature fusion modules and detection heads of the target detection model are fine-tuned.

[0015] The present invention also relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0016] Compared with the prior art, the beneficial effects of the present invention are: By employing a multimodal artificial intelligence model to generate open vehicle type names, rather than selecting from predefined, finite, fine-grained categories during the description generation stage, this invention can obtain natural language vehicle type descriptions adapted to different traffic scenarios. The descriptions are then categorized into preset target categories using mapping rules, thus balancing the adaptability of open descriptions with the consistency of traffic volume survey classification standards.

[0017] This invention utilizes the temporal continuity provided by target tracking technology to perform trajectory-level statistical aggregation of the description results of the same vehicle in different frames, thereby reducing the randomness and uncertainty of single-frame image description and improving the consistency of training labels.

[0018] This invention establishes a processing flow from traffic monitoring videos to fine-grained vehicle training datasets, eliminating the need to assign fine-grained manual category labels frame by frame, which helps reduce data preparation costs in specific traffic survey scenarios.

[0019] This invention utilizes auxiliary attributes such as vehicle body structure, estimated vehicle length, and cargo type to perform consistency verification and rule processing on vehicle type names, which can reduce the risk of misjudgment of labels based on a single name field and improve the consistency of label processing.

[0020] This invention establishes a correspondence between fine-grained target detection categories and existing original coarse categories through backtracking mapping, enabling fine-grained statistical results to be aggregated according to existing business criteria.

[0021] In the implementation of class balancing, the class distribution can be adjusted during the training dataset construction stage by prioritizing rare class frames, downsampling the entire frame of the dominant class, or using frame-level sampling weights. At the same time, the complete annotations of vehicle targets in the selected frames are preserved, avoiding false negative samples caused by deleting some annotation boxes.

[0022] The present invention can also use coarse-grained vehicle model weights in the same field to initialize fine-grained detection models, so that the models can be fine-tuned by using existing visual features of traffic monitoring scenarios. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating the overall process of the method of the present invention. Figure 2 A schematic diagram illustrating attribute consistency verification, label correction, and uncertain trajectory marking; Figure 3 A flowchart illustrating the mapping from natural language vehicle model names to target categories; Figure 4 This is a diagram showing the overall architecture of the system of the present invention; Figure 5 This is a flowchart for optional category balancing.

[0024] Figure 6 This is a schematic diagram illustrating the transfer training of a fine-grained vehicle target detection model. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] Terminology Explanation: Open vehicle type name: refers to the natural language vehicle type name generated autonomously by the multimodal artificial intelligence model based on the visible features of the vehicle in the input image, rather than a label selected from a pre-set limited set of vehicle types.

[0027] Structured description: refers to the data structure output by the multimodal artificial intelligence model, which contains multiple parsable fields, each corresponding to a semantically defined vehicle attribute. This data structure can be in JSON format.

[0028] Auxiliary attribute dimension: refers to the attribute fields in the structured description other than the vehicle type name, including at least one of body structure type (body_structure), estimated vehicle length range (estimated_length), and cargo type (cargo_type), used to perform attribute consistency verification and correction for the vehicle type name.

[0029] Track-level statistical aggregation: refers to performing statistical analysis on the description results of multiple representative frames of the same vehicle trajectory by field to obtain the field aggregation result of the vehicle trajectory.

[0030] Attribute consistency verification: This refers to determining whether the aggregation result of auxiliary attributes of the same vehicle trajectory meets the preset consistency rules with the aggregation result of vehicle type name, and performing label correction or uncertain trajectory marking accordingly.

[0031] Backtracking mapping: refers to the mapping relationship between fine-grained target detection categories and existing original coarse categories, used to convert fine-grained classification results into existing statistical standards.

[0032] Example This embodiment of the invention takes a traffic volume survey project in a certain city as an example, processing 23 traffic monitoring video segments with a resolution of 1920×1080, totaling 730,066 frames and a total duration of approximately 412 minutes, with a video frame rate of approximately 24.45 to 30.01 frames per second. This embodiment uses an object detection model, a multi-object tracking algorithm, and a multimodal artificial intelligence model to complete the processing; the specific equipment models, software versions, computational precision, and inference frameworks are only implementation examples and do not constitute a limitation of the invention.

[0033] like Figure 1 As shown, the traffic volume survey vehicle classification method based on multimodal description and trajectory voting proposed in this invention includes the following steps: Step S1: Object Detection and Multi-Object Tracking For each video frame of the aforementioned traffic monitoring video, the YOLO11L object detection model was used to detect vehicles, and the BoT-SORT multi-object tracking algorithm was used to assign a unique trajectory number to each detected vehicle target. This forms a continuous vehicle trajectory across frames; the detection confidence threshold in this embodiment is 0.15.

[0034] Specifically, each video frame acquired by a camera is input into a pre-trained object detection model to obtain a detection box. The model then extracts the vehicle appearance features of the corresponding vehicle target image. The BoT-SORT multi-object tracking algorithm assigns a unique trajectory number to each vehicle based on its appearance features and the corresponding detection box. While outputting each detection box, the object detection model directly provides the corresponding basic category prediction (i.e., the original coarse category) and a confidence score within the range of 0 to 1. This confidence score indicates the likelihood that the object detection model predicts the detection box to be a target of the corresponding category. Therefore, the output information for each target in each video frame includes the detection box coordinates [x1, y1, x2, y2], the confidence score, the original coarse category, and the trajectory number assigned by the multi-object tracking algorithm. The BoT-SORT multi-object tracking algorithm forms a single-camera multi-vehicle tracking trajectory based on the vehicle's appearance features and the unique trajectory number of the vehicle tracked by the detection box. The confidence score and the original coarse category are directly output by the object detection model without additional manual calculation and are reused in subsequent steps. (x1, y1) and (x2, y2) are the vertex coordinates of the two ends of a diagonal line of the detection box, respectively.

[0035] In this embodiment, a total of 730,066 video frames were processed, generating 71,302 vehicle trajectories.

[0036] Step S2: Trajectory-level target clipping Valid trajectories are selected from vehicle trajectories. For each valid trajectory, samples are uniformly taken from multiple video frames within the trajectory, with at least three representative frames sampled for each trajectory: the frame with the highest detection confidence, the frame with the largest detection box area, and the middle frame of the trajectory detection record. When at least two representative frames are actually the same video frame, the remaining one or two unsampled representative frames are supplemented with the first and / or last frame of the trajectory. Subsequently, the vehicle target image is cropped from the original representative frames according to the detection box coordinates, and the boundary around the detection box is expanded by approximately 10% to preserve the visual information of the vehicle's vicinity, resulting in the target image. The above selection of representative frames is not based on equal-interval frame sampling within the trajectory time span, nor does it dynamically change the number of representative frames according to the trajectory length; in this embodiment, the number of representative frames is fixed at three frames. This embodiment defines a vehicle trajectory with a continuous frame count of at least five frames and a maximum detection box area of ​​at least 4096 pixels squared as a valid vehicle trajectory. Following the actual processing order, firstly, 24,550 trajectories with insufficient continuous frame counts were removed from the 71,302 original vehicle trajectories; then, 19,643 trajectories with insufficient maximum detection box area were removed from the remaining trajectories that met the previous condition. The two removal counts were counted sequentially and without overlap, ultimately retaining 27,109 valid vehicle trajectories (71,302-24,550-19,643=27,109). The above thresholds are parameters for this embodiment and can be adjusted according to video frame rate, shooting distance, and image resolution.

[0037] Three representative frames are selected for each valid vehicle trajectory, so a total of 81,327 target images are cropped in this embodiment.

[0038] Step S3: VLM Multidimensional Open Description In this embodiment, an open-ended structured description is generated for each cropped target image using the Qwen3-VL-8B-Instruct multimodal artificial intelligence model. This multimodal artificial intelligence model is capable of image understanding and outputting a parsable JSON structure; the specific model, computational precision, and inference framework do not constitute a limitation of this invention.

[0039] The open-ended structured description includes a vehicle type name field and a structured description. The structured description consists of auxiliary attribute dimensions. The vehicle type name field, `vehicle_type`, is generated by the multimodal AI model using natural language based on the vehicle's visible features, without pre-defined categories, rather than selecting from a pre-defined, limited list of vehicle types. For example, it can generate names such as "semi-trailer container truck," "concrete mixer truck," or "light van." The multimodal AI model also outputs the structured auxiliary attributes shown in Table 1: body structure, estimated vehicle length, cargo configuration, main color, and free description.

[0040] Table 1

[0041] In Table 1, vehicle body structure, estimated vehicle length, and freight morphology are closed-ended restricted enumeration fields. The multimodal artificial intelligence model (VLM) can only choose one from the finite set of values ​​given in the prompt, and each finite set of values ​​includes the optional value "uncertain" as a catch-all. The main color and free description are open-ended natural language fields. The open-ended structured description uses a combination of open-ended vehicle type names and closed-ended restricted enumeration auxiliary attributes, which facilitates subsequent field-level statistical aggregation and attribute consistency verification.

[0042] In this embodiment, when the target image is input into the multimodal artificial intelligence model (VLM), a prompt is also input. The vehicle model name example in the prompt is only used to illustrate the descriptive granularity and does not constitute a closed candidate category list; the multimodal artificial intelligence model can output names other than the examples based on the visible features of the vehicle.

[0043] The prompt sets the model to a "traffic engineer," requiring it to carefully observe the vehicles in the image, without classification, and only describe the visible information. The model is required to output a single line of JSON containing the following fields: vehicle type name, vehicle body structure, estimated vehicle length, freight type, main color, and a free description. Example prompt: Please carefully observe this cropped image of the vehicle from the traffic surveillance camera and provide a detailed description. Do not categorize; only describe what you see. Please answer the following fields: Vehicle Type Name: The specific type of vehicle you see (e.g., sedan, SUV, van, pickup truck, minivan, light van, bus, coach, school bus, medium-duty truck, heavy-duty box truck, flatbed truck, tanker truck, dump truck, concrete mixer truck, semi-trailer container truck, semi-trailer flatbed truck, car carrier, etc., using the name you think is most accurate); Body Structure: Body structure (one-piece / articulated / tractor + trailer / uncertain); Estimated Vehicle Length: Estimated vehicle length (<4m / 4-6m / 6-10m / 10-14m / 14m+ / uncertain); Freight Type: Freight type (none / enclosed body / open flatbed / tank / dump / container / other / uncertain); Main Color: Main color; Free Description: Free description within 20 characters. Output a single line of JSON: {"vehicle_type":"...","body_structure":"...","estimated_length":"...","cargo_type":"...","color":"...","desc":"..."}.

[0044] This embodiment performs validity checks, name normalization, and unique mode checks on the output of the multimodal AI model. Validity checks include: if the multimodal AI model output cannot be parsed into a JSON object, lacks necessary fields, or has restricted field values ​​not included in the specified enumeration, it is marked as an invalid description and does not participate in the corresponding field voting; for vehicle type names, name normalization includes removing leading and trailing spaces, standardizing abbreviations, and using synonyms in the multimodal AI model output. Unique mode checks include: if a field in the multimodal AI model output does not have a unique highest number of votes in the target image cropped from the three representative frames corresponding to a valid vehicle trajectory, that field is marked as uncertain; when the valid descriptions of the vehicle type name in the target image cropped from the three representative frames corresponding to a valid vehicle trajectory are insufficient to form a preset number or cannot form a unique mode, the valid vehicle trajectory is marked as uncertain. The above validity checks, name normalization, and unique mode checks can also be implemented independently.

[0045] In this embodiment, valid vehicle trajectories initially categorized as large passenger vehicles, large trucks, and articulated vehicles are prioritized for inclusion. To supplement passenger vehicle samples, stratified sampling is performed based on video, resulting in a total of 7,250 sampled vehicle trajectories that enter the multimodal description processing. Thus, 21,750 target images corresponding to the 7,250 sampled vehicle trajectories are selected from the 81,327 target images generated in step S2 for multimodal description generation. Target images not included in this processing subset do not participate in subsequent voting statistics. The remaining valid vehicle trajectories not included in the multimodal description processing can generate expanded training labels in step S7 based on the original coarse category and fallback mapping rules output in step S1, and these labels are recorded as having not undergone excessive modal description and trajectory voting.

[0046] Step S4: Trajectory-level multi-field statistical aggregation The open-structured descriptions of the same sampled vehicle trajectory obtained from multiple representative frames of the target image are statistically aggregated by field to obtain the aggregated VLM description result of the sampled vehicle trajectory, thereby reducing the impact of single-frame description fluctuations on the sampled vehicle trajectory-level result. The specific rules are as follows: For the four fields of vehicle type name, body structure, estimated vehicle length, and freight mode, a mode vote is performed to obtain the frequency of the selected values ​​for each field. The selected value with the highest frequency in each field is extracted as the winning value for the corresponding field of the sampled trajectory. If there is no winning value, the aggregated vehicle type name of the valid trajectory is marked as "uncertain" to eliminate the randomness and errors in the VLM description results of a single sample frame. Suppose that after a sampled vehicle trajectory is output by a multimodal artificial intelligence model, a validity check is performed to filter out invalid values ​​and N valid descriptions. Necessary name normalization processing is performed, and then the frequency of each selected value in the valid descriptions is counted. The selected value with the highest frequency and unique highest vote is used as the aggregated result for that field. Each field is counted independently; if there is no unique highest vote, the field is marked as uncertain.

[0047] For the vehicle type name field, if the number of valid descriptions is less than 2, the normalized vehicle type name does not have a unique mode, or the number of valid descriptions is not less than 3 and the number of vehicle type name categories is not less than 3, the sampled vehicle trajectory is marked as uncertain. In this embodiment, when each sampled vehicle trajectory has 3 representative frames, this marking is triggered when three target images generate three different vehicle type names.

[0048] The aforementioned uncertainty criterion primarily applies to the vehicle type name field. The vehicle type name is an open-ended natural language field, potentially resulting in multiple vehicle type names due to synonyms or different perspectives. The vehicle body structure, estimated vehicle length, and freight type are closed-ended restricted enumeration fields, and their aggregation results are determined based on their respective unique mode checks. If a closed-ended restricted enumeration field shows a tie, that field is marked as uncertain, and the sampled vehicle trajectory can be excluded from the fine-grained training set accordingly.

[0049] The main color and free description are not included in the mode vote. To obtain a uniquely representative result, this embodiment uses the main color and free description output values ​​corresponding to the representative frame with the highest detection confidence.

[0050] The trajectory-level multi-field statistical aggregation can prevent isolated descriptions that deviate from the majority of frame results from being selected as aggregation results, thereby suppressing the impact of single-frame abnormal descriptions on trajectory-level labels.

[0051] like Figure 3 As shown, different representative frames of the same sampled vehicle trajectory generate open-ended structured descriptions, and then the trajectory-level aggregation result is obtained through field-level mode voting. In this embodiment, 7,250 sampled vehicle trajectories yielded 51 types of natural language vehicle model names.

[0052] The aforementioned 7,250 sampled vehicle trajectories each correspond to 3 representative frames that have been generated and described. Therefore, the 21,750 target images in step S3 correspond one-to-one with the 7,250 sampled vehicle trajectories in step S4.

[0053] Step S5: Attribute consistency verification, label correction, and uncertain trajectory marking The field aggregation results obtained in step S4 are used for attribute consistency verification. Vehicle type name and each auxiliary attribute are aggregated independently. When the aggregation result meets the preset consistency processing conditions, corresponding label correction is performed. For sampled vehicle trajectories marked as uncertain in step S4, the uncertain label is retained for exclusion when constructing the training dataset in step S7. Specific rules are as follows: Rule R1: If the vehicle type corresponding to the vehicle body structure is inconsistent with the vehicle model in the vehicle type name, and the original coarse category and at least one other independent auxiliary attribute of the sampled vehicle trajectory are consistent with the vehicle type corresponding to the vehicle body structure, then the vehicle model in the vehicle type name is forcibly corrected using the vehicle type corresponding to the vehicle body structure; otherwise, the original aggregation result is retained or marked as uncertain. For example: if the vehicle body structure contains "tractor" and "trailer" and the original coarse category is articulated vehicle, then the vehicle type name of the sampled trajectory is forcibly corrected to semi-trailer; when the vehicle body structure is "articulated", the correction is only performed when the vehicle type name does not contain passenger transport names such as bus, public transport, or school bus and the original coarse category is articulated vehicle; otherwise, the original aggregation result is retained or marked as uncertain.

[0054] Rule R2: If the vehicle type length corresponding to the estimated vehicle length is greater than the vehicle type length in the vehicle type name, and the original coarse category and at least one other independent auxiliary attribute of the sampled vehicle trajectory are consistent with the vehicle type corresponding to the body structure, then the vehicle type in the vehicle type name is corrected using the vehicle type corresponding to the estimated vehicle length; otherwise, the original aggregation result is retained or marked as uncertain. For example: when the vehicle type name is a medium-sized truck and the estimated vehicle length is "14m+", the vehicle type length in the vehicle type name is inconsistent with the estimated vehicle length; only when the freight form is "closed box" or "container", the body structure is "integral", and the original coarse category is large truck, is the vehicle type name corrected to heavy-duty box truck; otherwise, the original aggregation result is retained or marked as uncertain.

[0055] Rule R3: If the vehicle type length corresponding to the estimated vehicle length is less than the vehicle type length in the vehicle type name, and the original coarse category of the sampled vehicle trajectory and at least one other independent auxiliary attribute are consistent with the vehicle type corresponding to the vehicle body structure, then the vehicle type in the vehicle type name is corrected using the vehicle type corresponding to the estimated vehicle length; otherwise, the original aggregation result is retained or marked as uncertain. For example: when the vehicle type is a heavy-duty van and the estimated vehicle length is "4-6m" or "6-10m", the vehicle type length in the vehicle type name is inconsistent with the estimated vehicle length; if there are no other independent attributes to support this, the classification is not forced, and the original aggregation result is retained or marked as uncertain.

[0056] Rule R4: If the vehicle type name of the sampled vehicle trajectory is marked as "uncertain" in step S4, no fine-grained category label will be generated for it for training, and it will be excluded when constructing the training dataset in step S7.

[0057] It should be noted that R1 to R3 are consistency processing rules based on preset business category definitions; no single auxiliary attribute has absolute priority to forcibly overturn the vehicle type name in all scenarios. Correction is only performed when the original degree category and at least one other independent auxiliary attribute jointly support it; otherwise, the original aggregation result is retained or marked as uncertain.

[0058] like Figure 2 As shown, R1 to R3 are used to process trajectory-level labels that meet the preset contradiction conditions, and R4 is used to mark vehicle trajectories with inconsistent vehicle type names and descriptions, and is excluded when constructing the training dataset.

[0059] Step S6: Map the corrected vehicle type name to a fine-grained target detection category label using a preset mapping rule; For sampled vehicle trajectories not marked as uncertain, a preset mapping rule is used to map the corrected natural language vehicle model names to a finite number of fine-grained target detection categories. Table 2 lists examples of names, the number of name entries, and the backtracking coarse categories in the direct mapping configuration to illustrate the mapping rules; the number of experiments for each name is not listed. This embodiment maps natural language vehicle model names to seven fine-grained target detection categories and retains the backtracking mapping to the existing original coarse categories. The setting of fine-grained target detection categories comprehensively considers the purpose of traffic surveys, visual distinguishability, and the correspondence with existing business statistical standards.

[0060] Table 2

[0061] The preset mapping rules include: (a) Direct mapping: A direct mapping table between natural language vehicle names and fine-grained object detection categories is pre-established. When the aggregated vehicle type name contains a vehicle name from the direct mapping table, the vehicle type name output by the multimodal artificial intelligence VLM is directly mapped to the fine-grained object detection category label.

[0062] (b) Last resort mapping: When the vehicle type name output by the multimodal artificial intelligence VLM does not contain any vehicle model name in the direct mapping table, the following priority is used to search step by step until a match is found: first, the corresponding vehicle model name is searched in the direct mapping table by body structure; second, the corresponding vehicle model name is searched in the direct mapping table by freight mode; and third, the corresponding vehicle model name is searched in the direct mapping table by the original coarse category output in step S1. If no match is found in any of the above, it is classified into the default target detection category label.

[0063] (c) Backtracking mapping: For each fine-grained target detection category, a fixed mapping is maintained pointing to the current coarse-grained category standard of traffic volume survey, so that the fine-grained statistical results can be aggregated according to the existing business statistical standards.

[0064] The seven fine-grained target detection categories in this embodiment are: cls=0 passenger cars, cls=1 medium-sized trucks, cls=2 heavy-duty vans, cls=3 dump trucks, cls=4 tank trucks, cls=5 semi-trailers, and cls=6 buses. The direct mapping configuration table contains 44 natural language vehicle name entries, and the number of entries for each category is shown in Table 2; the 44 refers to the number of name entries in the mapping configuration, which is not the same as the number of natural language vehicle names obtained in step S4. Names that do not match the direct mapping table are classified according to the fallback mapping rule.

[0065] It should be noted that when the same vehicle type name both matches the direct mapping table and satisfies the triggering condition of rule R1 (e.g., names like "semi-trailer dump truck" and "semi-trailer tank truck," which combine a dedicated operating structure and an articulated structure), the structure determination result of rule R1 takes precedence. That is, step S5 first corrects the name to "semi-trailer," and then the category mapping is performed according to the corrected name. The entry for this type of name in the direct mapping table only applies when the triggering condition of rule R1 is not met. Thus, any vehicle type name has a unique and definite affiliation among the seven target detection categories.

[0066] like Figure 3 As shown, when the vehicle type name matches the direct mapping table, the target category is generated directly; when it does not match, the vehicle body structure, freight form and original coarse category are used for fallback determination in turn, and finally it can be converted into an existing coarse-grained statistical category through backtracking mapping.

[0067] Step S7: Training dataset construction and optional class balancing In one implementation, a training dataset is constructed based on the trajectory-level category labels and their corresponding detection boxes obtained in step S6; for valid vehicle trajectories that did not enter step S3, expanded training labels can be generated based on the original coarse categories output in step S1, and their sources are recorded separately from the multimodal labels; vehicle trajectories marked as uncertain are not included in the training dataset.

[0068] In another implementation, class balancing can be performed during the training dataset construction phase based on the vehicle category distribution and training objectives.

[0069] The category balancing process includes at least one of the following: Prioritizing the selection of representative frames containing rare target detection categories into the training set. Specifically, after steps S5 / S6, each sampled vehicle trajectory carries a fine-grained target detection category label. If the representative frame contains targets of rare target detection categories such as tank trucks, buses, and dump trucks (which have fewer trajectories), these will be preferentially retained and selected for the training set. If the representative frame only contains targets of dominant target detection categories such as passenger cars, the representative frame will be downsampled as a whole frame. Alternatively, frame-level sampling weights can be set for targets of dominant target detection categories. The category balancing process does not delete the bounding boxes of some vehicle targets in the same training image to avoid generating false negative samples inconsistent with the image content.

[0070] like Figure 5 As shown, the category balancing process is an optional implementation of step S7; when category balancing is not performed, the training dataset can be directly constructed; when category balancing is performed, representative frames are selectively retained or downsampled, but the complete annotations of each vehicle target in the selected frames are retained.

[0071] Whether or not category balancing is performed does not change the trajectory-level label generation and category mapping logic in steps S1 to S6.

[0072] Table 3 shows a comparison of the distribution of detection boxes in the training set before and after class balancing: Table 3

[0073] Step S8: Training the fine-grained vehicle target detection model The fine-grained vehicle target detection model is trained using the training dataset constructed in step S7. The training dataset can be implemented with or without class balancing. The training steps include: like Figure 6 As shown, the weights of the four target detection models of the basic detection model trained using coarse-grained data under the four standards of current traffic volume surveys (passenger cars, trucks, articulated vehicles, and buses) are used as initial weights. These initial weights are derived from existing coarse-grained vehicle detection models, rather than being directly retrained from the original coarse-grained categories output in step S1 of this embodiment. Through this training stage, the backbone network of the basic detection model can pre-learn visual features such as road conditions, viewing angles, vehicle scale, and imaging in traffic monitoring scenarios, thereby achieving domain adaptation.

[0074] Subsequently, the category output dimension of the detection head was adjusted according to the seven fine-grained object detection tasks, and fine-grained fine-tuning was performed. In a set of control experiments with all other conditions being the same, except for whether the attribute consistency processing in step S5 was performed on the training labels (both groups used the same training image set, the same initial weights, and the same training parameters; the training set had 6,000 frames, the validation set had 800 frames, and the training was conducted for 15 epochs, with both models evaluated on the same validation set), the model obtained by performing step S5 had an mAP50 of 0.5737 and an mAP50-95 of 0.5228, while the model obtained by not performing step S5 had an mAP50 of 0.5265 and an mAP50-95 of 0.4690. The above results exemplify the impact of the attribute consistency processing in step S5 on the quality of the training labels and the accuracy of the obtained model, and do not constitute a limitation on the scope of protection of this invention.

[0075] This embodiment uses the YOLO11L model. Following the network's topological order from input to output, the first 10 feature extraction modules (numbered 0 to 9) are designated as shallow feature extraction modules and frozen. The semantic feature extraction modules located after these frozen modules, at the end of the backbone network, are designated as high-level feature extraction modules and trained together with the multi-scale feature fusion network and the detection head. The freezing range is configured in this embodiment; other implementations can adjust this range based on the initial weights and the scale of the training data.

[0076] Other implementation methods The following embodiments are intended to illustrate that the present invention is not limited to the specific models and parameters described above. While maintaining the technical concepts of open structured description, trajectory-level statistical aggregation, attribute consistency processing, and category mapping, alternative technologies with corresponding functions can be employed.

[0077] like Figure 4 As shown, this invention also relates to a traffic volume survey vehicle classification system based on multimodal description and trajectory voting. The system is divided into a visual perception and target cropping subsystem, a trajectory-level label generation subsystem, and a training data and model generation subsystem according to data processing responsibilities, including: The target detection and tracking module is used to perform target detection and multi-target tracking on the input traffic monitoring video, output vehicle detection boxes, detection confidence scores and original coarse categories, and assign unique trajectory numbers to vehicle targets to form continuous vehicle trajectories across frames. The target cropping module is used to filter valid vehicle trajectories and select representative frames from multiple video frames contained in each valid vehicle trajectory, and crop the target image according to the detection box corresponding to the vehicle target. The multimodal description generation module is used to input the target image into a multimodal artificial intelligence model to generate an open multi-dimensional structured description. The structured description includes at least the fields generated by the multimodal artificial intelligence model based on the visible features of the vehicle, including: vehicle type name and at least one auxiliary attribute dimension. The auxiliary attribute dimension includes: body structure, estimated vehicle length, cargo form, main color, and free description. The trajectory-level aggregation module is used to perform mode voting on the structured descriptions corresponding to the target images of each representative frame of the same vehicle trajectory, and aggregate the winning values ​​of each field to obtain the field aggregation result of the vehicle trajectory. If there is no winning value, the vehicle trajectory is marked as an uncertain vehicle trajectory. The attribute consistency processing module is used to correct, retain, or mark uncertain vehicle type names according to the field aggregation results and preset consistency processing rules, and retain uncertain vehicle trajectory marks. The category mapping module is used to perform attribute consistency verification using the field aggregation results, correct, retain, or mark uncertain vehicle type names according to preset consistency processing rules, and retain the marking of uncertain vehicle trajectories; The dataset construction module is used to map the vehicle type names corrected in step S5 to a finite number of fine-grained target detection category labels for unlabeled vehicle trajectories according to a preset mapping rule; The model training module constructs a training dataset based on the fine-grained object detection category labels, but does not include vehicle trajectories labeled as uncertain in the training dataset. For valid vehicle trajectories that do not enter the multimodal description processing, a preset fallback mapping rule is used to search step by step according to priority. If a search is successful, expanded training labels are generated. If no search is successful, the trajectories are assigned to the default object detection category labels, and the source of the labels is recorded. The priority order is as follows: vehicle body structure, freight type, and the original coarse category output from step S1.

[0078] Implementation Method 1: Target Detection and Multi-Target Tracking The target detection model and multi-target tracking algorithm in step S1 are not limited to the models listed in this embodiment. As long as they can output vehicle detection boxes, detection confidence, original coarse-grained categories, and trajectory numbers that are consistent across frames, they can be used for subsequent trajectory-level processing.

[0079] Implementation Method 2: Multimodal Artificial Intelligence Model Step S3 can employ other multimodal artificial intelligence models with image understanding and structured output capabilities. The model used should be able to generate open-ended vehicle type names based on visible vehicle features and output at least one auxiliary attribute field for attribute consistency verification.

[0080] Implementation Method 3: Statistical Aggregation The statistical aggregation in step S4 is not limited to the mode voting. It can also adopt weighted voting based on detection confidence or voting based on the winning frequency threshold to reduce the impact of randomness and accidental errors in single-frame description on the aggregation results.

[0081] Implementation Method 4: Number of Target Categories The number of target categories in step S6 is not limited to seven categories. It can be set according to traffic volume survey needs, sample size, and vehicle type classification criteria, and corresponding direct mapping, catch-all mapping, and backtracking mapping rules can be established.

[0082] Implementation Method 5: Detection Model Training Method The fine-grained vehicle target detection model in step S8 is not limited to YOLO11L; model training can adopt full parameter fine-tuning, partial parameter fine-tuning or parameter efficient fine-tuning, and the specific method can be selected according to the initial weights, training data scale and computing resources.

[0083] The above description is merely an exemplary embodiment of the present invention and is not intended to limit the present invention. Any equivalent substitutions or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A vehicle classification method for traffic volume surveys based on multimodal description and trajectory voting, characterized in that, Includes the following steps: Step S1: Perform target detection and multi-target tracking on the input traffic monitoring video, output vehicle detection boxes, detection confidence and original coarse category, and assign a unique trajectory number to the vehicle target to form a continuous vehicle trajectory across frames; Step S2: Filter valid vehicle trajectories and select representative frames from the multiple video frames contained in each valid vehicle trajectory, and crop the target image according to the detection box corresponding to the vehicle target; Step S3: Input the target image into the multimodal artificial intelligence model to generate an open multi-dimensional structured description. The structured description includes at least the fields generated by the multimodal artificial intelligence model based on the visible features of the vehicle, including: vehicle type name and at least one auxiliary attribute dimension, including: body structure, estimated vehicle length, cargo form, main color, and free description. Step S4: For the structured descriptions corresponding to the target images of multiple representative frames of the same vehicle trajectory, perform mode voting for each field. Aggregate the winning values ​​selected for each field to obtain the field aggregation result of the vehicle trajectory. If there is no winning value, the vehicle trajectory is marked as an uncertain vehicle trajectory. Step S5: Perform attribute consistency verification using the field aggregation results, correct, retain, or mark uncertain vehicle type names according to preset consistency processing rules, and retain the mark of uncertain vehicle trajectories; Step S6: For vehicle trajectories that are not marked as uncertain, map the vehicle type names corrected in step S5 to a finite number of fine-grained target detection category labels according to the preset mapping rules; Step S7: Construct a training dataset based on the fine-grained object detection category labels, and do not include vehicle trajectories marked as uncertain in the training dataset; for valid vehicle trajectories that have not entered the multimodal description processing, use a preset fallback mapping rule to search step by step according to priority. If the search is successful, generate expanded training labels. If the search is unsuccessful, classify them into the default object detection category labels and record the label source. The priority order is: vehicle body structure, freight form, and the original coarse category output in step S1. Step S8: Train a fine-grained vehicle target detection model using the training dataset constructed in step S7.

2. The method according to claim 1, characterized in that, In step S2, the selection criteria for valid vehicle trajectories include: the number of consecutive frames of the trajectory is greater than or equal to a preset frame number threshold, and the area of ​​the largest detection box in the trajectory is greater than or equal to a preset area threshold; from each selected valid vehicle trajectory, the frame with the highest detection confidence, the frame with the largest detection box area, and the middle frame of the trajectory detection record are selected as representative frames. When at least two representative frames are the same video frame, the remaining one or two unsampled representative frames are supplemented by at least one frame from the first and last frames of the trajectory.

3. The method according to claim 1, characterized in that, The multimodal AI model in step S3 is a visual language model with image understanding and structured output capabilities; the vehicle type name is an open natural language field; the auxiliary attribute dimension includes an "uncertain" option; the structured description also includes the main color and free description; the vehicle type name example is only used to illustrate the granularity of description and does not constitute a closed set of candidate categories.

4. The method according to claim 1, characterized in that, Before step S4, the steps include: performing a validity check on the structured description, normalizing the vehicle type name into a string, and performing a unique mode check on the vehicle type name, body structure type, estimated vehicle length range, and freight mode respectively. If there is no unique highest vote, the field is marked as uncertain. In step S4, the value that appears most frequently in each field and has the highest number of unique votes is taken as the aggregation result of the corresponding field; The main color and free description are based on the output value corresponding to the representative frame with the highest detection confidence. When the number of valid descriptions in the vehicle type name field is less than the preset number, there is no unique highest vote after normalization, or at least three valid descriptions contain three different names, the corresponding vehicle trajectory will be marked as an uncertain vehicle trajectory, and the vehicle type name will be marked as uncertain.

5. The method according to claim 1, characterized in that, The consistency processing rules in step S5 include at least one of the following: Rule R1: If the vehicle type corresponding to the vehicle body structure is inconsistent with the vehicle model in the vehicle type name, and the original coarse category and at least one other independent auxiliary attribute of the sampled vehicle trajectory are consistent with the vehicle type corresponding to the vehicle body structure, then the vehicle type corresponding to the vehicle body structure is used to forcibly correct the vehicle model in the vehicle type name; otherwise, the original aggregation result is retained or marked as uncertain. Rule R2: If the vehicle model length corresponding to the estimated vehicle length is greater than the vehicle model length in the vehicle type name, and the original coarse category and at least one other independent auxiliary attribute of the sampled vehicle trajectory are consistent with the vehicle type corresponding to the vehicle body structure, then the vehicle model in the vehicle type name is corrected using the vehicle type corresponding to the estimated vehicle length; otherwise, the original aggregation result is retained or marked as uncertain. Rule R3: If the vehicle model length corresponding to the estimated vehicle length is less than the vehicle model length in the vehicle type name, and the original coarse category and at least one other independent auxiliary attribute of the sampled vehicle trajectory are consistent with the vehicle type corresponding to the vehicle body structure, then the vehicle model in the vehicle type name is corrected using the vehicle type corresponding to the estimated vehicle length; otherwise, the original aggregation result is retained or marked as uncertain. Rule R4: Retain the labels marked in step S4 for uncertain vehicle trajectories, but do not generate fine-grained object detection category labels for training on the uncertain vehicle trajectories.

6. The method according to claim 1, characterized in that, The preset mapping rules in step S6 include: direct mapping, where if the aggregated vehicle type name contains a vehicle model name in the direct mapping table, it is mapped to the corresponding fine-grained target detection category; fallback mapping, where if the vehicle type name does not contain any vehicle model name in the direct mapping table, the corresponding vehicle model name is retrieved from the direct mapping table according to the vehicle body structure type, freight form, and the original coarse category output in step S1 to determine the fine-grained target detection category. If none of the above are found, it is assigned to the default target detection category label; and backtracking mapping, which maps the fine-grained target detection category to the existing original coarse category. The fine-grained target detection categories include passenger cars, medium-sized trucks, heavy-duty vans, dump trucks, tank trucks, semi-trailers, and buses.

7. The method according to claim 1, characterized in that, Step S7 further includes: for valid vehicle trajectories that did not enter step S3, generating expanded training labels based on the original coarse category output in step S1 and the preset fallback mapping rule, and recording the label source; and performing category balancing processing, wherein the category balancing processing includes at least one of prioritizing the selection of representative frames containing rare category targets, performing whole-frame downsampling on representative frames containing only the dominant category targets, and performing whole-frame downsampling on representative frames of the dominant target detection category according to the set frame-level sampling weight, without deleting the bounding boxes of some vehicle targets in the same training image.

8. The method according to claim 1, characterized in that, In step S8, the weights of the target detection model trained with coarse-grained vehicle data in the same domain are used as initial weights. The category output dimension of the detection head is adjusted according to the number of fine-grained target detection categories. At least some of the feature extraction modules, feature fusion modules and detection heads of the target detection model are fine-tuned.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images