Traffic sign extraction and state judgment method and device based on large model fine tuning

Abstract

The application provides a traffic sign extraction and state judgment method and device based on large model fine tuning, which comprises the following steps: acquiring point cloud data of a road surface and an object containing a traffic sign, and converting the point cloud data into a two-dimensional feature map; processing the two-dimensional feature map through a target detection model fine tuned by a traffic sign data set to obtain a candidate region containing the traffic sign and initial class information of the traffic sign; segmenting the two-dimensional feature map based on the candidate region to obtain a segmentation mask of the traffic sign and an image region corresponding to the segmentation mask; inputting the image region and the initial class information into a fine tuned visual language model to judge the appearance state of the traffic sign according to the image region, and checking the semantic consistency of the initial class information and the image region.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method and apparatus for traffic sign extraction and status determination based on large model fine-tuning. Background Technology

[0002] With the development of intelligent transportation systems and autonomous driving technology, high-precision and timely road information perception has gradually become an important foundation for high-precision map updates and safe vehicle operation. As an important component of road information, the accurate acquisition and maintenance of traffic signs' spatial location, category information, and appearance condition (such as wear, damage, and obstruction) are receiving increasing attention from traffic asset management.

[0003] In related technologies, traffic sign extraction is typically based on factors such as point cloud height, reflection intensity, and geometric structure. This point cloud is then converted into a two-dimensional feature map, and deep learning is used for object detection to identify the location and category of the traffic sign. However, these methods are sensitive to acquisition conditions, occlusion, noise, and scene complexity, and the detection results are prone to false positives, false negatives, or misclassifications, leading to insufficient reliability. Furthermore, the assessment of the wear and tear on traffic signs still relies heavily on manual judgment, which is detrimental to the refined management of traffic assets. Summary of the Invention

[0004] This invention provides a method and apparatus for traffic sign extraction and status determination based on large model fine-tuning, which solves the problem that the existing technology only identifies traffic signs and lacks verification of the identification results and judgment of their appearance status.

[0005] This invention provides a traffic sign extraction and state determination method based on large model fine-tuning, comprising: acquiring point cloud data of a road surface and objects containing traffic signs, and converting the point cloud data into a two-dimensional feature map; processing the two-dimensional feature map using a target detection model fine-tuned with a traffic sign dataset to obtain candidate regions containing the traffic signs and initial category information of the traffic signs; segmenting the two-dimensional feature map based on the candidate regions to obtain a segmentation mask of the traffic signs and image regions corresponding to the segmentation masks; inputting the image regions and initial category information into a fine-tuned visual language model to determine the appearance state of the traffic signs based on the image regions, and verifying the semantic consistency between the initial category information and the image regions.

[0006] According to the present invention, a traffic sign extraction and state judgment method based on large model fine-tuning is provided. The fine-tuning process of the visual language model includes: constructing a training set including normal image-text pairs and abnormal image-text pairs; wherein, the normal image-text pairs include the image region of the traffic sign and the corresponding category information; the abnormal image-text pairs are generated by perturbing the image region or category information; and the visual language model is subjected to image and text semantic consistency learning based on the training set, so that the visual language model can distinguish between normal image-text pairs and abnormal image-text pairs, and determine the semantic consistency between the initial category information and the image region.

[0007] According to the present invention, a traffic sign extraction and state determination method based on large model fine-tuning is provided. The normal image-text pair is constructed based on the candidate region output by the fine-tuned target detection model and its corresponding initial category information, combined with the image region corresponding to the segmentation mask.

[0008] According to the present invention, a traffic sign extraction and state judgment method based on large model fine-tuning is provided. The fine-tuning process of the visual language model further includes: constructing training samples based on image regions of traffic signs with appearance state labels; converting the appearance state labels into state description text for input to the visual language model, so that the state description text and the image regions correspond to form appearance state image-text pairs; and training the visual language model based on the appearance state image-text pairs, so that the visual language model can judge the appearance state of traffic signs.

[0009] According to the present invention, a traffic sign extraction and state determination method based on large model fine-tuning is provided. The step of processing the two-dimensional feature map by a target detection model fine-tuned by a traffic sign dataset to obtain candidate regions containing the traffic signs and initial category information of the traffic signs includes: pre-selecting sample images including different types of traffic signs and labeling the traffic sign regions in the sample images to obtain sample sign regions corresponding to each traffic sign category; extracting visual features based on the sample sign regions to generate visual cue vectors for each traffic sign category, which are used by the target detection model to locate the traffic signs when detecting the two-dimensional feature map.

[0010] According to the present invention, a traffic sign extraction and state determination method based on large model fine-tuning is provided. The method involves processing the two-dimensional feature map using a target detection model fine-tuned with a traffic sign dataset to obtain candidate regions containing the traffic signs and initial category information of the traffic signs. The method further includes: detecting the two-dimensional feature map based on text prompts related to the category of the traffic signs to locate traffic sign regions in the two-dimensional feature map that semantically match the text prompts, determining the traffic sign regions as at least a part of the candidate regions, and generating corresponding initial category information for the traffic sign regions.

[0011] According to the present invention, a traffic sign extraction and state determination method based on large model fine-tuning is provided. The step of segmenting the two-dimensional feature map based on the candidate region to obtain a segmentation mask of the traffic sign and the image region corresponding to the segmentation mask includes: using the bounding box of the candidate region as a segmentation cue input to the segmentation model to generate a traffic sign segmentation mask located within the candidate region; and cropping the image region from the two-dimensional feature map according to the segmentation mask.

[0012] According to the present invention, a method for traffic sign extraction and state determination based on large model fine-tuning is provided. The method further includes vectorizing the segmentation mask to obtain vector data representing the shape boundary of the traffic sign.

[0013] This invention also provides a traffic sign extraction and state determination device based on large model fine-tuning, comprising: a feature map generation module, used to acquire point cloud data of road surface and objects containing traffic signs, and convert the point cloud data into a two-dimensional feature map; an initial category information determination module, used to process the two-dimensional feature map using a target detection model fine-tuned with a traffic sign dataset to obtain candidate regions containing the traffic signs and initial category information of the traffic signs; an image region determination module, used to segment the two-dimensional feature map based on the candidate regions to obtain a segmentation mask of the traffic signs and the image region corresponding to the segmentation mask; and a semantic verification module, used to input the image region and the initial category information into a fine-tuned visual language model to determine the appearance state of the traffic signs based on the image regions, and to verify the semantic consistency between the initial category information and the image regions.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the traffic sign extraction and status determination method based on large model fine-tuning as described above.

[0015] The present invention provides a method and apparatus for traffic sign extraction and state determination based on large model fine-tuning. This method converts acquired point cloud data into a two-dimensional feature map, and then uses a target detection model fine-tuned with a traffic sign dataset for identification, initially obtaining the initial category information of the traffic signs. A visual language model is then used to verify the semantic consistency between the initial category information and the image region, and the initial category information obtained from the target detection model is further validated to ensure the reliability of the detection results. Furthermore, the visual language model is used to determine the appearance state of the traffic signs, which is beneficial for traffic sign maintenance.

[0016] Compared with existing technologies, the traffic sign extraction and state determination method and apparatus based on large model fine-tuning provided by the present invention has at least the following technical effects: (1) By introducing semantic consistency verification in the traffic sign extraction process based on point cloud feature maps, the initial category information output by the target detection model is automatically reviewed, thereby reducing the probability of misclassification of traffic signs and improving the credibility of the detection results. (2) Input the image area and category information of the traffic sign into the visual language model in parallel. While completing the category verification, output the appearance status of the traffic sign to realize the integrated processing of traffic sign extraction and appearance status judgment, reducing manual intervention. (3) Through multi-stage collaborative processing of target detection, segmentation and semantic verification, the robustness and stability of traffic sign extraction results under complex acquisition conditions are improved, which is suitable for large-scale traffic asset management scenarios. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a traffic sign extraction and state determination method based on large model fine-tuning according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the acquisition of point cloud data according to an embodiment of the present invention; Figure 3 This is a flowchart of the fine-tuning process of a visual language model according to an embodiment of the present invention; Figure 4 This is a schematic diagram of a traffic sign extraction and status determination device based on large model fine-tuning according to an embodiment of the present invention; Figure 5 A schematic diagram of the physical structure of an electronic device is provided. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] In related technologies, the identification of traffic sign categories is often based on feature maps transformed from point cloud data of traffic signs obtained through deep learning. However, due to the lack of an automatic verification mechanism with classification results and the lack of judgment on appearance wear in existing technologies, it is not conducive to the refined management of traffic assets.

[0021] In view of this, the present invention provides a method for traffic sign extraction and status determination based on large model fine-tuning.

[0022] Figure 1 This is a flowchart of a traffic sign extraction and state determination method based on large model fine-tuning according to an embodiment of the present invention; Figure 2 This is a schematic diagram of acquiring point cloud data according to an embodiment of the present invention.

[0023] like Figure 1 As shown, the traffic sign extraction and status determination method based on large model fine-tuning includes steps S110~S140.

[0024] In step S110, point cloud data of the road surface and objects containing traffic signs are acquired, and the point cloud data is converted into a two-dimensional feature map.

[0025] According to an embodiment of the present invention, point cloud data can be acquired by vehicle-mounted mobile laser scanning, and the acquired point cloud can be segmented along the vehicle's driving trajectory.

[0026] Specifically, such as Figure 2 As shown, vehicle trajectory is the vehicle's trajectory, block ID is the block number, height is the block's slice length, width is the maximum width of the block, and overlap is the overlapping area between blocks.

[0027] For straight road sections, fixed-length slices (50 meters) are used to ensure efficient processing. For curved road sections, the slice length is dynamically adjusted according to the road's radius of curvature to ensure that the blocks adapt to the road's geometry. Each block has a maximum width of 25 meters to completely cover the road surface and traffic signs on both sides. To avoid truncating signs or markings that cross blocks, a 9-meter overlap area is set between blocks. In addition, to improve processing efficiency, point cloud data above vehicle trajectories (such as trees and buildings) is removed, retaining only the point cloud of the road surface and related objects (such as traffic signs and markings).

[0028] According to an embodiment of the present invention, a two-dimensional feature map is generated after segmentation. The features of each pixel in the two-dimensional feature map include the point's reflectance intensity, elevation variance, and distance from the point to the scanner, which are mapped to RGB channels respectively. Specifically, the point's reflectance intensity is used to distinguish road signs from the background and is mapped to the blue channel; the elevation variance is used to identify elevation changes and is mapped to the green channel; and the distance from the point to the scanner is used to compensate for the attenuation of point cloud reflectance intensity caused by greater distances and is mapped to the red channel. The vertical resolution of the two-dimensional feature map is set to 0.08 meters, and the horizontal resolution is set to 0.03 meters to better capture elongated signs.

[0029] According to an embodiment of the present invention, considering that although RGB three-channel feature maps are beneficial for providing richer feature representations, traffic signs are not prominent in them and may even affect the detection of subsequent large models, preferably, a two-dimensional feature map can be generated by assigning values ​​only according to intensity, so that traffic signs in the scene are more prominent.

[0030] In step S120, the two-dimensional feature map is processed by a target detection model fine-tuned with a traffic sign dataset to obtain candidate regions containing traffic signs and initial category information of traffic signs.

[0031] According to embodiments of the present invention, the object detection model can be a deep learning-based object detection model, for example, it can be based on a neural network model or a visual Transformer architecture.

[0032] In some embodiments, the object detection model can detect traffic signs based on cue information in a two-dimensional feature map. This cue information can include visual cues and textual cues. For example, visual cues characterize the appearance features of traffic signs, while textual cues characterize the category semantic information of traffic signs. By introducing these cue information, the object detection model is not limited to a fixed set of categories during detection and can locate and identify traffic signs in the two-dimensional feature map based on the input cue information, thus enabling traffic sign detection under open-category conditions.

[0033] During the detection process, the target detection model determines the region that matches the prompt information in the two-dimensional feature map based on the prompt information, and outputs the region as a candidate region.

[0034] Furthermore, the object detection model determines the traffic sign category corresponding to the candidate region based on the prompt information that matches the candidate region, and outputs the traffic sign category as the initial category information of the candidate region.

[0035] According to an embodiment of the present invention, for the object detection model, the fine-tuning process can adopt an end-to-end training approach, with the optimization objective being the detection loss function (such as classification loss and bounding box regression loss), so that the model learns the specific distribution pattern and contextual features of traffic signs in the two-dimensional feature map, thereby improving the robustness of the object detection model.

[0036] In step S130, the two-dimensional feature map is segmented based on the candidate region to obtain the segmentation mask of the traffic sign and the image region corresponding to the segmentation mask.

[0037] According to an embodiment of the present invention, a general segmentation model can be used to segment traffic signs in a two-dimensional feature map by taking candidate regions as input prompts, so as to generate a segmentation mask.

[0038] According to embodiments of the present invention, the segmentation model may be a semantic segmentation model, an instance segmentation model, or a segmentation model based on CNN or Transformer.

[0039] In step S140, the image region and initial category information are input into the fine-tuned visual language model to determine the appearance of the traffic sign based on the image region and to verify the semantic consistency between the initial category information and the image region.

[0040] According to an embodiment of the present invention, the initial category information can be input into the visual language model in the form of a category name or a text description generated from the category name. The visual language model then verifies the semantic consistency between the initial category information and the image region based on the association between the image region and the text description, and outputs the corresponding verification result.

[0041] For example, if the verification result shows that the initial category information output by the target detection model matches the category of the traffic sign, the detection result of the target detection model is considered correct. If the initial category information does not match the category of the traffic sign, the result is marked as abnormal and added to the list to be manually verified.

[0042] According to an embodiment of the present invention, the visual language model can judge the appearance status of traffic signs based on the appearance features of the image region. The appearance features include the edge integrity, pattern clarity, texture continuity, and overall appearance consistency of the traffic signs; the appearance status includes whether the traffic signs are worn, damaged, or have abnormal appearance.

[0043] According to embodiments of the present invention, the visual language model can be a multimodal pre-trained model (Contrastive Language-Image Pretraining, CLIP) or a generative pre-trained transformer (GPT).

[0044] Through the aforementioned setup, this invention extracts candidate regions for traffic signs from a two-dimensional feature map generated based on point cloud data. Based on this, it combines segmentation processing to obtain the image regions of the traffic signs, thus providing accurate target regions for subsequent semantic analysis. Furthermore, by inputting the image regions and initial category information into a finely tuned visual language model, the semantic consistency of the traffic sign categories is verified, and the appearance status of the traffic signs is simultaneously assessed, achieving automated review and status evaluation of the traffic sign detection results. Therefore, without increasing manual intervention, the reliability and stability of the traffic sign extraction results are improved, making it suitable for traffic asset management and maintenance applications in complex road scenarios.

[0045] In one illustrative embodiment, the fine-tuning process of the visual language model includes: A training set is constructed that includes normal image-text pairs and abnormal image-text pairs; wherein, normal image-text pairs include the image region of traffic signs and the corresponding category information; abnormal image-text pairs are generated by perturbing the image region or category information.

[0046] Based on the training set, the visual language model is used to learn the consistency between image and text semantics, so that the visual language model can distinguish between normal image-text pairs and abnormal image-text pairs, and judge the semantic consistency between the initial category information and the image region.

[0047] According to an embodiment of the present invention, a normal image-text pair refers to an image-text combination in which the image region and its corresponding category information are semantically consistent.

[0048] In some embodiments, normal image-text pairs can be constructed based on candidate regions output by the object detection model and their corresponding initial category information, wherein samples whose candidate regions are semantically consistent with their corresponding initial category information, as confirmed by manual verification, are considered normal image-text pairs.

[0049] In some embodiments, normal image-text pairs can also be obtained based on a pre-built traffic sign annotation dataset, which includes image regions of traffic signs and their corresponding category annotations.

[0050] According to embodiments of the present invention, abnormal image-text pairs can be obtained by replacing or mismatching the category information in normal image-text pairs, or by cropping, occluding, blurring, or noise processing the image region.

[0051] By using the above setup, the visual language model is fine-tuned using a training set containing both normal and abnormal image-text pairs. This enables the visual language model to distinguish between semantically consistent and semantically inconsistent image regions and category information during the inference phase. Consequently, when performing semantic consistency verification on the initial category information output by the object detection model, the visual language model can effectively identify detection results with category mismatches or semantic discrepancies, thereby improving the reliability of the traffic sign detection result verification process and reducing the risk of misjudgment.

[0052] In one illustrative embodiment, the normal image-text pair is constructed based on the candidate regions output by the fine-tuned object detection model and their corresponding initial category information, combined with the image regions corresponding to the segmentation mask.

[0053] According to an embodiment of the present invention, for each candidate region, an image region containing traffic signs is cropped from the two-dimensional feature map based on the segmentation mask corresponding to the candidate region, and the initial category information is converted into text information corresponding to the traffic sign category; the image region and the text information together constitute a normal image-text pair.

[0054] Figure 3 This is a flowchart of the fine-tuning process of a visual language model according to an embodiment of the present invention.

[0055] like Figure 3 As shown, the fine-tuning process of the visual language model includes steps S310 to S330.

[0056] In step S310, training samples are constructed based on the image regions of traffic signs with appearance status labels.

[0057] According to embodiments of the present invention, appearance status labels are used to characterize the appearance status of traffic signs. These labels can be obtained through manual annotation or generated based on existing traffic maintenance records or historical data. Appearance status labels may include status categories such as no wear, slight wear, moderate wear, or heavy wear.

[0058] In step S320, the appearance status label is converted into status description text for input to the visual language model, so that the status description text and the image region correspond to form an appearance status image-text pair.

[0059] According to an embodiment of the present invention, the appearance status image-text pair can be an image-text combination consisting of an image area of ​​a traffic sign and a status description text used to characterize the appearance status of the traffic sign.

[0060] In step S330, a visual language model is trained based on the appearance state image-text pair so that the visual language model can judge the appearance state of traffic signs.

[0061] In one illustrative embodiment, step S120 includes: Sample images containing different types of traffic signs are pre-selected, and the traffic sign areas in the sample images are labeled to obtain sample sign areas corresponding to each traffic sign category.

[0062] Visual features are extracted from the sample sign regions to generate visual cue vectors for each traffic sign category, which are then used by the target detection model to locate traffic signs when detecting two-dimensional feature maps.

[0063] According to an embodiment of the present invention, in order to enable the target detection model to detect traffic signs based on visual cues, sample images containing different types of traffic signs are pre-selected, and the traffic sign regions in the sample images are manually labeled to obtain sample sign regions corresponding to each traffic sign category. Based on the sample sign regions, visual cues vectors are obtained. The target detection model locates the region where the traffic sign is located in the two-dimensional feature map through the visual cues vectors, and finally outputs the location result as a candidate region.

[0064] In illustrative terms, the object detection model can perform open-set object detection based on visual cues. Specifically, images containing target traffic signs (such as straight, left turn, right turn, straight and left turn, straight and right turn, and diamond-shaped lines) are selected from the generated 2D feature map as samples. LabelMe is used to label the six signs, obtaining the coordinates of the bounding boxes and assigning them labels. After successfully obtaining the bounding box cues corresponding to the signs, the corresponding visual embeddings are generated sequentially, serving as visual cues for the subsequent zero-shot object detection model based on visual cues.

[0065] In one illustrative embodiment, step S120 further includes: detecting the two-dimensional feature map based on text cues related to the category of the traffic sign, so as to locate the traffic sign region in the two-dimensional feature map that matches the semantics of the text cues, and determining the traffic sign region as at least a part of the candidate region, and generating corresponding initial category information for the traffic sign region.

[0066] For example, the object detection model can perform detection based on a two-dimensional feature map using text prompts. Specifically, the text prompts used to represent traffic sign categories are compared with the two-dimensional features. Figure 1 Similar to the input object detection model, this model locates traffic sign regions in the 2D feature map that semantically match the text prompts based on the semantic matching relationship between the text prompts and the image content. The object detection model generates corresponding candidate regions in the 2D feature map according to the traffic sign category corresponding to the text prompts and outputs the corresponding initial category information for each candidate region.

[0067] According to embodiments of the present invention, the target detection model can simultaneously detect two-dimensional feature maps based on image cues and text cues, and determine candidate regions of traffic signs and corresponding initial category information based on the matching relationship between image cues, text cues and image content in the two-dimensional feature maps.

[0068] In one illustrative embodiment, step S130 includes: The bounding boxes of the candidate regions are used as segmentation cue inputs to the segmentation model to generate traffic sign segmentation masks located within the candidate regions; Image regions are cropped from the two-dimensional feature map based on the segmentation mask.

[0069] In one illustrative embodiment, the segmentation mask can also be vectorized to obtain vector data representing the shape boundaries of the traffic sign.

[0070] According to embodiments of the present invention, vector data is used to describe the precise shape, spatial coordinates, orientation, and other information of traffic signs in the real world. It can be used for real-world 3D modeling and can also provide vectorized results for high-precision maps.

[0071] The traffic sign extraction and state determination device based on large model fine-tuning provided by the present invention will be described below. The traffic sign extraction and state determination device based on large model fine-tuning described below can be referred to in correspondence with the traffic sign extraction and state determination method based on large model fine-tuning described above.

[0072] Figure 4 This is a schematic diagram of a traffic sign extraction and status determination device based on large model fine-tuning according to an embodiment of the present invention.

[0073] like Figure 4 As shown, the traffic sign extraction and status judgment device 400 based on large model fine-tuning provided in this embodiment of the invention includes a feature map generation module 410, an initial category information determination module 420, an image region determination module 430, and a semantic verification module 440.

[0074] The feature map generation module 410 is used to acquire point cloud data of the road surface and objects containing traffic signs, and convert the point cloud data into a two-dimensional feature map.

[0075] The initial category information determination module 420 is used to process the two-dimensional feature map through a target detection model finely tuned to the traffic sign dataset to obtain candidate regions containing traffic signs and initial category information of traffic signs.

[0076] The image region determination module 430 is used to segment the two-dimensional feature map based on the candidate region to obtain the segmentation mask of the traffic sign and the image region corresponding to the segmentation mask.

[0077] The semantic verification module 440 is used to input the image region and initial category information into the fine-tuned visual language model to determine the appearance of the traffic sign based on the image region and to verify the semantic consistency between the initial category information and the image region.

[0078] According to embodiments of the present invention, any plurality of modules among the feature map generation module 410, the initial category information determination module 420, the image region determination module 430, and the semantic verification module 440 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of the present invention, at least one of the feature map generation module 410, the initial category information determination module 420, the image region determination module 430, and the semantic verification module 440 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any one of the three implementation methods or a suitable combination of any of them. Alternatively, at least one of the feature map generation module 410, the initial category information determination module 420, the image region determination module 430, and the semantic verification module 440 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0079] Figure 5 A schematic diagram of the physical structure of an electronic device is provided.

[0080] like Figure 5As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540. The processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions from the memory 530 to execute a traffic sign extraction and status determination method based on large model fine-tuning.

[0081] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0082] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0083] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for traffic sign extraction and state determination based on large model fine-tuning, characterized in that, include: Acquire point cloud data of the road surface and objects containing traffic signs, and convert the point cloud data into a two-dimensional feature map; The two-dimensional feature map is processed by a target detection model finely tuned with a traffic sign dataset to obtain candidate regions containing the traffic signs and initial category information of the traffic signs. The two-dimensional feature map is segmented based on the candidate region to obtain the segmentation mask of the traffic sign and the image region corresponding to the segmentation mask; The image region and initial category information are input into a fine-tuned visual language model to determine the appearance of the traffic sign based on the image region, and to verify the semantic consistency between the initial category information and the image region.

2. The traffic sign extraction and state determination method based on large model fine-tuning according to claim 1, characterized in that, The fine-tuning process of the visual language model includes: A training set is constructed, comprising normal image-text pairs and abnormal image-text pairs; wherein, the normal image-text pairs include the image region of the traffic sign and the corresponding category information; the abnormal image-text pairs are generated by perturbing the image region or category information; Based on the training set, the visual language model is trained to learn the consistency between image and text semantics, so that the visual language model can distinguish between normal image-text pairs and abnormal image-text pairs, and determine the semantic consistency between the initial category information and the image region.

3. The traffic sign extraction and state determination method based on large model fine-tuning according to claim 2, characterized in that, The normal image-text pair is constructed based on the candidate regions output by the fine-tuned target detection model and their corresponding initial category information, combined with the image region corresponding to the segmentation mask.

4. The traffic sign extraction and state determination method based on large model fine-tuning according to claim 1 or 2, characterized in that, The fine-tuning process of the visual language model also includes: Training samples are constructed based on image regions of traffic signs with appearance status labels; The appearance status label is converted into a status description text for input to the visual language model, so that the status description text and the image region correspond to form an appearance status image-text pair; The visual language model is trained based on the appearance state image-text pair so that the visual language model can determine the appearance state of traffic signs.

5. The traffic sign extraction and state determination method based on large model fine-tuning according to claim 1, characterized in that, The process of processing the two-dimensional feature map using a target detection model fine-tuned with a traffic sign dataset to obtain candidate regions containing the traffic signs and initial category information of the traffic signs includes: Sample images containing different types of traffic signs are pre-selected, and the traffic sign areas in the sample images are labeled to obtain sample sign areas corresponding to each traffic sign category; Visual features are extracted based on the sample sign regions to generate visual cue vectors for each traffic sign category, which are then used by the target detection model to locate the traffic signs when detecting the two-dimensional feature map.

6. The traffic sign extraction and state determination method based on large model fine-tuning according to claim 5, characterized in that, The step of processing the two-dimensional feature map using a target detection model fine-tuned with a traffic sign dataset to obtain candidate regions containing the traffic signs and initial category information of the traffic signs also includes: The two-dimensional feature map is detected based on textual cues related to the category of the traffic sign, so as to locate the traffic sign region in the two-dimensional feature map that matches the semantics of the textual cues, and determine the traffic sign region as at least a part of the candidate region, and generate corresponding initial category information for the traffic sign region.

7. The traffic sign extraction and state determination method based on large model fine-tuning according to claim 1, characterized in that, The step of segmenting the two-dimensional feature map based on the candidate region to obtain a segmentation mask for traffic signs and the image region corresponding to the segmentation mask includes: The bounding box of the candidate region is used as a segmentation cue input to the segmentation model to generate a traffic sign segmentation mask located within the candidate region; The image region is obtained by cropping from the two-dimensional feature map based on the segmentation mask.

8. The traffic sign extraction and state determination method based on large model fine-tuning according to claim 7, characterized in that, The method further includes vectorizing the segmentation mask to obtain vector data representing the shape boundary of the traffic sign.

9. A traffic sign extraction and status determination device based on large model fine-tuning, characterized in that, include: The feature map generation module is used to acquire point cloud data of the road surface and objects containing traffic signs, and convert the point cloud data into a two-dimensional feature map. The initial category information determination module is used to process the two-dimensional feature map using a target detection model finely tuned to the traffic sign dataset to obtain candidate regions containing the traffic signs and the initial category information of the traffic signs. The image region determination module is used to segment the two-dimensional feature map based on the candidate region to obtain the segmentation mask of the traffic sign and the image region corresponding to the segmentation mask; The semantic verification module is used to input the image region and initial category information into a fine-tuned visual language model to determine the appearance state of the traffic sign based on the image region and to verify the semantic consistency between the initial category information and the image region.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the traffic sign extraction and status determination method based on large model fine-tuning as described in any one of claims 1 to 8.

Images