Guardrail detection method, device and equipment and storage medium

Abstract

The invention relates to the field of intelligent detection, and discloses a guardrail detection method, device and equipment and a storage medium, and the method comprises the steps: carrying out the preliminary detection of the guardrail state of an input image through a first guardrail detection model, and obtaining a first detection result and confidence; according to the first detection result, carrying out guardrail state secondary detection on the input image of which the confidence coefficient is lower than a confidence coefficient threshold value by utilizing a second guardrail detection model to obtain a second detection result; and combining the first detection result and the second detection result to obtain a target result. The problem of low accuracy of guardrail state detection in the prior art is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent detection, and in particular to a guardrail detection method, device, equipment and storage medium. BACKGROUND

[0002] The existing method for detecting the state of the guardrail generally has the following two methods: one is to extract the image contour in combination with the CANNY operator and merge the channels, and then train a guardrail classification model based on the ResNet50 network to realize the recognition of the safety state of the input image (including the conditions of the guardrail being installed too low, not being provided with a foot plate, being damaged, being removed and being tilted), and some model compression techniques are combined to improve the analysis speed and efficiency; or the other is to train a segmentation model to realize the extraction of the guardrail contour and the auxiliary line of the edge of the reference image and the real-time image, and then compare the intersection and union ratio of the guardrail contours of the two images and the included angle of the auxiliary lines to judge the collapse of the guardrail, and then train a classification model to realize the recognition of the damage degree of the guardrail.

[0003] The existing technical solution only detects the state of the guardrail through one model, and the detection accuracy is low. SUMMARY

[0004] The present application provides a guardrail detection method, device, computer equipment and storage medium to solve the problem of low accuracy of the existing technology in detecting the state of the guardrail.

[0005] In a first aspect, a guardrail detection method is provided, comprising: performing preliminary detection of the state of the guardrail on an input image by using a first guardrail detection model to obtain a first detection result and a confidence level; According to the first detection result, performing secondary detection of the state of the guardrail on the input image with a confidence level lower than a confidence threshold by using a second guardrail detection model to obtain a second detection result; obtaining a target result in combination with the first detection result and the second detection result.

[0006] Optionally, the training method of the first guardrail detection model comprises: obtaining a guardrail image and performing labeling to obtain first data; training a guardrail special detection model based on deep learning by using the first data to obtain the first guardrail detection model.

[0007] Optionally, the training of the guardrail special detection model based on deep learning by using the first data to obtain the first guardrail detection model comprises: performing data enhancement on the first data to obtain second data; training the guardrail special detection model based on deep learning by using the second data to obtain the first guardrail detection model.

[0008] Optionally, the acquiring guardrail images and labeling to obtain first data further comprises: Based on the labeled guardrail images, augmented images are obtained through image augmentation technology. The first data comprises the labeled guardrail images and augmented images.

[0009] Optionally, the acquiring guardrail images and labeling to obtain first data comprises segmenting and labeling single safety guardrails and gap regions between guardrails in the guardrail images, and the labeling comprises features of the gap regions and the fence regions on both sides.

[0010] Optionally, the using a second guardrail detection model to perform secondary detection of the guardrail state to obtain a second detection result comprises: The second guardrail detection model is fine-tuned through prompt words and examples.

[0011] Optionally, the prompt words comprise task description, analysis thought description, input format specification, and second detection result format definition.

[0012] In a second aspect, a guardrail detection device is provided, comprising: A preliminary detection module is configured to use a first guardrail detection model to perform preliminary detection of a guardrail state of an input image to obtain a first detection result and a confidence level. A secondary detection module is configured to use a second guardrail detection model to perform secondary detection of the guardrail state of an input image with a confidence level lower than a confidence level threshold to obtain a second detection result. An output module is configured to combine the first detection result and the second detection result to obtain a target result.

[0013] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above guardrail detection method when executing the computer program.

[0014] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program implements the steps of the above guardrail detection method when executed by a processor.

[0015] The scheme implemented by the guardrail detection method, device, equipment and storage medium can be widely applied to detection of safety guardrails in scenes such as road, highway, engineering construction protection and dangerous area isolation. The first guardrail detection model is used to perform preliminary detection on the input image to obtain a first detection result and a confidence level. According to the first detection result, the second guardrail detection model is used to perform secondary detection on the input image with a confidence level lower than a confidence level threshold to obtain a second detection result. The target result is obtained by combining the first detection result and the second detection result. In the present application, the first guardrail detection model performs preliminary detection, and the result with a high confidence level is directly included in the preliminary conclusion to ensure the detection accuracy in clear and typical scenes. For the fuzzy sample with a low confidence level (such as a small guardrail size, a blurred image, and shielding), the second guardrail detection model is used to perform secondary detection to avoid misjudgment or omission caused by the limitation of a single model. Therefore, the problem of low accuracy of guardrail state detection in the prior art is solved. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0017] Figure 1 is a flowchart of a guardrail detection method in an embodiment of the present application; Figure 2 is a structural diagram of a guardrail detection device in an embodiment of the present application. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0019] It should be understood that when used in the specification and the appended claims of the present application, the term "comprising" indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or sets thereof.

[0020] It should also be understood that the term "and / or" as used herein refers to any one of the associated listed items, or a combination of any and all of the associated listed items, and includes these combinations.

[0021] As used in the description of the application and the appended claims, the term "if' can be interpreted to mean "when" or "upon" or "in response to determining" depending on the context. Similarly, the phrase "if it is determined" or "if the [described condition or event] is met" can be interpreted to mean "upon determining" or "in response to determining" or "upon the [described condition or event] being met" or "in response to the [described condition or event] being met," depending on the context.

[0022] In addition, the terms "first", "second", "third", etc. as used in the description of the application and the appended claims are merely used to differentiate descriptions and cannot be understood as indicating or implying relative importance.

[0023] Reference in the specification to "one embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase "in one embodiment" or "in some embodiments" in various places in the specification are not necessarily all referring to the same embodiment, although it can. The terms "including," "containing," "having," and variations thereof are meant to encompass the terms "including but not limited to."

[0024] It should be understood that the size of the serial number of each step in the following embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the application.

[0025] In order to illustrate the technical solutions of the application, the following specific embodiments are described.

[0026] Please refer to Figure 1 as shown, Figure 1 A flowchart of a guardrail detection method provided by an embodiment of the application is shown in FIG. 1, which includes the following steps: S11: using a first guardrail detection model to perform preliminary detection of the state of the guardrail in the input image to obtain a first detection result and a confidence level.

[0027] This step is the basic link of the entire guardrail detection process, and its core function is to quickly screen the guardrail target in the input image and preliminarily judge the state, providing a basis for subsequent processing.

[0028] The first guardrail detection model is usually a lightweight model with the ability to quickly process images, and can efficiently complete preliminary screening while ensuring a certain detection accuracy. The design goal is to deal with large-scale and diversified image inputs and quickly lock the areas where the guardrails and guardrail state abnormalities may exist.

[0029] The content of the preliminary detection of the guardrail state mainly includes detecting whether there is a guardrail in the input image, the position of the guardrail (usually presented in the form of a coordinate box), and the basic state of the guardrail (such as whether it is complete, whether there is a significant loss, etc.). During the detection process, the model will learn the guardrail features (such as shape, color, texture, etc.) during training, and perform regional scanning and identification on the input image.

[0030] The first detection result generally includes the detected guardrail information, such as the coordinate range of each guardrail (for example, a certain guardrail in the image is located in the area from the upper left corner coordinate (x1, y1) to the lower right corner coordinate (x2, y2)), the type of the guardrail (such as road guardrail, bridge guardrail, etc.), and the preliminary determined state (such as complete, suspected loss, etc.).

[0031] Confidence is a quantitative evaluation of the reliability of the model's detection results, usually represented by a value between 0 and 1. The closer the value is to 1, the more reliable the model considers the detection result to be; the closer the value is to 0, the lower the model's confidence in the detection result. The calculation of confidence is based on the probability output inside the model and is related to factors such as the clarity of the features of the guardrail in the input image and the similarity to the training samples.

[0032] For example, a monitoring image of a road contains various guardrails and complex environments: Clear and complete guardrail scene: There is no obstruction in the image of a certain road guardrail, and the light is sufficient, and the shape, color, and other features of the guardrail are clear and identifiable. After the first guardrail detection model scans this area, it can accurately identify that it is a road guardrail by comparing it with the complete guardrail features in the training samples, and determine that its state is complete, with a high confidence value (such as 0.9). The first detection result is: guardrail coordinates (x1, y1) - (x2, y2), state complete, confidence 0.9. Guardrail scene with partial obstruction: Another segment of the guardrail in the image is partially obstructed by roadside shrubs, revealing only part of the guardrail. During model detection, the matching degree with the training samples decreases due to the obstruction, resulting in the loss of some features. The model can still identify it as a guardrail, but it preliminarily determines that there may be some state abnormalities (such as partial loss), with a relatively low confidence value (such as 0.6). The first detection result is: guardrail coordinates (x3, y3) - (x4, y4), state suspected partial loss, confidence 0.6. Fuzzy guardrail scene: due to low resolution of monitoring camera or weather reasons (such as rain), the guardrail profile in a certain area of the input image is fuzzy, and the features are not obvious. When the model detects, it is difficult to clearly extract the key features of the guardrail, and can only roughly judge that there may be a guardrail in the area, but cannot accurately determine its state. At this time, the output confidence is very low (such as 0.3), and the first detection result is: the guardrail coordinates (x5, y5) - (x6, y6), the state is unknown, and the confidence is 0.3.

[0033] Missing guardrail scene: in a section of monitoring image, the originally continuous central divider guardrail has a obvious broken gap, there is no guardrail structure in the gap, and the surrounding road surface is flat without any guardrail residual parts. The first guardrail detection model scans the area and finds that the guardrail features in the area are interrupted according to the continuous features of the complete guardrail in the training samples (such as continuous guardrail arrangement, fixed spacing, etc.). The model will first identify the guardrail parts on both sides of the gap, whose coordinates are (x7, y7) - (x8, y8) and (x9, y9) - (x 10 ,y 10 ), and determine that the states of the two sections of guardrails are complete, with confidence of 0.85 and 0.88 respectively. At the same time, the model detects that there is an obvious blank area between the two sections of guardrails by comparing the position relationship and continuous features of the two sections of guardrails, and preliminarily determines that the blank area is a missing guardrail area according to the "missing guardrail" features learned during training (such as the sudden interruption of continuous guardrails, the discordance between the blank area and the surrounding environment, etc.). The output of the first detection result is: the missing area coordinates (x 11 ,y 11 ) - (x 12 ,y 12 ), the state is missing guardrail, and the confidence is 0.75.

[0034] S12: according to the first detection result, for the input image with confidence lower than the confidence threshold, a second guardrail detection model is used for secondary detection of the guardrail state to obtain a second detection result.

[0035] This step is the key link to improve the accuracy in the whole detection process, and the core is to perform deep detection on the input image with low confidence by using the second guardrail detection model with higher analysis ability.

[0036] The confidence threshold is the critical point for judging whether secondary detection is needed (such as the threshold is set to 0.7), when the confidence of the first detection result is lower than the threshold, it means that the first guardrail detection model has insufficient reliability in grasping the result (may be caused by image blur, atypical features, complex interference, etc.). At this time, the second guardrail detection model is started, which is not to deny the first detection result, but to reduce the risk of misjudgment through more comprehensive analysis.

[0037] The second guardrail detection model is usually a large model with strong semantic understanding and multi-modal analysis capability (such as a visual language large model), which can combine global image information, target correlation and prior knowledge for reasoning. Unlike the first guardrail detection model, which focuses on rapid positioning, the second guardrail detection model is better at handling ambiguous and complex scenes, and improves judgment accuracy by integrating multi-dimensional information such as guardrail continuity logic and environmental context.

[0038] For example, in the above highway guardrail missing scene, the missing area confidence output by the first guardrail detection model is 0.65 (lower than the threshold of 0.7), triggering secondary detection, the specific process is as follows: Information input: The information received by the second guardrail detection model includes: the original monitoring image, the first detection result (the coordinates of the two sides of the gap (x7, y7) - (x8, y8), (x9, y9) - (x 10 , 10 ), the coordinates of the missing area (x 11 , 11 ) - (x 12 , 12 ), the state "guardrail missing", and the confidence 0.65).

[0039] Second guardrail detection model deep analysis process: The second guardrail detection model first observes the overall environment of the input image and finds that the area is a highway curve. The guardrail should be continuous to ensure driving safety, and there is no construction sign around the gap (excluding the possibility of temporary removal), and there is no guardrail debris on the road (excluding recent collision damage marks). By comparing the direction of the two sides of the guardrail, it is found that the extension line at the gap should be connected, and the gap width (about 3 meters) is much larger than the normal guardrail connection gap, which meets the spatial logic of "missing". At the same time, the second guardrail detection model combines the knowledge learned during training, such as "highway guardrail standard spacing" and "continuous protection requirements", to determine that the blank area does not meet the normal design specifications. For the points that the first guardrail detection model may misjudge (such as "false gap" caused by shadows / obstructions), the model magnifies the pixel features of the gap area and confirms that there is no guardrail texture or structure left, and it is consistent with the surrounding road features, excluding the possibility of "misjudging shadows". Output the second detection result: After comprehensive analysis by the second guardrail detection model, the output is "confirm the missing area coordinates (x 11 , 11 ) - (x 12 , 12 ), the state is guardrail missing, and the confidence is 0.92", and it is supplemented that "the two sides of the guardrail are continuous, the gap has no reasonable design basis, and the misjudgment possibility is excluded".

[0040] Through secondary detection, the low-confidence result originally in doubt is verified, avoiding the missed judgment caused by the first guardrail detection model's insufficient judgment of complex features, and enhancing the reliability of the result through the second guardrail detection model's enhanced logical reasoning ability.

[0041] S13: Obtain a target result by combining the first detection result and the second detection result.

[0042] This step is the final integration link of the entire guardrail detection process, and its core is to fuse the results of the two detections through scientific rules, fully utilizing the efficiency advantage of the high-confidence part in the first detection result, and accurately correcting the low-confidence part through the second detection result to avoid the limitations of a single result. For high-confidence results without secondary detection, they are directly included in the target result; for low-confidence results with secondary detection, the second detection result is used as the core to form the final target result.

[0043] For example, in a highway monitoring image, the first guardrail detection model outputs the following results: Region A: guardrail is complete, coordinates (x1, y1)-(x2, y2), confidence 0.85 (higher than the threshold 0.7, no need for secondary detection); Region B: suspected guardrail missing, coordinates (x3, y3)-(x4, y4), confidence 0.65 (lower than the threshold, secondary detection by the second model); Since the confidence of the first detection result of region B is lower than the threshold, the second guardrail detection model needs to be used for secondary detection; Second detection result: confirms that region B is a missing guardrail, coordinates are corrected to (x3', y3')-(x4', y4') (due to possible deviation of the first guardrail detection model), and the confidence is 0.92, which is higher than the threshold.

[0044] Target result: directly retain the first detection result of region A, as its confidence is high and no secondary correction is needed; use the second detection result of region B, and finally output: guardrail is complete, coordinates (x1, y1)-(x2, y2); guardrail is missing, coordinates (x3', y3')-(x4', y4').

[0045] In another case, if the confidence of the second detection result of region B is lower than the threshold, then both the first and second detection results of region B are inaccurate, and the guardrail state of region B is not output The guardrail detection method uses a first guardrail detection model to preliminarily detect the guardrail state of an input image to obtain a first detection result and a confidence level; according to the first detection result, for an input image with a confidence level lower than a confidence level threshold, a second guardrail detection model is used for secondary detection of the guardrail state to obtain a second detection result; and the target result is obtained by combining the first detection result and the second detection result. In the present application, the first guardrail detection model preliminarily detects, and the results with high confidence levels are directly included in the preliminary conclusion to ensure the detection accuracy in clear and typical scenes; and for fuzzy samples with low confidence levels (such as small guardrail size, image blur, occlusion, etc.), secondary detection is performed through the second guardrail detection model to avoid misjudgment or omission due to the limitations of a single model. Therefore, the problem of low accuracy of guardrail state detection in the prior art is solved.

[0046] In an embodiment, the training method of the first guardrail detection model comprises: obtaining guardrail images and performing labeling to obtain first data; and training a deep learning-based guardrail special detection model based on the first data to obtain the first guardrail detection model.

[0047] The obtained guardrail images need to cover diversified guardrail scenes, including different types of guardrails (such as road guardrails, bridge guardrails, and factory guardrails), different environmental conditions (sunny, rainy, night, backlight), different shooting angles (eye level, bird's eye view, side view), different states (complete, missing), and different interference factors (occlusion, blur, small size). In this way, the model can have strong generalization ability after training and adapt to complex situations in actual applications.

[0048] The obtained images are finely labeled, and the labeling content mainly includes the coordinates of the guardrail and the related area (usually in the form of a bounding box, such as (x1, y1, x2, y2) to represent a rectangular area) and the state label (such as "complete" and "missing"). The labeling needs to be accurate and standardized to provide reliable supervision information for model training. For example, for the guardrail missing scene, the coordinates of the missing area and the "missing" label need to be clearly labeled.

[0049] The deep learning-based guardrail special detection model generally selects a lightweight deep learning architecture as the basis, such as a yolov8-based m lightweight network structure.

[0050] The following constructs a loss function of a first guardrail detection model based on a yolov8-based m lightweight network structure: l t The total loss function represents an index for measuring the overall prediction error of the model for the input image, and the smaller the value, the closer the model prediction result is to the actual situation; l crepresents a category loss function, which is used to measure the difference between the predicted guardrail state in the input image and the true labeled guardrail state in the first data; l b represents a bounding box loss function, which is used to measure the difference between the predicted guardrail bounding box in the input image and the true labeled guardrail bounding box in the first data; m represents the total number of real targets, i.e. the total number of guardrails in the first data, reflecting the sample size of the model learning.

[0051] ω c represents a category loss weight, which is a preset value and can be adjusted according to actual conditions; y c represents a real single-class binary classification value, i.e. a labeled guardrail state value (such as "missing" being recorded as 1 and "complete" being recorded as 0); p represents a predicted single-class score, i.e. a guardrail state value predicted by the model, without size limitation; σ(p) represents a sigmoid activation function, which maps the predicted single-class score p to the interval of 0-1.

[0052] ω b represents a bounding box loss weight, which is a preset value and can be adjusted according to actual conditions; q represents the bounding box coordinates predicted by the model; g represents the true bounding box coordinates labeled in the first data; IOU(q, g) represents the intersection over union, which measures the overlapping degree of the predicted bounding box coordinates and the true bounding box coordinates.

[0053] The weight parameter gradient of the first guardrail detection model is calculated according to the total loss function, and then the SGDM algorithm, i.e. the stochastic gradient descent optimization algorithm with momentum, is used to help the first guardrail detection model converge to the optimal weight faster and improve the detection accuracy. This algorithm belongs to the known technology and will not be described here.

[0054] In an embodiment, the first guardrail detection model is obtained by training the guardrail special detection model based on deep learning using the first data, including: performing data enhancement on the first data to obtain second data; and training the guardrail special detection model based on deep learning using the second data to obtain the first guardrail detection model.

[0055] Data augmentation has the following benefits: first, expand the sample diversity, the original first data (labeled guardrail images) may have single scene, limited angle, incomplete state and other problems, through data augmentation, various complex situations (such as different light, occlusion, weather conditions) can be simulated, so that the model learns more comprehensive guardrail features; second, alleviate the lack of data, especially for the missing rare state of guardrail, the real sample may be less, data augmentation can artificially generate more samples of this kind, avoiding the model overfitting to common scenes; third, improve the robustness of the model, the enhanced data contains more noise and interference factors, forcing the model to learn more essential guardrail features, so as to have stronger adaptability to unseen complex scenes in actual application.

[0056] For example, to train a first guardrail detection model for urban road guardrail detection, first, the acquisition and labeling of first data are performed, guardrail images of different road sections in urban roads are collected, covering guardrails of main roads, secondary roads, branch roads, including sunny and well-lit, rainy and wet, night and light-illuminated, tree-shaded partial guardrails, small-size guardrails in the distance, and various other situations, a total of 100,000 images are collected. The images are labeled: for an image containing a complete guardrail, the boundary box coordinates (x1, y1, x2, y2) of the guardrail and the label "complete" are labeled; for an image with missing guardrails, the boundary box (x3, y3, x4, y4) of the missing area and the label "missing" are labeled, and the coordinates and label of the surrounding complete guardrails are labeled. After processing by professional annotators, 100,000 pieces of labeled data, i.e., first data, are obtained. Then, second data is obtained through data augmentation, and the data augmentation methods generally include: rotation and flipping, randomly rotating the original image by a certain angle (such as -15° to 15°), and performing horizontal / vertical flipping to generate new images. For example, a horizontal guardrail image is rotated by 90° to become a vertical one, simulating scenes taken from different angles; brightness and contrast adjustment: adjusting the brightness and contrast of the image to simulate guardrails under different lighting conditions, such as reducing the brightness of a sunny guardrail image to simulate an overcast or night scene; adding noise: adding random noise (such as Gaussian noise) to the image to simulate disturbances in the image acquisition process, for example, adding noise to a rainy guardrail image to simulate the impact of raindrops on image quality; occlusion simulation, using a rectangle or irregular shape to occlude part of the guardrail to simulate the occlusion of trees, vehicles, etc. to the guardrail, for example, occluding the middle part of the guardrail with a rectangular area to force the model to learn to recognize the complete guardrail from local features; combined enhancement: combining the above-mentioned multiple enhancement methods to generate more complex and diverse images, such as rotating a sunny guardrail image clockwise by 10°, then reducing the brightness, and finally adding Gaussian noise. Through these data augmentation operations, the 100,000 first data are expanded to 500,000 second data, greatly enriching the diversity of samples. It should be noted that the present application is mainly aimed at intelligent construction, supervision, and other business scenarios, and has been technically verified. The same technical solution can also be used for conventional urban roads.

[0057] In an embodiment, the obtaining of the guardrail image and the labeling to obtain the first data further include: based on the labeled guardrail image, an augmented image is obtained through image augmentation technology; and the first data include the labeled guardrail image and the augmented image.

[0058] The image augmentation is a new sample generated through artificial construction based on the existing labeled image, and the core purpose is to make up for the deficiency of the original data.

[0059] In the present application, the copypaste image augmentation technique can be generally used, by "copying" and "pasting" the guardrail region in the labeled image to other background images, or replacing the background of the original image, a new guardrail scene (especially the missing scene) is artificially constructed. From a labeled image of a complete guardrail, a part of the guardrail region is cropped, and after the part is deleted, a road surface, grassland or other background is replaced to generate a simulated sample of "guardrail missing"; different types of guardrails (such as metal guardrails and concrete guardrails) are copied to the same background to generate mixed scene samples.

[0060] For example, a real labeled guardrail image is "industrial park complete metal guardrail" (labeled: guardrail region bounding box and "complete" label), and the method for generating augmented images by image augmentation technology includes: Method 1: Construct a guardrail missing sample. From the labeled guardrail image, a section of the guardrail region (such as a 1-meter-long guardrail in the middle) is selected, and the copypaste technique is used to replace the region with the ground background of the industrial park, while retaining part of the features of the guardrails on both sides. When labeling, add a "missing region" bounding box and a "missing" label in the new image to generate different position guardrail missing samples.

[0061] Method 2: Cross-scene migration sample. The complete guardrail region in the labeled guardrail image is copied and pasted into different background images such as "rainy day highway" and "night construction site", and the size and angle of the guardrail are adjusted (such as being reduced to 80% of the original size and being inclined by 5°), to generate cross-scene sample images, which are labeled.

[0062] Method 3: Mixed type sample. The metal guardrail in the labeled guardrail image is copied into the same "factory entrance" background as the concrete guardrail in another labeled image to generate a sample image containing two types of guardrails, which are labeled.

[0063] In an embodiment, the guardrail image is obtained and labeled to obtain first data, which includes: segmenting and labeling a single safety guardrail and a gap region between the guardrails in the guardrail image, and the label includes the gap region and the features of the partial fence regions on both sides.

[0064] Taking the labeling process of the "urban road guardrail missing scene" as an example, the above labeling rules are described in detail: First, a monitoring image is collected, which shows that there is a gap about 2 meters wide in the isolation guardrail beside the urban trunk road, the left side of the gap is a continuous metal guardrail, the right side is another set of aligned metal guardrails, there is no guardrail structure in the gap, and the background is a sidewalk and a green belt; Second step, use purple contour line to outline the complete guardrail area on the left and right side of the gap respectively: left guardrail: start from the left edge of the image, end at the left end point of the gap, accurately enclose the pixel range of all the railings and columns, and label the label as "complete guardrail"; right guardrail: start from the right end point of the gap, end at the right end point of the image, also outline the complete contour, and label the label as "complete guardrail". Mark the gap related area with a red contour line: gap core area: cover a 2-meter-wide blank area (no guardrail part); two sides of the fence: extend to the last column of the left guardrail to the left, and extend to the first column of the right guardrail to the right (about 1 meter long complete guardrail is included). The overall marked range forms a "left partial guardrail + gap + right partial guardrail" continuous area, and the label is "guardrail gap".

[0065] In an embodiment, the second guardrail detection model is fine-tuned through a prompt word and an example to perform secondary detection of the guardrail state, and a second detection result is obtained.

[0066] In the guardrail state detection process of the application, the key of the secondary detection by the second guardrail detection model (such as a visual language large model) is to adapt the general model to the professional scene of guardrail detection through the Prompt Tuning fine-tuning technology. The key of this process lies in designing a targeted prompt word and an example to guide the model to understand the task logic and output structured results.

[0067] The visual language large model has strong cross-modal understanding ability, but needs to be guided by a prompt word to understand the specific task target. In the fine-tuning process, the original parameters of the model are not changed, but the model is guided to focus on specific features (such as guardrail continuity and gap features) by optimizing the prompt word.

[0068] When the visual language large model is used for secondary detection, it generally needs to combine visual features (such as gap morphology and guardrail structure) and domain knowledge (such as design specifications and installation standards). The prompt word needs to convert these professional requirements into instructions that the model can understand, for example: continuity judgment: distinguish between "design gap" (such as a pedestrian passageway) and "abnormal missing"; detail analysis: identify guardrail material, connection method, and whether there are signs of damage.

[0069] The example is a "demonstration case" used to help the second guardrail detection model (such as a visual language large model) understand the task logic. Through the pairing of "input + correct output", the second guardrail detection model learns how to apply the rules in the prompt word to the actual scene, thereby improving the accuracy and consistency of the secondary detection.

[0070] In an embodiment, the prompt word includes: task description, analysis thought description, input format specification, and second detection result format definition.

[0071] The purpose of the prompt words is to use structured instructions to clarify the task objectives, reasoning logic, and input / output specifications for the second guardrail detection model (usually a large visual language model), ensuring that the secondary detection results are accurate and can be directly reused.

[0072] For example, the task description is as follows: Based on the input image and the first detection result, analyze the continuity status of the guardrail (whether there is an interruption), confirm whether there is a real missing area, and determine the type of missing area (abnormal missing / design gap). The analysis approach is as follows: First, based on the first detection result, determine whether the orientation of the guardrails on both sides is consistent (e.g., whether straight lines are collinear, whether curves are smoothly connected); for suspected missing areas, check whether there are design markings at the gap (e.g., 'pedestrian crossing' signs, traffic barriers); compare the structural features of the guardrails on both sides of the gap (e.g., material, post spacing, height). If the features are consistent and there are no design markings, it is determined to be an abnormal missing area; if there are design markings or the features on both sides have a tapering design, it is determined to be a design gap. Input format description: image to be tested, first detection result, confidence level. Second detection result format definition: continuity status: continuous / interrupted; missing area confirmation: coordinates of the missing area.

[0073] In one embodiment, a guardrail detection device is provided. For example... Figure 2 As shown, the guardrail detection device includes a preliminary detection module 21, a secondary detection module 22, and an output module 23. Detailed descriptions of each functional module are as follows: The preliminary detection module 21 is used to perform preliminary detection of the guardrail status on the input image using the first guardrail detection model, and obtain the first detection result and confidence level. The secondary detection module 22 is used to perform secondary detection of the guardrail status using the second guardrail detection model on the input image with a confidence level lower than the confidence threshold, and obtain a second detection result. Output module 23 is used to combine the first detection result and the second detection result to obtain the target result.

[0074] This invention also provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor. The processor executes the program to implement the aforementioned guardrail detection method; to avoid repetition, this will not be described again here. Alternatively, the electronic device can implement the functions of each module in this embodiment of the guardrail detection device; this will not be described again here.

[0075] This invention also provides a readable storage medium storing a program, characterized in that, when executed by a processor, the program implements the aforementioned guardrail detection method; to avoid repetition, this will not be described again here. Alternatively, when executed by a processor, the program implements the functions of each module in this embodiment of the guardrail detection device; this will not be described again here.

[0076] The above-described embodiments are only used to illustrate the technical solutions of the present application, but not limit the present application; even though the present application has been described in detail with reference to the foregoing embodiments, ordinarily skilled in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

[0077] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0078] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

Claims

1. A method for detecting guardrails, characterized in that, include: The first guardrail detection model is used to perform preliminary guardrail status detection on the input image to obtain the first detection result and confidence level. Based on the first detection result, for the input image with a confidence level lower than the confidence threshold, the second guardrail detection model is used to perform a second detection of the guardrail status to obtain the second detection result; The target result is obtained by combining the first and second detection results.

2. The guardrail detection method according to claim 1, characterized in that, The training methods for the first guardrail detection model include: Acquire and annotate the guardrail image to obtain the first data; Using the first data, a deep learning-based guardrail detection model is trained to obtain the first guardrail detection model.

3. The guardrail detection method according to claim 2, characterized in that, The step of using the first data to train a deep learning-based guardrail detection model to obtain a first guardrail detection model includes: The first data is augmented to obtain the second data; Using the second data, a deep learning-based guardrail detection model is trained to obtain the first guardrail detection model.

4. The guardrail detection method according to claim 2, characterized in that, The step of acquiring and annotating the guardrail image to obtain the first data also includes: Based on the labeled guardrail image, an augmented image is obtained using image augmentation technology; The first data includes the labeled guardrail image and the augmented image.

5. The guardrail detection method according to claim 2, characterized in that, The process of acquiring and annotating the guardrail image to obtain the first data includes: segmenting and annotating a single safety guardrail and the gap area between guardrails in the guardrail image, and the annotation includes the gap area and the features of the fence area on both sides.

6. The guardrail detection method according to claim 1, characterized in that, The second detection result obtained by using the second guardrail detection model to perform secondary detection of the guardrail status includes: The second guardrail detection model was fine-tuned using prompts and examples.

7. The guardrail detection method according to claim 6, characterized in that, The prompts include: task description, analysis approach description, input format instructions, and second detection result format definition.

8. A guardrail detection device, characterized in that, include: The preliminary detection module is used to perform preliminary detection of the guardrail status on the input image using the first guardrail detection model, and obtain the first detection result and confidence level. The secondary detection module is used to perform secondary detection of the guardrail status on the input image with a confidence level lower than the confidence threshold using the second guardrail detection model to obtain a second detection result; The output module is used to combine the first detection result and the second detection result to obtain the target result.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the guardrail detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the guardrail detection method as described in any one of claims 1 to 7.

Images