Road disease review method, system, device, medium and program product based on visual language model

Abstract

The application discloses a road disease review method, system, device, medium and program product based on a visual language model, and the method comprises the following steps: acquiring a road inspection image and a candidate disease detection result; screening a to-be-reviewed candidate disease target from the candidate disease detection result, numbering each candidate target, and performing normalization processing on the pixel coordinates of the to-be-reviewed candidate target; drawing a rectangular frame of the candidate target in the road inspection image, generating a road inspection labeling image, and inputting a candidate review data group; inputting the to-be-reviewed road inspection labeling image, the candidate review data group and a road disease criterion into a preset visual language model for single inference, simultaneously performing candidate category review, category correction and zero-sample supplementary review, and generating a structured inference result; and generating a road disease list and a final road inspection labeling image according to the structured inference result. The application realizes efficient and reliable road disease review and supplementary review.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, system, device, medium, and program product for road defect verification based on a visual language model. Background Technology

[0002] The road inspection system can use an object detection model to output candidate defects such as cracks, potholes, road surface water accumulation, and road facility damage from road images acquired by vehicle-mounted or portable acquisition devices. Object detection results typically include candidate categories, candidate bounding boxes, detection confidence scores, and model source information. The Vision Language Model (VLM) in this invention refers to a multimodal model capable of jointly processing image and text inputs, understanding image content based on text commands, and outputting structured judgment results.

[0003] Due to interference such as shadows, reflections, construction joints, watermarks, perspective compression of distant views, and distortion of facility edges in road images, it is difficult to cover the three types of needs of verification, error correction, and supplementation of missed detections based solely on the detector output.

[0004] In engineering implementation, if candidate verification, category correction, and unlabeled defect detection are separated into multiple calls to the visual language model, the same image will be repeatedly transmitted and visual inference will be repeatedly executed. If the detection confidence, model source, and threshold are directly provided to the visual language model, the verification conclusion may be influenced by the prior knowledge of the upstream model, rather than being based on image evidence inside and outside the candidate box. Category correction also needs to maintain the geometric position of the original target, and zero-sample additions need to be able to be mapped back to the original image coordinates and undergo deterministic verification.

[0005] Therefore, there is an urgent need to invent a new method for verifying road defect detection results in order to solve the problems of low verification efficiency and reliability caused by the existing technology, which involves multiple calls to the visual language model and difficulty in maintaining the geometric position during correction. Summary of the Invention

[0006] In view of this, embodiments of the present invention provide a method, system, device, medium and program product for road defect verification based on a visual language model, which at least partially solves the problems existing in the prior art.

[0007] Other features and advantages of the invention will become apparent from the following detailed description, or may be learned in part by practice of the invention.

[0008] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0009] According to a first aspect of the present invention, a road defect verification method based on a visual language model is provided, the method comprising:

[0010] Acquire road patrol images and candidate disease detection results, wherein the candidate disease detection results include candidate bounding boxes and candidate categories;

[0011] From the candidate disease detection results, candidate disease targets to be reviewed are selected, each candidate disease target to be reviewed is numbered, and the pixel coordinates of the candidate disease targets to be reviewed are normalized.

[0012] Draw rectangles corresponding to the candidate defects to be reviewed in the road inspection images, generate road inspection annotation images to be reviewed, and construct candidate review input data groups;

[0013] The road inspection and annotation images to be reviewed, the candidate review input data group, and the road defect judgment criteria are input into the preset visual language model for single reasoning. Simultaneously, candidate category review, category correction, and zero sample supplementation are performed to generate structured reasoning results.

[0014] Based on the structured reasoning results, a list of road defects and a final road inspection and annotation image are generated.

[0015] In the candidate target preprocessing stage, candidate disease targets to be reviewed are selected from the candidate disease detection results, each candidate disease target to be reviewed is numbered, and the pixel coordinates of the candidate disease targets to be reviewed are normalized, including:

[0016] Based on the preset set of road defect categories, candidate defect targets to be reviewed are selected from the candidate defect detection results;

[0017] For each candidate disease target to be reviewed, a unique number is assigned to the candidate disease target to be reviewed, and the unique number corresponding to the candidate disease target to be reviewed is obtained.

[0018] The pixel coordinates of the candidate bounding boxes corresponding to the candidate disease targets to be reviewed are normalized to obtain the normalized candidate bounding boxes corresponding to the candidate disease targets to be reviewed.

[0019] Based on this, rectangular boxes corresponding to the candidate defects to be reviewed are drawn in the road inspection images, generating road inspection annotation images to be reviewed, and constructing candidate review input data sets, including:

[0020] Draw a rectangle and a unique number for each candidate defect target to be reviewed in the road inspection image to generate a complete road inspection annotation image to be reviewed.

[0021] For each candidate disease target to be reviewed, candidate review input items are generated using the unique number, candidate category, and normalized candidate box corresponding to the candidate disease target to be reviewed.

[0022] The candidate verification input items are aggregated into the candidate verification input data group, wherein the candidate verification input data group does not include the detection model identifier, model source, detection confidence, and detection threshold.

[0023] Subsequently, the road inspection and annotation images to be reviewed, the candidate review input data group, and the road defect judgment criteria are input into the preset visual language model for single-pass reasoning. Simultaneously, candidate category review, category correction, and zero-sample supplementation are performed to generate structured reasoning results, including:

[0024] The road inspection and annotation images to be reviewed, the candidate review input data group, and the road defect criteria are input into a preset visual language model for single-pass reasoning. The road defect criteria include positive examples of road defects and exclusion items of road defects.

[0025] In the same inference, the preset visual language model reviews the candidate categories corresponding to each candidate disease target based on the road inspection and annotation image to be reviewed, the candidate review input items corresponding to each candidate disease target to be reviewed in the candidate review input data group, and the road disease criterion, and outputs the review results. The review results include correct category, incorrect category, or uncertain category.

[0026] If the review result indicates that the category is correct, then the candidate category and candidate box of the candidate disease target to be reviewed are retained;

[0027] If the review result is a category error and the visual language model determines a corrected category within the preset road defect category set, then the candidate category corresponding to the candidate defect target to be reviewed is corrected to obtain the corrected category; if the review result is an uncertain category, then no automatic category correction is performed, the original candidate data is retained and marked as pending manual review.

[0028] Using the preset visual language model, zero-sample supplementary checks are performed on the areas in the road patrol annotation image to be reviewed that are not marked by any candidate boxes, resulting in zero-sample new items. The zero-sample new items include the verification category and verification coordinates.

[0029] After the reasoning is completed, based on the structured reasoning results, a list of road defects and a final road inspection and annotation image are generated, including:

[0030] The structured reasoning results are verified to obtain verified structured reasoning results.

[0031] For disease targets whose review results indicate that the category is correct, the original category and original candidate box of the disease target are retained as the final disease detection result;

[0032] For the disease targets after category correction, the corrected category of the disease target and the original candidate box are used as the final disease detection result;

[0033] The newly added item with zero samples will be used as the final disease detection result;

[0034] Based on the final defect detection results, annotations are made in the road inspection images to obtain the final road inspection annotated images, and a road defect list is generated.

[0035] According to a second aspect of the present invention, a road defect verification system based on a visual language model is provided, the system comprising:

[0036] The candidate result acquisition module is used to acquire road patrol images and candidate disease detection results, wherein the candidate disease detection results include candidate boxes and candidate categories;

[0037] The spatial anchoring module is used to filter out candidate disease targets to be reviewed from the candidate disease detection results, assign numbers to each candidate disease target to be reviewed, normalize the pixel coordinates of the candidate disease targets to be reviewed, and draw a rectangle corresponding to the candidate disease targets to be reviewed in the road inspection image to generate a road inspection annotation image to be reviewed.

[0038] The candidate review input data group construction module is used to generate candidate review input items based on the unique number, candidate category and normalized candidate box of each candidate disease target to be reviewed, and to collect them to form a candidate review input data group. The detection model identifier, model source, detection confidence and detection threshold are not written into the candidate review input data group.

[0039] The joint visual reasoning module is used to input the road inspection and annotation images to be reviewed, the candidate review input data group and the road defect judgment criteria into the preset visual language model for single reasoning, and simultaneously perform candidate category review, category correction and zero sample supplementation to generate structured reasoning results.

[0040] The result normalization and merging module is used to generate a list of road defects and a final road inspection and annotation image based on the structured reasoning results.

[0041] According to a third aspect of the present invention, a road defect verification device based on a visual language model is provided, the device comprising: a processor and a memory;

[0042] The memory is used to store one or more program instructions;

[0043] The processor is configured to run one or more program instructions to perform the steps of the road defect verification method based on the visual language model as described in any of the preceding claims.

[0044] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, wherein when executed by a processor, the computer program implements the steps of the road defect verification method based on a visual language model as described in any of the preceding claims.

[0045] According to a fifth aspect of the present invention, a computer program product is provided, the computer program product comprising a computing program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions that, when executed by a computer, cause the computer to implement the steps of the road defect verification method based on a visual language model as described in any of the preceding claims.

[0046] This invention provides a road defect verification method, system, device, medium, and program product based on a visual language model. The method includes: acquiring road inspection images and candidate defect detection results; selecting candidate defect targets to be verified from the candidate defect detection results, assigning numbers to each candidate target, and normalizing the pixel coordinates of the candidate targets; drawing rectangular boxes of the candidate targets in the road inspection images to generate road inspection annotation images and candidate verification input data groups; inputting the road inspection annotation images to be verified, the candidate verification input data groups, and road defect criteria into a preset visual language model for single-pass reasoning, simultaneously performing candidate category verification, category correction, and zero-sample supplementary inspection to generate structured reasoning results; and generating a road defect list and a final road inspection annotation image based on the structured reasoning results. This invention achieves efficient and reliable road defect verification and supplementary inspection. Attached Figure Description

[0047] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0048] Figure 1 A flowchart illustrating the road defect verification method based on a visual language model provided in this embodiment of the invention;

[0049] Figure 2 This is a schematic diagram illustrating the principle of pixel coordinate normalization and inverse conversion provided in an embodiment of the present invention.

[0050] Figure 3 A schematic diagram illustrating the principle of verification and merging of review results provided in this embodiment of the invention;

[0051] Figure 4This is a schematic diagram of the road defect verification system based on a visual language model provided in an embodiment of the present invention;

[0052] Figure 5 This is a flowchart illustrating the exception and degradation handling process provided in an embodiment of the present invention;

[0053] Figure 6 This is a schematic diagram of the data flow in an end-to-end embodiment of the present invention. Detailed Implementation

[0054] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0055] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0056] Figure 1 A flowchart of a road defect verification method based on a visual language model according to an embodiment of the present invention is shown.

[0057] like Figure 1 As shown, the road defect verification method based on a visual language model according to an embodiment of the present invention may include steps S100, S200, S300, S400 and S500.

[0058] In step S100, road patrol images and candidate disease detection results are acquired. The candidate disease detection results include candidate bounding boxes and candidate categories.

[0059] The process of acquiring the road patrol images and candidate defect detection results includes:

[0060] Acquire road patrol images and candidate defect detection results generated by upstream detectors.

[0061] Each candidate disease detection result contains at least a candidate category and a candidate bounding box.

[0062] The upstream detector may be one or more target detection models, but its specific brand, number of network layers and training method are not necessary limitations of this invention.

[0063] Next, in step S200, candidate disease targets to be reviewed are selected from the candidate disease detection results, each candidate disease target to be reviewed is numbered, and the pixel coordinates of the candidate disease targets to be reviewed are normalized.

[0064] The filtering, numbering, and coordinate normalization processes include:

[0065] Based on the preset set of road defect categories, candidate defects to be reviewed are selected from the candidate defect detection results.

[0066] For each candidate disease target to be reviewed, a unique number is assigned to it within this review request, thus obtaining the unique number corresponding to the candidate disease target to be reviewed.

[0067] The pixel coordinates of the candidate boxes corresponding to the candidate disease targets to be reviewed are normalized. The pixel coordinates of the candidate boxes are converted into normalized coordinates with the upper left corner as the origin and the horizontal and vertical coordinates as 0 to 1000. The coordinate values ​​are limited to this coordinate range to obtain the normalized candidate boxes corresponding to the candidate disease targets to be reviewed.

[0068] The structure of the review items for the above-mentioned candidate disease targets to be reviewed is shown in Table 1, including number, candidate category and normalized candidate boxes from 0 to 1000, namely {id, label, bbox_norm_0_1000}.

[0069] Table 1. Field Structure of Candidate Disease Targets

[0070]

[0071] For the above normalization process, let the width of the road patrol image be W and the height be H; let the pixel coordinates of the candidate box boundary be x and y; and let the normalized coordinates be u and v.

[0072] The conversion from pixel coordinates to the 0-1000 coordinate range is as follows: , , where round means taking the nearest integer and clip means restricting the result to the closed interval [0, 1000].

[0073] In step S300, a rectangular box corresponding to the candidate disease target to be reviewed is drawn in the road inspection image, a road inspection annotation image to be reviewed is generated, and a candidate review input data group is constructed.

[0074] The process of generating labeled images and constructing candidate verification input data sets includes:

[0075] Draw a rectangle and a unique number for each candidate defect to be reviewed in the road inspection image, and generate a complete road inspection annotation image with numbering.

[0076] For each candidate defect to be reviewed, candidate review input items are generated using its unique ID, candidate category, and normalized candidate bounding box. All candidate review input items from the same review request are then aggregated into a candidate review input data group. This candidate review input data group is a structured list of candidate data, excluding detection evaluation metadata such as detection model identifier, model source, detection confidence level, and detection threshold, to reduce the influence of explicit priors on the visual language model's review judgment.

[0077] In step S400, the road inspection and annotation images to be reviewed, the candidate review input data group, and the road defect judgment criteria are input into the preset visual language model for single reasoning. Simultaneously, candidate category review, category correction, and zero sample supplementation are performed to generate structured reasoning results.

[0078] The single reasoning process includes:

[0079] The road inspection and annotation images to be reviewed, the candidate review input data group, and the road defect judgment criteria are input into the preset visual language model for a single inference. In the same inference, the model is required to give the review or category correction results for each candidate defect target to be reviewed, and output the zero-sample new defects that are not marked by the candidate box within the preset defect category set.

[0080] The above criteria for determining road defects include positive examples of road defects and exclusion criteria for road defects.

[0081] In the same inference, the pre-defined visual language model reviews the candidate categories corresponding to each candidate disease target in the road inspection and annotation images to be reviewed, the candidate review input items corresponding to each candidate disease target in the candidate review input data group, and the road disease criteria, and outputs the review results, which include category correct, category incorrect, or category uncertain.

[0082] If the review result indicates that the category is correct, then the candidate category and candidate box of the candidate disease target to be reviewed are retained.

[0083] If the review result is a category error and the visual language model determines a corrected category within the preset road defect category set, then the candidate category corresponding to the defect to be reviewed is corrected to obtain the corrected category; if the review result is an uncertain category, then no automatic category correction is performed, the original candidate data is retained and marked as pending manual review.

[0084] Using a pre-defined visual language model, zero-sample supplementary checks are performed on areas in the road patrol annotation images that are not marked by any candidate boxes, resulting in zero-sample new items. The zero-sample new items include the verification category and verification coordinates.

[0085] The judgment scope of the visual language model is jointly limited by a pre-defined set of road defect categories and their positive examples and exclusions. Positive examples describe the shape, texture, spatial relationships, and typical scenes of the target; exclusions describe common interferences such as shadows, reflections, construction joints, water stains, normal road surface textures, perspective compression of distant views, detection bounding boxes, and numbers. When both positive and negative evidence appear simultaneously or the image clarity is insufficient, an uncertain state is output and the sufficiency of evidence is reduced.

[0086] Zero-sample supplementation is not an unlimited open-category recognition process, but rather a process that supplements obvious defects not covered by the original candidate boxes under the constraints of a pre-defined category set and visual criteria. The visual language model must not treat later-added bounding boxes, numbers, or text itself as evidence of defects.

[0087] It should be noted that server-side whitelist verification and coordinate validity verification for newly identified zero-shot items are essential technical features, not optional steps: only new items belonging to the preset road defect category set, with valid normalized coordinates, and satisfying coordinate validity after reverse conversion are allowed to enter the drawing results; new items that fail either verification are considered invalid boxes and discarded. It is this server-side mandatory verification that distinguishes the zero-shot supplementary inspection of this invention from the open output of general zero-shot target detection, ensuring that the supplementary inspection results are controlled and do not introduce targets outside the set or with invalid coordinates.

[0088] The model's output verification item structure is fixed at {id, task_type, label, bbox_norm_0_1000, correctness, verdict, evidenceSufficiency}. Here, task_type can be review, edit_after_review, or new_recognize (zero-sample new recognition); correctness can be correct, incorrect, or uncertain, with new items taking none; evidenceSufficiency is a value between 0 and 1, describing the sufficiency of image evidence, and is not equivalent to model confidence. The model inference output structure is shown in Table 2.

[0089] Table 2 Model Inference Output Structure

[0090]

[0091] Preferably, when the visual language model is unavailable, the returned content cannot be parsed, or the candidate number is missing, the abnormal result is not disguised as a completed review. When a call fails, a clear fallback source and uncertain status are returned; when the original candidate number is missing, an uncertain item is added to the corresponding candidate target. Retrying different service providers according to priority is an optional fault-tolerant implementation.

[0092] Figure 5 The flowchart of exception and degradation handling is shown, demonstrating that three types of exceptions—call failure, unresolvable return, and missing candidate number—are uniformly output as uncertain states while retaining the original candidate data.

[0093] This invention combines verification, category correction, and zero-sample supplementation into a single visual language model inference, not merely to reduce the number of calls or save bandwidth, but because these three tasks share the same judgment premise:

[0094] First, all three types of tasks share the same numbered complete image and the same set of positive and negative visual criteria. The model judges the original candidate target and the unlabeled area together in the same visual context, avoiding discrepancies in judgment caused by inconsistent inputs during multiple inferences.

[0095] Secondly, all three types of results are uniformly expressed in the coordinate range of 0 to 1000, eliminating the need to repeatedly convert pixel coordinates to normalized coordinates between multiple inferences, thus avoiding the accumulation of rounding errors introduced by multiple conversions.

[0096] Third, single-pass reasoning allows the three types of outputs—review, correction, and addition—to be determined simultaneously in a single forward pass. The category definitions, coordinate benchmarks, and evidence standards of these three types are naturally consistent, eliminating the need for the server-side deterministic merging to reconcile contradictions between multiple independent results. Therefore, merged reasoning does not degrade the judgment quality of each task; on the contrary, it improves the overall consistency of the three types of results by sharing context, unifying the coordinate domain, and maintaining consistent standards.

[0097] Finally, in step S500, a list of road defects and a final road inspection and annotation image are generated based on the structured reasoning results.

[0098] The verification and output process of the structured reasoning results includes:

[0099] The task type, candidate number, category, normalized candidate box, correctness status, and evidence sufficiency of the structured reasoning results are verified to obtain the verified structured reasoning results.

[0100] For disease targets whose review results indicate the category is correct, the original category and original candidate box of the disease target are retained as the final disease detection result.

[0101] For disease targets after category correction, if the corrected category belongs to the set of allowed categories, the original category is replaced and the original candidate box is used. The corrected category and the original candidate box of the disease target are used as the final disease detection result.

[0102] For candidate targets whose task type is review and whose correctness status is incorrect or uncertain, they will not be included in the final road defect list and the final road inspection annotation map, and the original candidate data will be retained for manual review.

[0103] Figure 3 The diagram illustrates the principle of verification and merging of review results.

[0104] The processing results for the review, correction, and zero-sample additions are shown in Table 3.

[0105] Table 3. Rules for Review, Category Correction, and Handling of New Items with Zero Samples

[0106]

[0107] Preferably, when the original candidate number is missing, an uncertain completion item is generated to avoid the silent loss of candidate targets.

[0108] The zero-sample new items are used as the final disease detection results. Based on the final disease detection results, their normalized candidate boxes are converted into pixel candidate boxes and added to the drawing list. They are then labeled in the road inspection images to obtain the road disease list and the final road inspection labeled image.

[0109] When plotting zero-sample new items, the candidate bounding box for zero-sample new items is inversely converted as follows: , Perform the above conversion on the top left and bottom right corners of the candidate box, and require that the horizontal and vertical coordinates of the bottom right corner after conversion are greater than those of the top left corner; otherwise, the candidate box is considered invalid and will not be included in the drawing results.

[0110] Figure 2 A schematic diagram illustrating the principle of pixel coordinate normalization and inverse conversion is shown.

[0111] The following examples are used to further illustrate the present invention, but do not constitute a limitation on the scope of protection of the present invention.

[0112] End-to-end implementation: Taking a road inspection image with a width W = 1920 pixels and a height H = 1080 pixels as an example. The upstream detector provides a candidate target of "pothole" numbered 1, whose pixel candidate box is converted to [436, 612, 528, 704]. Number 1 is drawn in the corresponding area of ​​the complete road image, and number 1, the candidate category "pothole", and the normalized candidate box are encoded as candidate verification input items and added to the candidate verification input data group. The candidate verification input data group is shown in Table 4.

[0113] Table 4. Example of candidate review input data groups

[0114]

[0115] The same request can also include other candidate defects. Submit the numbered complete image, the candidate review input data set, and the positive and negative visual criteria for road defects to the visual language model. The visual language model returns multiple review items at once: if the area corresponding to number 1 has clear concave edges and internal shadow levels, it can return task_type=review, correctness=correct; if there is a defect in a candidate box but the category direction is incorrect, it can return task_type=edit_after_review and the corrected category; if obvious road water accumulation within a preset set is found in an unlabeled area, it can return task_type=new_recognize, correctness=none, and its normalized candidate box.

[0116] The server normalizes the returned items. Category correction items are forcibly restored to the original candidate boxes to prevent the visual language model from changing the geometric position of the original target; zero-sample additions must pass the validity checks of allowed categories and candidate boxes before being converted into pixel candidate boxes. During final rendering, correctly verified items are retained, category correction items that meet the conditions replace the original categories, and zero-sample additions that meet the conditions are appended as new boxes. An example of structured returns is shown in Table 5.

[0117] Table 5. Example of Structured Returns

[0118]

[0119] Figure 6 A schematic diagram of the data flow in an end-to-end embodiment is shown.

[0120] In addition, embodiments of the present invention also provide a road defect verification system based on a visual language model, the system comprising:

[0121] The candidate result acquisition module is used to acquire road patrol images and candidate disease detection results, which include candidate bounding boxes and candidate categories.

[0122] The spatial anchoring module is used to filter out candidate disease targets to be reviewed from the candidate disease detection results, assign numbers to each candidate disease target to be reviewed, normalize the pixel coordinates of the candidate disease targets to be reviewed, and draw the rectangle corresponding to the candidate disease targets to be reviewed in the road inspection image to generate the road inspection annotation image to be reviewed.

[0123] The candidate review input data group construction module is used to generate candidate review input items based on the unique number, candidate category and normalized candidate box of each candidate disease target to be reviewed, and to collect them to form a candidate review input data group. The detection model identifier, model source, detection confidence and detection threshold are not written into the candidate review input data group.

[0124] The joint visual reasoning module is used to input the road patrol annotation images to be reviewed, the candidate review input data group and the road defect judgment criteria into the preset visual language model for single reasoning, and simultaneously perform candidate category review, category correction and zero sample supplementation to generate structured reasoning results.

[0125] The result normalization and merging module is used to generate a list of road defects and the final road inspection and annotation images based on the structured reasoning results.

[0126] Figure 4 The diagram shows the structure of the road defect verification system based on a visual language model provided in this embodiment of the invention, illustrating the data flow relationship of six modules: candidate result acquisition, spatial anchoring, construction of candidate verification input data groups, joint visual reasoning, result normalization, and result merging.

[0127] In addition, this embodiment of the invention also provides a road defect verification device based on a visual language model. The device includes: a processor and a memory; the memory is used to store one or more program instructions; the processor is used to run one or more program instructions to perform the steps of the road defect verification method based on a visual language model as described above.

[0128] In addition, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the road defect verification method based on a visual language model as described above.

[0129] In addition, embodiments of the present invention also provide a computer program product, which includes computer program instructions that, when executed by a processor, implement the steps of the road defect verification method based on a visual language model as described above.

[0130] This invention processes original category verification, category correction, and zero-sample supplementation simultaneously through a single visual language model inference, reducing the repetitive organization and transmission of the same road images in three independent processes.

[0131] This invention establishes a one-to-one correspondence between visual targets and structured candidate items by using a complete image with numerical numbering and normalized coordinates from 0 to 1000. This allows candidate positions across image resolutions to be expressed uniformly and enables newly added lesion boxes to be mapped back to the original image.

[0132] This invention constructs a candidate verification input data set and does not provide the visual language model with detection evaluation metadata such as detection model identifier, model source, detection confidence, and detection threshold. This allows the model to perform verification by combining image evidence, candidate category, spatial location, and road defect criteria, thereby reducing the influence of explicit upstream detection priors on verification judgment.

[0133] This invention uses category correction terms to force the use of the original candidate boxes, thus maintaining the original geometric position of the target; and generates uncertain completion terms by using missing numbers to avoid the silent loss of candidate targets due to missing terms returned by the model.

[0134] This invention uses a server to perform deterministic verification of task type, allowed category, and candidate box validity, restricting visual language model outputs that do not meet the constraints from directly entering the road defect annotation results.

[0135] This invention can be deployed in road patrol backends, edge devices and cloud-based visual language model collaborative systems, or other image processing systems capable of acquiring road patrol images and candidate bounding boxes. Upstream candidate bounding boxes can come from single-model or multi-model detection chains; the visual language model can be provided by a local service or a remote service. The aforementioned deployment location and service brand are not essential limitations of this invention.

[0136] In this embodiment of the invention, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in this embodiment of the invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in this embodiment of the invention can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor reads information from the storage medium and, in conjunction with its hardware, completes the steps of the above methods. The storage medium can be memory, for example, volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM).The storage media described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable types of memory. Those skilled in the art will recognize that the functions described in one or more of the above examples can be implemented using a combination of hardware and software. When applied software, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of computer programs from one place to another. Storage media can be any available medium accessible to general-purpose or special-purpose computers. Although the invention has been described in detail above with general description and specific embodiments, modifications or improvements can be made to it, which will be apparent to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the invention are within the scope of protection claimed by this invention.

[0137] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent changes or alterations made by those skilled in the art using the disclosed technical content shall fall within the protection scope of the present invention.

Claims

1. A method for reviewing road defects based on a visual language model, characterized in that, The method includes: Acquire road patrol images and candidate disease detection results, wherein the candidate disease detection results include candidate bounding boxes and candidate categories; From the candidate disease detection results, candidate disease targets to be reviewed are selected, each candidate disease target to be reviewed is numbered, and the pixel coordinates of the candidate disease targets to be reviewed are normalized. Draw rectangles corresponding to the candidate defects to be reviewed in the road inspection images, generate road inspection annotation images to be reviewed, and construct candidate review input data groups; The road inspection and annotation images to be reviewed, the candidate review input data group, and the road defect judgment criteria are input into the preset visual language model for single reasoning. Simultaneously, candidate category review, category correction, and zero sample supplementation are performed to generate structured reasoning results. Based on the structured reasoning results, a list of road defects and a final road inspection and annotation image are generated.

2. The road defect verification method based on a visual language model according to claim 1, characterized in that, From the candidate disease detection results, candidate disease targets to be reviewed are selected, each candidate disease target is assigned a number, and the pixel coordinates of the candidate disease targets to be reviewed are normalized, including: Based on the preset set of road defect categories, candidate defect targets to be reviewed are selected from the candidate defect detection results; For each candidate disease target to be reviewed, a unique number is assigned to the candidate disease target to be reviewed, and the unique number corresponding to the candidate disease target to be reviewed is obtained. The pixel coordinates of the candidate bounding boxes corresponding to the candidate disease targets to be reviewed are normalized to obtain the normalized candidate bounding boxes corresponding to the candidate disease targets to be reviewed.

3. The road defect verification method based on a visual language model according to claim 2, characterized in that, Draw rectangular boxes corresponding to the candidate defects to be reviewed in the road inspection images, generate road inspection annotation images to be reviewed, and construct candidate review input data sets, including: Draw a rectangle and a unique number for each candidate defect to be reviewed in the road inspection image to generate a complete road inspection annotation image to be reviewed. For each candidate disease target to be reviewed, candidate review input items are generated using the unique number, candidate category, and normalized candidate box corresponding to the candidate disease target to be reviewed. The candidate verification input items are aggregated into the candidate verification input data group, wherein the candidate verification input data group does not include the detection model identifier, model source, detection confidence, and detection threshold.

4. The road defect verification method based on a visual language model according to claim 3, characterized in that, The road inspection and annotation images to be reviewed, the candidate review input data sets, and the road defect judgment criteria are input into a preset visual language model for single-pass reasoning. Simultaneously, candidate category review, category correction, and zero-sample supplementation are performed to generate structured reasoning results, including: The road inspection and annotation images to be reviewed, the candidate review input data group, and the road defect judgment criteria are input into a preset visual language model for single reasoning. The road defect judgment criteria include positive examples of road defects and exclusion items of road defects. In the same inference, the preset visual language model reviews the candidate categories corresponding to each candidate disease target based on the road inspection and annotation image to be reviewed, the candidate review input items corresponding to each candidate disease target to be reviewed in the candidate review input data group, and the road disease criterion, and outputs the review results. The review results include correct category, incorrect category, or uncertain category. If the review result indicates that the category is correct, then the candidate category and candidate box of the candidate disease target to be reviewed are retained; If the review result is a category error and the visual language model determines a corrected category within the preset road defect category set, then the candidate category corresponding to the candidate defect target to be reviewed is corrected to obtain the corrected category; if the review result is a category uncertainty, then no automatic category correction is performed, the original candidate data is retained and marked as pending manual review. Using the preset visual language model, zero-sample supplementary checks are performed on the areas in the road patrol annotation image to be reviewed that are not marked by any candidate boxes, resulting in zero-sample new items. The zero-sample new items include the verification category and verification coordinates.

5. The road defect verification method based on a visual language model according to claim 4, characterized in that, Based on the structured reasoning results, a list of road defects and final road inspection and annotation images are generated, including: The structured reasoning results are verified to obtain verified structured reasoning results. For disease targets whose review results indicate that the category is correct, the original category and original candidate box of the disease target are retained as the final disease detection result; For the disease targets after category correction, the corrected category of the disease target and the original candidate box are used as the final disease detection result; The newly added item with zero samples will be used as the final disease detection result; Based on the final defect detection results, annotations are made in the road inspection images to obtain the final road inspection annotated images, and a road defect list is generated.

6. A road defect verification system based on a visual language model, characterized in that, The system includes: The candidate result acquisition module is used to acquire road patrol images and candidate disease detection results, wherein the candidate disease detection results include candidate boxes and candidate categories; The spatial anchoring module is used to filter out candidate disease targets to be reviewed from the candidate disease detection results, assign numbers to each candidate disease target to be reviewed, normalize the pixel coordinates of the candidate disease targets to be reviewed, and draw a rectangle corresponding to the candidate disease targets to be reviewed in the road inspection image to generate a road inspection annotation image to be reviewed. The candidate review input data group construction module is used to generate candidate review input items based on the unique number, candidate category and normalized candidate box of each candidate disease target to be reviewed, and to collect them to form a candidate review input data group. The detection model identifier, model source, detection confidence and detection threshold are not written into the candidate review input data group. The joint visual reasoning module is used to input the road inspection and annotation images to be reviewed, the candidate review input data group and the road defect judgment criteria into the preset visual language model for single reasoning, and simultaneously perform candidate category review, category correction and zero sample supplementation to generate structured reasoning results. The result normalization and merging module is used to generate a list of road defects and a final road inspection and annotation image based on the structured reasoning results.

7. A road defect verification device based on a visual language model, characterized in that, The device includes: a processor and a memory; The memory is used to store one or more program instructions; The processor is configured to run one or more program instructions to perform the steps of the road defect verification method based on a visual language model as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the road defect verification method based on a visual language model as described in any one of claims 1 to 5.

9. A computer program product, characterized in that, The computer program product includes computer program instructions that, when executed by a processor, implement the steps of the road defect verification method based on a visual language model as described in any one of claims 1 to 5.

Images