Method, device and equipment for determining non-standard covering of freight vehicle and storage medium

Abstract

The embodiment of the application provides a method, device and equipment for non-standard covering of a freight vehicle, and a storage medium, including obtaining a single frame image, the single frame image including a target freight vehicle; identifying the target freight vehicle in the single frame image according to a first model to determine a corresponding candidate region of interest; inputting the candidate region of interest and a corresponding preset prompt word into a second model to determine a first determination result, the preset prompt word being used to determine whether the covering is non-standard; determining a law violation probability according to the first determination result and a region not covered in the candidate region of interest; determining a second determination result according to the law violation probability, and generating an evidence image corresponding to the second determination result and a record. The combination of the review mechanism of multiple models and the determination mechanism is realized, and high-precision, explainable and recordable law violation auxiliary determination of the covering state of the freight vehicle under the condition of the single frame image is realized.

Description

Technical Field

[0001] This application relates to the field of computers, and in particular to a method, apparatus, equipment, and storage medium for determining whether a freight vehicle's covering is not up to standard. Background Technology

[0002] In road freight transport, bulk materials such as sand, gravel, coal, slag, and mineral powder are commonly transported in open wagons or flatbed wagons. Regulations require that necessary measures be taken during transport to prevent cargo from falling off, scattering, or spilling. Common compliant practices include covering the cargo with a complete tarpaulin or covering material and reliably securing the edges and overlaps with ropes, netting, or weight bars to keep the cargo in a closed or semi-closed controlled state. However, in actual operation, incomplete covering, damaged tarpaulins, loose or curled overlapping edges, slack ropes, or no covering at all still frequently occur. This leads to dust and particulate matter being stirred up by high-speed airflow, and even material sliding off the sides and rear of the wagon, creating road debris. This not only pollutes the environment but also forces following vehicles to swerve, potentially causing scrapes, rear-end collisions, and secondary accidents, posing a significant threat to road safety. Summary of the Invention

[0003] This application provides a method, apparatus, equipment, and storage medium for determining non-standard covering of freight vehicles.

[0004] This application provides a method for determining whether a freight vehicle is improperly covered, comprising: acquiring a single-frame image, the single-frame image including a target freight vehicle; identifying the target freight vehicle in the single-frame image according to a first model to determine a corresponding candidate region of interest; inputting the candidate region of interest and a corresponding preset prompt word into a second model to determine a first determination result, the preset prompt word being used to determine whether the covering is improper; determining the probability of violation based on the first determination result and the uncovered area in the candidate region of interest; determining a second determination result based on the probability of violation, and generating an evidence image and record corresponding to the second determination result.

[0005] This application provides a device for determining whether a freight vehicle is improperly covered, comprising: an acquisition module for acquiring a single-frame image, the single-frame image including a target freight vehicle; a first determination module for identifying the target freight vehicle in the single-frame image according to a first model and determining a corresponding candidate region of interest; a second determination module for inputting the candidate region of interest and a corresponding preset prompt word into a second model to determine a first determination result, the preset prompt word being used to determine whether the covering is improper; a third determination module for determining a violation probability based on the first determination result and the uncovered area in the candidate region of interest; and a generation module for determining a second determination result based on the violation probability and generating an evidence image and record corresponding to the second determination result.

[0006] This application provides a device, including: a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program for: acquiring a single-frame image, the single-frame image including a target freight vehicle; identifying the target freight vehicle in the single-frame image according to a first model, and determining a corresponding candidate region of interest; inputting the candidate region of interest and a corresponding preset prompt word into a second model to determine a first judgment result, the preset prompt word being used to determine whether the covering is not up to standard; determining the probability of violation based on the first judgment result and the uncovered area in the candidate region of interest; determining a second judgment result based on the probability of violation, and generating an evidence image and record corresponding to the second judgment result.

[0007] This application provides a computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by one or more processors, it causes the one or more processors to implement the steps in the above-described method.

[0008] The technical solution disclosed herein involves acquiring a single-frame image, which includes a target freight vehicle; identifying the target freight vehicle in the single-frame image using a first model to determine the corresponding candidate region of interest; inputting the candidate region of interest and its corresponding preset prompt word into a second model to determine a first judgment result, wherein the preset prompt word is used to determine whether the covering is not up to standard; determining the probability of violation based on the first judgment result and the uncovered area in the candidate region of interest; determining a second judgment result based on the probability of violation, and generating an evidence image and record corresponding to the second judgment result.

[0009] This system employs a multi-model verification mechanism. By acquiring a single-frame image, a first model identifies the target freight vehicle within the image, determining the corresponding candidate region of interest (ROI). The ROI and corresponding preset prompts are then input into a second model to determine the first judgment result. Furthermore, the system determines the probability of violation based on the first judgment result and the uncovered areas within the ROI candidates. A second judgment result is then determined based on this probability, generating an evidence image and record corresponding to the second judgment result. This combination of multi-model verification and judgment mechanisms enables high-precision, interpretable, and verifiable auxiliary judgment of freight vehicle covering status under single-frame image conditions. Moreover, it does not rely on continuous video streams and is applicable to scenarios such as fixed-image capture, roadside monitoring, and off-site enforcement, demonstrating significant engineering deployment value and promising application prospects. Attached Figure Description

[0010] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0011] Figure 1 This is a flowchart illustrating an exemplary method for determining non-standard covering of freight vehicles according to this application;

[0012] Figure 2 This is a schematic diagram illustrating the process of identifying a target freight vehicle, as exemplified in an embodiment of this application.

[0013] Figure 3 This is a schematic diagram of the architecture of a first model as an exemplary embodiment of this application;

[0014] Figure 4 A schematic diagram of the structure of a device for determining non-standard covering of freight vehicles, which is an exemplary embodiment of this application;

[0015] Figure 5 This is a schematic diagram of the structure of a device that is an exemplary embodiment of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] Manual visual inspections or on-board verifications of trucks are conducted at loading points, toll stations, weight control stations, or during roadside enforcement. While this method provides strong on-site enforcement, it is inherently dependent on manpower density and enforcement window hours. The sampling rate is low during periods of high traffic volume, coverage is insufficient at night or on remote road sections, and the inspection process poses safety risks to both enforcement personnel and drivers. Furthermore, manual judgment regarding whether coverings are inadequate or exposed is subjective and inconsistent, making it difficult to standardize into a verifiable chain of evidence.

[0018] Vehicle images or videos are captured by deploying cameras at key road nodes, such as ramps, toll plazas, and checkpoints, and then monitored by personnel or reviewed later. However, this method is still limited by the fact that when relying solely on visible light imaging, factors such as backlighting, uneven nighttime lighting, rain, fog, dust, vehicle shadows, and similar colors between tarpaulins and cargo can make it difficult to reliably identify exposed areas or tarpaulin edges. Single-view, especially side-view, methods are weak in judging the continuity of the roof coverage and whether the rear upper edge is suspended, easily misjudging what appears to be tarpaulin as adequate coverage. Even when suspicious photos are captured, manual secondary verification is often required before they can be used as evidence of violations or rectification, resulting in low automation and slow response times.

[0019] Furthermore, for highway traffic incidents, it is necessary to rely on continuous video streams captured by cameras to detect suspected incidents, extract video clips, and then confirm and generate reports. However, in off-site capture-based law enforcement scenarios, only single frames or a few keyframes are often available, making it difficult to directly apply continuous video analysis methods.

[0020] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0021] Figure 1 This is a flowchart illustrating a method for determining non-standard covering of freight vehicles, as an exemplary embodiment of this application. The method 100 provided in this embodiment can be deployed on a smart terminal, such as a computer, or on a server, such as a server, service cluster, or cloud platform. The method 100 includes the following steps:

[0022] Step 101: Obtain a single frame image.

[0023] Each single-frame image includes the target freight vehicle.

[0024] Step 102: Identify the target freight vehicle in the single frame image based on the first model and determine the corresponding candidate region of interest.

[0025] Step 103: Input the candidate regions of interest and their corresponding preset prompts into the second model to determine the first judgment result.

[0026] Among them, the preset prompts are used to determine whether the covering is not done properly.

[0027] Step 104: Determine the probability of violation based on the first judgment result and the uncovered areas in the candidate region of interest.

[0028] Step 105: Determine the second judgment result based on the probability of violation, and generate evidence images and records corresponding to the second judgment result.

[0029] This method is applicable to scenarios such as fixed-point capture, roadside non-on-site law enforcement, and video patrol frame interception. By locating the target freight vehicle in a single frame image, selecting candidate regions, performing semantic verification of multimodal large models, and quantifying rules, it can automatically identify and assist in determining the illegality of freight vehicle covering behavior.

[0030] The above steps are explained in detail below:

[0031] Step 101: Obtain a single frame image.

[0032] Each single-frame image includes the target freight vehicle.

[0033] Methods for acquiring single-frame images include: capturing a single-frame image containing the target freight vehicle from a monitoring device. For example, capturing a single-frame image containing the target freight vehicle from a monitoring device can originate from a fixed capture camera, keyframe capture of a video stream, roadside monitoring equipment, or a mobile law enforcement terminal.

[0034] After acquiring a single-frame image, image preprocessing can be performed. Specifically, the acquired single-frame image is processed in a preset format, and the processed image undergoes image quality evaluation. If the preset quality standard is met, the first model is executed to identify the target freight vehicle in the single-frame image and determine the corresponding candidate region of interest.

[0035] The preset format processing methods include format conversion, resolution unification, and color space standardization. It may also include channel transformation to a preset channel format, normalization to a preset numerical range, and image scaling to a preset scaling value.

[0036] For example, after acquiring the original single-frame image Iraw, it undergoes preset format conversion, resolution unification, and color space standardization to obtain a standardized image. Where H and W are the image height and width, respectively. Let be the set of real numbers.

[0037] Image quality assessment includes image sharpness detection and exposure detection, and may also include occlusion detection. For example, quality assessment of the standardized image mentioned above includes sharpness detection, exposure detection, and occlusion detection. Let the image sharpness score be Sclear and the exposure score be Sexp. When the sharpness score is greater than or equal to a threshold... ,Right now And the exposure score is within the preset range, that is If the image quality is deemed acceptable, it proceeds to subsequent processing. Otherwise, it is marked as unacceptable and processing terminates; another single-frame image is then selected for the same processing. The lower threshold within the interval, This represents the high threshold within the range.

[0038] Image sharpness can be calculated using Laplacian variance and then mapped to a score. Exposure can be calculated using histogram overflow analysis and then mapped to a score. If the occlusion score is less than a threshold, the image quality is considered acceptable. This occlusion score can be determined by the occlusion ratio of the bounding box and mapped to the corresponding score.

[0039] Because event detection relies on continuous video streams, and in off-site enforcement scenarios, only single frames or a few keyframes are often available, it's difficult to directly apply continuous video analysis methods. This implementation addresses the issue of reliance on continuous video streams in related technologies, enabling effective analysis of single-frame captured images. It is applicable to scenarios such as fixed-point capture and roadside off-site enforcement.

[0040] Step 102: Identify the target freight vehicle in the single frame image based on the first model and determine the corresponding candidate region of interest.

[0041] The first model includes an object detection model, which is a lightweight detection model, such as YOLOv8. The object detection model uses convolutional neural networks to locate and classify objects in an image, and can include the YOLO series, Faster R-CNN, etc., outputting the bounding box coordinates and class confidence scores of the objects. It can also perform image segmentation, further dividing pixel-level regions, including semantic segmentation and instance segmentation, for accurately extracting object contours and region masks.

[0042] As mentioned above, a standardized image of acceptable quality is input into the vehicle localization model to identify the main area of ​​the freight vehicle, output the vehicle target box, and generate candidate ROIs (Regions of Interest).

[0043] Specifically, the identification of target freight vehicles in a single frame image based on the first model and the determination of corresponding candidate regions of interest include: inputting the single frame image into the first model, enabling the first model to perform forward inference, identifying vehicles in the single frame image, and determining the target bounding boxes of the vehicles and the category confidence of the vehicles; and filtering the target bounding boxes of target freight vehicles from the vehicles based on the category confidence and the category of freight vehicles, and determining candidate regions of interest.

[0044] A lightweight object detection model is used to perform forward inference on a single frame of acceptable quality image to identify freight vehicle targets in the image and output a set of vehicle target boxes. Each target bounding box bi has its coordinates and class confidence score, which can be specifically... , (x i y i ) represents the coordinates of the top-left corner, w i h i These are the width and height, respectively, c i For category confidence. Based on category confidence c i Filter by vehicle type, retaining those that meet the category confidence level c. i Target bounding boxes greater than or equal to the confidence threshold ,Right now Candidate targets are those whose vehicle type is freight vehicle. Let the set of filtered target boxes be... ,in, The selected target bounding boxes can be directly used as candidate regions of interest.

[0045] like Figure 2 As shown, the original single-frame image 201 is acquired, and the image is preprocessed as described above to obtain the preprocessed image 202. The preprocessed image 202 is then input into the first model 203. The first model 203 may include a backbone network, a neck, and a prediction head. The backbone network is used for feature extraction. The neck is used for feature fusion. The prediction head is used for prediction results. The corresponding prediction results, i.e., the set of vehicle target boxes and the corresponding class confidence, are obtained through forward inference of the first model 203. Figure 2 The output image 204 shows the bounding boxes for each vehicle and their corresponding category confidence scores. For example, the confidence score for "truck" is 0.96, meaning it represents a freight vehicle.

[0046] like Figure 3As shown, the backbone network includes convolutional networks, a C2f (Cross Stage Partial with 2-branch and fast variant) module, and a pooling module SPPF (Spatial Pyramid PoolingFast). The backbone network's entry point consists of two downsampling convolutional networks with 3×3 kernels and a stride s=2. Each time, the image's width and height are halved, the area is reduced by a factor of four, and the number of channels is doubled. Several downsampling convolutional networks and the C2f module form a feature extraction module, used for feature extraction and fusion, responsible for transforming simple edge and color features into complex object shape features. The pooling module SPPF is located at the very end of the backbone network, used to expand the receptive field, utilizing pooling kernels of different sizes, such as 5×5, 9×9, and 13×13, to discover a wider range of context. After the first set of C2f, P3 is obtained, with a maximum resolution of 80×80, preserving the most detailed information, used for detecting small objects. After the second set of C2f, P4 is obtained. A moderate resolution of 40×40 is used for detecting medium-sized targets. After SPPF, the output is P5. The lowest resolution is 20×20, but it provides the strongest semantic information, used for detecting large targets. The first C2f Block × 3 is used for initial feature extraction. At this point, the feature map resolution is relatively high at 320×320, and stacking 3 layers is sufficient to extract basic edge and texture information. The second C2f Block × 6 is used for core feature enhancement. The feature map resolution drops to 160×160, which is a key level for detecting medium-sized targets. Stacking 6 layers maximizes feature extraction capability, ensuring the model can learn complex object shapes. The last C2f Block × 3 is used for deep semantic aggregation. The feature map resolution drops to 80×80, mainly used to capture global semantic information of large targets. Since the resolution is already very low, more layers are unnecessary; 3 layers are sufficient, while also saving computation.

[0047] The input is a 640×640 color single-frame image. After a series of downsampling Convs and convolutions with a stride of 2, it is finally transformed into a feature map with a very small size but a high number of channels, which is then fed into the subsequent Neck and Head.

[0048] As mentioned earlier, the C2f module includes an upsampling module and a concatenation module. The bottom layer P5 is first upsampled × 2, doubling its size back to 40 × 40. The enlarged P5 is then concatenated with the P4 originally from the backbone network, and finally fed into the C2f module for feature fusion processing, outputting a new 40 × 40 feature map as input for the next layer.

[0049] The newly generated 40×40 feature map is taken and upsampled again by Upsample×2, resulting in a resolution of 80×80. It is then merged with the P3 from the outermost layer of the backbone network at the Concat node, and finally fused through the C2f module to generate a high-resolution 80×80 feature map.

[0050] The fused 80×80 high-resolution feature map is downsampled by a Conv3×3 convolution with a stride of s=2, halving its resolution to 40×40. It is then concatenated with the newly generated 40×40 feature map from the left side using a Concat convolution, followed by fusion using a C2f module to output a medium-resolution P4 feature map. The final generated P4 feature map is then downsampled again using a Conv3×3 convolution with s=2, reducing its resolution to 20×20. This is concatenated with the bottom-level P5 feature map using a Concat convolution, and finally fused using a C2f module to output the final P5 feature map. This process generates a high-resolution 80×80 feature map.

[0051] It is drawn directly to the right as the final P3 feature map.

[0052] The three feature maps P3, P4, and P5 from the Neck network are used by the prediction head to determine two types of information: classification and location. The classification branch determines whether an object exists within the bounding box and what type of object it is. This branch consists of two convolutional layers (Conv×2) to extract the features needed for classification. The regression branch determines the object's bounding box, i.e., its coordinates. This branch also consists of two convolutional layers (Conv×2) to learn the localized coordinate information.

[0053] In addition, the method 100 further includes: expanding the boundaries of the selected target boxes so that the expanded target boxes include each edge of the cargo box of the target freight vehicle; cropping the corresponding image from the single frame image based on the expanded target boxes, and determining the candidate regions of interest.

[0054] Each filtered target bounding box can also be expanded to include the top and edge areas of the cargo box, generating candidate ROI regions Rj. The expansion calculation formula is:

[0055]

[0056] in, The boundary spread coefficient has a range of values ​​of 100.

[0057] .

[0058] Based on the boundary coordinates of the candidate ROIs, the corresponding regions are cropped from the original image and normalized to a uniform size S×S, such as 640×640, using padding or resizing, resulting in a set of candidate ROI images. .

[0059] In addition, the method 100 also includes: determining whether the target freight vehicle is improperly covered based on the candidate region of interest and the first model, and determining the corresponding confidence level.

[0060] Specifically, after obtaining candidate regions of interest, the candidate regions of interest are input into the lightweight tarpaulin determination module in the first module to analyze the tarpaulin status of the cargo box area of ​​the target freight vehicle. The lightweight tarpaulin determination module may include a classification network, an object detection network, or an image segmentation network to identify cargo areas, covered areas, and exposed areas, and output the tarpaulin status and the corresponding confidence level.

[0061] Specifically, based on the candidate regions of interest and the first model, it is determined whether the target freight vehicle's covering is not up to standard. This includes: inputting the candidate regions of interest into the first model, allowing the first model to extract features from the candidate regions of interest to obtain the corresponding feature maps. The feature maps include the texture features, color features, and edge features of the cargo box area.

[0062] The feature extraction can be performed using the backbone network described above, or the classification network, object detection network, or image segmentation network in the lightweight cover determination module mentioned above can include a backbone network, and the structure of the backbone network is the same as described above, so it will not be described again.

[0063] Based on the feature map, the candidate regions of interest are segmented at the pixel level to obtain the covered area, the cargo area, and the background area. The covered area includes the areas corresponding to tarpaulins, dust nets, and covering cloths; the cargo area includes the exposed areas of goods such as sand, coal, slag, and mineral powder.

[0064] A fully convolutional network can be used to segment candidate regions of interest (ROIs) pixel-wise based on feature maps, resulting in covered regions, cargo regions, and background regions. For example, this can be achieved using the mask head in Mask R-CNN. This can include a 3×3 convolution stack to extract local features, a transposed convolution to restore spatial resolution, and a 1×1 convolution to map the number of channels to the number of region categories (here, 3), corresponding to the covered region, cargo region, and background region, respectively. The output is an H×W×3 tensor. Each channel uses a sigmoid function to predict the pixel-wise probability of its corresponding region, thus obtaining the segmentation masks for the three regions. Then, based on the obtained masks, average pooling is performed to obtain the corresponding covering probability and covering category, such as uncovered, covered, and partially covered.

[0065] Determine the cargo-bearing area Ac and the cargo-exposed area Ae, where Ac is the sum of the covered area and the cargo area, and Ae is the cargo area. Based on these two areas, determine the exposure ratio r.

[0066] r = Ae / Ac.

[0067] Specifically, semantic component segmentation is performed on the candidate region of interest image Rj to obtain a set of semantic component masks M; based on the set of semantic component masks, the confidence score ck of each semantic component is calculated; based on the confidence score, a splitting decision is independently performed on each semantic component to obtain the first determination result.

[0068] The probability of violation can be calculated based on the initial judgment result.

[0069] Specifically, in addition to the target detection head, such as the prediction head, the first model also includes a component segmentation head, such as the mask head.

[0070] The component segmentation head adopts a lightweight design and may include: a feature fusion module for upsampling the multi-scale feature maps extracted by the backbone network of the first model to a uniform resolution; a lightweight segmentation decoder composed of several depthwise separable convolutional layers stacked together; and a component prediction head that maps the feature maps to K channels through 1×1 convolutions and outputs the foreground probability of each pixel belonging to each component category through a Sigmoid activation function.

[0071] That is, when the first model above performs forward inference on a single frame image, it executes two sub-tasks simultaneously.

[0072] As mentioned above, the first subtask is vehicle target detection. Consistent with the previous embodiments, it outputs a set of vehicle target boxes B={b1,b2,...,bm}, where each target box bi contains coordinate information and category confidence.

[0073] As mentioned above, the second subtask is semantic component segmentation. For the candidate region of interest image Rj obtained after boundary expansion and cropping, the component segmentation head outputs a set of semantic component masks M={M1,M2,...,MK} corresponding to the candidate region of interest.

[0074] Where K is the number of semantic component categories, and in this embodiment K=3 or 5, etc.

[0075] When K=5, the specific features include: a cargo box area mask (Mcargo) to identify the cargo box carrying area of ​​the target freight vehicle; a tarpaulin area mask (Mcover) to identify the area covered by tarpaulin or tarpaulin material; a cargo exposed area mask (Mexposed) to identify the area where the cargo body is directly exposed; an edge area mask (Medge) to identify the area where the edge of the cargo box overlaps with the tarpaulin material; and a fastening device area mask (Mfastener) to identify the area where fastening devices are located. Fastening devices include tie-down components and pressure-down components. Tie-down components are used to secure the edge of the tarpaulin to the edge of the cargo box using ropes or straps, and include at least one of ropes, straps, rope tensioners, ratchet tighteners, and elastic ropes. Pressure-down components are used to press the tarpaulin down from above using rods or netting, and include at least one of pressure rods, pressure strips, netting, and arches.

[0076] Therefore, the training loss function of the first model described above can be a multi-task joint loss, and is determined by the following formula:

[0077]

[0078] in, To detect the loss, CIoU loss and classification cross-entropy loss are used; For component segmentation loss, a weighted combination of Dice Loss and Focal Loss is used; This is an edge-aware loss function used to enhance the model's segmentation accuracy in part boundary regions. Weight coefficients. , and The range of values ​​is .

[0079] After obtaining the masks of each semantic component, the component confidence score ck for each semantic component is calculated to characterize the reliability of the first model's segmentation result for component k. The formula for calculating the component confidence score is:

[0080]

[0081] Where Mk is the set of predicted mask pixels of component k, pk(i,j) is the predicted probability that pixel (i,j) belongs to component k, and IoUk is the intersection-union ratio of the predicted mask of component k in the current frame and the historical moving average mask.

[0082] Based on component confidence, traffic splitting decisions are performed independently for each semantic component:

[0083]

[0084] in, The high confidence threshold can take values ​​ranging from [value range missing]. ; The low confidence threshold can take values ​​ranging from [value range missing]. .

[0085] As mentioned above, when the confidence level of a component is higher than... When the component's confidence level is between [a certain value], the decision result is directly taken from the output of the first model, without needing to be input into the second model. and During this period, only a partial view of the component and its corresponding preset prompt are input into the second model for verification. When the component's confidence level is lower than... First, the partial view of the component is repaired, and then the repaired partial view and the corresponding preset prompt are input into the second model for verification.

[0086] Image inpainting is achieved through a lightweight image inpainting network. The network takes the original local view of the component and its predicted mask as input, and outputs the completed local view of the component. The image inpainting network includes an edge generation sub-network and an image completion sub-network. The edge generation sub-network generates the edge map of the completed component, and the image completion sub-network generates the final completed image based on the original local view, the predicted mask, and the completed edge map. The image inpainting network is activated only when the first model determines that inpainting triggering conditions are met. These triggering conditions include at least one of the following: the proportion of blurred areas in the local view of the component exceeds a preset blur threshold or the proportion of occluded areas exceeds a preset occlusion threshold.

[0087] Furthermore, a coverage continuity analysis is performed on the edge areas of the cargo box. The top edge, left and right edges, and rear edge of the cargo box are divided into multiple detection areas, and the coverage continuity coefficient of each detection area is calculated. When the coverage area ratio within a detection area is lower than a preset threshold, the corresponding edge is determined to have a coverage anomaly.

[0088] The process involves discretizing each edge of the cargo box into multiple detection regions using the aforementioned covering material mask, including the covered area, cargo area, and background area. The coverage rate of each region is calculated, which is the ratio of the number of pixels covered by the covering material to the total number of pixels in that region, and also the ratio of the area covered by the covering material to the total area of ​​that region. A continuity coefficient is calculated based on the degree of abrupt changes in coverage rates between adjacent regions or the proportion of the total length of the break. For example, the difference in coverage rates between adjacent regions is calculated, and the normalized opposite value of this difference (i.e., the opposite value to 1) is determined. If the opposite value is greater than a threshold, the region is considered continuous; otherwise, it is considered discontinuous. If the region is further divided into multiple smaller regions, the average of the differences between these smaller regions is determined, and the opposite value corresponding to the edge is also determined. The mean of the opposite values ​​between the same edge can be used to determine if there is a coverage anomaly at that edge. Alternatively, the mean of the final opposite values ​​of all the edges can be used as the overall coverage continuity criterion for the cargo box, i.e., continuity, or the smallest opposite value among the different edges can be used as the coefficient.

[0089] Based on the exposed ratio, edge coverage continuity, and the covering category probability output by the first model mentioned above, a preliminary judgment score S is determined:

[0090] S = αPc + βr + γ(1-Ce)

[0091] Where Pc represents the cover category probability output by the first model, Ce represents the edge cover continuity coefficient, and α, β, and γ are weight coefficients that satisfy:

[0092] α+β+γ=1.

[0093] The preliminary judgment score S determines whether the target freight vehicle has improper covering behavior. When the preliminary judgment score is greater than or equal to a preset threshold, the target freight vehicle is judged to have improper covering behavior; when the preliminary judgment score is less than the preset threshold, the target freight vehicle is judged to have proper covering behavior.

[0094] Simultaneously, the confidence score conf for the corresponding category is output according to the first model described above. When the confidence score conf is greater than or equal to the first threshold, the determination of whether the target freight vehicle is improperly covered is directly based on the determination result of the first model; when the confidence score conf is less than the first threshold, step 103 is executed, and the candidate region of interest and the corresponding preset prompt words are input into the second model for semantic verification to determine the first judgment result.

[0095] It should be noted that the first model mainly makes a preliminary judgment on the target freight vehicle by the proportion of exposed cargo, the continuity of edge coverage, and the characteristics of the coverage area. It can quickly complete the identification of most compliant scenarios and obvious violations. For low-confidence scenarios such as backlighting, shadow occlusion, cargo color similar to tarpaulin color, and unclear coverage, the second model is triggered to perform deep semantic analysis, thereby realizing a collaborative mechanism of rapid screening by the small model and accurate verification by the large model.

[0096] It should be noted that when the confidence level of the first model's output is less than the threshold, there are uncertainties, such as unclear exposure of goods or unclear type definition. In this case, the intermediate result between the image and the bounding box output by the first model is stored in the verification queue, triggering the large model to perform secondary recognition. Thus, through the mechanism of rapid initial judgment by the small model, autonomous decision-making with high confidence, and verification by the large model with low confidence, the real-time processing efficiency of high-deterministic scenarios is ensured, while the overall recognition accuracy is improved by leveraging the deep reasoning of the large model, achieving collaborative cooperation between the small and large models.

[0097] Step 103: Input the candidate regions of interest and their corresponding preset prompts into the second model to determine the first judgment result.

[0098] Among them, the preset prompts are used to determine whether the covering is not done properly.

[0099] The preset prompts include a task definition layer, a standard judgment layer, and an output format requirement layer. The task definition layer is used to clarify the task object and limit the analysis scope. The standard judgment layer is used to define the characteristics of non-standard covering according to relevant laws and regulations. The output format requirement layer is used to output preset structured results.

[0100] Construct structured prompts to clarify the task definition, judgment criteria, and output format requirements. The prompts include the following content: The task definition layer clarifies that the object to be identified is the cargo covering area of ​​freight vehicles, limiting the analysis scope to the truck cargo box area within the ROI image. The judgment criteria layer defines specific manifestations of non-standard covering based on the "Road Transport Regulations of the People's Republic of China" and enforcement practices, including typical violations such as tarpaulins not covering all cargo, missing tarpaulin fixing devices, and failure to use dust nets to cover bulk sand and gravel. The output format requirements layer requires the model to return structured results, including the probability of complete coverage, the probability of edge anomalies, the probability of fixed anomalies, and a confidence score.

[0101] For example, the specific content of the prompt could be: "You are the assistant for the intelligent traffic monitoring cover problem analysis model. The core task is executed in two stages. The judgment logic is fixed. No additional violation judgments are made throughout the process. Priority is given to the middle and lower areas of the image. Any uncertain situation is judged as no. Misjudgment is strictly prohibited."

[0102] The middle and lower regions of the image refer to the core driving area of ​​the vehicle.

[0103] The first stage refers to screening trucks whose covering issues need to be determined.

[0104] Define the target vehicles and filter for open-top or flatbed trucks, i.e., trucks with cargo boxes that do not have a fixed enclosed roof and whose cargo can be directly exposed. Specifically exclude vehicles with blue license plates, flatbed trucks with fixed enclosed roofs, and trucks with removable roofs that are completely closed.

[0105] Exclusion criteria that do not require further analysis include non-freight vehicles such as tricycles, private cars, buses, coaches, ambulances, and police cars. Also excluded are non-open-top or flatbed trucks, such as box trucks (including enclosed vans, refrigerated vans, tank trucks, oil tankers, powder tankers, container trucks, and completely enclosed containers). Invalid trucks include those located at the edge or far from the image, causing blurred cargo details (e.g., unclear roof), those severely obscured by trees or other vehicles (e.g., obscuration exceeding half the cargo area, making cargo condition indeterminate), and those showing only the front or rear, missing key cargo areas, making it impossible to confirm whether they are open-top or flatbed.

[0106] The second stage refers to analyzing the improper covering behavior of vehicles.

[0107] The analysis scope is limited to trucks within the bounding boxes of the first-stage screening, excluding any vehicles outside this range to avoid duplicate screening. The criteria for determining non-compliant vehicle covering include whether the cargo is exposed and at risk of scattering. A violation is defined as cargo loaded on a truck not being effectively and completely covered by tarpaulins, waterproof sheets, or other covering materials, resulting in the cargo itself being directly exposed. Effective and complete coverage means the covering material completely covers all cargo areas within the cargo compartment, with no obvious gaps or openings, and no partial leakage of cargo due to inadequate covering. The violation is determined by whether the cargo itself is clearly visible. If only the tarpaulin is slightly loose or fluttering but does not expose the cargo, it is not considered a violation. Priority is given to the upper and middle parts of the cargo compartment, where cargo is easily scattered by wind; the edges of the cargo compartment, where cargo is easily exposed due to bumps; and large areas of exposed cargo exceeding one-third of the cargo compartment. The type of cargo and whether it extends beyond the roof area are also crucial factors.

[0108] Compliance exceptions that are not considered violations include loading large, non-flying, fixed goods that have been effectively secured with ropes, fasteners, etc. For example, this can include large machinery such as excavators, loaders, steel structural components such as steel beams, steel columns, metal or concrete pipes with a diameter ≥30cm that need to be bundled and secured, and wind turbine blades secured with special brackets.

[0109] License plate recognition for trucks that have been determined to be in violation. The recognition priority is to first recognize the license plate at the front of the truck cab to achieve the clearest recognition effect, then recognize the license plate at the rear of the truck and the enlarged number plate sprayed on the side of the truck body. The license plates identified from different positions must be consistent with the license plate number at the front of the truck. The license plate formats include standard civilian truck license plates, such as the Chinese character for provincial abbreviation + 1 uppercase letter city code + 5 alphanumeric combinations, and new energy vehicle license plates are 6 digits, such as "Beijing A1XXX5", "Shanghai D1XXX5". The license plate recording rules include: if the license plate is complete without occlusion and the characters are clear and distinguishable, the full uppercase number shall be recorded. If the character edges are blurred, more than 2 characters are occluded, or the position is hidden, it shall be recorded as unrecognized. For multiple violating trucks, only the license plate of the first recognized one shall be recorded.

[0110] The output format includes: No eligible trucks in the first stage: {"result":"no","truck_boxes": [],"violation":{"result": "no"}}. Eligible trucks exist in the first stage but no violation: {"result":"yes","truck_boxes":[[xmin1,ymin1,xmax1,ymax1],...],

[0111] "violation":{"result":"no"}}. Eligible trucks exist in the first stage and violations are present: {"result":"yes","truck_boxes":[[xmin1,ymin1,xmax1,ymax1],...],

[0112] "violation":{"result":"yes","bounding_boxes":[[xmin1,ymin1,xmax1,ymax1],...],"license_plate":"number / unrecognized"}}.

[0113] Herein, truck_boxes are marked as the bounding box of the entire eligible truck, and bounding_boxes are marked as the exposed cargo area. All coordinates are normalized to a 1000x1000 pixel space and kept as integers. Each eligible truck corresponds to 1 truck_boxes bounding box.

[0114] In addition, the preset prompt words may also be component-level exclusive prompt words configured for each semantic component. Each component's exclusive prompt word includes a task definition layer, a specification judgment standard layer, and an output format requirement layer, and the specific content of each layer is customized differently according to the judgment task of different components.

[0115] For example, taking the covered area component as an example, its task definition layer is: "You are a freight vehicle covered area specification judgment expert. The current task is to analyze the coverage status of the covered area. The analysis scope is strictly limited to the covered area of ​​the input image." Its specification judgment standard layer is: "The specification judgment standards for the covered area include: 1. Whether the tarpaulin completely covers the top of the cargo box without gaps or cracks; 2. Whether there are holes with a diameter exceeding 10 cm on the surface of the tarpaulin; 3. Whether the tarpaulin has large-area wrinkles that cause the coverage to fail; 4. Whether the color and texture of the tarpaulin are consistent with the standard tarpaulin material."

[0116] Taking the fixed device area components as an example, its task definition layer is: "You are a freight vehicle covering specification judgment expert. The current task is to analyze the fastening and constraint status of the fixed device area. The analysis scope is strictly limited to the fixed device area in the input image." Its specification judgment standard layer is: "The specification judgment standards for the fixed device area include: 1. Securing components: Whether the rope or bandage is intact and unbroken, whether the rope tension is sufficient and there is no obvious sagging, whether the rope tensioner or ratchet tightener is in the locked state, whether the fixing point, hook or anchor point is in place and undamaged, and whether the number of securing points on each side is not less than 3; 2. Covering components: Whether the pressure bar or pressure strip is in place and both ends are reliably fixed, whether the netting is intact and undamaged, and whether the arch rod is deformed or displaced; 3. Whether the securing components and covering components cooperate and are not missing."

[0117] The output format requirements for the dedicated prompt words of each semantic component require that the layer output structured results, specifically including component name, whether it is abnormal, abnormal type, abnormal bounding box, severity, and confidence score.

[0118] The second model includes a multimodal large model, which is not a lightweight model. The multimodal large model achieves the unification of image understanding and text generation by fusing a visual encoder and a large language model. It includes visual encoders such as ViT, projection alignment layers, and large language models. Large language models, such as GPT and the Qwen series, can receive image and text inputs and output semantic understanding and reasoning results.

[0119] As mentioned above, when the initial confidence level of non-standard covering is below a threshold, the image frame corresponding to that confidence level, i.e., the corresponding candidate ROI image, is cached. The candidate ROI image and preset judgment prompts are jointly input into a multimodal large model to perform semantic verification of the freight vehicle covering status, and output semantic judgment results such as coverage integrity, edge anomalies, and fixation anomalies.

[0120] By introducing a multimodal large model for high-level semantic understanding, the system outputs semantic judgment results related to law enforcement rules, such as coverage integrity, edge anomalies, and fixed anomalies, thereby improving the interpretability of the judgments.

[0121] Specifically, the candidate regions of interest and their corresponding preset prompts are input into the second model to determine the first judgment result. This includes sending the candidate regions of interest and the preset prompts to the cloud server so that the candidate regions of interest and the preset prompts are jointly input into the preset second model deployed on the cloud server to obtain the semantic judgment result of the second model, thereby determining whether the target freight vehicle is improperly covered.

[0122] The semantic determination results include coverage integrity probability, edge anomaly probability, and fixed anomaly probability. Explanatory text may also be included.

[0123] The candidate ROI image Rj and the prompt word P are jointly input into the multimodal large model M to achieve accurate verification of the large model. The large model optimizes the reasoning logic based on the prompt word, mainly focusing on whether the covering is complete and compliant, and outputs the semantic judgment result, i.e. the covering recognition result.

[0124]

[0125] in, To cover the integrity probability, For marginal anomaly probability, To fix the anomaly probability, To interpret the text.

[0126] For non-on-site enforcement scenarios, this implementation method creates a violation determination format adapted to single-frame images, which does not rely on continuous video streams and solves the problem of video segment dependence in related technologies. Simultaneously, by combining candidate ROIs with large-model semantic verification, it achieves effective analysis of single-frame images, making it suitable for practical scenarios such as fixed-point capture and roadside non-on-site enforcement.

[0127] Applying a multimodal large model to the determination of tarpaulin compliance overcomes the technical limitations of relying on grayscale segmentation and pixel-level processing in related technologies. By guiding the large model to output semantic elements related to law enforcement rules, such as coverage integrity, edge anomalies, and fixation anomalies, through prompt word engineering, the determination results have clear semantic interpretation and meet the interpretability requirements of law enforcement applications.

[0128] As mentioned above, traditional visual segmentation of the carriage area can be performed using methods such as grayscale image segmentation, gradient fluctuation analysis, pixel-level probability calculation, and Otsu's method to identify uncovered areas. This approach primarily relies on grayscale images and pixel-level image processing, employing traditional algorithms such as segmentation, gradient analysis, neighborhood relationship analysis, and Otsu's method. Its focus is on segmenting uncovered areas, and its ability to determine high-level semantics in complex scenarios is limited. It can only identify uncovered areas and cannot further determine whether it constitutes non-standard covering, nor can it output semantic elements related to law enforcement rules such as coverage integrity, edge anomalies, and fixation anomalies.

[0129] In addition, the method 100 also includes: keeping the weight parameters of the second model unchanged and setting a low-rank matrix; and fine-tuning the second model by training the low-rank matrix.

[0130] To improve the adaptability of the large model to traffic enforcement scenarios, LoRA (Low-Rank Adaptation) is used to fine-tune the large model. The original weights of the large model are frozen, and only the low-rank adaptation matrix is ​​trained.

[0131]

[0132] in, It is a low-rank matrix. Here, d represents the dimension of the original weight matrix W, and r is the rank parameter.

[0133] The fine-tuning objective function is Lfine:

[0134]

[0135] Here, Lfine is used to measure the difference between the model's current prediction and the standard answer. During training, the model's goal is to continuously adjust the parameters to minimize this loss value. T refers to the length of the target sequence. Pθ refers to the model's conditional probability distribution. θ represents the model's parameters. Tk refers to the k-th token in the target sequence. <k It refers to all historical tokens before the k-th token.

[0136] By leveraging the precise tuning capabilities of the finely adjusted large model, feature matching is refined, resulting in a corrected cover recognition result. This leads to the final recognition outcome.

[0137] Through a collaborative mechanism of pre-filtering with a small model and precise verification with a large model, the small model is responsible for rapid vehicle localization and ROI generation, while the large model is responsible for refined semantic verification. A confidence-driven triage strategy directly outputs results for high-confidence scenarios and triggers verification by the large model for low-confidence scenarios. This significantly reduces computational overhead while maintaining recognition accuracy, achieving a balance between efficiency and accuracy.

[0138] Specifically, the second model can also be configured with multiple component-level LoRA adapters, with each semantic component corresponding to an independent LoRA adapter. Each LoRA adapter shares the basic weight parameters of the second model but has independent low-rank matrix parameters. As mentioned earlier, when dealing with the... When reviewing the semantic components, the first one is dynamically loaded. LoRA adapters for each semantic component are used to unload LoRA adapters for other components. Each LoRA adapter is trained independently using the labeled data for its corresponding component.

[0139] The forward propagation of a LoRA adapter is represented as follows:

[0140]

[0141] in, The original weight matrix frozen for the second model. and It is a trainable low-rank matrix. It is a rank parameter and , This is the scaling factor. As input features, For output features.

[0142] Step 104: Determine the probability of violation based on the first judgment result and the uncovered areas in the candidate region of interest.

[0143] The first determination result refers to the recognition result of the second model mentioned above.

[0144] The determined approach may include constructing a comprehensive risk score based on the estimated exposed areas in candidate ROIs and the semantic results output by the large model, and generating the final probability of violation through probability mapping.

[0145] Specifically, based on the first judgment result and the uncovered areas in the candidate region of interest, the probability of violation is determined, including: determining the cargo-carrying area and the cargo-exposed area of ​​the truck corresponding to the candidate region of interest, and determining the exposure ratio; determining the comprehensive score based on the exposure ratio and the coverage integrity probability, edge anomaly probability, and fixed anomaly probability in the semantic judgment result; and mapping the comprehensive score to the corresponding probability of violation.

[0146] Based on image segmentation or edge detection, estimate the cargo-carrying area Ac and the exposed area Ae in the candidate ROI, and calculate the exposed ratio r:

[0147]

[0148] The semantic probability output by the large model is combined with the exposure ratio, and a comprehensive risk score s is constructed by weighted summation:

[0149]

[0150] in, Let be the weighting coefficient, satisfying The range of values ​​is . To cover the integrity probability, For marginal anomaly probability, For fixed anomaly probability

[0151] The final probability of violation, pfinal, is obtained by mapping the comprehensive risk score using the Sigmoid function.

[0152]

[0153] in, For the Sigmoid function, is the translation parameter, and T is the temperature parameter.

[0154] Based on the comparison between the final probability of violation and the threshold, the judgment result is output as follows:

[0155]

[0156] Where θ is the threshold for determining illegality, θ low The low confidence threshold has a range of values. .

[0157] If the covering is deemed improper, an alarm and evidence collection will be automatically triggered. If it is deemed a suspected violation, the data will be submitted to the manual review center. If the transportation is deemed compliant, the data will be archived and no alarm will be triggered.

[0158] This implementation constructs a method for calculating a comprehensive risk score and the final probability of violation, combining semantic output with geometric region information to achieve a leap from qualitative judgment to quantitative assessment. Through Sigmoid probability mapping and a threshold determination mechanism, the judgment results possess quantifiable and threshold-controllable characteristics, facilitating engineering deployment and the standardization of law enforcement standards.

[0159] In addition, the probability of violation can be determined based on the aforementioned first determination result and the uncovered areas in the candidate region of interest, including: establishing a component spatial relationship matrix; calculating the basic weighted score; calculating the topology consistency penalty term; and calculating the final probability of violation.

[0160] Establish a component spatial relationship matrix by predefining the spatial topological relationships between each semantic component. , among which, element Indicates components With components Spatial relationship types, as shown below:

[0161]

[0162] Based on the judgment results of each semantic component, the basic weighted score is calculated using the following formula:

[0163]

[0164] in, For components The violation score is obtained by mapping the confidence level and severity level output by the second model as described above, where the severity level is the quantification result of the violation category; For components The weighting coefficients satisfy The range of values ​​is .

[0165] Based on the component spatial relationship matrix and the violation scores of each semantic component, the topology consistency penalty term is calculated using the following formula:

[0166]

[0167] in, The penalty for violating topological relations is determined by the following formula:

[0168]

[0169] It should be noted that when the spatial relationship is a covering relationship or an inclusion relationship, if the judgment result of the covering party or the included party is normal and the judgment result of the covered party or the included party is abnormal, it is judged as a semantic contradiction and a positive penalty is imposed; when the spatial relationship is an adjacency relationship, if the score difference of the adjacent components exceeds the preset difference threshold, it is judged as a spatial inconsistency and a penalty proportional to the difference magnitude is imposed.

[0170] The overall risk score is obtained by combining the basic weighted score with the topology consistency penalty.

[0171]

[0172] in, The constraint coefficient has a range of values. .

[0173] The overall risk score is mapped to the final probability of violation using the Sigmoid function:

[0174]

[0175] Where μ is the translation parameter, with a value range of [0,1]; T is the temperature parameter, with a value range of [0.01,1.0].

[0176] Step 105: Determine the second judgment result based on the probability of violation, and generate evidence images and records corresponding to the second judgment result.

[0177] Based on the judgment results, evidence images and structured records are generated, supporting background evidence storage and manual review.

[0178] Specifically, a second determination result is determined based on the probability of violation, and evidence images and records corresponding to the second determination result are generated, including: determining whether non-standard covering constitutes a violation based on the probability of violation and the violation threshold; if it is determined that non-standard covering constitutes a violation, then generating corresponding evidence images and records based on the target frame corresponding to the target freight vehicle and the location of the non-standard covering.

[0179] For images determined to be improperly covered, an evidence image is generated, overlaid with the following information, which may include timestamp and location information, device number and lane number, vehicle target box annotation, highlight mark of violation area, and abnormal description text.

[0180] Generate a structured record in JSON format, containing the following fields:

[0181] {

[0182] "capture_time":"YYYY-MM-DD HH:MM:SS",

[0183] "capture_location":"Road segment name / station number",

[0184] "device_id":"device number",

[0185] "vehicle_type":"Type of freight vehicle",

[0186] "bbox":[x1,y1,x2,y2],

[0187] "judgment_result":"Incorrect covering / Standardized / Manual review",

[0188] "risk_score": 0.85,

[0189] "final_probability": 0.92,

[0190] "semantic_output":{

[0191] "coverage_probability": 0.15,

[0192] "edge_anomaly_probability":0.78,

[0193] "fixation_anomaly_probability":0.65

[0194] },

[0195] "evidence_image_path":" / path / to / evidence.jpg"

[0196] }

[0197] Evidence images and structured records are uploaded to the back-end management system to support subsequent review and processing by law enforcement officers.

[0198] Evidence reports with superimposed spatiotemporal metadata are generated, and difficult cases are automatically fed back to a private sample pool to continuously evolve the multimodal large model.

[0199] This implementation can generate evidence images with overlaid time, location, device number, target box annotation and anomaly description, as well as structured records containing capture time, location, vehicle type, judgment result, risk score and semantic output, forming a complete chain of evidence. It supports background evidence storage and manual review, making up for the lack of support for law enforcement evidence collection in related technologies.

[0200] As mentioned above, on-site early warning can be the primary method, employing cameras, multi-angle capture, analysis modules, and audible and visual alarm devices to identify the covering status of large trucks and issue alerts. However, by focusing on on-site alarm devices and emphasizing hardware structure and alarm components, the main output is alerts and warnings, which is insufficient for supporting law enforcement evidence collection and structured violation determination. It cannot generate interpretable and verifiable violation auxiliary conclusions and structured records.

[0201] Furthermore, traditional pixel-level segmentation methods are poorly adapted to complex backgrounds, backlighting, shadows, and partially occluded scenes, easily resulting in missed detections and false detections, and lacking robustness. This implementation method addresses the limited discriminative ability of a single small model in complex scenes, while directly applying a large model to all images incurs excessive computational overhead and lacks an effective collaborative mechanism to balance efficiency and accuracy. It simultaneously considers the efficient screening capability of the small model and the high-level semantic understanding capability of the large model, ensuring both recognition accuracy and processing efficiency. After obtaining the judgment results of each semantic component, a cross-component result consistency check can be performed before calculating the probability of violation. The consistency check includes the following rules: if the judgment result of the covered area component is complete coverage, while the judgment result of the exposed goods area component is exposed, it is judged as a semantic contradiction, and the contradiction information is recorded in the judgment log. If the judgment result of the fixed device area component is normal, while the judgment result of the edge area component is that there is an edge overhang, it is judged as a semantic inconsistency, and the inconsistency information is recorded in the judgment log. If the abnormal boundary frame of a component in the exposed area of ​​the cargo exceeds the cargo box area indicated by the cargo box area mask, it is determined to be a spatial error, and the spatial error information is recorded in the determination log. If the determination result of the fastening components in the fixing device area is normal, but the determination result of the cover components is missing or malfunctioning, it is determined to be a missing component, and the missing information is recorded in the determination log; conversely, if the cover components are normal but the fastening components are malfunctioning, it is also determined to be a failure of the entire fixing device, and the failure information is recorded in the determination log.

[0202] When semantic contradictions or inconsistencies are detected, the confidence levels of the components of both sides of the contradiction are compared, and the judgment result with the higher confidence level is retained. When the difference in the confidence levels of the components of both sides of the contradiction is less than the preset difference threshold, the local views of both sides of the contradiction are stitched together and re-input into the second model for joint judgment. The preset prompt for joint judgment includes the instruction "Please combine the information of the following two areas to judge whether there is a contradiction in the covering status".

[0203] In a specific example, the input image is captured by a fixed camera with a resolution of 1920×1080. The vehicle localization module uses the YOLOv8n lightweight object detection model with a confidence threshold of 0.5 to identify the main region of the freight vehicle and output the vehicle bounding box. The candidate ROI generation module expands the boundaries based on the vehicle bounding box, with an expansion coefficient of [value missing]. The system generates candidate ROI regions including the top and edges of the cargo box and normalizes them to a size of 448×448. The multimodal large-scale model semantic verification module uses the Qwen2-VL-7B-Instruct model, fine-tuned via LoRA, with rank r=8 and α=16, to adapt to traffic enforcement scenarios. The prompt words include a three-layer structure of task definition, judgment criteria, and output format. The model output covers integrity probability, edge anomaly probability, and fixed anomaly probability. The rule quantization module calculates a comprehensive risk score based on the semantic output and exposure ratio, with weight coefficients set to... The final probability of violation is generated by mapping μ=0.5, T=0.1 using a Sigmoid algorithm, with a threshold θ=0.75. low =0.55. When the final probability of violation... When outputting non-standard covering results, it generates forensic images with timestamps, locations, device numbers, target bounding boxes, and anomaly descriptions, as well as structured records in JSON format. When, output the results of manual review; when When that happens, output the standard results.

[0204] In another specific example, a small-model collaborative mechanism is used for optimization. The small model, YOLOv8n, performs initial recognition on each frame of the image and outputs the confidence score. When the confidence level is high, the judgment result is output directly without entering the large model stage. At that time, the large model is triggered to perform secondary recognition.

[0205] Experimental results show that the collaborative mechanism reduces the average processing latency by about 60% while maintaining a recognition accuracy of 91.5%, significantly improving the real-time processing capability of the server system.

[0206] In the third specific example, the input image was captured by a fixed snapshot camera with a resolution of 1920×1080. The first model uses YOLOv8n-MT, which adds a part segmentation head to YOLOv8n. The part segmentation head has approximately 1.2M parameters, and the total model parameters are approximately 4.3M. The part mask output by the part segmentation head has a resolution of 160×160, and the number of part categories K=5. A high confidence threshold is used. Low confidence threshold The second model uses Qwen2-VL-7B-Instruct, configuring 5 component-level LoRA adapters, with each LoRA adapter having a rank... Scaling factor The target modules are query projection and value projection. The component weight vector is set to... constraint coefficient Translation parameters Temperature parameters The threshold for determining illegality is set at 0.75, and the threshold for determining low confidence is set at 0.55.

[0207] Experimental results show that, while maintaining an accuracy rate of 91.5%, this embodiment reduces the average processing latency from 105ms in the original scheme to approximately 72ms, the large model call rate from 62% to 38%, and the contradiction determination rate from 4.3% to 1.1%. Furthermore, since the determination results of each component can be output independently and superimposed onto the evidence image, the interpretability of the determination is significantly improved compared to the original scheme.

[0208] like Figure 4 As shown, this application also proposes a device for determining improper covering of freight vehicles, applied to service equipment. The device 400 includes:

[0209] The acquisition module 401 is used to acquire a single frame image, wherein the single frame image includes the target freight vehicle.

[0210] The first determining module 402 is used to identify the target freight vehicle in the single frame image according to the first model and determine the corresponding candidate region of interest.

[0211] The second determining module 403 is used to input the candidate region of interest and the corresponding preset prompt words into the second model to determine the first determination result. The preset prompt words are used to determine whether the covering is not standardized.

[0212] The third determining module 404 is used to determine the probability of violation based on the first determination result and the uncovered areas in the candidate region of interest.

[0213] The generation module 405 is used to determine a second judgment result based on the probability of violation, and to generate evidence images and records corresponding to the second judgment result.

[0214] The first model includes an object detection model, which is a lightweight detection model; the second model includes a multimodal large model, which is a non-lightweight model.

[0215] The acquisition module 401 is specifically used to acquire a single-frame image containing the target freight vehicle from the monitoring equipment. The device 400 further includes a preprocessing module, used to process the acquired single-frame image in a preset format, evaluate the image quality of the processed image, and, if the preset quality standard is met, execute the step of identifying the target freight vehicle in the single-frame image according to the first model and determining the corresponding candidate region of interest.

[0216] Specifically, the first determining module 402 includes: an inference unit, used to input the single-frame image into the first model, so that the first model performs forward inference, identifies vehicles in the single-frame image, and determines the target bounding box of the vehicle and the category confidence of the vehicle; and a filtering unit, used to filter the target bounding box of the target freight vehicle from the vehicles according to the category confidence and the category of the freight vehicle, and determine the candidate region of interest.

[0217] In addition, the device 400 also includes: an expansion module for expanding the boundaries of the selected target boxes so that the expanded target boxes include each edge of the cargo box of the target freight vehicle; and a cropping module for cropping the corresponding image from the single frame image based on the expanded target boxes and determining the candidate region of interest.

[0218] In addition, the first determining module 402 is also used to determine whether the target freight vehicle is improperly covered based on the candidate region of interest and the first model, and to determine the confidence level of whether it is properly covered; when the confidence level of whether it is properly covered is less than a threshold, the step of inputting the candidate region of interest and the corresponding preset prompt word into the second model to determine the first determination result is executed; otherwise, the determination of whether the target freight vehicle is improperly covered is directly based on the determination result of the first model.

[0219] Specifically, the second determining module 403 includes: a sending module, used to send the candidate region of interest and the preset prompt words to the cloud server, so that the candidate region of interest and the preset prompt words are jointly input into a preset second model deployed on the cloud server to obtain the semantic determination result of the second model, so as to determine whether the target freight vehicle is improperly covered.

[0220] The semantic determination results include coverage integrity probability, edge anomaly probability, and fixed anomaly probability.

[0221] The preset prompts include a task definition layer, a standard judgment layer, and an output format requirement layer. The task definition layer is used to clarify the task object and limit the analysis scope. The standard judgment layer is used to define the characteristics of non-standard covering according to relevant laws and regulations. The output format requirement layer is used to output preset structured results.

[0222] In addition, the device 400 also includes: a holding module for keeping the weight parameters of the second model unchanged and setting a low-rank matrix; and a fine-tuning module for fine-tuning the second model by training the low-rank matrix.

[0223] Specifically, the third determining module 404 is used to determine the cargo-carrying area and cargo-exposed area of ​​the truck corresponding to the candidate region of interest, and to determine the exposure ratio; based on the exposure ratio and the coverage integrity probability, edge anomaly probability and fixed anomaly probability in the semantic judgment result, a comprehensive score is determined; and the comprehensive score is mapped to the corresponding violation probability.

[0224] Specifically, the generation module 405 is used to determine whether the non-standard covering constitutes a violation based on the violation probability and violation threshold; if it is determined that the non-standard covering constitutes a violation, it generates corresponding evidence images and records based on the target frame corresponding to the target freight vehicle and the location of the non-standard covering.

[0225] This will not be elaborated upon further here; for any content that has not been covered in detail, please refer to the content mentioned above.

[0226] like Figure 5 As shown, this application also proposes a device 500, including: a memory 501 and a processor 502; the memory 501 is used to store a computer program; the processor 502 is used to execute the computer program for:

[0227] Acquire a single-frame image, wherein the single-frame image includes the target freight vehicle;

[0228] The target freight vehicle in the single frame image is identified based on the first model, and the corresponding candidate region of interest is determined.

[0229] The candidate region of interest and the corresponding preset prompt words are input into the second model to determine the first judgment result. The preset prompt words are used to determine whether the covering is not standardized.

[0230] Based on the first determination result and the uncovered areas in the candidate region of interest, the probability of violation is determined;

[0231] A second determination result is determined based on the probability of violation, and evidence images and records corresponding to the second determination result are generated.

[0232] This will not be elaborated upon further here; for any content that has not been covered in detail, please refer to the content mentioned above.

[0233] This application also proposes a computer-readable storage medium storing a computer program, which, when executed by one or more processors, causes the one or more processors to implement the steps in method 100. Further details are omitted here; please refer to the foregoing descriptions for any omissions or omissions.

[0234] Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations appearing in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0235] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0236] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform, or by a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0237] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable multimedia data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable multimedia data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0238] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable multimedia data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0239] These computer program instructions may also be loaded onto a computer or other programmable multimedia data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0240] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0241] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0242] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0243] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for determining non-standard covering of freight vehicles, characterized in that, include: Acquire a single-frame image, wherein the single-frame image includes the target freight vehicle; The target freight vehicle in the single frame image is identified based on the first model, and the corresponding candidate region of interest is determined. The candidate region of interest and the corresponding preset prompt words are input into the second model to determine the first judgment result. The preset prompt words are used to determine whether the covering is not standardized. Based on the first determination result and the uncovered areas in the candidate region of interest, the probability of violation is determined; A second determination result is determined based on the probability of violation, and evidence images and records corresponding to the second determination result are generated.

2. The method according to claim 1, characterized in that, The first model includes an object detection model, which is a lightweight detection model; the second model includes a multimodal large model, which is a non-lightweight model.

3. The method according to claim 1, characterized in that, The acquisition of a single frame image includes: Capture single-frame images containing the target freight vehicle from monitoring equipment; The method further includes: The acquired single-frame image is processed in a preset format, and the image quality of the processed image is evaluated. If the preset quality standard is met, the step of identifying the target freight vehicle in the single-frame image according to the first model and determining the corresponding candidate region of interest is executed.

4. The method according to claim 1, characterized in that, The step of identifying the target freight vehicle in the single-frame image according to the first model and determining the corresponding candidate region of interest includes: The single-frame image is input into the first model, which then performs forward inference to identify the vehicle in the single-frame image and determine the target box of the vehicle and the category confidence of the vehicle. Based on the category confidence level and the category of the freight vehicle, target boxes for the target freight vehicle are filtered from the vehicles, and the candidate regions of interest are determined.

5. The method according to claim 4, characterized in that, The method further includes: The selected target box is expanded to include all edges of the cargo box of the target freight vehicle. Based on the expanded target bounding box, the corresponding image is cropped from the single-frame image, and the candidate region of interest is determined.

6. The method according to claim 1 or 4, characterized in that, The method further includes: Based on the candidate region of interest and the first model, determine whether the target freight vehicle is improperly covered, and determine the confidence level of whether it is properly covered; If the confidence level of whether it is normal is less than the threshold, then the step of inputting the candidate region of interest and the corresponding preset prompt word into the second model to determine the first judgment result is executed; Otherwise, the determination of whether the target freight vehicle is improperly covered is directly based on the determination result of the first model.

7. The method according to claim 1 or 5, characterized in that, The step of inputting the candidate region of interest and the corresponding preset prompt word into the second model to determine the first determination result includes: The candidate region of interest and the preset prompt words are sent to the cloud server so that the candidate region of interest and the preset prompt words are jointly input into the preset second model deployed on the cloud server to obtain the semantic judgment result of the second model, so as to determine whether the target freight vehicle is improperly covered; The semantic determination results include coverage integrity probability, edge anomaly probability, and fixed anomaly probability.

8. The method according to claim 1 or 7, characterized in that, The preset prompts include a task definition layer, a standard judgment layer, and an output format requirement layer. The task definition layer is used to clarify the task object and limit the analysis scope. The standard judgment layer is used to define the characteristics of non-standard covering according to relevant laws and regulations. The output format requirement layer is used to output preset structured results.

9. The method according to claim 7, characterized in that, The method further includes: Keeping the weight parameters of the second model unchanged, set a low-rank matrix; The second model is fine-tuned by training the low-rank matrix.

10. The method according to claim 1, characterized in that, The step of determining the probability of violation based on the first determination result and the uncovered areas in the candidate region of interest includes: Determine the cargo-carrying area and cargo-exposed area of ​​the truck corresponding to the candidate region of interest, and determine the exposure ratio; The comprehensive score is determined based on the exposure ratio and the coverage integrity probability, edge anomaly probability, and fixed anomaly probability in the semantic judgment results. The comprehensive score is mapped to the corresponding probability of violation.

11. The method according to claim 1, characterized in that, The step of determining a second judgment result based on the probability of violation and generating evidence images and records corresponding to the second judgment result includes: Based on the aforementioned probability of violation and the violation threshold, determine whether the act constitutes improper covering. If it is determined that the tarpaulin is improperly covered, then corresponding evidence images and records are generated based on the target frame corresponding to the target freight vehicle and the location of the improper tarpaulin.

12. A device for determining non-standard covering of freight vehicles, characterized in that, include: The acquisition module is used to acquire a single-frame image, wherein the single-frame image includes the target freight vehicle; The first determining module is used to identify the target freight vehicle in the single frame image according to the first model and determine the corresponding candidate region of interest; The second determining module is used to input the candidate region of interest and the corresponding preset prompt words into the second model to determine the first determination result. The preset prompt words are used to determine whether the covering is not standardized. The third determining module is used to determine the probability of violation based on the first determination result and the uncovered areas in the candidate region of interest; The generation module is used to determine a second judgment result based on the probability of violation, and to generate evidence images and records corresponding to the second judgment result.

13. A device, characterized in that, include: Memory, processor; The memory is used to store computer programs; The processor is configured to execute the computer program for: Acquire a single-frame image, wherein the single-frame image includes the target freight vehicle; The target freight vehicle in the single frame image is identified based on the first model, and the corresponding candidate region of interest is determined. The candidate region of interest and the corresponding preset prompt words are input into the second model to determine the first judgment result. The preset prompt words are used to determine whether the covering is not standardized. Based on the first determination result and the uncovered areas in the candidate region of interest, the probability of violation is determined; A second determination result is determined based on the probability of violation, and evidence images and records corresponding to the second determination result are generated.

14. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by one or more processors, it causes the one or more processors to perform the steps of the method according to claims 1-11.

Images