Protective equipment wearing detection and model training method and device, equipment and medium

Abstract

The present specification provides a protective equipment wearing detection and model training method, device, equipment and medium, the detection method comprises: using a pre-trained lightweight target detection model to perform three-level cascade initial detection on an image, and detecting in turn whether the image contains a portrait, whether the image contains protective equipment and the wearing state; determining whether review is needed according to whether the confidence of the initial detection result of each detection task falls into the corresponding uncertainty interval; for the detection task that needs to be reviewed, constructing targeted prompt data to guide a multimodal large model to perform review detection; and determining the final detection result by comprehensively determining the results of each detection task. Through the task-level shunting mechanism combining the initial screening of the lightweight model and the review of the multimodal large model, the present specification improves the accuracy of protective equipment wearing detection while controlling the computing cost.

Description

Technical Field

[0001] This specification relates to the fields of computer vision and artificial intelligence technology, and in particular to methods, devices, equipment and media for detecting the wearing of protective equipment and training models. Background Technology

[0002] In scenarios such as on-demand delivery, construction, and industrial production, wearing protective equipment (such as helmets, safety helmets, goggles, and protective gloves) is a basic requirement for ensuring personal safety. Currently, the industry mainly uses deep learning-based object detection models to detect and identify people and protective equipment in images. However, traditional detection solutions have high false positive and false negative rates in complex scenarios. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this manual provides methods, devices, equipment and media for protective equipment wearing detection and model training.

[0004] According to a first aspect of the embodiments of this specification, a method for detecting the wearing of protective equipment based on a multimodal large model is provided, comprising: Acquire one or more frames of images captured in this detection; A pre-trained lightweight object detection model is used to perform an initial detection on the image. This initial detection includes the following cascaded detection tasks: detecting whether the image contains a human figure; if a human figure is clearly or possibly detected, detecting whether protective equipment is being worn; if protective equipment is clearly or possibly detected, detecting the wearing status of the protective equipment. Based on whether the confidence level of the initial detection result falls within the uncertainty interval corresponding to that detection task, it is determined whether the detection task needs to be reviewed. The uncertainty interval characterizes the range of values ​​within which the initial detection result of the detection task is insufficient to form a definitive conclusion. If the testing task does not require verification, the initial testing result shall be used as the judgment result for the testing task. If the detection task requires verification, then construct prompt data to guide the multimodal large model to perform protective equipment wearing detection related to the detection task, and use the multimodal large model to detect the image based on the prompt data to obtain the judgment result of the detection task; Based on the judgment results of each testing task, the final protective equipment wearing test result is determined.

[0005] According to a second aspect of the embodiments of this specification, a method for training an object detection model is provided, comprising: The image is acquired and input into the target detection model; the target detection model is used to perform the following cascaded detection tasks: detect whether the image contains a human image; if a human image is clearly or suspected to be contained, detect whether protective equipment is being worn; if protective equipment is clearly or suspected to be being worn, detect the wearing status of the protective equipment. After determining the total loss based on the recognition result of the image by the target detection model, the target detection model is trained based on the total loss; The total loss of the target detection model during training is determined as follows: based on the task weights corresponding to each detection task, the cascaded perception loss of each detection task is weighted and summed to obtain the total loss; wherein: the task weight of each detection task is configured according to the degree of influence of the level to which the detection task belongs on the final protective equipment wearing detection result; the cascaded perception loss includes at least a focus loss term.

[0006] According to a third aspect of the embodiments of this specification, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the aforementioned first or second aspect method embodiments.

[0007] According to a fourth aspect of the embodiments of this specification, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the aforementioned first or second aspect method embodiments.

[0008] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the methods of the first or second aspect described above.

[0009] The technical solutions provided in the embodiments of this specification may include the following beneficial effects: In the embodiments described in this specification, an initial detection of images is performed using a three-level cascaded target detection model. This allows for the rapid screening of cases with clear conclusions with low computational overhead, while only questionable cases with confidence levels falling within the uncertainty range are submitted to a multimodal large-scale model for verification. This approach, combining lightweight model coarse screening with multimodal large-scale model fine review, enables simple cases to be detected quickly, while complex cases undergo detailed verification by the multimodal large-scale model. This significantly improves detection accuracy while controlling overall processing costs, and leverages the semantic understanding capabilities of the multimodal large-scale model to effectively determine the compliance of wearing practices.

[0010] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0011] Figure 1A This specification is a flowchart illustrating a method for detecting the wearing of protective equipment based on a multimodal large model, according to an exemplary embodiment. Figure 1B This is a flowchart illustrating another protective equipment wearing detection method based on a multimodal large model, according to an exemplary embodiment of this specification; Figure 1C This is a flowchart illustrating another protective equipment wearing detection method based on a multimodal large model, according to an exemplary embodiment of this specification.

[0012] Figure 1D This is a schematic diagram illustrating an application scenario of the protective equipment wearing detection method according to an exemplary embodiment of this specification.

[0013] Figure 2 This is a flowchart illustrating a method for detecting the wearing of protective equipment based on a multimodal large model, according to an exemplary embodiment of this specification.

[0014] Figure 3 This is a hardware structure diagram of a computer device illustrated in this specification according to an exemplary embodiment.

[0015] Figure 4 This is a block diagram illustrating a protective equipment wearing detection device according to an exemplary embodiment.

[0016] Figure 5 This is a block diagram illustrating a training apparatus for an object detection model according to an exemplary embodiment of this specification. Detailed Implementation

[0017] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification as detailed in the appended claims.

[0018] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant regions, and corresponding operation portals shall be provided for users to choose to authorize or refuse.

[0019] Currently, the industry mainly uses deep learning-based object detection models, such as the YOLO (You Only LookOnce) series, SSD (Single Shot MultiBox Detector) series, and Faster R-CNN (Faster Region-based Convolutional Neural Network), to detect and identify people and protective equipment in images. However, these traditional detection solutions face significant technical bottlenecks in practical deployment: On the one hand, limited by model capacity and the coverage of training data, traditional object detection models have high false positive and false negative rates in complex scenarios (such as occlusion, insufficient lighting, and non-standard wearing postures). The overall error rate in actual deployment is usually between 15% and 25%, which is difficult to meet the high accuracy requirements of safety management. On the other hand, traditional object detection models can essentially only output the object category and bounding box, and it is difficult to make effective semantic judgments on whether the wearing is standardized (such as whether the helmet is crooked or the straps are fastened). The high false positive rate forces platforms to invest a lot of human resources in secondary review, which is costly per manual review and often takes days to process, affecting operational efficiency and the experience of employees.

[0020] To improve detection accuracy, an intuitive approach is to directly use a Multimodal Large Language Model (MLLM) to analyze and judge all images to be detected. MLLMs, with their powerful visual understanding and semantic reasoning capabilities, can indeed provide more accurate judgment results and natural language interpretations in complex scenes. However, applying MLLMs to the detection of all images one by one is not feasible in engineering practice: a single MLLM inference call incurs costs and inference latency. In business scenarios with millions of detections per day, calling the entire MLLM would generate enormous computational costs and time overhead, failing to meet real-time or near-real-time business requirements. Another approach is to use a larger-scale, deeper object detection model to replace the lightweight model, but this solution can only improve detection accuracy to a limited extent; it is an incremental optimization and cannot address the inherent shortcomings of traditional detection models, such as insufficient judgment ability on uncertain samples and a lack of semantic understanding and reasoning capabilities.

[0021] In summary, how to improve the accuracy of protective equipment wearing detection and identify wearing compliance by leveraging the semantic understanding capabilities of multimodal large models while controlling computational costs has become an urgent technical problem to be solved.

[0022] Based on this, the embodiments of this specification propose a protective equipment wearing detection scheme based on a multimodal large model. By combining a lightweight target detection model for initial detection and a multimodal large model for secondary verification of questionable cases, along with a cascaded uncertainty diversion mechanism and a dynamic threshold adjustment strategy, high-accuracy wearing compliance detection is achieved while controlling computational costs. Specifically, as shown... Figure 1A The diagram shown is a flowchart of a protective equipment wearing detection method based on a multimodal large model provided in an embodiment of this specification, which may include the following steps: Step S110: Obtain one or more frames of images captured in this detection.

[0023] Step S120: Use a pre-trained lightweight object detection model to perform initial detection on the image.

[0024] The initial detection includes the following cascaded detection tasks: Detect whether the image contains a human figure. If a person's image is clearly detected or suspected to be present, the system will check whether protective equipment is being worn. If it is detected that someone is clearly wearing protective equipment or is suspected of wearing protective equipment, check the wearing status of the protective equipment. Step S130: Based on whether the confidence level of the initial detection result of the detection task falls within the uncertainty range corresponding to the detection task, determine whether the detection task needs to be reviewed. If no review is required, the initial detection result is used as the judgment result. If review is required, construct prompt data to guide the multimodal large model to perform protective equipment wearing detection related to the detection task, and use the multimodal large model to detect the image based on the prompt data to obtain the judgment result.

[0025] Step S140: Determine the final protective equipment wearing test result based on the judgment results of each test task.

[0026] In real-world scenarios such as on-demand delivery, construction site operations, and factory workshops, the wearing of protective equipment (PEG) is typically required to ensure the safety of workers, with safety helmets being the most typical example. The PEG-based PE detection method based on a multimodal large model provided in this specification can be applied to various work scenarios requiring automated detection of PEG wearing by workers. Application scenarios can be diverse, including delivery capacity scenarios, construction site operations, and factory workshop scenarios. PE includes, but is not limited to, helmets, safety helmets, goggles, face shields, and reflective vests; the specific type of PEG to be detected can be configured according to the actual application scenario.

[0027] Taking delivery scenarios as an example, the main protective equipment is a helmet. Accordingly, the wearing status can include various categories such as whether the helmet is correctly worn on the head and whether the helmet strap (chin strap) is fastened. For example, helmet wearing status can be further subdivided into: correctly worn (helmet on top of the head and strap fastened), not fastened (helmet on top of the head but strap not fastened or not fastened), and not worn (helmet not detected on the head). Detecting the strap status is crucial for delivery safety because during riding, a helmet with an unfastened strap is prone to falling off in a collision and will not provide protection.

[0028] Besides delivery scenarios, the methods described in this specification can also be applied to other scenarios requiring the detection of protective equipment (PE) wearing. For example, in a construction site scenario, fixed cameras can be installed at the site entrance or on-site to detect whether personnel are wearing safety helmets; in a factory workshop scenario, fixed cameras can be installed at the workshop entrance or key workstations to monitor workers' PE wearing status in real time. In these fixed camera scenarios, the cameras are typically installed above door frames or on the side of passageways, with a top-down or eye-level viewing angle. The method framework described in this specification is applicable to the above-mentioned different viewing angles and scenarios, and the detection parameters are adjusted through a scenario adaptive mechanism to adapt to different scenario characteristics.

[0029] In step S110, detection events can be set according to specific scenarios. The number of image frames captured in a single detection event is not limited. For example, in a delivery scenario, a delivery rider fixes their phone to the handlebar bracket while riding, and the client can randomly trigger snapshots in the background. Each detection event can continuously capture multiple frames (the specific number is set according to actual needs). In a construction site scenario, a fixed camera can capture images of passing pedestrians at a predetermined frequency, and each detection event can capture one or more frames. In this embodiment, the application is based on the authorization and consent of the detected object.

[0030] Single-shot detection schemes based on traditional object detection models have revealed several limitations in practical operation. From the perspective of detection accuracy, due to the complex and varied image acquisition conditions—including blurred helmet outlines caused by backlighting or low light, occlusion caused by riders looking down or turning to the side, significant differences in appearance between different helmet styles and colors, and perspective distortion caused by shooting from a specific angle—the model's confidence level for boundary samples often falls into an indeterminate middle ground. From the perspective of detection capability, the design goal of traditional object detection models is to output the object category and its bounding box. It can answer whether an object resembling protective equipment exists in the image, but it cannot understand whether the protective equipment is worn correctly (e.g., whether the helmet is tilted or the chin strap is fastened). These fine-grained semantic judgments involving proper wearing exceed the capabilities of traditional object detection models.

[0031] In the embodiments described in this specification, a two-stage detection method is designed, consisting of an initial detection using a lightweight target detection model and a secondary detection using a multimodal large model. Furthermore, a refined interaction mechanism between the two stages and a special architectural design are implemented for protective equipment wearing detection scenarios: by decomposing the detection process into multiple cascaded classification detection tasks and independently configuring an uncertainty threshold range for each task, the granularity of uncertainty judgment is advanced from the image level to the task level; furthermore, by dynamically constructing structured prompt data based on the specific detection task triggering the secondary detection, the multimodal large model is guided to perform focused secondary detection on specific issues, rather than generalized image description or general detection.

[0032] The object detection model is a lightweight deep learning model with multi-classification capabilities, such as a model optimized based on architectures like YOLO, MobileNet, or EfficientDet. This model plays a rapid initial screening role in the initial detection stage, aiming to complete basic image classification with low computational overhead. Cascaded detection tasks refer to multiple detection tasks with logical sequential dependencies and conditional triggering relationships, where the execution of a later detection task is predicated on a specific output result of a previous detection task. The three cascaded detection tasks in this embodiment are examples; in practical applications, there may be many more cascaded detection tasks.

[0033] The uncertainty interval can refer to a pre-determined confidence level range for each detection task. When the confidence level of the initial detection result falls within this interval, it indicates that the target detection model lacks sufficient certainty in its judgment of the detection task, and the initial detection result is insufficient to form a definitive conclusion, requiring verification using a multimodal large model. This verification, also known as secondary detection, allows for independent configuration of the uncertainty interval for each detection task based on factors such as the task's difficulty, sample distribution, and business tolerance. This embodiment does not impose such limitations. Furthermore, the uncertainty interval for each detection task can be fixed or dynamically adjusted as needed in different detection events; this embodiment also does not impose such limitations.

[0034] The Multimodal Large Language Model (MLLM) refers to a deep learning model that simultaneously possesses visual information understanding and natural language processing capabilities. It can receive image and text prompts as input and generate natural language output including semantic analysis, judgment conclusions, and reasoning explanations. In practical applications, specific models can be selected from open-source models, self-built models, or fine-tuned models based on open-source models, as needed. This embodiment does not impose any limitations on these options. The prompt data can be constructed using a fixed template or dynamically assembled based on the specific content of the current detection task to be reviewed. This allows the MLLM to understand the specific problem to be judged, the image region or features to focus on, and the desired output format. The specific settings can be configured according to actual needs, and this embodiment does not impose any limitations on these settings.

[0035] In step S110, the image can be one frame or multiple frames. If the image is a single frame, the final result can be directly output after performing the above detection process on that frame. If the image is multiple frames, the detection process can be performed on each frame separately to obtain the final protective equipment wearing detection result. The method for determining this final result can be set as needed; for example, it can be determined by combining the detection results of multiple frames through mechanisms such as voting or weighted fusion. The multi-frame mode can effectively reduce the detection deviation caused by accidental factors such as shooting angle, instantaneous occlusion, and motion blur in a single frame image.

[0036] As an example, this specification provides a method for detecting the wearing of protective equipment based on a multimodal large model, which can be applied to the scenario of on-demand delivery capacity management. In this scenario, the delivery personnel can fix a mobile terminal (such as a mobile phone) to the handlebar bracket of an electric vehicle or motorcycle during riding. The delivery personnel client can randomly trigger image acquisition events during riding. For example, when the delivery personnel client detects that the rider is in a delivery riding state, it can trigger a detection event at random time intervals (e.g., every 15 to 45 minutes). In each detection event, the front or rear camera of the mobile phone continuously acquires multiple frames of images to be detected within a set time window. Since the mobile phone is fixed to the handlebar bracket, the camera's acquisition angle can be an upward angle, which can capture image information of the rider's upper body and head area. In a preferred embodiment, each detection event can acquire 4 frames of images, and the time interval between adjacent frames can be 200 milliseconds to 500 milliseconds to ensure that at least one frame of usable quality image is obtained under riding vibration conditions.

[0037] As an example, refer to Figure 1BThe diagram shown is a flowchart of another protective equipment wearing detection method based on a multimodal large model, as illustrated in an embodiment of this specification. The initial detection in this embodiment can employ a three-level cascaded detection approach, including sequentially executed human image detection (level 1), wearing determination (level 2), and wearing status detection (level 3), with an independent early termination mechanism at each level. The core design idea of ​​this three-level cascaded architecture is to decompose the complex protective equipment wearing detection task into three sub-tasks, progressing from coarse to fine and from easy to difficult. Each sub-task focuses only on its corresponding detection elements and terminates the subsequent detection process early when the confidence level is sufficiently certain, thereby significantly reducing computational overhead while ensuring detection accuracy.

[0038] Based on this, the initial detection of the image in step S120 mentioned above may include: The first-level task is to detect whether the image contains a human figure. If the confidence level of this detection task falls within the preset deterministic range corresponding to the absence of a human figure, the subsequent cascaded detection tasks are terminated, and the initial detection result of the absence of a human figure is output. Otherwise, the second-level task of detecting whether protective equipment is being worn is performed. Perform the second-level detection task to determine whether protective equipment is being worn; if the confidence level of this detection task falls within the preset certainty interval corresponding to not wearing protective equipment, then terminate the subsequent cascaded detection tasks and output the initial detection result of not wearing protective equipment; otherwise, proceed to the third-level detection task to determine the wearing status of protective equipment. Perform the Level 3 test to determine the wearing status of protective equipment, and output the initial test results of whether the equipment is worn correctly or incorrectly, along with its confidence level.

[0039] The first-level portrait detection task is used to determine whether a human figure exists in the image to be detected. The target detection model performs portrait detection on the image and outputs the portrait detection confidence score. If the confidence score falls into the "non-human figure" determination interval (e.g., the confidence score is lower than the first lower threshold), it is determined that there is no human figure in the current image, and the subsequent detection process is immediately terminated, directly outputting the "non-human figure" detection result. For example, when the first lower threshold is set to 0.35, if the portrait detection confidence score is 0.12, then the confidence score clearly falls into the non-human figure determination interval, and the system does not need to perform subsequent wearing determination and wearing status detection, directly outputting the non-human figure result. Similarly, if the portrait detection confidence score is higher than the first upper threshold (e.g., 0.85), it is confirmed that there is a human figure in the image, and the detection box and cropping region are passed to the second level. If the confidence score is between the first lower threshold and the first upper threshold (i.e., between 0.35 and 0.85), it is determined to be in the uncertain interval, and the image will be marked as requiring multimodal large model verification.

[0040] The second-level wearing determination task is used to determine whether a human target is correctly wearing protective equipment. After receiving the human detection bounding box and cropped area from the first level, the target detection model performs a wearing determination on the cropped area and outputs a wearing determination confidence score. If the confidence score falls within the "not wearing" determination range (i.e., the confidence score is below the second lower threshold, for example, 0.50), it is determined that the human target is not wearing protective equipment, and the subsequent detection process is immediately terminated, directly outputting the detection result of "not wearing". If the confidence score is above the second upper threshold (for example, 0.90), it is confirmed that the human target is wearing protective equipment, and the detection area is passed to the third level. If the confidence score is between 0.50 and 0.90, it is marked as needing review.

[0041] The third-level wearing status detection task can be used to determine whether the protective equipment is worn correctly. In the helmet delivery scenario, this level mainly detects whether the straps are fastened tightly. The target detection model performs fine detection on the head area and outputs the wearing status category (correctly worn, straps not fastened, etc.) and the corresponding confidence score. If the confidence score falls within the corresponding defined interval (e.g., lower threshold 0.45, upper threshold 0.85), the final detection result is directly output; otherwise, it is marked as needing review.

[0042] In the embodiments of this specification, each level of detection task can be set with an independent threshold pair, and the thresholds of different levels can be configured independently according to the detection difficulty and false judgment risk of the task at that level.

[0043] As an example, the threshold pair for Level 1 portrait detection is (0.35, 0.85), the threshold pair for Level 2 wearing determination is (0.50, 0.90), and the threshold pair for Level 3 wearing status detection is (0.45, 0.85). The design logic for these thresholds is as follows: Level 1 portrait detection is relatively simple, and the distinction between the background and the portrait is high, so the lower threshold can be set relatively low (0.35) to allow most non-portrait images to be quickly excluded; Level 2 wearing determination involves detailed analysis of the head region, and the risk of misjudgment is high, so a higher lower threshold (0.50) and upper threshold (0.90) can be set to ensure the reliability of the determination; Level 3 lacing detection is the most detailed task, and the detection target (lacing) occupies a very small proportion in the image, so the lower threshold can be set to 0.45 to accommodate more uncertain samples in the review process.

[0044] In the embodiments of this specification, a differentiated processing strategy can be adopted for each level of detection task in the intermediate zone of uncertainty. Specifically: in the first-level portrait detection, if the confidence level falls into the uncertainty zone, the frame can be marked as a "suspected portrait" and subsequent level detection can continue (i.e., marked as verification); in the second-level wearing determination, if the confidence level falls into the uncertainty zone, a multi-frame fusion mechanism can be triggered to jointly determine the image using other frames in the same detection event; in the third-level wearing status detection, if the confidence level falls into the uncertainty zone, the frame with the highest resolution among the multiple frames can be selected as the verification input to improve the recognition of fine features.

[0045] In some examples, the uncertainty interval can be determined in the following way: Obtain scene information corresponding to the image, wherein the scene information includes at least one of the following: lighting conditions, weather conditions, degree of target occlusion in the image, and shooting angle of the image acquisition device; Based on the scenario information, the width of the uncertainty interval corresponding to the detection task is dynamically adjusted so that the detection task in different scenarios has an uncertainty interval that is adapted to the scenario. The scene complexity represented by the scene information is negatively correlated with the width of the uncertainty interval.

[0046] As an example, scene information corresponding to the image can be obtained. This scene information may include at least one of lighting conditions, weather conditions, degree of occlusion, and shooting angle. Then, the scene complexity is determined based on the scene information, and the width of the uncertainty interval is dynamically adjusted according to the scene complexity. Notably, scene complexity and the width of the uncertainty interval are negatively correlated; that is, the more complex the scene, the narrower the uncertainty interval; and the simpler the scene, the wider the uncertainty interval.

[0047] The principle behind the aforementioned negative correlation design is that under complex scene conditions (such as low light at night, rainy days, partially occluded targets, extreme elevation or depression angles), the detection capability of the target detection model decreases, and the reliability of its output confidence level reduces. If a wide uncertainty interval is maintained in such conditions, a large number of samples will be marked as "uncertain" and enter the multimodal large model review process, leading to a sharp increase in review costs. Therefore, in complex scenes, narrowing the uncertainty interval (i.e., raising the confidence threshold for triggering review) ensures that only samples truly near the decision boundary enter the review process, thereby controlling review costs. Conversely, under simple scene conditions (such as sufficient daylight, sunny days, unobstructed targets, and frontal eye-level viewing angles), the model confidence level is relatively reliable. In this case, appropriately widening the uncertainty interval allows more boundary samples to be included in the review scope, further improving detection accuracy.

[0048] In one implementation, scene complexity can be assessed using a multi-dimensional weighted scoring mechanism. As an example: Lighting conditions can be divided into three levels of average image brightness: low light (brightness < 80), normal light (brightness 80-180), and strong light (brightness > 180). Both low light and strong light increase the complexity score.

[0049] Weather conditions can be obtained through the meteorological API. Current weather information (sunny, cloudy, light rain, heavy rain, foggy, etc.) can be obtained, and different weather conditions correspond to different complexity increases.

[0050] Geographic location can be determined via GPS, which can be used to determine the city level (first-tier city, second-tier city, third-tier city and below). The city level can reflect the differences in road environment and traffic complexity.

[0051] The degree of occlusion can be scored based on whether there are large areas of obstruction within the detection frame.

[0052] The shooting angle can be scored based on the deflection angle estimated from the facial pose.

[0053] The scene complexity score can be obtained by weighted summation of each dimension. The width W of the uncertainty interval can be adjusted according to the following relationship: W = × (1 - β × ),in β is the baseline interval width, and β is the adjustment coefficient (e.g., 0.3). Normalize to the [0,1] interval. When When =0 (the simplest scenario), the interval width is ;when When =1 (most complex scenario), the interval width narrows to ×(1-β).

[0054] In some examples, the width of the uncertainty interval corresponding to different detection tasks is configured differently according to the level to which the detection task belongs; wherein, the level of the detection task is positively correlated with the width of the corresponding uncertainty interval.

[0055] In the embodiments of this specification, the width of the uncertainty interval can be positively correlated with the level of the detection task; that is, the higher the level of the detection task, the wider its uncertainty interval. For example, the uncertainty interval of the third-level wearing status detection (e.g., strap detection) is the widest, the uncertainty interval of the second-level wearing determination is the second widest, and the uncertainty interval of the first-level portrait detection is the narrowest.

[0056] The principle behind the aforementioned positive correlation design is that the higher the level of the detection task, the more refined the detection target, the greater the difficulty in judgment, and the weaker the model's confidence level's discriminative ability. Taking the third-level frenulum detection as an example, the frenulum typically occupies only a few pixels in the image and is easily obscured by the chin, collar, etc. The model's confidence distribution for this task is often relatively concentrated, with a large overlap between the confidence levels of correct and incorrect predictions. Therefore, a wider uncertainty interval is needed to capture more potential misclassified samples. Conversely, the detection target (complete human image) in the first-level portrait detection task occupies a large proportion of the image and has significant features. The model's confidence distribution has good bimodal characteristics, and the separation between the confidence levels of correct and incorrect predictions is high. Therefore, a narrower uncertainty interval is sufficient to meet the accuracy requirements.

[0057] As an example, the uncertainty interval width ratio for a three-level detection task can be set to 1:1.5:2. Using the baseline interval width of 0.50 for the first-level face detection (i.e., threshold 0.35-0.85, interval width 0.50) as a reference, the interval width for the second-level wearing determination is 0.40 (threshold 0.50-0.90), and the interval width for the third-level wearing status detection is 0.40 (threshold 0.45-0.85). It should be noted that the "interval width" here refers to the difference between the upper and lower thresholds. A larger difference means more samples are marked as uncertain and enter the review process. In actual deployment, the interval widths for each level can be optimized based on the confidence distribution statistics on the validation set, so that the review rate for each level can be controlled within the required range (e.g., first-level review rate 5%-10%, second-level review rate 10%-15%, third-level review rate 15%-25%).

[0058] In some examples, embodiments of this specification can construct targeted prompt data based on the specific detection task triggering the review and the initial detection results, to guide the multimodal large model to perform accurate detection on the detection elements of interest for that task. The construction of the prompt data can follow a "task-driven" principle, that is, generating prompt data content related to the detection elements of the task based on the detection level triggering the review and the specific task type. Specifically, the construction of prompt data for the detection task requiring review may include: Based on the detection task that triggers the review and the initial detection result of the detection task, the prompt data is constructed; The prompt data is used to guide the multimodal large model to perform protective equipment wearing detection for the detection elements of interest in the detection task, and to provide the initial detection results and their confidence levels as contextual information to the multimodal large model so that the multimodal large model can refer to the initial detection results for verification or correction during review.

[0059] If the detection task triggering the review is a Level 1 human face detection task, the suggested data can guide the multimodal large model to focus on whether there is a human face target in the image, including checking whether there is human body contours, facial features, limb structures, and other human face evidence in the image. If the detection task triggering the review is a Level 2 wearing determination task, the suggested data can guide the multimodal large model to focus on whether there is a helmet in the head area of ​​the human face, including checking whether there is a helmet outline in the top of the head area, whether the helmet color is distinguishable from the background, and whether the helmet shape conforms to the characteristics of a standard safety helmet. If the detection task triggering the review is a Level 3 wearing status detection task, the suggested data can guide the multimodal large model to focus on the helmet strap area, including checking whether there is a strap passing through the chin area, and whether the strap is in a tight state (fitting the chin) or a loose state (hanging or invisible).

[0060] Optionally, embodiments of this specification may include the initial detection result and its confidence level as contextual information in the prompt data. For example, the prompt data may include text data, such as prompt words, which may contain the following contextual information: the detection category given by the initial detection model for the frame image (e.g., "suspected not wearing"), the confidence level value (e.g., 0.62), and the interval position of the confidence level (e.g., "close to the boundary of the confirmed not wearing interval"). This contextual information enables the multimodal large model to understand the judgment tendency and uncertainty of the initial detection model, thereby giving appropriate attention during review. For example, when the initial detection confidence level is 0.52 (just exceeding the lower threshold of 0.50), the prompt data may indicate "The initial detection model judged it as not wearing with a low confidence level; please focus on confirming whether the helmet is indeed not present in the head area."

[0061] In some examples, constructing the prompt data may include: Construct prompt data that includes a description of the detection target, judgment criteria, and output instructions corresponding to the detection task; The output instruction is used to guide the multimodal large model to output the judgment conclusion of the detection task and the judgment basis corresponding to the judgment conclusion.

[0062] As an example, the detection target description section can inform the multimodal large model of the target objects and detection elements to be focused on in this detection, based on the specific type of the current review task. For a human face detection task, the detection target description could be "determine whether a real human face target exists in the image, including checking for the presence of complete or partial human body contours, facial features, and limb structures." For a helmet wearing determination task, the detection target description could be "determine whether the head area of ​​the human face target in the image is wearing a helmet, including checking for the presence of a cover on the top of the head that conforms to the characteristics of a safety helmet." For a helmet wearing status detection task, the detection target description could be "determine whether the chin strap of the helmet worn by the human face target in the image is in a tight state, including checking whether the chin strap passes through and fits snugly."

[0063] The judgment criteria section can provide clear judgment basis and classification standards for multimodal large models. Taking helmet wearing status detection as an example, the judgment criteria can include the following dimensions: helmet coverage (whether the helmet covers the top of the head and forehead area), helmet position (whether the helmet is worn upright rather than tilted or leaning back), strap status (whether the strap passes under the chin and is tight, and whether it is loose or hanging), and helmet integrity (whether the helmet has obvious damage or cracks). The judgment criteria can give clear definitions for each category, such as: "Correctly worn - the helmet covers the top of the head and forehead, is in an upright position, and the strap passes under the chin and is tight and fits snugly; Straps not fastened - the helmet covers the top of the head but no strap is seen in the chin area or the strap is obviously loose and hanging; not worn - no helmet covering is detected in the head area."

[0064] The output instructions section can specify the output format requirements for the multimodal large model, instructing it to output detection results in a structured format. As an example, the output instructions can require the multimodal large model to output structured results containing the following fields: detection category (values ​​can be one of correctly_worn, not_worn, strap_unfastened, or non_portrait), confidence (value range 0 to 1), and reasoning (a description of the judgment basis in natural language). This structured output format ensures that the output of the multimodal large model can be automatically parsed and processed subsequently, avoiding the parsing difficulties and ambiguities caused by free text output.

[0065] In some examples, constructing the prompt data may further include: Based on the confidence level of the judgment results of previous levels of inspection tasks that require review, the guidance intensity of the prompt data is dynamically adjusted; wherein, when the confidence level is close to the boundary of the uncertainty interval, guidance information for key inspection areas is added to the prompt data words; Based on the scene information during image acquisition, scene-adaptive prompts are added to the prompt data; The detection box position information output by the target detection model and the close-up image of the target area obtained by cropping based on the detection box position information are added to the prompt data.

[0066] As an example, the prompt data construction in the embodiments of this specification may also include dynamic guidance intensity adjustment and scene adaptation.

[0067] Regarding dynamic guidance intensity, when the confidence level of the initial detection approaches the boundary value of the uncertainty interval (i.e., the difference between the confidence level and the upper or lower threshold is less than the preset boundary neighborhood range), the prompt data can include focused guidance statements for key detection areas. For example, when the confidence level of the third-level lace-up detection is 0.48 (close to the lower threshold of 0.45), the prompt data can include focused guidance such as "Please pay special attention to the area directly below the chin, carefully observe whether there is a band-like structure (i.e., lace-up) with a width of about 2-3 pixels, and pay attention to distinguishing the lace-up from interfering factors such as collars and shadows." This dynamic guidance intensity mechanism allows the multimodal large model to obtain more accurate detection guidance near the decision boundary where the initial detection model is most "hesitant."

[0068] Regarding scene adaptation, the embodiments of this specification can inject scene-adaptive prompts into the prompt data based on the acquired scene information. For example, in low-light scenes, the prompt data can include "The current image is dimly lit. Please pay attention to the reflective strips or fluorescent markings on the helmet as auxiliary criteria, and pay attention to distinguishing between a dark helmet and a dark background." In rainy scenes, the prompt data can include "The current environment is rainy. There may be raindrops or water mist interference in the image. Please ignore the water droplets on the lens surface and focus on the actual wearing condition of the head area of ​​the portrait." In strong backlight scenes, the prompt data can include "The current image has strong backlight. The portrait area may appear as a silhouette. Please judge the presence of the helmet based on the outline shape." These scene-adaptive prompts enable the multimodal large model to correctly understand the cause of image quality degradation and avoid misjudging environmental interference as abnormal wearing.

[0069] In a preferred embodiment, the cue data may also include data in other formats, such as providing the detection box coordinates obtained from the initial detection as part of the cue data to the multimodal large model; optionally, the cue data may also include image data, such as a cropped close-up image of an image, that is, the input of the multimodal large model may also include a cropped close-up image of an image. For example, the detection box coordinates may be in a normalized form (e.g., […]). , , , The values ​​[0.32, 0.05, 0.78, 0.65] are annotated in the cue data, enabling the multimodal large model to accurately locate the region of interest. The cropped close-up image can be a magnified local image cropped from the original image based on the detection box. It can be provided as an additional image input along with the original full image to the multimodal large model, allowing it to observe both the global context and focus on local details.

[0070] In a preferred embodiment, the detection box coordinates can be overlaid on the input image as visual annotations (e.g., drawing a rectangle on the original image), while a cropped close-up image is provided as a second input image. The multimodal large model simultaneously receives both the annotated full image and the close-up image, enabling it to understand the detected target at both global and local scales, thus improving the accuracy of the re-detection.

[0071] In some examples, the prompt data may include one or more of the following information: Role setting information is used to instruct the multimodal large model to perform detection as a protective equipment wearing detection expert; Output format constraint information is used to constrain the multimodal large model to output the verification results in a preset structured format, wherein the structured format includes at least the verification classification category and the verification confidence level; The judgment criteria description information describes the judgment criteria for each category of the current testing task that needs to be reviewed.

[0072] In this embodiment, the prompt data is enhanced individually or in combination through three methods: role setting, output format constraints, and judgment criterion description, in order to improve the standardization and reliability of multimodal large model verification and detection.

[0073] In terms of role setting, the initial part of the prompt data can assign a professional role to the multimodal large model, for example: "You are a professional safety equipment testing expert with extensive experience in industrial and traffic safety protection. Your task is to accurately determine the protective equipment worn by the target object based on the provided images." This role setting activates the multimodal large model's prior knowledge in the field of safety testing, enabling it to automatically invoke professional knowledge related to protective equipment such as helmets, seat belts, and reflective vests during the reasoning process, thereby improving the professionalism and accuracy of the testing and judgment.

[0074] Regarding output format constraints, the prompt data explicitly requires the multimodal large model to output detection results in a JSON structured format, specifically as follows: {"category": "Detection category", "confidence": confidence score, "sub_scores": {"coverage": coverage score, "position": position score, "chin_strap": chin-strap score, "integrity": integrity score}, "reasoning": "reasoning explanation text"}. The `sub_scores` field can contain sub-scores for each evaluation dimension, enabling the system to understand the specific judgments of the multimodal large model in each dimension, providing fine-grained data for subsequent quality analysis and model optimization. Output format constraints ensure consistent output structure for each review, facilitating automatic system parsing and batch processing.

[0075] Regarding the description of judgment criteria, the suggested data provides detailed descriptions of the criteria for each detection category, including positive and negative example features. For example, for the "correctly worn" category, the judgment criteria could be described as follows: "The helmet fully covers the top of the head and forehead area, the front edge of the helmet is about 2-3 cm above the eyebrows, the helmet is not significantly tilted (left or right tilt not exceeding 15 degrees), the straps pass under the chin from both sides of the ears and are fastened at the chin, and there is no obvious gap between the straps and the chin (not exceeding one finger width)." For the "straps not fastened" category, the judgment criteria could be described as follows: "The helmet covers the top of the head area, but no straps are observed under the chin, or the straps are present but are obviously loose and hanging (the gap between the straps and the chin exceeds two finger widths), or the straps are only connected on one side and the other side is detached." These detailed judgment criteria descriptions provide clear judgment references for the multimodal large model, reducing inconsistencies in judgments caused by ambiguity in the criteria.

[0076] In some examples, acquiring one or more frames of images captured in this detection includes: Acquire multiple frames of images continuously acquired by the same image acquisition device for the same target object within a set time window during the same detection event; The determination of the final protective equipment wearing test result based on the judgment results of each test task includes: The final result of the protective equipment wearing detection is determined based on the judgment results of each detection task corresponding to multiple frames of images of the same detection event.

[0077] Taking the delivery scenario as an example, the same detection event can refer to the complete image acquisition process executed by the delivery capacity client at a single trigger moment. During this process, the same mobile phone camera can continuously acquire 4 frames of images within a time window of about 1 second. Since the 4 frames of images are acquired by the same device in a very short time, their shooting angle, lighting conditions, and target posture are basically the same. The differences mainly come from the slight displacement and changes in blur caused by the shaking of the rider.

[0078] Based on multiple frames of images for the same detection event, the embodiments of this specification can perform detection tasks on each frame separately to obtain the detection results and confidence levels corresponding to each frame. Then, the final detection result is determined based on the judgment results of each detection task across multiple frames. Specifically, for four frames of images for the same detection event, each frame can undergo three levels of cascaded detection to obtain its own judgment result (including detection category and confidence level). Then, a multi-frame joint voting mechanism (detailed below) is used to synthesize the judgment results of the four frames to arrive at the final detection conclusion for the event. This multi-frame joint judgment mechanism can effectively reduce the risk of misjudgment caused by jitter, blurring, or accidental occlusion in a single frame image.

[0079] In an extended implementation, the time window of this specification embodiment can be extended from a second-level window for the same detection event to a supplementary evidence window of ±5 minutes. Specifically, when the same rider has multiple detection events within 5 minutes before and after (e.g., consecutive or repeated triggering), images from adjacent detection events can be included as supplementary evidence in the comprehensive judgment. For images spanning multiple detection events, this specification embodiment can introduce a time decay factor to reduce the weight of images with a greater time distance. The time decay factor can adopt an exponential decay model: weight(t) = exp(-0.0058 × dt), where dt is the time difference from the current detection event (in seconds), and its half-life is set to approximately 2 minutes (i.e., the weight of an image 2 minutes before the current detection event decays to 50% of its original weight, and the weight of an image 4 minutes before the current detection event decays to 25%). This design allows images that are closer in time to have a greater influence on the final judgment, while still allowing the use of recent historical images to provide supplementary evidence.

[0080] In some examples, the images acquired in this detection consist of multiple frames; the step of using the multimodal large model to detect the images based on the cue data and obtaining the judgment result of the detection task includes: The multimodal large model is used to detect multiple frames of images collected in this detection based on the prompt data, and the verification results of each frame of image in this detection task are obtained. Obtain the image quality assessment results for each frame, and then perform a weighted vote on the review results of each frame in this detection task based on the image quality assessment results for each frame. The judgment result of the testing task is determined based on the weighted voting results.

[0081] like Figure 1C The diagram shown is a flowchart illustrating another protective equipment wearing detection method based on a multimodal large model according to an exemplary embodiment of this specification. As an example, in the initial detection (three-level cascaded detection), if one or more tasks require verification, this embodiment can employ a multimodal large model (MLLM) to perform frame-by-frame verification detection on multiple frames of the detection event, and synthesize the verification results of multiple frames through a quality-weighted voting mechanism to arrive at a final judgment. For example, the multimodal large model can perform independent detection analysis on each frame of the same detection event, outputting the detection category, confidence level, and inference description corresponding to each frame.

[0082] In some examples, the multimodal large model is used to detect multiple frames of images collected in this detection based on the cue data, and the verification results of each frame of images in this detection task are obtained, including: For each frame of image, multiple independent detections are performed on the frame of image based on the cue data using multiple different multimodal large models to obtain multiple detection results for the frame of image in the detection task; Multiple detection results corresponding to the frame image are fused to determine the verification result of the frame image in this detection task.

[0083] In a preferred embodiment, for the verification detection of a single frame image, a multi-inference fusion strategy can be adopted to improve the stability of a single judgment. Specifically, for the same frame image, multiple multimodal large models are used to perform multiple independent inferences (e.g., N times, N≥2) based on the same cue data. Each inference outputs a detection category and its corresponding confidence score. Then, the multiple inference results of the frame image are fused (e.g., the mode of the category is taken as the judgment category of the frame, and the median or weighted mean of the confidence scores is taken as the fusion confidence score of the frame) to obtain a single judgment result for the frame image. This single-frame-level fusion mechanism is used to eliminate the random fluctuations of single inference by the large model and improve the stability of the single-frame judgment result. Based on this, the judgment results and fusion confidence scores of each frame image after single-frame-level fusion can be combined with the quality weights of each frame image for cross-frame weighted voting to obtain the final judgment conclusion of this detection. That is, the verification detection in the embodiments of this specification adopts a two-level fusion architecture: the first level is multi-inference fusion within a single frame to solve the problem of instability in single inference; the second level is cross-frame quality weighted voting to solve the problem of poor angle or quality of a single frame image.

[0084] For example, in this embodiment, to ensure the reliability of the review results, a multi-model redundancy mechanism is adopted. For instance, 2-3 different multimodal large models (e.g., models from different manufacturers or different versions) are simultaneously invoked to perform review detection on the same frame image in parallel, and a majority vote strategy is used to determine the final review conclusion. When multiple model results differ, a conservative strategy can be adopted (i.e., a tendency is to determine "requires manual review" rather than directly giving a conclusion). If a model call times out (e.g., no result is returned after more than 10 seconds), the frame is marked as "model timeout" and can be transferred to manual review as needed.

[0085] Before the quality-weighted voting, the embodiments in this specification may first perform a quality assessment on each frame of the image to determine the quality weight of each frame. The quality assessment dimensions may include: sharpness (measured using the variance of the Laplacian operator, the larger the variance, the sharper the image), illumination quality (based on brightness histogram analysis to assess whether there is overexposure or underexposure), shooting angle (based on face pose estimation, a frontal angle scores the highest, and a larger deflection angle scores the lowest), and occlusion degree (assessing whether there are obstructions in the detection area, with no occlusion scoring the highest).

[0086] Optionally, the quality scores for each dimension can be normalized to the [0,1] interval using the following formula: Sharpness score = sqrt(min(var / 2000, 1.0)); Where var is the variance of the Laplace operator; Illumination score = exp(-((brightness-128) / 80)^2), where brightness is the average brightness of the image; Angle fraction = max(0, 1-|yaw| / 60), where yaw is the horizontal yaw angle of the face (in degrees); Occlusion score = 1 - Occlusion ratio, where the occlusion ratio is the percentage of the detected area that is occluded.

[0087] After normalization, the overall quality weights can be calculated using the following weighting formula: = 0.4× + 0.3× + 0.2× + 0.1× . Wherein the definition weight has the highest value (0.4), because a blurred image has the most significant impact on detection accuracy; the occlusion degree weight has the lowest value (0.1), because partial occlusion usually still allows a correct judgment to be made through the visible area. The above weights are only examples, and specific weight values may be set as required in practical applications, which is not limited in this embodiment.

[0088] In the voting mechanism, the embodiments of the present specification may assign different voting scores to different detection categories: correctly_worn is assigned +1, not_worn is assigned -1, strap_unfastened is assigned -0.5, and non_portrait is assigned 0. It should be noted that although the non-portrait category is assigned a value of 0, it is still counted in the voting denominator (i.e., participating in the normalization calculation), and the effect thereof is to dilute the concentration of valid votes. For example, among 4 frames, if 1 frame is determined as non-portrait and 3 frames are determined as correctly worn, the comprehensive score is (1+1+1+0) / 4=0.75 instead of (1+1+1) / 3=1.0, which embodies the design intention of reducing the confidence level for a detection event with non-portrait frames. For each frame of image, the voting score corresponding to its detection category is multiplied by the quality weight of the frame to obtain a weighted voting value of the frame. The weighted voting values of all frames are summed and normalized to obtain a comprehensive voting score S.

[0089] A final result is determined according to a comparison between the comprehensive voting score and a preset threshold; for example, when the comprehensive score S>0.6, it is determined as pass (correctly worn); when the comprehensive score S<-0.6, it is determined as fail (not worn), and a corresponding processing procedure may be triggered, which is not limited in this embodiment. When the comprehensive score -0.6<S<-0.2, it is determined as a strap reminder, for example, processing such as pushing a reminder to fasten the strap to a rider may be performed; when the comprehensive score -0.2<S<0.6, it is determined that manual review is required.

[0090] Optionally, embodiments of this specification may introduce a consistency check as a supplementary quality control mechanism to multi-frame voting. For example, taking Fleiss Kappa as an example, the FleissKappa coefficient between the detection results of multiple frames of the same detection event can be calculated (where the number of evaluators k = the number of images collected, for example, 4 frames; the number of categories = 4, i.e., correctly_worn, not_worn, strap_unfastened, and non_portrait). If the Kappa coefficient is greater than 0.6, it indicates that there is a high degree of consistency between the multi-frame results, and the voting results can be directly accepted; if the Kappa coefficient is less than 0.5 (indicating poor consistency between the multi-frame results), an additional review process is triggered, such as increasing the number of review frames, increasing the guidance strength of the multimodal large model, or directly transferring it to manual review. This consistency check mechanism can identify abnormal situations where the multi-frame results contradict each other (e.g., 2 frames out of 4 frames determine correct wearing, and 2 frames determine not wearing), avoiding forcibly outputting a potentially incorrect final judgment when the results are inconsistent.

[0091] In some examples, the method further includes: Based on the judgment results and judgment criteria obtained by the multimodal large model for the detection tasks that need to be reviewed, an interpretable report on the detection of protective equipment wearing is generated. The interpretable report marks the key areas in the image that correspond to the judgment criteria and includes anomaly descriptions for the anomaly detection task.

[0092] The embodiments in this specification can generate an interpretable test report based on the review test results and their reasoning explanations after the multimodal large model has completed the review test. This interpretable report not only includes the final test conclusion, but also the complete test reasoning process and visual evidence annotations, enabling reviewers to quickly understand the test basis and conduct manual review when necessary. It can also be output to the users who were tested for review as needed.

[0093] For example, the process of generating an interpretable report may include the following steps: First, extract the detection category, confidence score, sub-scores of each dimension, and inference description text from the review output of the multimodal large model; second, annotate key detection regions on the original image, including drawing bounding boxes on the image using the detection box coordinates obtained from the initial detection, labeling the detection category and confidence score values, and performing local magnification annotation on key parts such as the frenulum region; third, associate and organize the inference description text output by the multimodal large model with the annotated image to form a detection report with both text and graphics.

[0094] An interpretable report can include the following structure: frame-by-frame detection results (detection category, confidence level, and inference explanation for each frame), multi-frame consistency index (Fleiss-Kappa coefficient and consistency / disagreement status of results across frames), overall voting score and final judgment, and anomaly explanation (if anomalies exist, such as inconsistent results across frames or low confidence levels, provide an analysis of the causes and suggested solutions). The anomaly explanation section is crucial for quality control. For example, if 3 out of 4 frames indicate correct wearing and 1 frame indicates the strap is not fastened, the report could state, "The 3rd frame image is slightly blurred due to riding vibrations, which may affect the accuracy of the strap area judgment. It is recommended to use the results of the other 3 frames as the standard."

[0095] In a preferred embodiment, the interpretable report may also include an overall score and a confidence trend graph. The overall score is a composite score of weighted votes across multiple frames (e.g., +0.82 indicates high confidence in correct wearing), and the confidence trend graph displays the distribution of confidence levels for each of the four frames, allowing reviewers to intuitively understand the stability of the test results. This interpretable report can be stored in the testing records of the delivery management platform as electronic evidence for safety management, and can also serve as a basis for appeal review when the tested user raises objections to the test results.

[0096] In practical applications, protective equipment wearing detection data may suffer from class imbalance. For example, in delivery scenarios, the vast majority of riders wear helmets correctly (e.g., the "wearing" category accounts for over 85% of the samples), while "not wearing" and "not fastened" are minority categories, and the proportion of the "non-human" category depends on the installation location of the image acquisition device. Traditional cross-entropy loss functions, in class imbalance scenarios, cause the model's training gradient to be dominated by majority class samples, resulting in insufficient recall for the minority class. In safety detection scenarios, the risk of missing a helmet wearing is far higher than the risk of misreporting correctly worn helmets as not worn; therefore, insufficient recall for the minority class may be unacceptable. Based on this, this embodiment pre-trains the object detection model and designs a cascaded perception loss function to address the aforementioned issues.

[0097] In some examples, the total loss of the object detection model during training can be determined in the following ways: Based on the task weights corresponding to each detection task, the cascaded perception losses of each detection task are weighted and summed to obtain the total loss; wherein: The task weight of each detection task is configured according to the degree of influence of the level to which the detection task belongs on the final protective equipment wearing detection result; The cascaded sensing loss includes at least a focus loss term.

[0098] The target detection model in the embodiments of this specification may employ the Cascade-Aware Focal Loss (CAF). The model is trained using a structured training loss function designed to address the specific characteristics and interrelationships of each task in the three-level cascaded detection architecture. This allows the model to fully perceive the impact of the cascaded structure on the final detection results during training.

[0099] The total loss function in the embodiments of this specification can be defined as the weighted sum of the loss terms of each stage of the connected sensing process: Where k is the cascading level index, k=1 corresponds to the human face detection task, k=2 corresponds to the wearing determination task, and k=3 corresponds to the wearing status detection task.

[0100] The weight coefficient for the k-th level task is... This represents the cascaded perception loss term for the k-th level task. Task weights. The configuration can be based on the degree of impact of this level of task on the final detection results.

[0101] As an example, the weight of the second-level wearing determination task can be set to the highest ( = 1.0), because this level directly determines the core safety judgment of "wearing / not wearing", and a misjudgment will directly lead to an incorrect safety score; the third level of wearing status detection task has the next highest weight ( = 0.7), because the condition of the strap is a refined judgment, and its error impact is lower than that of completely missing the detection of not wearing the strap; the first level of facial recognition task has the lowest weight ( = 0.4), because this level mainly plays a filtering role, excluding non-human images from subsequent detection. Its misjudgment (misjudging non-human images as human images) only leads to invalid calculations in subsequent levels and will not directly produce incorrect security judgments.

[0102] As an example, each cascaded sensing loss term It can include at least a focal loss term, which can be implemented using the standard focal loss in existing technologies, and can be used to solve the common problems of imbalance between positive and negative samples and imbalance between easy and difficult samples in object detection.

[0103] In some examples, embodiments of this specification may also modify the standard focus loss to suit the specific needs of a cascaded detection architecture. For instance, the focus loss term includes focus parameters, with different focus parameter values ​​corresponding to different detection tasks, and the focus parameter values ​​increasing progressively with the level.

[0104] As an example, the focal parameter γ in the cascaded sensing loss function can be configured using a progressively increasing strategy, i.e. < < In practical applications, the specific value of the focus parameter γ can be set as needed, and this embodiment does not limit it.

[0105] As a concrete example, = 1.5, = 2.0, = 2.5. The focus parameter γ controls the degree of attention the loss function pays to both easy and difficult samples: the larger γ is, the more the model focuses on "difficult samples" (i.e., the predicted probability). The higher the attention given to samples close to 0.5, the more attention is paid to "easy samples" (i.e., samples with a value close to 0.5). The closer the sample is to 1, the stronger the suppression of the loss contribution.

[0106] In some examples, the focus loss term includes a class balance factor, which is used to weight the focus loss term based on the proportion of samples of each class in the training dataset, so that the weight of each class sample is negatively correlated with its quantity.

[0107] In the embodiments of this specification, the class balance factor α can be negatively correlated with the number of samples in each category. Specifically, for categories with a small number of samples (for example, the "lacing not fastened" category usually accounts for a much smaller proportion in the training data than the "correctly worn" category), a higher class balance factor α can be assigned to compensate for its insufficient presence in the training data.

[0108] In one implementation, the category balance factor can be expressed as follows: in, For category In the The level balance weight, also known as the category balance factor; N total The total number of samples in the training set. Let be the number of samples in class j of task k. For the first The number of categories at each level.

[0109] This inverse frequency weighting strategy ensures that minority class samples (such as "not wearing" or "not fastened") receive higher weights in the loss calculation, preventing the model from neglecting the learning of the minority class due to the gradient dominance of the majority class.

[0110] In some examples, the cascaded sensing loss also includes a confidence calibration regularization term, which is used to align the distribution of confidence output by the target detection model with the uncertainty interval corresponding to the detection task.

[0111] As an example, each cascaded sensing loss term can be represented as: in, For the sample The actual label (one-hot encoded) takes a value of 0 or 1; For the model to sample Category The predicted probability, Sigmoid / Softmax output, ; The regularization coefficient is used to calibrate the confidence level; it can be an empirical value, such as β = 0.1. The regularization term is used to calibrate the confidence level.

[0112] The embodiments in this specification can introduce a confidence calibration regularization term into the cascaded sensing loss. This is used to align the confidence distribution of the model output with the threshold design of the uncertainty interval. For example, During training, the model is constrained to ensure that the confidence distribution for correct predictions is as far away from the uncertainty interval as possible (i.e., the confidence of correct predictions should be as high as possible above the upper threshold or as low as possible below the lower threshold), while the confidence distribution for incorrect predictions should fall within the uncertainty interval as much as possible. This regularization term ensures that the confidence of the model output after training has a good match with the threshold interval design, enabling the early termination mechanism and the review triggering mechanism to operate at the expected accuracy.

[0113] In some examples, the cascaded sensing loss further includes a confidence calibration regularization term, which includes: The first penalty term is used to penalize erroneous predictions with a confidence level higher than the upper boundary of the uncertainty interval; The second penalty term is used to penalize correct predictions with a confidence level below the lower boundary of the uncertainty interval.

[0114] As an example, traditional focus loss only focuses on whether a pair is correctly classified, but ignores whether the confidence level is accurate. This embodiment introduces a confidence level calibration regularization term to ensure that the probability values ​​output by the model truly reflect the reliability of its judgment. Optionally, the mathematical expression of the confidence level calibration regularization term can be as follows: in: Model on samples Predict the confidence probability of the category; : No. The upper bound threshold of the level (the lower boundary of the deterministic interval); : No. The lower bound threshold of the level (upper boundary of the deterministic interval / lower boundary of the deterministic interval); Indicator function, retrieves the value when the condition is true. Otherwise take .

[0115] In the formula, the term before the plus sign is the first penalty term. and The first penalty term can be used to penalize high-confidence erroneous predictions. That is, when the model makes an incorrect class determination for a sample, but its output confidence is very high (exceeding the upper limit threshold), the first penalty term applies a larger loss penalty.

[0116] The design principle behind this penalty term is that high-confidence erroneous predictions are a dangerous type of misjudgment because such errors will trigger an early termination mechanism that directly outputs the erroneous result, bypassing subsequent levels of detection and review processes. By applying the first penalty term, the training process "teachs" the model to remain "humble" (i.e., output lower confidence) when making mistakes, making erroneous predictions more likely to fall into the uncertainty range and be captured by the review mechanism.

[0117] The term after the plus sign in the formula is the second penalty term. and The second penalty term can be used to penalize correct predictions with low confidence. That is, when the model makes the correct class determination for a sample, but its output confidence is very low (below the lower limit threshold or within the uncertainty range), the second penalty term applies a moderate loss penalty.

[0118] The design principle behind this penalty term is that while correct predictions with low confidence do not lead to incorrect outputs, they trigger unnecessary verification processes, increasing the cost of calling large multimodal models. By applying a second penalty term, the training process "encourages" the model to output sufficiently high confidence when making correct predictions, reducing unnecessary verification triggers and lowering system operating costs.

[0119] The first and second penalty terms can be correlated and work together to achieve the following technical effects: the Focal Loss term mainly addresses the "classification accuracy" problem (i.e., enabling the model to correctly distinguish between categories), while the confidence calibration regularization term... The main problem addressed is "confidence calibration" (i.e., ensuring the model is confident when correct and cautious when incorrect). The two work together to give the trained model not only high classification accuracy but also good confidence calibration characteristics, thus ensuring the reliable operation of the early termination mechanism and the verification triggering mechanism in the three-level cascaded detection architecture.

[0120] In a preferred embodiment, the weight coefficient of the first penalty term can be set to 2 to 3 times that of the second penalty term, reflecting the design priority of "avoiding high-confidence errors" over "reducing low-confidence correctness".

[0121] In some cases, the task weight for detecting whether protective equipment is being worn is greater than the task weight for detecting the wearing status of protective equipment; and the task weight for detecting the wearing status of protective equipment is greater than the task weight for detecting whether an image contains a human figure.

[0122] In the embodiments of this specification, the weight relationship of the three-level cascaded detection tasks can satisfy: λ_wearing determination > λ_wearing state > λ_face detection, that is, the training weight of the second-level wearing determination task is the highest, the third-level wearing state detection task is the second highest, and the first-level face detection task is the lowest.

[0123] In one specific implementation, λ_wearing determination = 1.0, λ_wearing status = 0.7, and λ_face detection = 0.4. The reason for setting the weight of the second-level wearing determination task to the highest is that it directly determines the crucial security judgment of "wearing / not wearing," and its misjudgment will directly lead to errors in the security score, significantly impacting the detected person. Therefore, it needs to be given the highest loss weight during training to ensure the model prioritizes the accuracy of this level of judgment. The weight of the third-level wearing status detection task is next (0.7) because wearing status is a refined judgment, and its error impact is lower than completely missing the detection of not wearing. The weight of the first-level face detection task is the lowest (0.4) because this task is the most basic filtering task, with the lowest detection difficulty, and its misjudgment can be corrected by subsequent cascading corrections.

[0124] It should be noted that in practical applications, the task weights for each detection task can be customized as needed. For example, in another implementation, the weight of the third-level wearing status detection task can be set to the highest (e.g., λ_wearing status = 1.0, λ_wearing determination = 0.7, λ_face detection = 0.4). The principle is that the third-level lacing detection is the most refined and difficult task in the three-level cascade, and its detection target (lacing) occupies a very small proportion in the image (usually only 0.1%-0.5% of the image area). The model needs higher loss weights to fully learn the decision boundary of this level. Both weight configuration schemes are within the protection scope of the embodiments in this specification.

[0125] In some examples, the class balance factor is dynamically adjusted during training as the training process progresses in the following manner: In the early stages of training, by controlling the class balance factor, the weight of samples from the class with fewer samples is increased to that of samples from the class with more samples. As the number of training rounds increases, the weight differences between samples of each category in the category balancing factor gradually decrease, so that the weights obtained by samples of each category gradually approach a uniform distribution.

[0126] In the embodiments described in this specification, the class balance factor α is not fixed throughout the entire training process, but can be dynamically adjusted with each training round. Specifically, in the early stages of training, the minority class (e.g., "belt not fastened" or "not wearing") can be assigned a higher weight to accelerate the model's learning of the minority class decision boundary; as the training rounds increase, the weight difference between the minority and majority classes can gradually decrease; in the later stages of training, the weights of each class can approach a uniform distribution.

[0127] The mathematical expression for this dynamic adjustment strategy can be: in Let be the initial class balance factor for class j in the k-th level task (calculated by inverse frequency), t be the current training round, T be the total training rounds, and δ be the decay control parameter (preferred value is 0.3). When t=0 (training begins), = That is, the complete inverse frequency weights; when t=T (training ends), = • (1-δ) = 0.7 • The weight difference was reduced to 70% of the initial value.

[0128] The motivation behind the dynamic adjustment strategy described above is as follows: In the early stages of training, the model has not yet established decision boundaries for each class. At this time, the minority class requires a larger weight difference to provide sufficient gradient signals, enabling the model to quickly learn the basic discriminative features of the minority class (such as the basic visual pattern of "tassel not fastened"). If the initial weight difference is insufficient, the gradient signals of the minority class will be overwhelmed by the majority class, causing the model to form a decision bias towards the majority class in the early stages of training, which is difficult to correct later. However, in the later stages of training, the model has basically established decision boundaries for each class. If a large weight difference is still maintained at this point, the model will over-focus on noisy samples in the minority class (such as mislabeled samples), leading to overfitting. Therefore, the weight difference is gradually reduced in the later stages, allowing the model to achieve a more balanced optimization among the classes and improve the overall generalization ability.

[0129] Furthermore, the embodiments in this specification can employ temperature scaling technology to calibrate the original confidence level output by the object detection model. For example, after model training is complete, a scalar temperature parameter T can be learned using a validation set. The original logits are divided by T, and then the result is processed through a softmax function to obtain the calibrated confidence level. In one specific implementation, the validation set can be 10% of the total labeled data (approximately 50,000 images), and the search range of the temperature parameter T can be set to [0.5, 5.0]. The optimal T value is determined by minimizing the negative log-likelihood loss (NLL) on the validation set, typically converging in the interval [1.2, 2.5]. Temperature scaling allows the confidence level output by the model to more accurately reflect the probability of a correct prediction. For example, a calibrated confidence level of 0.85 means that the prediction has a statistically significant probability of being correct (approximately 85%), providing an interpretable basis for threshold setting.

[0130] The cascaded sensing loss function in the embodiments of this specification differs from the loss function schemes in the prior art in the following ways: Compared to standard cross-entropy loss, which applies equal weight to all samples and biases the model towards the majority class in imbalanced scenarios, the embodiments in this specification improve the recall rate of the minority class by jointly adjusting the focus parameter γ and the class balance factor α, thereby giving higher attention weight to the minority class (e.g., "not wearing" or "not fastened") during training.

[0131] Compared to standard focal loss, which focuses solely on classification accuracy and lacks the ability to calibrate the confidence level of the model output, the embodiments in this specification add a confidence level calibration regularization term to the focal loss, aligning the confidence level of the model output with the threshold design in the cascaded detection architecture, thus achieving consistency between the training and inference phases.

[0132] Compared to cascade-insensitive schemes, if the same loss function and weights are used for all three cascade levels, the importance of the wearing decision level cannot be reflected during training. The embodiments in this specification, through differentiated configuration of task weights, allocate model training resources towards key decision stages, ensuring that the level with the greatest impact on the final detection result receives sufficient training signals.

[0133] like Figure 1D The diagram shown is an application scenario diagram of protective equipment wearing detection according to an exemplary embodiment of this specification. Figure 1D It includes delivery capacity (riders and their electric bikes, with mobile phones fixed to the handlebar brackets), delivery capacity client, detection server (with target detection models and multimodal large models deployed internally), and delivery management platform.

[0134] In this scenario, the delivery personnel are riding their vehicles and their mobile phones are fixed to the handlebar bracket of the electric vehicle. The delivery personnel's client can trigger a protective equipment wearing detection event according to fixed rules or randomly.

[0135] Step 1: Multi-frame image acquisition. After the delivery capacity client triggers the detection event, the phone's rear camera can continuously acquire multiple frames of images within a set time window. Due to slight vibrations during riding, each frame may have minor differences in angle and clarity.

[0136] Step Two: Three-Level Cascaded Initial Detection. Multiple image frames are input into the target detection model in the detection server to perform three-level cascaded detection. Taking one frame as an example: the confidence score of the first-level portrait detection output is higher than its upper threshold, confirming the presence of a portrait and outputting a detection box; the confidence score of the second-level wearing determination output is within its uncertainty interval, and is marked as needing verification; the third-level wearing status detection outputs a certain wearing status category and its confidence score is within its uncertainty interval, also marked as needing verification. In the initial detection results of each frame, the confidence scores of the second and third levels can both fall within their respective uncertainty intervals, thus triggering the multimodal large model verification process.

[0137] Step 3: Quality Assessment and Weight Calculation. The detection server performs quality assessment on each frame of the image, calculating scores for dimensions such as sharpness, occlusion level, shooting angle, and lighting conditions. It then calculates the quality weight of each frame according to preset weight coefficients for each dimension, and normalizes the quality weights of each frame.

[0138] Step 4: Data Construction and Multimodal Model Verification. Since the verification tasks include Level 2 helmet wearing determination and Level 3 helmet wearing status detection, the detection server constructs targeted prompt data. The prompt data may include: role settings; detection targets (e.g., determining whether the rider is wearing a helmet and focusing on checking if the strap is tight); initial detection context (including the confidence level of the initial detection and key areas requiring confirmation); scene adaptation prompts (e.g., current lighting conditions and riding vibration); and output format constraints. Subsequently, multiple frames of images (with bounding box annotations and close-up cropped sections of key areas) and the prompt data are input into the multimodal model on the detection server.

[0139] Step 5: Frame-by-frame verification of the multimodal large model. The multimodal large model outputs structured verification results for each frame of image, including the detection category, confidence score, sub-scores for each dimension (e.g., coverage, location, lacing status, integrity, etc.), and inference description text. The inference description can describe the key visual evidence observed by the multimodal large model, such as the helmet's coverage of the top of the head and whether lacing structures are observed in the jaw area.

[0140] Step Six: Quality-Weighted Voting. When the review results of all frames point to the same detection category, a quality-weighted vote is performed based on the voting scores corresponding to each frame to calculate the comprehensive score. When the comprehensive score falls within the judgment interval corresponding to the detection category, the corresponding judgment conclusion is determined according to the threshold rule. At the same time, the detection server calculates the consistency index of the results of multiple frames. When the consistency requirements are met and the confidence of each frame meets the preset conditions, the final judgment result is confirmed according to the consistency supplementary rules.

[0141] Step 7: Generate and output an interpretable report. The detection server generates an interpretable detection report containing multi-frame image annotations, frame-by-frame detection results, inference explanations, consistency indicators, and a final judgment, and uploads it to the delivery management platform. Delivery riders receive safety alerts pushed by the delivery capacity client. The delivery management platform can use this detection record as electronic evidence for safety management, or provide complete detection evidence when riders file appeals; samples from successful appeals can be fed back to the detection server for incremental training of the model.

[0142] like Figure 2 As shown in the embodiments of this specification, a method for training an object detection model is also provided, which may include the following steps: Step S210: Acquire the image and input it into the object detection model.

[0143] The target detection model is used to perform the following cascaded detection tasks: detect whether the image contains a human image; if a human image is clearly or possibly detected, detect whether protective equipment is being worn; if protective equipment is clearly or possibly detected, detect the wearing status of the protective equipment. Step S220: After determining the total loss based on the recognition result of the image by the target detection model, the target detection model is trained based on the total loss.

[0144] The total loss of the target detection model during training is determined in the following way: Based on the task weights corresponding to each detection task, the cascaded perception losses of each detection task are weighted and summed to obtain the total loss; wherein: The task weight of each detection task is configured according to the degree of influence of the level to which the detection task belongs on the final protective equipment wearing detection result; The cascaded sensing loss includes at least a focus loss term.

[0145] In this embodiment, the training sample set can contain a large number of labeled images. Each labeled image contains a real human image detection bounding box, a wearing status label, and a wearing status category label. In the delivery scenario, the training samples can be derived from historical images collected by delivery riders, which are manually labeled to form the training set. The labeling content can include: human image bounding box coordinates, a binary label for wearing / not wearing, and a multi-class label for wearing status (correctly worn, strap not fastened, etc.). The training sample set can be organized in a cascaded manner. The first-level training samples contain human / non-human image labels, the second-level training samples contain wearing / not wearing labels (only for images containing human images), and the third-level training samples contain detailed labeling of wearing status (only for images where the image is worn).

[0146] The training samples are input into the object detection model, and a three-stage cascaded forward propagation is performed. For each training image, the model can sequentially perform the first stage of portrait detection (outputting portrait confidence and detection box), the second stage of wearing determination (outputting wearing confidence), and the third stage of wearing state detection (outputting state category and confidence). In training mode, all three stages are executed completely (without triggering early termination) to obtain complete prediction outputs for each stage for loss calculation.

[0147] Based on the predicted outputs at each level and the actual labels, calculate the total loss of the cascaded sensing: Each cascaded sensing loss term It can include a focus loss term (with progressively increasing focus parameters). ), category balance factor (It has the characteristic of dynamic decay with training rounds) and confidence calibration regularization term. (Including the first and second penalty items).

[0148] Based on total loss Perform backpropagation to update the parameters of the object detection model. Stochastic gradient descent (SGD) or the Adam optimizer can be used to update the model weights according to the calculated gradient. During training, the learning rate can employ a cosine annealing strategy, gradually decaying from an initial learning rate (e.g., 0.01) to a final learning rate (e.g., 0.0001). See the description in the foregoing embodiments for further details.

[0149] In the embodiments of this specification, the cascaded sensing loss function described above is not simply an application of the standard focal loss to each task level, but rather a modification tailored to the cascaded detection architecture. Specifically, while the standard focal loss uses globally uniform γ and α parameters, the cascaded sensing loss in the embodiments of this specification has the following differences: (1) The focal parameter γ increases progressively according to the cascade level. < < This reflects the progressive relationship between the difficulty levels of tasks at each level; (2) The class balance factor α decreases dynamically with each training round to avoid overfitting minority class noise in the later stages; (3) Introduce confidence calibration regularization term The model's confidence output is aligned end-to-end with the threshold design of the cascaded architecture. These structural modifications enable the trained model to not only perform well in classification accuracy but also match the early termination and verification triggering mechanisms of the cascaded detection architecture in terms of confidence calibration characteristics.

[0150] As an example, the object detection model in the embodiments of this specification can adopt a single-model architecture with a shared backbone network and multi-level detection heads. For example, the object detection model may include a feature extraction backbone network and three detection heads corresponding to three levels of detection tasks. The feature extraction backbone network can be used to perform multi-level feature extraction on the input image and output feature maps at different scales.

[0151] In one example, the backbone network can employ a lightweight convolutional neural network to strike a balance between feature representation capability and inference speed. The backbone network can contain multiple downsampling stages, each outputting a feature map at a corresponding resolution for use by different levels of detection heads.

[0152] In one example, the three detection heads correspond to a three-level cascaded detection task: the first-level detection head is the portrait detection head, which outputs the portrait confidence score and the coordinates of the portrait bounding box; the second-level detection head is the wearing determination head, which outputs the wearing confidence score; and the third-level detection head is the wearing state classification head, which outputs the wearing state category and the corresponding confidence score. The three detection heads share the feature extraction parameters of the backbone network, but each maintains independent classification and regression parameters. Each detection head can contain several convolutional and fully connected layers to map the feature map output by the backbone network to the corresponding detection result. During cascaded inference, the detection box output by the first-level detection head is used to crop the target region containing the portrait from the original image. This cropped region, after scaling, serves as the input to the second-level detection head; the detection box output by the second-level detection head is further cropped to extract the head region, which then serves as the input to the third-level detection head.

[0153] In training mode, all three levels of detection heads can perform forward propagation in its entirety (without triggering early termination). The losses of each level of detection head are weighted and summed using the aforementioned cascaded sensing loss function before being uniformly propagated backward, while simultaneously updating the parameters of the backbone network and each detection head.

[0154] In this embodiment, the training sample set can contain a large number of labeled images. In the delivery scenario, the training samples can be derived from historical images collected by delivery capacity, which are then manually labeled to form the training set. The labeled content can include: human image bounding box coordinates (for training the first-level human image detection task), wearing / not wearing binary classification labels (for training the second-level wearing determination task), and wearing status multi-classification labels (for training the third-level wearing status detection task). In the helmet delivery scenario, the wearing status multi-classification labels include four categories: correctly worn (helmet is on top of the head and the strap is fastened), not worn (helmet not detected on the head), strap not fastened (helmet is on top of the head but the strap is not fastened or is not fastened), and non-human image (the image does not contain a human image target).

[0155] The training samples are categorized into three levels: Level 1 uses a binary classification label for human / non-human images; Level 2 uses a binary classification label for worn / not worn images, applicable only to images containing human figures; and Level 3 uses the aforementioned four-category label, applicable only to images where the device is worn. The training sample set is organized in a cascading manner, with each level of training sample being filtered and correlated based on the labeling results of the previous level.

[0156] Optionally, the object detection model can be initialized based on model weights pre-trained on a general object detection dataset (such as the COCO dataset or ImageNet dataset), and then fine-tuned using the training sample set; alternatively, it can start with randomly initialized weights and directly perform full training using the training sample set. When using pre-trained weight initialization, the convolutional layer parameters of the backbone network can be loaded with pre-trained weights to accelerate convergence, and the classification and regression parameters of the three detection heads can be randomly initialized.

[0157] The training samples are input into the object detection model, and a three-stage cascaded forward propagation is performed. For each training image, the model sequentially performs the first stage of portrait detection (outputting portrait confidence and detection box), the second stage of wearing determination (outputting wearing confidence), and the third stage of wearing state detection (outputting state category and confidence). In training mode, all three stages are executed completely to obtain complete prediction outputs for each stage for loss calculation.

[0158] The embodiments in this specification may also include a feedback loop mechanism for continuous optimization of the detection system's performance. For example, when a person being tested appeals a detection result and the appeal is successful, the sample can be included in the training set as incremental training data. Model updates can employ a tiered strategy: incremental fine-tuning can be performed monthly (using recently accumulated successful appeal samples and newly labeled data for incremental fine-tuning), while full retraining can be performed quarterly (using the complete training set for retraining from scratch or retraining from pre-trained weights). The adjustment step size of the dynamic threshold can be set to ±0.02 to 0.05 (i.e., each adjustment does not exceed 0.05) to avoid drastic threshold fluctuations that could lead to abrupt changes in system behavior. A cooling-off period (e.g., 7 days) is set after threshold adjustment, during which no further threshold adjustments are made to observe the adjustment effect and accumulate sufficient statistical data.

[0159] The protective equipment wearing detection method based on a multimodal large model provided in the embodiments of this specification has the following significant technical advantages compared with the prior art: Regarding detection accuracy, when using only the target detection model for initial detection, the accuracy rate for helmet wearing status detection in delivery scenarios is approximately 85%-90% (affected by factors such as riding vibration, shooting at an upward angle, and changes in lighting). After the multimodal large model verification mechanism in the embodiments of this specification, the overall detection accuracy can be improved to 95%-98%. Specifically, the false judgment rate of the initial detection (including misjudging correct wearing as not wearing, misjudging unfastened straps as correct wearing, etc.) can be reduced from 15%-25% to below 5%. The multimodal large model verification performs accurate secondary judgment on samples falling into the uncertainty range in the initial detection, effectively correcting the false judgments of the target detection model on boundary samples.

[0160] Regarding cost control, the embodiments in this specification, through a three-level cascaded early termination mechanism and dynamic adjustment of the uncertainty interval, can keep the proportion of samples requiring multimodal large-scale model review below 10% of the total detection volume (i.e., the manual / large-scale model intervention rate is less than 10%). Most well-defined samples (confidence levels falling within the certainty interval) are determined during the initial detection stage, without the need to call the multimodal large-scale model. For samples requiring review, the cost of a single multimodal large-scale model call is approximately 0.1-0.3 yuan (based on the current pricing of the multimodal large-scale model API), far lower than the cost of manual review (5-10 yuan per call). Based on an average of 1 million detection events per day, the method in the embodiments of this specification can save more than 99% of the review cost compared to full manual review.

[0161] In terms of real-time performance, the initial detection time for the three-level cascaded detection is approximately 50-100 milliseconds (per frame), and the total initial detection time for 4 frames is approximately 200-400 milliseconds, meeting the real-time detection requirements in delivery scenarios. The multimodal large model verification time is approximately 2-5 seconds (4 frames), and it is only executed when uncertain samples are triggered, without affecting the detection timeliness of most samples.

[0162] Regarding interpretability, the embodiments in this specification utilize reasoning explanations and interpretable reports generated by a multimodal large model to provide detection results with a complete reasoning chain and visual evidence support, thus solving the problem of the difficulty in interpreting the "black box" judgments of traditional target detection models. This interpretability is of high value for appeal handling, security auditing, and model optimization in security management.

[0163] The embodiments of the protective equipment wearing detection device / target detection model training device described in this specification can be applied to computer devices, such as servers or terminal devices. The device embodiments can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by its processor reading the corresponding computer program instructions from non-volatile memory into memory and executing them. From a hardware perspective, such as... Figure 3 The diagram shown is a hardware structure diagram of a computer device housing the training device for the protective equipment wearing detection device / target detection model according to an embodiment of this specification. Except for... Figure 3 In addition to the processor 310, memory 330, network interface 320, and non-volatile memory 340 shown, the electronic device in which the protective equipment wearing detection device 331 is located in the embodiment may also include other hardware depending on the actual function of the electronic device, which will not be described in detail here.

[0164] like Figure 4 As shown, Figure 4This is a block diagram illustrating a protective equipment wearing detection device according to an exemplary embodiment of this specification. The device includes: an image acquisition module 41, used to acquire one or more frames of images collected for the current detection; an initial detection module 42, used to perform an initial detection on the image using a pre-trained lightweight target detection model; the initial detection includes the following cascaded detection tasks: detecting whether the image contains a human image; if a human image is clearly or possibly detected, detecting whether protective equipment is being worn; if protective equipment is clearly or possibly detected, detecting the wearing status of the protective equipment; and a verification and judgment module 4. 3, used for: determining whether a detection task needs to be reviewed based on whether the confidence level of the initial detection result falls within the uncertainty range corresponding to the detection task; if the detection task does not need to be reviewed, the initial detection result is used as the judgment result of the detection task; if the detection task needs to be reviewed, prompt data for guiding the multimodal large model to perform protective equipment wearing detection related to the detection task is constructed, and the multimodal large model is used to detect the image based on the prompt data to obtain the judgment result of the detection task; the result determination module 44 is used for: determining the final protective equipment wearing detection result based on the judgment result of each detection task.

[0165] like Figure 5 As shown, Figure 5 This is a block diagram of a training device for an object detection model according to an exemplary embodiment of this specification. The device includes: a data acquisition module 51, configured to: acquire an image and input it into the object detection model; the object detection model is configured to perform the following cascaded detection tasks: detecting whether the image contains a human image; if a human image is clearly or possibly detected, detecting whether protective equipment is being worn; if protective equipment is clearly or possibly detected, detecting the wearing status of the protective equipment; a loss determination module 52, configured to: determine the total loss based on the recognition result of the image by the object detection model; wherein the total loss is determined by: weighted summing of the cascaded perceptual losses of each detection task based on the task weights corresponding to each detection task; the task weight of each detection task is configured according to the degree of influence of the level to which the detection task belongs on the final detection result of protective equipment wearing; the cascaded perceptual loss includes at least a focus loss term; and a training module 53, configured to: train the object detection model based on the total loss.

[0166] This specification also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the aforementioned method embodiments. This specification also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the aforementioned method embodiments. The above descriptions are merely preferred embodiments of this specification and are not intended to limit this specification. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this specification should be included within the scope of protection of this specification.

Claims

1. A method for detecting the wearing of protective equipment based on a multimodal large model, comprising: Acquire one or more frames of images captured in this detection; The image is initially detected using a pre-trained lightweight object detection model; the initial detection includes the following cascaded detection tasks: Detect whether the image contains a human figure. If a person's image is clearly detected or suspected to be present, the system will check whether protective equipment is being worn. If it is detected that someone is clearly wearing protective equipment or is suspected of wearing protective equipment, check the wearing status of the protective equipment. Whether a detection task needs to be reviewed depends on whether the confidence level of the initial detection result falls within the uncertainty interval corresponding to that detection task; wherein, the uncertainty interval is used to characterize the range of values ​​in which the initial detection result of the detection task is insufficient to form a definitive conclusion. If the testing task does not require verification, the initial testing result shall be used as the judgment result for the testing task. If the detection task requires verification, then construct prompt data to guide the multimodal large model to perform protective equipment wearing detection related to the detection task, and use the multimodal large model to detect the image based on the prompt data to obtain the judgment result of the detection task; Based on the judgment results of each testing task, the final protective equipment wearing test result is determined.

2. The method according to claim 1, wherein the initial detection of the image comprises: Perform the first-level task of detecting whether the image contains a human figure; If the confidence level of the detection task falls within the preset deterministic range that does not include the human image, the subsequent cascaded detection tasks are terminated, and the initial detection result that does not include the human image is output; otherwise, the second-level detection task of determining whether protective equipment is being worn is entered. Perform the second-level detection task to determine whether protective equipment is being worn; if the confidence level of this detection task falls within the preset certainty interval corresponding to not wearing protective equipment, then terminate the subsequent cascaded detection tasks and output the initial detection result of not wearing protective equipment; otherwise, proceed to the third-level detection task to determine the wearing status of protective equipment. Perform the Level 3 test to determine the wearing status of protective equipment, and output the initial test results of whether the equipment is worn correctly or incorrectly, along with its confidence level.

3. The method according to claim 1, wherein the total loss of the target detection model during training is determined in the following manner: Based on the task weights corresponding to each detection task, the cascaded perception losses of each detection task are weighted and summed to obtain the total loss; wherein: The task weight of each detection task is configured according to the degree of influence of the level to which the detection task belongs on the final protective equipment wearing detection result; The cascaded sensing loss includes at least a focus loss term.

4. The method according to claim 3, wherein the focus loss term includes a focus parameter, different detection tasks correspond to different focus parameter values, and the focus parameter value increases progressively with the level; and / or, The focus loss term includes a class balance factor, which is used to weight the focus loss term based on the proportion of samples of each class in the training dataset, so that the weight of each class sample is negatively correlated with its quantity; and / or, The cascaded sensing loss also includes a confidence calibration regularization term, which is used to align the distribution of the confidence output by the target detection model with the uncertainty interval corresponding to the detection task.

5. The method according to claim 3, wherein the cascaded sensing loss further includes a confidence calibration regularization term, the confidence calibration regularization term comprising: The first penalty term is used to penalize erroneous predictions with a confidence level higher than the upper boundary of the uncertainty interval; The second penalty term is used to penalize correct predictions with a confidence level below the lower boundary of the uncertainty interval.

6. According to the method of claim 3, the task weight corresponding to the detection task of determining whether protective equipment is being worn is greater than the task weight corresponding to the detection task of determining the wearing status of protective equipment; the task weight corresponding to the detection task of determining the wearing status of protective equipment is greater than the task weight corresponding to the detection task of determining whether the image contains a human figure.

7. A method for training an object detection model, the method comprising: Acquire images and input them into the object detection model; The target detection model is used to perform the following cascaded detection tasks: Detect whether the image contains a human figure. If a person's image is clearly detected or suspected to be present, the system will check whether protective equipment is being worn. If it is detected that someone is clearly wearing protective equipment or is suspected of wearing protective equipment, check the wearing status of the protective equipment. After determining the total loss based on the recognition result of the image by the target detection model, the target detection model is trained based on the total loss; The total loss of the target detection model during training is determined in the following way: Based on the task weights corresponding to each detection task, the cascaded perception losses of each detection task are weighted and summed to obtain the total loss; wherein: The task weight of each detection task is configured according to the degree of influence of the level to which the detection task belongs on the final protective equipment wearing detection result; The cascaded sensing loss includes at least a focus loss term.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

9. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images