Intelligent multimodal detection method, device and computer equipment in the field of safety supervision

Abstract

The embodiment of the present invention discloses an intelligent multimodal detection method, device and computer equipment in the field of safety supervision. The method includes: obtaining an image to be detected in the field of electric power safety supervision; inputting the image to be detected into a small visual model for personnel detection to obtain a detection result; judging whether there is a person in the detection result; if there is a person in the detection result, combining the image to be detected and the detection result, using a large multimodal model to detect violations to obtain a violation detection result; generating a final output result based on the violation detection result; and outputting the final output result. By implementing the method of the embodiment of the present invention, it is possible to utilize the advantages of a small model and a large multimodal model to achieve rapid preliminary judgment and accurate logical reasoning in the field of electric power safety supervision, thereby improving the accuracy and efficiency of the model.

Description

Technical Field

[0001] The present invention relates to a multimodal detection method, and more specifically to an intelligent multimodal detection method, device and computer equipment in the field of safety supervision. Background Art

[0002] In recent years, multimodal large models have attracted extensive research and attention in the field of machine learning and have been widely used in various industries. However, there are still some defects that need to be overcome when applying these models to specific fields. First, the inference speed of multimodal large models is slow, which is difficult to meet the needs of real-time applications, especially when processing large amounts of data, the inference speed may become a limiting factor. Second, the detection effect for small target objects may be poor. Due to the need to process smaller image inputs, the model may not be able to effectively capture and identify small targets, which is particularly evident in some scenarios that require fine detection, such as security inspection scenarios. In addition, increasing the input size of multimodal large models usually increases computing resources and storage requirements, which may become a cost factor that limits the deployment and application scope of models in resource-limited application scenarios. Therefore, although multimodal large models have high potential and broad application prospects in theory, there are still technical challenges and room for improvement in actual vertical field applications to improve the performance and applicability of the models. At the same time, small models of vision have been widely used in various industries. These small models are lightweight and efficient, can process image data quickly, and have achieved good performance in many fields. They perform well in scenarios with high real-time requirements, such as video surveillance and intelligent transportation systems. However, although small models perform well in simple scenarios, their performance may be limited in some complex scenarios and applications that require logical analysis of multiple targets in the image. Small models usually lack a deep understanding of image content and logical thinking capabilities, which makes their performance in complex scenarios less accurate. For example, in security inspection scene analysis, different areas and personnel status in the image need to be accurately analyzed and identified, and small models may not be able to fully understand the subtle differences in the image, resulting in inaccurate diagnosis results. Therefore, although small models are highly timely and efficient in some areas, they have limitations in handling complex scenarios and logical analysis, and cannot fully meet the requirements of practical applications.

[0003] At present, in the field of power safety supervision, the main focus is on monitoring the safety of personnel operations, such as whether they are wearing seat belts or safety ropes. Deep learning methods are often used in the detection process. There are two main types: small visual models and multimodal large models. Small visual models such as Yolo and Deeplab can perform accurate image classification, target detection, and object segmentation tasks. These models have small weights and fast reasoning speeds, and can achieve real-time reasoning. However, they cannot autonomously perform logical analysis of multiple targets in the image, and their accuracy may be low in some complex scenarios. On the other hand, multimodal large models such as LLaVA and InstructBLIP can combine the background information of the image and the relationship between the targets for logical analysis, and output the required conclusions and descriptions. These models have large weights, slow reasoning speeds, high resource consumption, and cannot output accurate target location information. In addition, since multimodal large models need to increase the image size, they will bring expensive computing costs, which limits the accurate analysis of smaller targets.

[0004] Therefore, it is necessary to design a new method to realize the advantages of small models and multimodal large models to achieve rapid preliminary judgment and precise logical reasoning in the field of power safety supervision, thereby improving the accuracy and efficiency of the model. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the prior art and provide an intelligent multi-modal detection method, device and computer equipment in the field of safety supervision.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: an intelligent multi-modal detection method in the field of safety supervision, comprising:

[0007] Acquire images to be tested in the field of power safety supervision;

[0008] Inputting the image to be detected into the small visual model to perform personnel detection to obtain a detection result;

[0009] Determine whether there is a person in the detection result;

[0010] If the detection result shows that a person exists, then combining the image to be detected and the detection result, a multimodal large model is used to perform violation detection to obtain a violation detection result;

[0011] Generate a final output result according to the violation detection result;

[0012] The final output result is outputted.

[0013] Its further technical solution is: the small visual model is a model obtained by training and testing the small model framework by collecting and annotating training data containing positive and negative scenes; the multimodal large model is a model obtained by formulating data collection and annotation specifications, collecting and annotating data in the field of power safety and supervision, and fine-tuning and testing the general multimodal large model.

[0014] A further technical solution is: combining the image to be detected and the detection result, using a multimodal large model to detect illegal behavior to obtain illegal behavior detection results, including:

[0015] Performing scaling and standardization operations on the image to be detected;

[0016] The manipulated image is input into the multimodal large model for violation detection to obtain the violation detection result.

[0017] A further technical solution is: combining the image to be detected and the detection result, using a multimodal large model to detect illegal behavior to obtain illegal behavior detection results, including:

[0018] Deleting target frames that do not meet the requirements from the person detection frames in the detection result according to the area threshold to obtain a deletion result;

[0019] Matching the person detection frame and the seat belt detection frame of the deletion result, selecting matching items whose confidence meets the requirements, and cropping the image to be detected using the matching items to obtain a cropping result;

[0020] Performing scaling and standardization operations on the cropping result;

[0021] The manipulated image is input into the multimodal large model for violation detection to obtain the violation detection result.

[0022] A further technical solution is: matching the person detection frame and the seat belt detection frame of the deletion result, selecting matching items with confidence meeting the requirements, and cropping the image to be detected using the matching items to obtain a cropping result, including:

[0023] Using the person detection frame in the deletion result as a reference object, matching the seat belt detection frame and the safety rope detection frame that intersect with the reference object, when the same person detection frame is matched to multiple seat belts or multiple safety rope detection frames at the same time, selecting the matching item with the highest confidence;

[0024] storing the matching items;

[0025] The image to be detected is cropped using the matching item and the person detection frame to obtain a cropping result.

[0026] A further technical solution is: combining the image to be detected and the detection result, using a multimodal large model to detect illegal behavior to obtain illegal behavior detection results, including:

[0027] Using the detection result to construct prompt information with location information;

[0028] The prompt information and the image to be detected are input into a multimodal large model to perform violation detection to obtain a violation detection result.

[0029] A further technical solution is: generating a final output result according to the violation detection result includes:

[0030] When the illegal behavior detection result is that there are illegal behaviors in the image to be detected, the illegal target is drawn through the prompt information to obtain the final output result; when the illegal behavior detection result is that there are illegal behaviors of multiple persons in the image to be detected, the final output result is generated according to the levels of illegal behaviors corresponding to the detection frames of different persons.

[0031] A further technical solution is: generating a final output result according to the violation detection result includes:

[0032] When the illegal behavior detection result is that there are multiple persons in the detected image who have violated the rules, the corresponding illegal behavior detection results, positions and judgment bases are output according to the person detection frames to obtain the final output result.

[0033] The present invention also provides an intelligent multi-modal detection device in the field of safety supervision, comprising:

[0034] An image acquisition unit, used to acquire images to be detected in the field of power safety supervision;

[0035] A small model detection unit, used for inputting the image to be detected into the small visual model to perform personnel detection to obtain a detection result;

[0036] A person judging unit, used for judging whether there is a person in the detection result;

[0037] A violation detection unit, for, if the detection result shows that a person exists, combining the image to be detected and the detection result, and using a multimodal large model to perform violation detection to obtain a violation detection result;

[0038] A final result generating unit, used to generate a final output result according to the violation detection result;

[0039] An output unit is used to output the final output result.

[0040] The present invention further provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the above method when executing the computer program.

[0041] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention first screens the persons in the images to be detected through a small visual model, then combines these images and the detection results, uses a large multimodal model to detect violations, and finally generates and outputs the final result; it utilizes the advantages of small models and large multimodal models to achieve rapid preliminary judgment and precise logical reasoning in the field of power safety supervision, thereby improving the accuracy and efficiency of the model.

[0042] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0044] Figure 1 A schematic diagram of an application scenario of an intelligent multimodal detection method in the field of safety supervision provided by an embodiment of the present invention;

[0045] Figure 2 A schematic diagram of a process flow of an intelligent multi-modal detection method in the field of safety supervision provided by an embodiment of the present invention;

[0046] Figure 3 A schematic diagram of a sub-process of an intelligent multi-modal detection method in the field of safety supervision provided by an embodiment of the present invention;

[0047] Figure 4 A schematic diagram of a sub-process of an intelligent multi-modal detection method in the field of safety supervision provided by an embodiment of the present invention;

[0048] Figure 5 A schematic diagram of a sub-process of an intelligent multi-modal detection method in the field of safety supervision provided by an embodiment of the present invention;

[0049] Figure 6 A schematic diagram of a sub-process of an intelligent multi-modal detection method in the field of safety supervision provided by an embodiment of the present invention;

[0050] Figure 7 A schematic block diagram of an intelligent multi-modal detection device in the field of safety supervision provided by an embodiment of the present invention;

[0051] Figure 8A schematic block diagram of a violation detection unit of an intelligent multi-modal detection device in the field of safety supervision provided by an embodiment of the present invention;

[0052] Fig. 9 A schematic block diagram of a violation detection unit of an intelligent multi-modal detection device in the field of safety supervision provided by an embodiment of the present invention;

[0053] Fig.10 A schematic block diagram of a cutting subunit of an intelligent multi-modal detection device in the field of safety supervision provided by an embodiment of the present invention;

[0054] Fig.11 A schematic block diagram of a violation detection unit of an intelligent multi-modal detection device in the field of safety supervision provided by an embodiment of the present invention;

[0055] Fig.12 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0058] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0059] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0060] See also Figure 1 and Figure 2 , Figure 1 A schematic diagram of an application scenario of the intelligent multimodal detection method in the field of safety supervision provided by an embodiment of the present invention. Figure 2A schematic flow chart of an intelligent multimodal detection method in the field of safety and supervision provided by an embodiment of the present invention. The intelligent multimodal detection method in the field of safety and supervision is applied to a server. The server interacts with the terminal for data, obtains the image to be detected in the field of electric power safety and supervision, and introduces a small visual model for pre-reasoning, thereby filtering out irrelevant images, increasing the proportion of small targets in the input of the large model, and obtaining local features, thereby improving the reasoning speed and performance of the multimodal large model in the safety and supervision scenario, and greatly promoting the implementation of the multimodal large model in the field of safety and supervision. The advantages of small models and multimodal large models are utilized to achieve rapid preliminary judgment and precise logical reasoning in the field of electric power safety and supervision, thereby improving the accuracy and efficiency of the model.

[0061] Figure 2 FIG. 1 is a flow chart of an intelligent multi-modal detection method in the field of safety supervision provided by an embodiment of the present invention. Figure 2 As shown, the method includes the following steps S110 to S170.

[0062] S110, obtaining an image to be detected in the field of power safety supervision.

[0063] In this embodiment, the images to be detected in the field of power safety supervision refer to images used for detection and analysis acquired during the process of power safety supervision and management. These images usually include photos or video frames of the power operation site, and the captured content may include power workers at work, equipment used, environmental conditions, safety devices (such as safety belts, safety ropes), etc. Through these images, it is possible to detect and evaluate whether there are safety violations on site to ensure the safety and standardization of the operation.

[0064] S120: input the image to be detected into a small visual model to perform personnel detection to obtain a detection result.

[0065] In this embodiment, the detection result refers to whether there is a person in the image to be detected. When there is a person, the detection result includes a person detection frame, a seat belt detection frame and corresponding coordinates.

[0066] The visual mini-model is a model obtained by training and testing the mini-model framework by collecting and annotating training data containing positive and negative example scenes.

[0067] Specifically, the data collection phase requires the collection of a large amount of image data related to whether workers at heights are wearing safety belts in compliance with regulations. These data include not only images of illegal scenes, but also scenes of compliance and scenes without people, etc., to ensure that the model can make accurate judgments in various situations; by collecting rich data, the generalization ability of the model can be improved, making it more robust and reliable in different scenarios. Next, the collected new data is annotated, mainly by drawing rectangular boxes for key elements such as people and safety belts that appear in the image, and annotating the corresponding category names, so as to train the model to recognize these elements; data annotation helps to provide supervised learning signals for machine learning algorithms, guide the model to learn the correct features and patterns, and improve the accuracy of the model in detecting people and safety belts. Finally, the selected small model framework (such as RTDETR) is used on the annotated data sets (Data1 and Data2) for training to obtain a small model (Model2) for the safety and supervision field. After the training is completed, the model is tested for performance to evaluate its accuracy and stability in detecting whether workers at heights are wearing safety belts in compliance with regulations; through the training and testing phases, the small model can learn and understand the specific visual features in the safety and supervision field, thereby improving its detection accuracy and efficiency. Performance testing can verify the effectiveness of the model and provide a reference for subsequent deployment and application.

[0068] S130, determining whether there is a person in the detection result;

[0069] S140: If the detection result shows that a person exists, then combining the image to be detected and the detection result, a multimodal large model is used to perform violation detection to obtain a violation detection result.

[0070] In this embodiment, the violation detection result refers to the judgment and identification result of the behavior that violates regulations or does not meet safety requirements in the field of power safety supervision after analyzing and processing the image to be detected through the multimodal large model. Specifically, it can include the following:

[0071] Identification of violations: The multimodal large model can analyze elements such as scenes, personnel, and equipment in the image to be detected, and identify violations, such as workers working at heights not wearing safety belts and operators using unqualified equipment.

[0072] Calibration and location of violations: The multimodal large model can calibrate and locate the specific location where the violation occurred in the image, that is, determine the area where the violation occurred or the location of the target.

[0073] Classification and description of violations: Multimodal large models can classify and describe different types of violations, such as not wearing a seat belt, using improper equipment, etc., thereby explaining the nature and circumstances of the violation in more detail.

[0074] Confidence or score of violations: Multimodal large models can provide a confidence or score for each detected violation, indicating the degree of credibility of the model's judgment on the violation, which helps in subsequent prioritization or decision-making of violations.

[0075] In summary, the violation detection results provide a comprehensive analysis and identification of the violations appearing in the detected images, providing an important reference for safety management and decision-making in the field of power safety supervision.

[0076] In this embodiment, the multimodal large model is a model obtained by formulating data collection and annotation specifications, collecting and annotating data in the field of power safety and supervision, and fine-tuning and testing the general multimodal large model.

[0077] First, clearly define the number of data to be collected, scenes, types of safety belts, and the status of workers. For example, determine the scenes of normal wearing of safety belts, scenes of not wearing safety belts, and samples of different types of safety belts that need to be collected. According to the specifications, collect a data set of images containing the above elements. It may be necessary to take pictures at different construction sites and environments to cover various actual scenes. Secondly, formulate annotation specifications to clarify the content, format, and labeling system of annotations. For example, determine the category name of rectangular box annotations, the content and format of descriptive annotations, etc.; then, annotate the collected image data. Including rectangular box annotations for personnel, safety belts, etc., and specify the category name of the rectangular box; descriptive annotations for the entire picture, including the main scene of the picture, whether the personnel are in a high-altitude working state, whether the safety belts are worn in accordance with regulations, and the type and color of the safety belts. Secondly, prepare a general multimodal large model as the basic model, which may be a model pre-trained on a large-scale data set with good feature extraction and generalization capabilities. Then, use the annotated data set (Data1) to fine-tune the basic model and adjust the model parameters to adapt to specific violation detection tasks. The fine-tuning process may take some time and computing resources. Perform performance tests on the fine-tuned multimodal large model (Model1). This includes evaluating indicators such as accuracy and recall on independent test sets to ensure the effectiveness and robustness of the model in actual scenarios. Analyze the performance test results and evaluate the performance of the model in the violation detection task. Determine the model's accuracy, recall and other indicators, as well as possible error types and causes. Finally, optimize the model based on the analysis results. This includes further adjusting the model structure, increasing training data, adjusting hyperparameters, and other means to improve the model's detection and generalization capabilities.

[0078] A multi-modal large model for the wearing of safety belts by workers working at heights is constructed to provide efficient and accurate detection capabilities for violations. At the same time, continuous optimization of the model will also help improve the performance and reliability of the model in practical applications.

[0079] In one embodiment, see Figure 3 , the above-mentioned step S140 may include steps S141~S142.

[0080] S141, scaling and standardizing the image to be detected.

[0081] In this embodiment, the image is scaled to a size suitable for the multimodal large model input. For example, if the multimodal large model requires the input image size to be 224x224 pixels, the image to be detected is scaled to this size. This operation can be achieved using bilinear interpolation or other image scaling algorithms.

[0082] Standardize the image. Usually includes: Normalization: scaling pixel values ​​from 0-255 to 0-1. Mean and standard deviation standardization: subtracting a specific mean from each channel of the image (such as RGB channels) and dividing by the standard deviation. These means and standard deviations can be pre-calculated means and standard deviations of training sets.

[0083] Scaling and normalization operations ensure that all input images have consistent sizes and value ranges, which is critical to the performance and stability of the model. Through normalization, the brightness and contrast differences in the image are balanced, allowing the model to focus more on recognizing the actual content in the image rather than being disturbed by factors such as lighting changes. Normalized image data can adapt to the input requirements of the model more quickly, thereby improving the efficiency of the training and inference process.

[0084] S142: Input the operated image into the multimodal large model for violation detection to obtain a violation detection result.

[0085] Large multimodal models are usually trained with a large amount of data and can accurately capture and identify violations in images, thereby improving detection accuracy. Large multimodal models can combine information from different data sources to provide more comprehensive and comprehensive violation detection results. Automated violation detection can significantly reduce the time and cost of manual inspection, and can achieve rapid response and early warning in real-time monitoring systems.

[0086] In another embodiment, see Figure 4 The above-mentioned step S140 may include steps S140a~S146.

[0087] S140a, deleting target frames that do not meet the requirements from the person detection frames in the detection result according to the area threshold, so as to obtain a deletion result.

[0088] In this embodiment, the deletion result refers to the person detection frame that meets the requirements.

[0089] Specifically, the area of ​​each person detection frame is calculated; the percentage of the area of ​​each detection frame to the total area of ​​the image is compared; if the area of ​​the detection frame is less than the set threshold (such as 5%), the target frame is deleted, and the remaining person detection frames are the deletion results. By deleting the detection frames that are too small, misjudgments can be reduced, the accuracy of subsequent processing steps can be improved, and the number of targets that need to be processed is reduced, thereby reducing the computational burden.

[0090] S140b, matching the person detection frame and the seat belt detection frame of the deletion result, selecting matching items whose confidence meets the requirements, and using the matching items to crop the image to be detected to obtain a cropping result.

[0091] In this embodiment, the cropping result refers to a person detection frame that matches a single seat belt detection frame.

[0092] In one embodiment, see Figure 5 , the above-mentioned step S140b may include steps S140b1~S140b3.

[0093] S140b1, using the person detection frame in the deletion result as a reference object, matching the seat belt detection frame and the safety rope detection frame that intersect with the reference object, and when the same person detection frame is matched to multiple seat belts or multiple safety rope detection frames at the same time, selecting the matching item with the highest confidence;

[0094] S140b2, storing the matching items;

[0095] S140b3. Crop the image to be detected using the matching item and the person detection frame to obtain a cropping result.

[0096] In this embodiment, the deleted person detection frame is used as a reference; the seat belt and safety rope detection frames that intersect with it are found; when the same person detection frame matches multiple seat belt or safety rope detection frames at the same time, the match with the highest confidence is selected; the determined matching relationship is saved for subsequent cropping and use; using the stored matching items and person detection frames, a sub-image containing the person and the matching safety equipment is cropped from the image to be detected. Specifically, the person in the image is cropped according to certain rules (if the person has a matching seat belt, etc., it needs to be cropped on one picture), and all the cropped images are used as input to the multimodal large model.

[0097] Ensure that each person detection frame has a corresponding safety equipment detection frame to improve the rigor of the overall detection process; the cropped image focuses more on key information and reduces the interference of irrelevant background on large model reasoning.

[0098] S140c, scaling and standardizing the cropping result.

[0099] In this embodiment, the cropped sub-images are scaled to a uniform size; and the images are standardized in terms of color, brightness, etc.

[0100] Ensure that the images input to the large model have uniform size and normalized properties to improve the stability of model processing; normalization helps improve the recognition accuracy and generalization ability of the large model.

[0101] S140d. Input the operated image into the multimodal large model for violation detection to obtain a violation detection result.

[0102] The processed images are input into the multimodal large model one by one; the large model analyzes the input images to detect whether there are safety violations (such as not wearing a seat belt); the model outputs the detection results, including whether there are violations and related descriptions.

[0103] The multimodal large model can integrate multiple types of information in the image to provide more accurate violation detection; the entire process is automated without the need for human intervention, which improves efficiency and practicality.

[0104] Through phased processing, the accuracy and efficiency of detection are effectively improved. The design of each step helps to optimize the overall process and ensure reliable detection of high-altitude work safety belt wearing in complex scenarios. This can not only reduce missed detections and misjudgments, but also achieve comprehensive monitoring of violations and improve the safety level of the workplace through standardized data processing and powerful multi-modal large model analysis.

[0105] In another embodiment, see Figure 6 , the above-mentioned step S140 may include steps S141'~ S142'.

[0106] S141', construct prompt information with location information using the detection result.

[0107] In this embodiment, a prompt with location information is constructed, that is, prompt information. Specifically, the sorted visual mini-model output (Output2) is used to extract the location information (x1, y1, x2, y2) of the person; the location information is combined with the prompt information to form a description content for each person.

[0108] The prompt information content requirements: determine whether each person is working at height; confirm whether each person is wearing a safety belt in compliance with regulations; describe each person's position, actions, whether they are working at height, and whether there are any violations.

[0109] Combined with location information, it can accurately describe the specific situation of each person, including location, action and wearing conditions. It provides specific prompt information for the multimodal large model, which is conducive to the subsequent violation detection and detailed description output.

[0110] S142', input the prompt information and the image to be detected into the multimodal large model to perform violation detection to obtain a violation detection result.

[0111] In this embodiment, the constructed prompt information with location information is used as input together with the image to be detected; the input is comprehensively analyzed using a multimodal large model to detect violations; and the violation status of each person is output, including a specific description and conclusion.

[0112] The multimodal large model can comprehensively consider image information and location information to provide more comprehensive violation detection results; the output results include the violation description and conclusion of each person, which is conducive to subsequent processing and recording.

[0113] Through the above steps S141' and S142', the location information can be fully utilized to construct specific prompt information, and a comprehensive violation detection can be achieved through a multi-modal large model. Such a process design is helpful for comprehensive monitoring and analysis of the safety of high-altitude operations in complex scenarios, and improves the safety management level of the workplace.

[0114] S150: Generate a final output result according to the violation detection result.

[0115] In this embodiment, when the illegal behavior detection result is that there is illegal behavior in the image to be detected, the illegal target is drawn according to the prompt information to obtain the final output result.

[0116] Specifically, when the illegal behavior detection result shows that there is illegal behavior in the image to be detected, the following method is used to generate the final output result:

[0117] Illegal behavior determination: Analyze the output results of the large model to determine whether there is any illegal behavior and the specific basis for the judgment.

[0118] Location information acquisition: Get the accurate location information of people in the image through the output results of the small model.

[0119] Draw Target: Determine whether to draw the violation target on the image based on the conclusion to visualize the violation.

[0120] By combining the violation judgment of the large model and the location information of the small model, the violation target can be accurately mapped out; providing specific basis for violation judgment is helpful for subsequent analysis and decision-making.

[0121] In another embodiment, when the illegal behavior detection result is that the image to be detected shows that multiple persons have violated the law, the final output result is generated according to the levels of illegal behaviors corresponding to the detection frames of different persons.

[0122] Specifically, when multiple persons appear in the image to be detected:

[0123] If there is only one person in the image, processing is performed according to "when the illegal behavior detection result is that there is an illegal behavior in the image to be detected, the illegal target is drawn according to the prompt information to obtain the final output result".

[0124] For multiple personnel scenarios, analyze the output of each cropped image, retain the output of the large model according to a certain priority (for example: violation exists > uncertain whether violation exists > no violation exists); obtain an overall conclusion and judgment basis; draw the highest priority target in the original image to highlight the most important violation.

[0125] Ability to handle multi-person scenarios to ensure every violation is accounted for; through a prioritization mechanism, the most important violations are highlighted.

[0126] In another embodiment, when the illegal behavior detection result is that multiple persons in the detected image have violated the law, the corresponding illegal behavior detection results, positions and judgment bases are output respectively according to the person detection frame to obtain the final output result.

[0127] When the illegal behavior detection results involve multiple targets:

[0128] Detailed output analysis: The output includes the conclusion and basis for each goal, for example:

[0129] {Conclusion: Person [x1, y1, x2, y2] violated the rules, while person [x3, y3, x4, y4] did not violate the rules};

[0130] {Description: The position, status, and violation judgment logic of personnel [x1, y1, x2, y2]. Personnel [x3, y3, x4, y4]....};

[0131] Directly parse the required target type conclusion, location, judgment basis and other information from the illegal behavior detection results; based on the analysis results, draw the corresponding target in the image and mark the violation type and judgment basis.

[0132] The detailed conclusions and judgment basis of each target are clearly displayed to facilitate subsequent processing; the information of each target is output independently, which is more suitable for violation management in complex scenarios.

[0133] When the illegal behavior detection result is that there is no illegal behavior, the final output result is set as no illegal behavior and outputted as such.

[0134] S160, outputting the final output result;

[0135] S170: If no person exists in the detection result, output the detection result.

[0136] For example: suppose we are monitoring an electrical maintenance site to detect any violations of not wearing safety belts when working at height.

[0137] An image of an aerial work in progress is obtained from a surveillance camera; the image is input into a small visual model, and two workers are detected in the image; it is determined whether there are people, and the detection result shows that there are workers in the image; one of the methods is used to detect violations, for example, the area of ​​each person detection frame is calculated; the percentage of the area of ​​each detection frame to the total area of ​​the image is compared; the detection frames with an area less than 5% are deleted, leaving two detection frames that meet the requirements; the person detection frame is matched with the safety belt detection frame and the safety rope detection frame; the match with the highest confidence is selected; the sub-image containing the person and the matching safety equipment is cropped using the match; the cropped image is scaled to 224x224 pixels; normalize and mean-standardize the image; input the manipulated image into the multimodal large model for detection; the large model analyzes the image to determine whether there is any violation; construct prompt information; extract personnel location information: determine whether they are working at height, whether they are wearing safety belts in compliance, and describe their location and actions; input the prompt information together with the image to be detected into the multimodal large model; output the violation of each worker, including description and conclusion; generate the final output result: parse the output of the large model; determine whether there is any violation; obtain the location information of each worker; draw the violation target and provide a visual conclusion; specific violation detection type and final output result

[0138] Suppose the following is detected in the image:

[0139] Worker 1 (detection box is located at [50, 50, 150, 300]): not wearing a safety belt; Worker 2 (detection box is located at [200, 100, 350, 400]): wearing a safety belt and the status is compliant.

[0140] Draw a detection frame of worker 1 on the image and label it as “Not wearing a safety belt, violation”; label the detection frame of worker 2 as “Wearing in compliance, no violation”;

[0141] The final output result is explained as follows: Worker 1: There is a violation because he did not wear a safety belt when working at height. Worker 2: There is no violation, he wore a safety belt and the status is compliant.

[0142] In this embodiment, surveillance cameras will continuously capture a large number of images in actual scenes, many of which do not contain the objects that need to be detected (such as people and equipment). If each image is directly processed using a large multimodal model, it will result in high computational cost and low efficiency. To solve this problem, a small visual model is introduced for front-end reasoning.

[0143] These specially trained small visual models can efficiently identify specific types of targets, such as people and tools, and quickly filter out images containing relevant targets. This approach not only reduces the number of images that need to be input into the multimodal large model, but also significantly reduces the computational burden and improves overall detection efficiency.

[0144] Multimodal large models are good at handling complex tasks with rich contextual information, but in practical applications, target objects (such as aerial workers and safety belts) often occupy a small area in the image, making it difficult for the model to extract enough features for accurate judgment. To this end, the small visual model can effectively detect the target in the image and generate the corresponding detection frame. Through these detection frames, sub-images containing the target can be cropped and preprocessed (such as scaling and normalization) on these sub-images.

[0145] In addition, the multimodal large model needs to combine global and local features to make accurate judgments. Global features provide overall context, while local features focus on the details of specific targets. The visual small model can provide local features of the target by detecting the target position, which is crucial for judging violations (such as not wearing a seat belt or wrong operation, etc.).

[0146] The method of this embodiment can complete preliminary screening in a short time through the lightweight and fast characteristics of the small visual model, greatly reducing the number of images that need to be input into the multimodal large model. Through cropping and scaling operations, the image size and complexity processed by the large model can also be reduced, further speeding up the reasoning speed. In addition, focusing on images and areas containing targets improves the detection accuracy of the model; the extraction and analysis of local features make the model perform better in identifying details and complex behaviors.

[0147] Through the above optimization, the practicality and work efficiency of the multimodal large model in the power safety supervision scenario are significantly improved, accelerating its promotion and popularization in practical applications; at the same time, it reduces the consumption of computing resources and improves the economy and feasibility of enterprise applications.

[0148] In this embodiment, a traditional small visual model is first used to perform a preliminary screening of a given image to detect whether the image contains a specific target (such as a person or a vehicle) and its location information; then the positive sample containing the target is input into the multimodal large model, and its powerful logical reasoning ability is used to output accurate conclusions and descriptions. At the model fusion level, three different methods are proposed. Method 1: The original image corresponding to the positive sample, that is, the image to be detected, is directly input into the multimodal large model for reasoning, so that the reasoning can be fast and the complete global information can be retained. Method 2: The target location information provided by the small visual model is used to crop the target area, and these cropped images are input into the multimodal large model one by one to improve the reasoning performance on small targets. Method 3: When constructing the prompt words of the multimodal large model, the detection results of the small visual model and the target location information are combined, so that the multimodal large model pays attention to the local information of the target and the global information of the image at the same time, thereby outputting more accurate conclusions and descriptions.

[0149] The above-mentioned intelligent multimodal detection method in the field of safety supervision first screens the people in the image to be detected through a small visual model, then combines these images and the detection results, uses a large multimodal model to detect violations, and finally generates and outputs the final result; it utilizes the advantages of small models and large multimodal models to achieve rapid preliminary judgment and precise logical reasoning in the field of power safety supervision, thereby improving the accuracy and efficiency of the model.

[0150] Figure 7 is a schematic block diagram of an intelligent multi-modal detection device 300 in the field of safety supervision provided by an embodiment of the present invention. Figure 7 As shown, corresponding to the above intelligent multimodal detection method in the field of safety and supervision, the present invention also provides an intelligent multimodal detection device 300 in the field of safety and supervision. The intelligent multimodal detection device 300 in the field of safety and supervision includes a unit for executing the above intelligent multimodal detection method in the field of safety and supervision, and the device can be configured in a server. Specifically, please refer to Figure 7 The intelligent multimodal detection device 300 in the field of safety supervision includes an image acquisition unit 301, a small model detection unit 302, a personnel judgment unit 303, a violation detection unit 304, a final result generation unit 305 and an output unit 306.

[0151] An image acquisition unit 301 is used to acquire an image to be detected in the field of power safety supervision; a small model detection unit 302 is used to input the image to be detected into a small visual model for personnel detection to obtain a detection result; a personnel judgment unit 303 is used to judge whether there is a person in the detection result; a violation detection unit 304 is used to combine the image to be detected and the detection result, and use a multimodal large model to perform violation detection if there is a person in the detection result to obtain a violation detection result; a final result generation unit 305 is used to generate a final output result according to the violation detection result; an output unit 306 is used to output the final output result.

[0152] In one embodiment, the above-mentioned device further includes:

[0153] The result output unit is used to output the detection result if no person exists in the detection result.

[0154] In one embodiment, if Figure 8 As shown, the violation detection unit 304 includes a first operation subunit 3041 and a first detection subunit 3042 .

[0155] The first operation subunit 3041 is used to scale and standardize the image to be detected; the first detection subunit 3042 is used to input the operated image into the multimodal large model for violation detection to obtain a violation detection result.

[0156] In one embodiment, if Fig. 9 As shown, the violation detection unit 304 includes a deletion subunit 3043 , a cropping subunit 3044 , a second operation subunit 3045 and a second detection subunit 3046 .

[0157] The deletion subunit 3043 is used to delete the target frame that does not meet the requirements from the person detection frame in the detection result according to the area threshold to obtain a deletion result; the cropping subunit 3044 is used to match the person detection frame and the seat belt detection frame of the deletion result, select the matching items with the confidence meeting the requirements, and use the matching items to crop the image to be detected to obtain a cropped result; the second operation subunit 3045 is used to scale and standardize the cropped result; the second detection subunit 3046 is used to input the operated image into the multimodal large model for violation detection to obtain a violation detection result.

[0158] In one embodiment, if Fig.10 As shown, the cropping subunit 3044 includes a matching module 30441 , a storage module 30442 and an image cropping module 30443 .

[0159] A matching module 30441 is used to use the person detection frame in the deletion result as a reference object, match the seat belt detection frame and the safety rope detection frame that have an intersection with the reference object, and when the same person detection frame is matched to multiple seat belts or multiple safety rope detection frames at the same time, select the match with the highest confidence; a storage module 30442 is used to store the match items; an image cropping module 30443 is used to crop the image to be detected using the match items and the person detection frame to obtain a cropping result.

[0160] In one embodiment, if Fig.11 As shown, the violation detection unit 304 includes an information construction subunit 3047 and a third detection subunit 3048 .

[0161] The information construction subunit 3047 is used to use the detection result to construct prompt information with location information; the third detection subunit 3048 is used to input the prompt information and the image to be detected into the multimodal large model for violation detection to obtain the violation detection result.

[0162] In one embodiment, the final result generating unit 305 is used to draw the violation target through the prompt information to obtain the final output result when the illegal behavior detection result is that there is illegal behavior in the image to be detected; when the illegal behavior detection result is that there are multiple persons in the image to be detected, the final output result is generated according to the levels of illegal behavior corresponding to the detection boxes of different persons.

[0163] In one embodiment, the final result generating unit 305 is used to output the corresponding illegal behavior detection results, positions and judgment basis according to the person detection frame respectively when the illegal behavior detection result is that multiple persons have illegal behaviors in the image to be detected, so as to obtain the final output result.

[0164] It should be noted that technical personnel in the relevant field can clearly understand that the specific implementation process of the above-mentioned intelligent multimodal detection device 300 in the field of safety and supervision and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and conciseness of the description, it will not be repeated here.

[0165] The intelligent multi-modal detection device 300 in the field of safety supervision can be implemented in the form of a computer program. The computer program can be used in Fig.12 Runs on the computer device shown.

[0166] See also Fig.12 , Fig.12 5 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 may be a server, wherein the server may be an independent server or a server cluster composed of multiple servers.

[0167] See also Fig.12 The computer device 500 includes a processor 502 , a memory and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .

[0168] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions, and when the program instructions are executed, the processor 502 can execute an intelligent multi-modal detection method in the field of safety supervision.

[0169] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .

[0170] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute an intelligent multi-modal detection method in the field of safety supervision.

[0171] The network interface 505 is used to communicate with other devices over the network. Fig.12 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0172] The processor 502 is used to run the computer program 5032 stored in the memory to implement the following steps:

[0173] Acquire an image to be detected in the field of electric power safety supervision; input the image to be detected into a small visual model for personnel detection to obtain a detection result; determine whether there is a person in the detection result; if there is a person in the detection result, combine the image to be detected and the detection result, and use a multimodal large model to detect violations to obtain a violation detection result; generate a final output result based on the violation detection result; and output the final output result.

[0174] Among them, the visual small model is a model obtained by training and testing the small model framework by collecting and annotating training data containing positive and negative example scenes; the multimodal large model is a model obtained by formulating data collection and annotation specifications, collecting and annotating data in the field of power safety and supervision, and fine-tuning and testing the general multimodal large model.

[0175] In one embodiment, when the processor 502 implements the step of combining the image to be detected and the detection result and using a multimodal large model to perform violation detection to obtain a violation detection result, the processor 502 specifically implements the following steps:

[0176] The image to be detected is scaled and standardized; the operated image is input into a multimodal large model for violation detection to obtain a violation detection result.

[0177] In one embodiment, when the processor 502 implements the step of combining the image to be detected and the detection result and using a multimodal large model to perform violation detection to obtain a violation detection result, the processor 502 specifically implements the following steps:

[0178] The target frames that do not meet the requirements are deleted from the person detection frames in the detection results according to the area threshold to obtain a deletion result; the person detection frames and the seat belt detection frames are matched with the deletion results, and matching items with confidence levels that meet the requirements are selected, and the image to be detected is cropped using the matching items to obtain a cropped result; the cropped result is scaled and standardized; and the operated image is input into a multimodal large model for violation detection to obtain a violation detection result.

[0179] In one embodiment, when the processor 502 implements the step of matching the person detection frame and the seat belt detection frame of the deletion result, selecting a matching item whose confidence meets the requirement, and using the matching item to crop the image to be detected to obtain a cropped result, the following steps are specifically implemented:

[0180] The person detection frame in the deletion result is used as a reference object, and the seat belt detection frame and the safety rope detection frame that have an intersection with the reference object are matched. When the same person detection frame is matched to multiple seat belts or multiple safety rope detection frames at the same time, the matching item with the highest confidence is selected; the matching item is stored; and the image to be detected is cropped using the matching item and the person detection frame to obtain a cropping result.

[0181] In one embodiment, when the processor 502 implements the step of combining the image to be detected and the detection result and using a multimodal large model to perform violation detection to obtain a violation detection result, the processor 502 specifically implements the following steps:

[0182] The detection result is used to construct prompt information with location information; the prompt information and the image to be detected are input into a multimodal large model to perform violation detection to obtain a violation detection result.

[0183] In one embodiment, when the processor 502 implements the step of generating a final output result according to the violation detection result, the processor 502 specifically implements the following steps:

[0184] When the illegal behavior detection result is that there are illegal behaviors in the image to be detected, the illegal target is drawn through the prompt information to obtain the final output result; when the illegal behavior detection result is that there are illegal behaviors of multiple persons in the image to be detected, the final output result is generated according to the levels of illegal behaviors corresponding to the detection frames of different persons.

[0185] In one embodiment, when the processor 502 implements the step of generating a final output result according to the violation detection result, the processor 502 specifically implements the following steps:

[0186] When the illegal behavior detection result is that there are multiple persons in the detected image who have violated the rules, the corresponding illegal behavior detection results, positions and judgment bases are output according to the person detection frames to obtain the final output result.

[0187] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0188] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0189] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein when the computer program is executed by a processor, the processor executes the following steps:

[0190] Acquire an image to be detected in the field of electric power safety supervision; input the image to be detected into a small visual model for personnel detection to obtain a detection result; determine whether there is a person in the detection result; if there is a person in the detection result, combine the image to be detected and the detection result, and use a multimodal large model to detect violations to obtain a violation detection result; generate a final output result based on the violation detection result; and output the final output result.

[0191] Among them, the visual small model is a model obtained by training and testing the small model framework by collecting and annotating training data containing positive and negative example scenes; the multimodal large model is a model obtained by formulating data collection and annotation specifications, collecting and annotating data in the field of power safety and supervision, and fine-tuning and testing the general multimodal large model.

[0192] In one embodiment, when the processor executes the computer program to implement the step of combining the image to be detected and the detection result, using a multimodal large model to perform violation detection to obtain a violation detection result, the processor specifically implements the following steps:

[0193] The image to be detected is scaled and standardized; the operated image is input into a multimodal large model for violation detection to obtain a violation detection result.

[0194] In one embodiment, when the processor executes the computer program to implement the step of combining the image to be detected and the detection result, using a multimodal large model to perform violation detection to obtain a violation detection result, the processor specifically implements the following steps:

[0195] The target frames that do not meet the requirements are deleted from the person detection frames in the detection results according to the area threshold to obtain a deletion result; the person detection frames and the seat belt detection frames are matched with the deletion results, and matching items with confidence levels that meet the requirements are selected, and the image to be detected is cropped using the matching items to obtain a cropped result; the cropped result is scaled and standardized; and the operated image is input into a multimodal large model for violation detection to obtain a violation detection result.

[0196] In one embodiment, when the processor executes the computer program to implement the steps of matching the person detection frame and the seat belt detection frame of the deletion result, selecting matching items whose confidence meets the requirements, and cropping the image to be detected using the matching items to obtain the cropped result, the following steps are specifically implemented:

[0197] The person detection frame in the deletion result is used as a reference object, and the seat belt detection frame and the safety rope detection frame that have an intersection with the reference object are matched. When the same person detection frame is matched to multiple seat belts or multiple safety rope detection frames at the same time, the matching item with the highest confidence is selected; the matching item is stored; and the image to be detected is cropped using the matching item and the person detection frame to obtain a cropping result.

[0198] In one embodiment, when the processor executes the computer program to implement the step of combining the image to be detected and the detection result, using a multimodal large model to perform violation detection to obtain a violation detection result, the processor specifically implements the following steps:

[0199] The detection result is used to construct prompt information with location information; the prompt information and the image to be detected are input into a multimodal large model to perform violation detection to obtain a violation detection result.

[0200] In one embodiment, when the processor executes the computer program to implement the step of generating a final output result according to the violation detection result, the processor specifically implements the following steps:

[0201] When the illegal behavior detection result is that there are illegal behaviors in the image to be detected, the illegal target is drawn through the prompt information to obtain the final output result; when the illegal behavior detection result is that there are illegal behaviors of multiple persons in the image to be detected, the final output result is generated according to the levels of illegal behaviors corresponding to the detection frames of different persons.

[0202] In one embodiment, when the processor executes the computer program to implement the step of generating a final output result according to the violation detection result, the processor specifically implements the following steps:

[0203] When the illegal behavior detection result is that there are multiple persons in the detected image who have violated the rules, the corresponding illegal behavior detection results, positions and judgment bases are output according to the person detection frames to obtain the final output result.

[0204] The storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.

[0205] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0206] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0207] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0208] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0209] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An intelligent multi-modal detection method in the field of safety supervision, characterized in that: include: Acquire images to be tested in the field of power safety supervision; Inputting the image to be detected into the small visual model to perform personnel detection to obtain a detection result; Determine whether there is a person in the detection result; If the detection result shows that a person exists, then combining the image to be detected and the detection result, a multimodal large model is used to perform violation detection to obtain a violation detection result; Generate a final output result according to the violation detection result; Outputting the final output result; The method of combining the image to be detected and the detection result and using a multimodal large model to perform violation detection to obtain a violation detection result includes: Deleting target frames that do not meet the requirements from the person detection frames in the detection result according to the area threshold to obtain a deletion result; Matching the person detection frame and the seat belt detection frame of the deletion result, selecting matching items whose confidence meets the requirements, and cropping the image to be detected using the matching items to obtain a cropping result; Performing scaling and standardization operations on the cropping result; Inputting the manipulated image into the multimodal large model for violation detection to obtain a violation detection result; The visual small model is a model obtained by training and testing the small model framework by collecting and annotating training data containing positive and negative examples; the multimodal large model is a model obtained by formulating data collection and annotation specifications, collecting and annotating data in the field of power safety supervision, and fine-tuning and testing the general multimodal large model; The method of combining the image to be detected and the detection result and using a multimodal large model to perform violation detection to obtain a violation detection result includes: Using the detection result to construct prompt information with location information; The prompt information and the image to be detected are input into a multimodal large model to perform violation detection to obtain a violation detection result.

2. The intelligent multi-modal detection method in the field of safety supervision according to claim 1 is characterized in that: The method of combining the image to be detected and the detection result and using a multimodal large model to perform violation detection to obtain a violation detection result includes: Performing scaling and standardization operations on the image to be detected; The manipulated image is input into the multimodal large model for violation detection to obtain the violation detection result.

3. The intelligent multi-modal detection method in the field of safety supervision according to claim 1 is characterized in that: The matching of the person detection frame and the seat belt detection frame of the deletion result, selecting matching items with confidence meeting the requirement, and cropping the image to be detected by using the matching items to obtain a cropping result, includes: Using the person detection frame in the deletion result as a reference object, matching the seat belt detection frame and the safety rope detection frame that intersect with the reference object, when the same person detection frame is matched to multiple seat belts or multiple safety rope detection frames at the same time, selecting the matching item with the highest confidence; storing the matching items; The image to be detected is cropped using the matching item and the person detection frame to obtain a cropping result.

4. The intelligent multi-modal detection method in the field of safety supervision according to claim 1 is characterized in that: Generating a final output result according to the violation detection result includes: When the violation detection result is that there is a violation in the image to be detected, the violation target is drawn through the prompt information to obtain the final output result; when the violation detection result is that there are multiple people in the image to be detected who have violated the rules, the final output result is generated according to the levels of the violations corresponding to the detection boxes of different people.

5. The intelligent multi-modal detection method in the field of safety supervision according to claim 1 is characterized in that: Generating a final output result according to the violation detection result includes: When the violation detection result is that there are multiple persons in the detected image who have violated the rules, the corresponding violation detection results, positions and judgment bases are output respectively according to the person detection frames to obtain the final output result.

6. An intelligent multi-modal detection device in the field of safety supervision, characterized in that: include: An image acquisition unit, used to acquire images to be detected in the field of power safety supervision; A small model detection unit, used for inputting the image to be detected into the small visual model to perform personnel detection to obtain a detection result; A person judging unit, used for judging whether there is a person in the detection result; A violation detection unit, for, if the detection result shows that a person exists, combining the image to be detected and the detection result, and using a multimodal large model to perform violation detection to obtain a violation detection result; A final result generating unit, used to generate a final output result according to the violation detection result; An output unit, used for outputting the final output result; The violation detection unit includes a deletion subunit, a cropping subunit, a second operation subunit and a second detection subunit; A deletion subunit, configured to delete target frames that do not meet the requirements from the person detection frames in the detection result according to an area threshold, so as to obtain a deletion result; A cropping subunit, used for matching the person detection frame and the seat belt detection frame of the deletion result, selecting matching items with confidence meeting the requirements, and cropping the image to be detected using the matching items to obtain a cropping result; A second operation subunit, used for performing scaling and normalization operations on the cropping result; The second detection subunit is used to input the operated image into the multimodal large model for violation detection to obtain a violation detection result; The visual small model is a model obtained by training and testing the small model framework by collecting and annotating training data containing positive and negative examples; the multimodal large model is a model obtained by formulating data collection and annotation specifications, collecting and annotating data in the field of power safety supervision, and fine-tuning and testing the general multimodal large model; The violation detection unit includes an information construction subunit and a third detection subunit; An information construction subunit, used to construct prompt information with location information using the detection result; The third detection subunit is used to input the prompt information and the image to be detected into the multimodal large model to perform violation detection to obtain a violation detection result.

7. A computer device, characterized in that: The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 6 when executing the computer program.

Images