Illegal behavior automatic identification and reconfirmation method, device and medium

By using deep learning models and identity identification modules in the drone inspection system, the problem of the lack of re-confirmation of mobile targets by the drone is solved, and real-time and accurate target filtering effect is achieved.

CN119672564APending Publication Date: 2025-03-21SHANDONG SYNTHESIS ELECTRONICS TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411598560.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-11-11
Publication Date
2025-03-21

AI Technical Summary

Technical Problem

During the inspection, drones lack the means to re-confirm the identity and behavior of mobile targets, resulting in the inability to accurately filter confirmed targets in real time.

Method used

The automatic identification method based on the deep learning model is adopted to improve the YoLov8 and ResNet-18 models to identify violations in the inspection screen, and combine the identity identification module to match the suspicious targets. For unconfirmed targets, perform a secondary detection and record to the confirmed list.

Benefits of technology

Real-time accurate identity and behavior confirmation of mobile targets is achieved, the efficiency and accuracy of drone inspections are improved, and false alarms and missed reports are avoided.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119672564A_ABST
    Figure CN119672564A_ABST
Patent Text Reader

Abstract

The invention discloses an illegal behavior automatic identification and reconfirmation method and device, and a medium, and belongs to the technical field of unmanned aerial vehicle intelligent inspection. The method comprises the following steps: inspecting a preset inspection area based on an inspection unmanned aerial vehicle to obtain an inspection picture; processing the inspection picture based on a deep learning model to identify whether an illegal behavior exists in the inspection picture; when the illegal behavior exists, marking a person in the inspection video frame corresponding to the illegal behavior as a suspicious target, and processing the suspicious target and the confirmed list based on an identity recognition module so as to judge whether the suspicious target is recorded by the confirmed list or not; if not, marking the suspicious target as a to-be-confirmed target and recording the to-be-confirmed target to a to-be-confirmed list; and carrying out secondary detection on the to-be-confirmed target, and recording the to-be-confirmed target to a confirmed list when the to-be-confirmed target has an illegal behavior. According to the method, the effect that the unmanned aerial vehicle confirms the identity and behavior of the moving target again is achieved, and the effect that the confirmed target is accurately filtered in real time is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of drone intelligent inspection, and in particular to a method, device and medium for automatically identifying and reconfirming illegal behaviors. Background Art

[0002] Currently, drone inspection technology is gradually being widely used in various fields, but there are still significant challenges in the post-identification and re-confirmation stages of detecting illegal behaviors such as smoking and making phone calls. Traditional methods are mostly only applicable to static target detection. Once the target is in motion, repeated detection is prone to occur, which not only reduces the inspection efficiency, but may also cause false alarms and missed reports. Especially when drones are inspecting dangerous factory areas such as flammable and explosive areas, once a suspicious person is detected, the drone lacks effective means to further confirm the identity and illegal behavior of the suspicious person, and cannot accurately filter the confirmed targets in real time. Summary of the invention

[0003] The embodiments of the present application provide a method, device and medium for automatically identifying and reconfirming illegal behaviors, so as to solve the following technical problems: UAVs lack the means to reconfirm the identity and behavior of mobile targets, and are unable to filter confirmed targets in real time and accurately.

[0004] In a first aspect, an embodiment of the present application provides a method for automatic identification and reconfirmation of violations, which is used for a patrol drone inspection system, and the method includes: inspecting a preset inspection area based on a patrol drone to obtain an inspection screen; wherein the inspection screen is composed of inspection video frames; processing the inspection screen based on a preset deep learning model to identify whether there are any violations in the inspection screen, and marking the persons who have violated the rules as suspicious targets; wherein the deep learning model includes an improved YoLov8 model and an improved ResNet-18 model; processing suspicious targets and a preset confirmed list based on a preset identity recognition module to determine whether the suspicious target has been recorded in the confirmed list; if not, marking the suspicious target as a target to be confirmed, and recording the target to be confirmed to the preset list to be confirmed; adjusting the patrol drone to aim at the target to be confirmed to perform a secondary detection of the violation on the target to be confirmed, and recording the target to be confirmed to the confirmed list when the target to be confirmed has a violation.

[0005] In one implementation of the present application, the inspection screen is processed based on a preset deep learning model to identify whether there are any violations in the inspection screen, and the persons with violations are marked as suspicious targets, specifically including: obtaining inspection video frames of the inspection screen; processing the inspection video frames based on the improved YoLov8 model to identify the persons in the inspection video frames and mark them as identified persons, and generating identified person target frames; processing the identified person target frames according to a preset capture ratio and preset capture rules to determine the key target frames; processing the key target frames based on the improved ResNet-18 model to mark at least one operation behavior of the key target frames; wherein the operation behavior includes normal behavior and at least one violation; marking the persons in the key target frames with violations as suspicious targets; combining the key target frames with violations, the inspection video frames, the identified person target frames and the operation behaviors to output the video frame detection results.

[0006] In one implementation of the present application, before processing the key target box based on the improved ResNet-18 model to mark at least one operation behavior of the key target box, the method also includes: constructing a standard ResNet-18 model; setting the category of the standard ResNet-18 model to normal behavior and violation behavior set; wherein the violation behavior set includes at least one violation behavior; substituting the preset loss function wn-BCEloss into the standard ResNet-18 model to construct an improved ResNet-18 model.

[0007] In one implementation of the present application, the loss function wn-BCEloss is as follows:

[0008]

[0009] Where N is the number of batches, B is the number of samples in a batch, and w(n ij ) is the weighting function, n ij is the number of sample labels in a batch, α is the weighting coefficient one, β is the weighting coefficient two, is the binary cross entropy loss.

[0010] In one implementation of the present application, a suspicious target and a preset confirmed list are processed based on a preset identity recognition module to determine whether the suspicious target has been recorded in the confirmed list, specifically including: the identity recognition module includes an image feature extraction module and a position feature extraction module; the suspicious target and the video frame detection results in the confirmed list are processed based on the image feature extraction module to obtain the features to be confirmed and the confirmed features; the cosine similarity of the features to be confirmed and the confirmed features is calculated to determine the image similarity; the suspicious target and the identified person target frame in the confirmed list are processed based on the position feature extraction module, and the suspicious target and the identified person target frame in the confirmed list are processed based on the position feature extraction module, and the coordinates to be processed and the processed coordinates are obtained with the center point of the suspicious target and the geometric center of the identified person target frame as coordinate points; when the image similarity is greater than the preset image threshold and the Euclidean distance is less than the preset distance threshold, it is determined that the suspicious target has been recorded in the confirmed list.

[0011] In one implementation of the present application, before processing the video frame detection results of suspicious targets and confirmed lists based on the image feature extraction module to obtain the features to be confirmed and the confirmed features, the method also includes: based on the preset standard ResNet-50 model as the backbone network, using cross entropy loss to construct the classification layer in the standard ResNet-50 model, and using triplet loss to construct the feature layer in the standard ResNet-50 model to construct a ReID deep learning module; based on the ReID deep learning module as the image feature extraction module.

[0012] In one implementation of the present application, an inspection drone is adjusted to aim at a target to be confirmed so as to conduct a secondary inspection of the target to be confirmed for violations, and the target to be confirmed is recorded in a confirmed list when a violation exists in the target to be confirmed, specifically including: querying the list to be confirmed to inquire whether there is a target to be confirmed; if there is a target to be confirmed, calculating an offset based on the center of the picture taken by the inspection drone and the center of the target to be confirmed; adjusting the camera module of the inspection drone based on the offset so that the center of the target to be confirmed is at the center of the camera module; adjusting the focal length value of the camera module to a preset pixel value to obtain a secondary image frame of the target to be confirmed; processing the inspection picture based on a deep learning model to identify whether there is a violation in the secondary image frame; if so, recording the target to be confirmed in the confirmed list.

[0013] In an implementation of the present application, the method further includes: when there is no target to be confirmed, adjusting the camera module to a preset initial posture.

[0014] In a second aspect, an embodiment of the present application also provides an automatic identification and reconfirmation device for violations, characterized in that the device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can: inspect a preset inspection area based on a patrol drone to obtain an inspection screen; wherein the inspection screen is composed of patrol video frames; process the inspection screen based on a preset deep learning model to identify whether there are any violations in the inspection screen, and mark the persons with violations as suspicious targets; wherein the deep learning model includes a YoLov8 model and a ResNet-18 model; process suspicious targets and a preset confirmed list based on a preset identity recognition module to determine whether the suspicious target has been recorded in the confirmed list; if not, mark the suspicious target as a target to be confirmed, and record the target to be confirmed to the preset list to be confirmed; adjust the patrol drone to aim at the target to be confirmed to perform a secondary detection of the violation on the target to be confirmed, and record the target to be confirmed to the confirmed list when there is a violation in the target to be confirmed.

[0015] On the third aspect, the embodiment of the present application also provides a non-volatile computer storage medium for automatic identification and reconfirmation of violations, storing computer executable instructions, characterized in that the computer executable instructions are set to: based on the inspection drone patrolling the preset inspection area to obtain the inspection screen; wherein the inspection screen is composed of inspection video frames; based on the preset deep learning model, the inspection screen is processed to identify whether there are any violations in the inspection screen, and the persons with violations are marked as suspicious targets; wherein the deep learning model includes the YoLov8 model and the ResNet-18 model; based on the preset identity recognition module, the suspicious targets and the preset confirmed list are processed to determine whether the suspicious targets have been recorded in the confirmed list; if not, the suspicious targets are marked as targets to be confirmed, and the targets to be confirmed are recorded in the preset list to be confirmed; the inspection drone is adjusted to aim at the target to be confirmed, so as to perform secondary detection of the violations on the target to be confirmed, and the targets to be confirmed are recorded in the confirmed list when there are violations in the target to be confirmed.

[0016] The embodiment of the present application provides a method, device and medium for automatic identification and reconfirmation of illegal behaviors. The inspection drone inspects the preset area, obtains the inspection screen in real time, and uses YoLov8 and ResNet-18 to accurately identify the illegal behaviors in the screen, effectively marking suspicious targets. At the same time, combined with the preset identity recognition module, the identity of the suspicious target is matched to avoid repeated confirmation and reduce resource waste. For suspicious targets that are not recorded in the confirmed list, they are marked as targets to be confirmed, and the drone is adjusted to align with the targets to be confirmed for secondary detection to ensure the accuracy of identification. During the secondary detection process, if it is confirmed that there is a violation, it will be recorded in the confirmed list for subsequent tracking and management. At the same time, the zoom capability of the drone is fully utilized, and the focal length is adjusted to magnify the target during the secondary detection, so that the recognition effect is more guaranteed. Therefore, the present application achieves the effect of the drone reconfirming the identity and behavior of the mobile target, and achieves the effect of real-time and accurate filtering of confirmed targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 A flow chart of a method for automatically identifying and reconfirming illegal behaviors provided in an embodiment of the present application;

[0019] Figure 2 A schematic diagram of the internal structure of an automatic violation identification and reconfirmation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0021] The embodiments of the present application provide a method, device and medium for automatically identifying and reconfirming illegal behaviors, so as to solve the following technical problems: UAVs lack the means to reconfirm the identity and behavior of mobile targets, and are unable to filter confirmed targets in real time and accurately.

[0022] The technical solution proposed in the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0023] Figure 1A flowchart of automatic identification and reconfirmation of illegal behavior provided in the embodiment of the present application. Figure 1 As shown, an automatic identification and reconfirmation method of illegal behavior provided by an embodiment of the present application specifically includes the following steps:

[0024] Step 1: Inspect a preset inspection area based on an inspection drone to obtain an inspection picture; wherein the inspection picture is composed of inspection video frames.

[0025] Inspection drone: refers to a drone equipped with camera modules, sensors, flight control systems and communication equipment, which is used for regular or irregular patrols and inspections of specific areas.

[0026] Preset inspection area: refers to the specific geographical range or path that the drone needs to patrol and inspect according to actual needs before the inspection mission begins.

[0027] Inspection picture: refers to the visual information of the inspection area captured in real time by the camera module carried by the inspection drone during its flight.

[0028] Inspection video frames: The inspection screen is presented in the form of continuous video frames, and each frame is a static image of the inspection screen at a certain moment.

[0029] Before the inspection mission begins, the inspection area is preset in the control system of the drone or the ground control station according to the inspection objectives and requirements. The inspection area is a specific flight path set according to the factory layout, pipeline direction, etc.

[0030] After receiving the take-off command, the inspection drone takes off autonomously according to the preset flight plan. With the help of GPS navigation system or other positioning technology, the drone flies according to the preset inspection route or area. During the flight of the drone, the camera module on board starts working to capture the images of the inspection area in real time to obtain the inspection images. The inspection images are continuously generated in the form of video frames, and each frame contains detailed visual information of the inspection area at a certain moment. It is understandable that the capture frequency and resolution of the video frames should be comprehensively considered and set according to the inspection requirements, drone performance, and the ability of subsequent image processing.

[0031] At the same time, the captured inspection images are transmitted to the ground control station or cloud server in real time through wireless transmission technology (such as 4G, 5G or Wi-Fi) for storage and processing. The local storage device on the drone will also back up some key images (suspicious persons in subsequent steps, targets to be confirmed, confirmed lists and other image data) to prevent data loss or transmission interruption. The operator of the ground control station can view the inspection images in real time through the monitoring screen to detect abnormal situations in time and make corresponding treatments.

[0032] In a specific case, there are inspection drones in the petrochemical plant area, and the preset inspection areas are key areas such as the production equipment area, storage tank area, and dangerous goods transportation channel of the plant area. After receiving the take-off command, the inspection drone takes off autonomously and patrols these areas according to the preset flight route. During the flight, the high-definition camera module carried by the drone captures the inspection screen in real time at a capture frequency of 25 frames per second, and transmits it to the ground control station in real time through the 4G network. The operator of the ground control station can view the inspection screen in real time and make preliminary judgments and records of possible violations or safety hazards. At the same time, the inspection screen is also stored in the cloud server and the local storage device of the drone for subsequent analysis and processing.

[0033] Step 2: Process the inspection screen based on a preset deep learning model to identify whether there are any violations in the inspection screen, and mark the persons with violations as suspicious targets; wherein the deep learning model includes an improved YoLov8 model and an improved ResNet-18 model.

[0034] Deep learning model: An algorithm or network structure based on deep learning, which simulates the working mode of human brain neural network to learn large amounts of data, thereby realizing automatic recognition and extraction of complex patterns or features.

[0035] Improved YoLov8 model: An improved target detection model based on the YoLov8 algorithm, which is used to quickly and accurately identify human targets in inspection video frames.

[0036] Improved ResNet-18 model: A behavior recognition model based on the improved ResNet-18 algorithm, which is used to further classify the behavior of the identified personnel targets and determine whether they have violated the regulations.

[0037] Suspicious target: refers to a person who is identified by the deep learning model as possibly having violated regulations.

[0038] Step 21: Obtain inspection video frames of the inspection screen.

[0039] From the inspection images sent back by the inspection drone in real time, inspection video frames are extracted at a certain time interval or frame rate. These video frames are the basic data for subsequent deep learning model processing.

[0040] Step 22: Process the inspection video frames based on the improved YoLov8 model to identify the persons in the inspection video frames and mark them as identified persons, and generate identified person target frames.

[0041] The improved YoLov8 model is used to process the extracted inspection video frames, quickly identify the human targets in the picture, and generate an identified human target frame for it. The identified human target frame is a rectangular frame that represents the position and size of the person.

[0042] Step 23: Process the identified person target frame according to a preset interception ratio and preset interception rules to determine the key target frame.

[0043] According to the preset interception ratio and interception rules, the identified person target frame is processed to determine the key target frame. The key target frame is the area in the person target frame that may contain violations, such as the upper body or hand area of ​​the person.

[0044] Step 24: Process the key target frame based on the improved ResNet-18 model to mark at least one operation behavior of the key target frame; wherein the operation behavior includes a normal behavior and at least one illegal behavior.

[0045] The improved ResNet-18 model is used to process the key target frame, identify the operation behavior of the person in the target frame, and determine whether it belongs to the violation behavior set. If a violation behavior is identified, the corresponding person target is marked as a suspicious target.

[0046] Construct a standard ResNet-18 model, set the categories of the standard ResNet-18 model to normal behavior and violation behavior set; wherein the violation behavior set includes at least one violation behavior, substitute the preset loss function wn-BCEloss into the standard ResNet-18 model to construct an improved ResNet-18 model.

[0047] First, build a standard ResNet-18 model, which is a classic convolutional neural network with strong feature extraction capabilities. Set the output categories of the standard ResNet-18 model to normal behavior and violation behavior sets. The violation behavior set contains all violations that need to be detected, such as smoking, making phone calls, etc. Substitute the preset loss function wn-BCEloss into the standard ResNet-18 model to optimize the model training process. wn-BCEloss is a weighted binary cross entropy loss function that can handle sample imbalance and multi-label classification problems. The various parameters in its formula (such as the number of batches, the number of samples, the weighting function, the weighting coefficient, etc.) are set according to the actual situation.

[0048] Among them, the formula of the loss function wn-BCEloss is as follows:

[0049]

[0050]

[0051] Where N is the number of batches, B is the number of samples in a batch, and w(n ij ) is the weighting function, n ij is the number of sample labels in a batch, α is the weighting coefficient one, β is the weighting coefficient two, is the binary cross entropy loss.

[0052] Step 25: Mark the persons in the key target box who have committed violations as suspicious targets.

[0053] All persons in the key target box identified as having violated regulations by the improved ResNet-18 model are marked as suspicious targets.

[0054] Step 26: Combine the key target frame with illegal behavior, the inspection video frame, the identified person target frame and the operation behavior to output the video frame detection result.

[0055] The key target frames with violations, inspection video frames, identified personnel target frames, and identified operation behaviors are combined to form a complete video frame detection result. These results will be used for further analysis and processing.

[0056] In a specific example, in a drone inspection mission in a chemical park, the drone patrols key areas in the park according to a preset inspection route. After obtaining the inspection screen, the inspection video frame is first processed using the improved YoLov8 model to identify the human target in the screen and generate an identified human target frame for it. Next, the key target frame is determined according to the preset interception rules and interception ratios (such as 30% of the upper part of the target frame). Then, the key target frame is processed using the improved ResNet-18 model to identify the operating behavior of the personnel and determine whether they have any violations (such as smoking). For the identified suspicious targets, they are marked and annotated in the video frame detection results.

[0057] Step 3: Process the suspicious target and the preset confirmed list based on the preset identity recognition module to determine whether the suspicious target has been recorded in the confirmed list.

[0058] Identity recognition module: A module that integrates multiple feature extraction and comparison algorithms to match the identity of suspicious targets with records in the confirmed list, including image feature extraction module and location feature extraction module.

[0059] Suspicious target: A person who is initially identified as having violated regulations in the inspection image.

[0060] Confirmed list: A list that stores targets that have been previously confirmed to have violated regulations and their related information.

[0061] Image feature extraction module: a module used to extract feature information from images to facilitate subsequent comparison and recognition.

[0062] Position feature extraction module: a module used to extract target position information from an image, usually including the coordinates and size of the target.

[0063] Step 31, the identity recognition module includes an image feature extraction module and a position feature extraction module. The identity recognition module consists of two core submodules: an image feature extraction module and a position feature extraction module. These two modules are responsible for extracting the image features and position features of the target respectively.

[0064] Step 32: Process the video frame detection results of the suspicious target and the confirmed list based on the image feature extraction module to obtain the features to be confirmed and the confirmed features.

[0065] Among them, the image feature extraction module is constructed by improving the standard ResNet-50 model as the backbone network, as shown below: based on the preset standard ResNet-50 model as the backbone network, the cross entropy loss is used to construct the classification layer in the standard ResNet-50 model, and the triplet loss is used to construct the feature layer in the standard ResNet-50 model to construct the ReID deep learning module, which is used as the image feature extraction module based on the ReID deep learning module.

[0066] ReID deep learning module: A re-identification module based on deep learning technology, used to improve the accuracy of image feature extraction.

[0067] The image feature extraction module uses the ReID deep learning module, which uses the standard ResNet-50 model as the backbone network. In the training phase, the classification layer of the standard ResNet-50 model is trained using cross entropy loss to improve classification accuracy; the feature layer is trained using triplet loss to enhance the ability of feature representation. In the inference phase, for each target in the suspicious target and confirmed list, the image feature extraction module extracts its image features to form features to be confirmed and confirmed features.

[0068] Step 33: Calculate the cosine similarity between the feature to be confirmed and the confirmed feature to determine the image similarity.

[0069] Use cosine similarity as a metric to calculate the similarity between the feature to be confirmed and the confirmed feature. The value range of cosine similarity is [-1,1]. The closer the value of the pre-similarity is to 1, the more similar the two features are.

[0070] Step 34: Process the suspicious target and the identified person target frame in the confirmed list based on the position feature extraction module, and obtain the coordinates to be processed and the processed coordinates using the center point of the suspicious target and the geometric center of the identified person target frame as coordinate points.

[0071] The location feature extraction module extracts the location information of each target in the suspicious target and confirmed list, specifically taking the geometric center of the target box as the coordinate point.

[0072] Get the pending coordinates of the suspicious target and the processed coordinates of each target in the confirmed list.

[0073] Step 35: Calculate the Euclidean distance between the coordinate point to be processed and the processed coordinate point.

[0074] Using Euclidean distance as a metric, calculate the distance between the coordinate to be processed and each processed coordinate point. The smaller the Euclidean distance, the closer the two targets are in spatial position.

[0075] Step 36: When the image similarity is greater than a preset image threshold and the Euclidean distance is less than a preset distance threshold, it is determined that the suspicious target has been recorded in the confirmed list.

[0076] An image similarity threshold and a Euclidean distance threshold are preset as the basis for judging whether the suspicious target has been confirmed.

[0077] When the cosine similarity between the feature to be confirmed and a confirmed feature is greater than the image similarity threshold, and the Euclidean distance between the coordinate to be processed and the processed coordinate point is less than the distance threshold, the suspicious target is considered to be the same target as the confirmed target, that is, the suspicious target has been recorded in the confirmed list.

[0078] In a specific example, a patrol drone in a chemical park found a suspicious person smoking (illegal behavior) during the patrol process, and the person was initially identified as a suspicious target. Then, the identity recognition module started working:

[0079] The image feature extraction module uses the ReID deep learning model to extract feature vectors from the image of the suspicious target and compares them with each target in the confirmed list. By calculating the cosine similarity, it is found that the suspicious target has a high similarity with a target in the confirmed list (for example, the similarity is 0.95, which is greater than the preset image similarity threshold of 0.9).

[0080] At the same time, the location feature extraction module extracts the location information of the suspicious target and compares it with the targets in the confirmed list. By calculating the Euclidean distance, it is found that the suspicious target is very close to a target in the confirmed list in spatial position (for example, the distance is 5 pixels, which is less than the preset distance threshold of 10 pixels).

[0081] Based on the judgment results of image similarity and Euclidean distance, it can be determined that the suspicious target is actually a previously confirmed target. Therefore, the drone will no longer perform repeated detection and processing on the target, thereby improving the inspection efficiency and accuracy.

[0082] Step 4: If not, mark the suspicious target as a target to be confirmed, and record the target to be confirmed into a preset list to be confirmed.

[0083] Targets to be confirmed: refers to those targets that are initially identified as having violated regulations but have not yet been confirmed by the identity recognition module and require further detection and processing.

[0084] Pending confirmation list: A list used to store pending confirmation targets and their related information for easy subsequent management and tracking.

[0085] Referring to step 3, if the identity recognition module fails to confirm that the suspicious target matches any record in the confirmed list, this step is executed. The unconfirmed suspicious target is marked as a pending target, indicating that it requires further detection and processing, and the relevant information of the pending target (such as image, location, detection time, etc.) is recorded in the preset pending list for subsequent management and tracking.

[0086] In another specific example, take the inspection of the substation by the power inspection drone as an example. When the drone detects a person not wearing a helmet (violation) during the inspection, the person is initially identified as a suspicious target. Subsequently, the identity recognition module begins to compare the suspicious target with the records in the confirmed list, but fails to find a match. Based on this result, the drone determines that the suspicious target is a newly discovered target and needs to further confirm its identity and behavior. Therefore, the drone marks the suspicious target as a target to be confirmed and records its image, location, detection time and other related information in the preset list to be confirmed. Subsequently, the drone will perform secondary inspection and processing on the target to be confirmed according to the preset process to confirm whether it has violated the rules. If the secondary inspection confirms that the target does have a violation, it will be moved from the list to be confirmed to the confirmed list, and will be processed and recorded accordingly.

[0087] Step 5: Adjust the inspection drone to aim at the target to be confirmed, so as to conduct a secondary inspection on the target to be confirmed for violations, and record the target to be confirmed in the confirmed list if there is any violation in the target to be confirmed.

[0088] Secondary detection: A re-detection of a target that is initially identified as suspicious and not recorded in the confirmed list to confirm whether it has any violations.

[0089] Confirmed list: A list of targets that have been confirmed to have violations through secondary detection and their related information.

[0090] Step 51: query the pending confirmation list to check whether there is a pending confirmation target.

[0091] Before starting the secondary detection, first query the pending confirmation list to confirm whether there is a pending confirmation target. If the list is empty, no subsequent operations are required; if the list is not empty, continue to perform subsequent steps.

[0092] Step 52: If there is a target to be confirmed, calculate an offset based on the center of the image captured by the inspection drone and the center of the target to be confirmed.

[0093] When there is a target in the list to be confirmed, the camera module of the inspection drone needs to be adjusted to aim at the target to be confirmed. First, the offset between the center of the image taken by the drone and the center of the target to be confirmed is calculated to determine the direction and distance that the camera module needs to be adjusted.

[0094] Step 53: Adjust the camera module of the inspection drone based on the offset so that the center of the target to be confirmed is at the center of the camera module.

[0095] According to the offset calculated in step 52, the camera module of the inspection drone is adjusted so that the center of the target to be confirmed is at the center of the camera module. This step ensures that the target can appear clearly in the camera image during the secondary inspection.

[0096] Step 54 : adjusting the focal length value of the camera module to a preset pixel value to obtain a secondary image frame of the target to be confirmed.

[0097] After the camera module is aligned with the target to be confirmed, its focal length is adjusted to a preset pixel value to obtain a higher definition secondary image frame. This step helps to improve the accuracy of secondary detection.

[0098] Step 55: Process the inspection image based on the deep learning model to identify whether there is any violation in the secondary image frame.

[0099] The secondary image frames are processed using the deep learning model from step 2 to identify any violations. This step is similar to the initial detection, but for higher resolution image frames.

[0100] Step 56: If yes, record the target to be confirmed into the confirmed list.

[0101] If the secondary inspection confirms that the target to be confirmed has violated the rules, it will be recorded in the confirmed list. At this point, the target has been confirmed as a violator and can be processed and managed accordingly as needed.

[0102] In addition, the present application also includes the following method: when there is no target to be confirmed, the camera module is adjusted to a preset initial posture. If the list to be confirmed is empty, that is, there is no target to be confirmed, in order to avoid unnecessary energy consumption and wear, the camera module of the inspection drone is adjusted back to the preset initial posture and waits for the next inspection task.

[0103] In a specific example, a large petrochemical enterprise uses drones for daily safety inspections to monitor violations within the plant, such as smoking and not wearing a helmet. During one inspection, the drone identified a person suspected of smoking. The suspicious target was then sent to the identity recognition module for comparison, but it was not confirmed that the person had been recorded in the confirmed list, so it was marked as a target to be confirmed and recorded in the pending confirmation list.

[0104] Next, the drone executes step 5 for secondary detection. First, the list of objects to be confirmed is queried to confirm the existence of the object to be confirmed. Then, the drone calculates the offset between the center of the current camera image and the center of the object to be confirmed, and adjusts the camera module according to the calculation result so that the center of the object to be confirmed is aligned with the center of the camera image. Then, the drone adjusts the focal length of the camera module to a preset pixel value to obtain a higher-definition secondary image frame.

[0105] After acquiring the secondary image frame, the drone uses the deep learning model in step 2 to process it and identify again whether the target has violated the rules. After analysis, it is confirmed that the target is indeed smoking.

[0106] Finally, the drone records the information of the target to be confirmed (including identity information, violation type, time and location of occurrence, and secondary inspection images, etc.) to the confirmed list for subsequent tracking and management. At this point, the secondary inspection process is completed, and the drone can continue to perform other inspection tasks or further handle the violation according to the company's safety regulations.

[0107] If the list to be confirmed is empty, the drone will adjust the camera module back to the initial posture and wait for the next inspection task.

[0108] The above is an embodiment of the method proposed in this application. Based on the same inventive concept, the embodiment of this application also provides an automatic identification and reconfirmation device for illegal behavior, the structure of which is as follows: Figure 2 shown.

[0109] Figure 2 A schematic diagram of the internal structure of a device for automatically identifying and reconfirming illegal behavior provided in an embodiment of the present application. Figure 2 As shown, the device includes:

[0110] at least one processor 201;

[0111] and, a memory 202 communicatively connected to the at least one processor;

[0112] The memory 202 stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor 201 to enable at least one processor 201 to:

[0113] Based on the inspection drone, the preset inspection area is inspected to obtain the inspection picture; wherein the inspection picture is composed of the inspection video frames; based on the preset deep learning model, the inspection picture is processed to identify whether there are any violations in the inspection picture, and the persons with violations are marked as suspicious targets; wherein the deep learning model includes the YoLov8 model and the ResNet-18 model; based on the preset identity recognition module, the suspicious targets and the preset confirmed list are processed to determine whether the suspicious targets have been recorded in the confirmed list; if not, the suspicious targets are marked as targets to be confirmed, and the targets to be confirmed are recorded in the preset list to be confirmed; the inspection drone is adjusted to aim at the targets to be confirmed, so as to conduct secondary detection of the violations on the targets to be confirmed, and the targets to be confirmed are recorded in the confirmed list when the targets to be confirmed have violations.

[0114] Some embodiments of the present application provide corresponding Figure 1 A non-volatile computer storage medium for automatic identification and reconfirmation of illegal behaviors, storing computer executable instructions, wherein the computer executable instructions are set to:

[0115] Based on the inspection drone, the preset inspection area is inspected to obtain the inspection picture; wherein the inspection picture is composed of the inspection video frames; based on the preset deep learning model, the inspection picture is processed to identify whether there are any violations in the inspection picture, and the persons with violations are marked as suspicious targets; wherein the deep learning model includes the YoLov8 model and the ResNet-18 model; based on the preset identity recognition module, the suspicious targets and the preset confirmed list are processed to determine whether the suspicious targets have been recorded in the confirmed list; if not, the suspicious targets are marked as targets to be confirmed, and the targets to be confirmed are recorded in the preset list to be confirmed; the inspection drone is adjusted to aim at the targets to be confirmed, so as to conduct secondary detection of the violations on the targets to be confirmed, and the targets to be confirmed are recorded in the confirmed list when the targets to be confirmed have violations.

[0116] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the IoT device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0117] The system and medium provided in the embodiments of the present application correspond one-to-one to the method. Therefore, the system and medium also have similar beneficial technical effects to the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the system and medium will not be repeated here.

[0118] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0119] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0120] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0121] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0122] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0123] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0124] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0125] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0126] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for automatic identification and reconfirmation of illegal behavior, used in an inspection drone inspection system, characterized in that: The method comprises: Based on the inspection drone inspecting the preset inspection area, an inspection picture is obtained; wherein the inspection picture is composed of inspection video frames; Processing the inspection screen based on a preset deep learning model to identify whether there are any violations in the inspection screen, and marking the persons with violations as suspicious targets; wherein the deep learning model includes an improved YoLov8 model and an improved ResNet-18 model; Processing the suspicious target and the preset confirmed list based on a preset identity recognition module to determine whether the suspicious target has been recorded in the confirmed list; If not, the suspicious target is marked as a target to be confirmed, and the target to be confirmed is recorded in a preset list to be confirmed; The inspection drone is adjusted to aim at the target to be confirmed, so as to conduct a secondary inspection on the target to be confirmed for violations, and when the target to be confirmed has violations, the target to be confirmed is recorded in the confirmed list.

2. The method for automatic identification and reconfirmation of illegal behavior according to claim 1, characterized in that: The inspection screen is processed based on a preset deep learning model to identify whether there is any violation in the inspection screen, and the person who has violated the rules is marked as a suspicious target, specifically including: Acquire the inspection video frame of the inspection picture; Processing the inspection video frame based on the improved YoLov8 model to identify the person in the inspection video frame and mark them as identified persons, and generating an identified person target frame; Processing the identified person target frame according to a preset interception ratio and a preset interception rule to determine a key target frame; Processing the key target frame based on the improved ResNet-18 model to mark at least one operation behavior of the key target frame; wherein the operation behavior includes a normal behavior and at least one illegal behavior; Mark the persons in the key target box who have committed the said violations as suspicious targets; The key target frame with illegal behavior, the inspection video frame, the identified person target frame and the operation behavior are combined to output a video frame detection result.

3. The method for automatic identification and reconfirmation of illegal behavior according to claim 2, characterized in that: Before processing the key target frame based on the improved ResNet-18 model to mark at least one operation behavior of the key target frame, the method further includes: Build a standard ResNet-18 model; Setting the categories of the standard ResNet-18 model to normal behavior and violation behavior set; wherein the violation behavior set includes at least one violation behavior; Substitute the preset loss function wn-BCEloss into the standard ResNet-18 model to construct an improved ResNet-18 model.

4. The method for automatic identification and reconfirmation of illegal behavior according to claim 3 is characterized in that: The loss function wn-BCEloss is as follows: Where N is the number of batches, B is the number of samples in a batch, and w(n ij ) is the weighting function, n ij is the number of sample labels in a batch, α is the weighting coefficient one, β is the weighting coefficient two, is the binary cross entropy loss.

5. The method for automatic identification and reconfirmation of illegal behavior according to claim 2, characterized in that: Processing the suspicious target and the preset confirmed list based on the preset identity recognition module to determine whether the suspicious target has been recorded in the confirmed list specifically includes: The identity recognition module includes an image feature extraction module and a position feature extraction module; Processing the suspicious target and the video frame detection results in the confirmed list based on the image feature extraction module to obtain features to be confirmed and confirmed features; Calculating the cosine similarity between the feature to be confirmed and the confirmed feature to determine the image similarity; Processing the suspicious target and the identified person target frame in the confirmed list based on the position feature extraction module, obtaining the coordinates to be processed and the processed coordinates with the center point of the suspicious target and the geometric center of the identified person target frame as coordinate points; Calculate the Euclidean distance between the coordinate point to be processed and the processed coordinate point; When the image similarity is greater than a preset image threshold and the Euclidean distance is less than a preset distance threshold, it is determined that the suspicious target has been recorded in the confirmed list.

6. The method for automatic identification and reconfirmation of illegal behavior according to claim 5, characterized in that: Before processing the suspicious target and the video frame detection results in the confirmed list based on the image feature extraction module to obtain the features to be confirmed and the confirmed features, the method further includes: Based on the preset standard ResNet-50 model as the backbone network, the classification layer in the standard ResNet-50 model is constructed using the cross entropy loss, and the feature layer in the standard ResNet-50 model is constructed using the triplet loss to construct a ReID deep learning module; The ReID deep learning module is used as the image feature extraction module.

7. The method for automatic identification and reconfirmation of illegal behavior according to claim 1, characterized in that: The inspection drone is adjusted to aim at the target to be confirmed, so as to conduct a secondary inspection on the target to be confirmed for any violation, and when the target to be confirmed has any violation, the target to be confirmed is recorded in the confirmed list, specifically including: Query the pending confirmation list to check whether there is a pending confirmation target; If there is a target to be confirmed, an offset is calculated based on the center of the image captured by the inspection drone and the center of the target to be confirmed; Adjusting the camera module of the inspection drone based on the offset so that the center of the target to be confirmed is at the center of the camera module; Adjusting the focal length value of the camera module to a preset pixel value to obtain a secondary image frame of the target to be confirmed; Processing the inspection picture based on the deep learning model to identify whether there is any violation in the secondary image frame; If so, record the target to be confirmed into the confirmed list.

8. The method for automatic identification and reconfirmation of illegal behavior according to claim 7, characterized in that: The method further comprises: When there is no target to be confirmed, the camera module is adjusted to a preset initial posture.

9. An automatic identification and reconfirmation device for illegal behavior, characterized in that: The device comprises: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: Based on the inspection drone inspecting the preset inspection area, an inspection picture is obtained; wherein the inspection picture is composed of inspection video frames; Processing the inspection screen based on a preset deep learning model to identify whether there are any violations in the inspection screen, and marking the persons with violations as suspicious targets; wherein the deep learning model includes a YoLov8 model and a ResNet-18 model; Processing the suspicious target and the preset confirmed list based on a preset identity recognition module to determine whether the suspicious target has been recorded in the confirmed list; If not, the suspicious target is marked as a target to be confirmed, and the target to be confirmed is recorded in a preset list to be confirmed; The inspection drone is adjusted to aim at the target to be confirmed, so as to conduct a secondary inspection on the target to be confirmed for violations, and when the target to be confirmed has violations, the target to be confirmed is recorded in the confirmed list.

10. A non-volatile computer storage medium for automatic identification and reconfirmation of illegal behavior, storing computer executable instructions, characterized in that: The computer executable instructions are configured to: Based on the inspection drone inspecting the preset inspection area, an inspection picture is obtained; wherein the inspection picture is composed of inspection video frames; Processing the inspection screen based on a preset deep learning model to identify whether there are any violations in the inspection screen, and marking the persons with violations as suspicious targets; wherein the deep learning model includes a YoLov8 model and a ResNet-18 model; Processing the suspicious target and the preset confirmed list based on a preset identity recognition module to determine whether the suspicious target has been recorded in the confirmed list; If not, the suspicious target is marked as a target to be confirmed, and the target to be confirmed is recorded in a preset list to be confirmed; The inspection drone is adjusted to aim at the target to be confirmed, so as to conduct a secondary inspection on the target to be confirmed for violations, and when the target to be confirmed has violations, the target to be confirmed is recorded in the confirmed list.