Dangerous area intrusion rapid detection method based on multi-scale feature fusion

Abstract

The invention discloses a dangerous area intrusion rapid detection method based on multi-scale feature fusion, and belongs to the technical field of computer vision, and the method comprises the following steps: firstly, collecting a video and an image in which a normal behavior and a violation behavior coexist; preprocessing, screening and marking the collected video images, and dividing the collected video images into a training set, a verification set and a test set; inputting the training set into the multi-scale feature fusion convolutional network for training, verifying the validity of the algorithm in the iteration process by the verification set until the network converges, and obtaining a weight file; and then detecting a dangerous area entering behavior on a test set and in a real-time scene by using the weight file, selecting a detection result, and transmitting and displaying the result in a front-end interface in real time. According to the method, the problems of low scene adaptability and low efficiency of identifying the dangerous area invaded by the worker in the real scene are solved, and the safety and reliability of the worker in the production process are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a method for rapid detection of dangerous area intrusion based on multi-scale feature fusion. Background Art

[0002] With the advent of the Industrial 4.0 era, smart manufacturing and automation technologies are becoming increasingly mature, and industrial safety has become one of the key factors for the sustainable development of enterprises. In modern factories and other production spaces, the technology of detecting workers intruding into dangerous areas is particularly important. This technology can effectively prevent safety accidents caused by people mistakenly entering dangerous areas and ensure the safety and efficiency of the production environment.

[0003] The rapid development of deep learning technology has led to the widespread use of algorithms such as convolutional neural networks (CNN) in image recognition and behavior analysis. By learning a large number of worker operations and dangerous area intrusion scenes, the system can automatically identify and classify different human behavior patterns, including normal operations, mistaken entry into dangerous areas, etc., thereby achieving efficient and accurate detection. Once the system detects that a worker is about to or has entered a dangerous area, it will immediately trigger an intelligent early warning mechanism to prevent potential safety accidents from occurring in a timely manner through sound and light alarms, sending alarm information to safety managers, and automatically shutting down related machines or equipment. At the same time, the system can also be linked with other factory safety systems (such as access control systems, fire protection systems, etc.) to build a comprehensive safety protection system.

[0004] In order to detect workers who have strayed into dangerous areas, Zhu Penghao et al. (Zhu Penghao, Li Jun, Zhang Shiyi, et al. Dynamic dangerous area intrusion detection method based on deep learning [J]. China Safety Science and Technology, 2024, 20(07): 170-178.) designed a dynamic dangerous area personnel intrusion detection method based on deep learning. This method first uses binocular vision technology to complete the two tasks of heavy object lifting judgment and overhead crane hoisting dynamic dangerous area detection. Then, the proposed PDConv lightweight convolution module and convolution attention mechanism module are used to improve the YOLOv5 basic model to complete the detection of personnel intrusion in the dangerous area below the hoisting height; Li Da et al. People (Li Da. Research on port dangerous area intrusion detection algorithm based on deep learning [D]. Yanshan University, 2023. DOI: 10.27440 / d.cnki.gysdu.2023.002007.) Based on YOLOv4, the feature fusion part of the network model was rebuilt, downsampling connections and jump connections were added, and the Transformer basic module was introduced. The dangerous area to be detected was circled on the basis of placing triangular cones as anchor points, and the detection of unrelated personnel entering the dangerous area during port operations was completed; Zhang Yixiang et al. (Zhang Yixiang, Lin Song, Li Xue. Based on CenterNet-GhostNet et al. Detection of personnel in dangerous areas of coal preparation plants [J]. Industrial and Mining Automation, 2022, 48(04): 66-71. DOI: 10.13272 / j.issn.1671-251x.2021080058.) proposed a model CenterNet-GhostNet for detecting personnel in dangerous areas of coal preparation plants. The model uses the CenterNet network as the basic framework and combines the lightweight network GhostNet with the feature pyramid as a feature extraction network to improve the network feature expression ability and detection accuracy. However, although these existing methods have completed the recognition and detection of personnel entering dangerous areas, they are not adaptable to the scene and the processing speed cannot keep up with the real-time requirements. Many dangerous occurrences require timely response and alarm. Summary of the invention

[0005] The present invention provides a method for rapid detection of intrusion into dangerous areas based on multi-scale feature fusion. On the basis of a multi-scale feature fusion convolutional neural network, it combines a fast traditional algorithm, has good adaptability to different scenarios of behaviors of mistakenly entering dangerous areas, has a high accuracy rate, and can respond to dangerous behaviors in real time, effectively preventing safety accidents caused by human mistakenly entering dangerous areas.

[0006] To achieve the above objectives, the present invention adopts the following technical solutions:

[0007] A rapid detection method for dangerous area intrusion based on multi-scale feature fusion includes the following steps:

[0008] Step 1: Image collection process of intrusion into dangerous areas: First, place isolation poles around the determined simulated dangerous area to form a circle. Then, use industrial cameras placed around the factory to record the simulated behavior of workers entering and leaving the dangerous area, forming videos and images of normal behavior and illegal behavior.

[0009] Step 2: Dataset preparation process: The collected videos and images are screened, preprocessed, and labeled according to the categories of people and isolation poles. Data augmentation is performed using various methods such as rotation and light and dark changes. The data is randomly divided and a random number generator is used to allocate the data into training sets, validation sets, and test sets, with allocation ratios of 70%, 10%, and 20%;

[0010] Step 3: Algorithm training and debugging process: Input the training set into the multi-scale feature fusion convolutional neural network for training, and use the validation set to verify the effectiveness of the algorithm during the iteration process until the network converges and the weight file is obtained;

[0011] Step 4: Test the algorithm detection effect process: Use the trained weight file to detect the behavior target of workers entering the dangerous area on the test set and in the real-time scenario, select the detection results, and transmit the results in real time to the front-end interface for display.

[0012] In the above steps, step 1 specifically includes the following steps:

[0013] Step 1-1: Install industrial cameras at a height of 6 to 8 meters around the factory walls to ensure full coverage of dangerous areas;

[0014] Step 1-2: Multiple workers simulate entering and leaving dangerous areas in the factory. To improve the training effect of the model, normal and illegal behaviors need to appear in the same camera image at the same time.

[0015] Step 1-3: Collect images of factory violations at multiple time periods, including morning, noon, afternoon, and evening, to improve the model's adaptability to various lighting conditions.

[0016] Step 2 specifically includes the following steps:

[0017] Step 2-1: Pre-process the captured video and convert it into a picture set by taking one frame out of every five frames, from which clearer pictures are selected;

[0018] Step 2-2: Use image enhancement algorithms on the dataset, including random illumination enhancement, random rotation, random color jitter, and random noise;

[0019] Step 2-3: Label the dataset images according to the two categories of people and isolation rods, and divide them into training set, validation set and test set in the ratio of 8:1:1.

[0020] Step 3 specifically includes the following steps:

[0021] Step 3-1: Input the divided training set and validation set images into the multi-scale feature fusion neural network for training; set the loss function to be the sum of coordinate loss, target confidence loss and classification loss. Coordinate loss is used to measure the coordinate error between the predicted box and the true box, using CIOU Loss as the function. Target confidence loss is used to measure the confidence of whether the predicted box contains the target, using the binary cross entropy loss function. Classification loss is used to measure the confidence of the target category in the predicted box, also using the binary cross entropy loss function, so that the loss function continues to converge during the training iteration process;

[0022] Step 3-2: Observe the experimental results during training. If the loss function value does not converge or increase, reduce the learning rate and continue training. Observe the size of the video memory. If the conditions are met, try to increase the batch size.

[0023] Step 3-3: During the complete neural network process, the effect will be tested based on the validation set, and the model weights of the best and most recent neural network will be saved.

[0024] Step 4 specifically includes the following steps:

[0025] Step 4-1: Use the weight data obtained through training to complete the detection of people and isolation poles in the test set and real scenes;

[0026] Step 4-2: Obtain the center point of the isolation rod detection frame through the coordinates of the isolation rod detection frame, and take the average of the coordinates of the center points of all isolation rods to obtain the center point position of all points. Based on this point, sort all the isolation rod center points in a clockwise order to form a list;

[0027] Step 4-3: Obtain the position of the person's center point through the coordinates of the person detection frame, and use the PointPolygonTest algorithm to determine whether the person's center point is within the danger zone surrounded by the isolation rod center point list. If it is a positive value or 0, the worker is in the danger zone; if it is a negative value, the worker is away from the danger zone.

[0028] Step 4-2 includes detailed processing for errors in determining dangerous areas caused by isolation poles being blocked by pedestrians: by memorizing the number and position of isolation poles in 20 consecutive frames of the video, if the isolation pole status in the current frame is different from the isolation pole status in most of the 20 consecutive frames, the isolation pole status in most of the 20 consecutive frames is still used as the determination of the dangerous area. At the same time, the memorized status of the first frame of the 20 consecutive frames is removed, and the isolation pole status in the current frame is added to the end of the 20 consecutive frames to prevent the isolation poles from being actually moved or evacuated, thereby completing the update of the isolation pole status.

[0029] Beneficial effects: The present invention proposes a method for rapid detection of intrusion into dangerous areas based on multi-scale feature fusion. On the basis of a multi-scale feature fusion convolutional neural network, it combines a fast traditional algorithm to complete the real-time detection of workers intruding into dangerous areas marked by isolation poles and isolation belts, thereby improving the scene adaptability and efficiency of such violation detection. It has good adaptability to different scenarios of misentering dangerous areas, has a high accuracy rate, and can respond to dangerous behaviors in real time, effectively preventing safety accidents caused by human misentering dangerous areas, and improving the safety and efficiency of the production environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is an overall flow chart of an embodiment of the present invention.

[0031] Figure 2 This is a flow chart of step 1 of an embodiment of the present invention.

[0032] Figure 3 This is a flow chart of step 2 of an embodiment of the present invention.

[0033] Figure 4 This is a flow chart of step 3 of an embodiment of the present invention.

[0034] Figure 5 This is a flow chart of step 4 of an embodiment of the present invention.

[0035] Figure 6 This is a diagram of a multi-scale convolutional feature fusion module in an embodiment of the present invention.

[0036] Figure 7 This is a diagram showing the detection results of the method for rapid detection of intrusion into dangerous areas in an embodiment of the present invention. Specific implementation methods

[0038] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments:

[0039] like Figure 1 As shown, a method for rapid detection of dangerous area intrusion based on multi-scale feature fusion includes the following steps:

[0040] like Figure 2 As shown, step 1 is the process of collecting images of intrusion behavior in dangerous areas:

[0041] Step 101: Install industrial cameras at a height of 6 to 8 meters around the factory walls so that the industrial cameras can fully cover the dangerous areas;

[0042] Step 102: Multiple workers simulate entering and leaving dangerous areas in the factory. To improve the training effect of the model, normal behaviors and illegal behaviors need to appear simultaneously in the same camera image.

[0043] Step 103: Collect images of factory violations at multiple time periods, including morning, noon, afternoon and evening, to improve the adaptability of the model to various lighting conditions.

[0044] like Figure 3 As shown, step 2 dataset preparation process:

[0045] Step 201: pre-process the captured video, convert it into a picture set by taking one frame out of every five frames, and select clearer pictures from them;

[0046] Step 202: applying image enhancement algorithms to the data set, including random illumination enhancement, random rotation, random color jitter, and random noise;

[0047] Step 203: Label the dataset images according to the two categories of people and isolation rods, use rotation, light and dark changes and other methods to augment the data, and use a random number generator to allocate the data into training sets, validation sets and test sets using a random partitioning method, with allocation ratios of 70%, 10% and 20%.

[0048] like Figure 4 As shown, step 3 algorithm training and debugging process:

[0049] Step 301: Input the divided training set and validation set images into Figure 6 The multi-scale feature fusion neural network shown in the figure is trained. The multi-scale of the multi-scale feature fusion convolutional network is realized by convolution kernels of different sizes, including 1x1, 5x5 and 9x9. In order to reduce the amount of calculation caused by the large convolution kernel, the large convolution kernel is decomposed into a 3x3 stack by increasing the depth. During the training process, the original image will go through multiple stacks such as Figure 6The module shown in the figure performs feature extraction, and then sends it to the loss function to calculate the loss value by comparing with the label. The loss function is set to the sum of coordinate loss, target confidence loss and classification loss. The coordinate loss is used to measure the coordinate error between the predicted box and the real box. CIOU Loss is used as the function. The target confidence loss is used to measure the confidence of whether the predicted box contains the target. The binary cross entropy loss function is used. The classification loss is used to measure the confidence of the target category in the predicted box. The binary cross entropy loss function is also used to make the loss function converge continuously during the training iteration process.

[0050] LOSS=cls_BCE(y,y(x))+obj_BCE(y,y(x))+loc_CIoU(y,y(x))

[0051] Among them, cls_BCE is the binary cross entropy loss for classification, obj_BCE is the binary cross entropy loss for target confidence, loc_CIoU is the intersection-over-union loss for the positive sample bounding box, y is the true value, and y(x) is the predicted value.

[0052] Step 302: Observe the experimental results during the training process. If the loss function value does not converge or increase, reduce the learning rate and continue training. Observe the size of the video memory. If the conditions are met, increase the batch size as much as possible.

[0053] Step 303: During the neural network completion process, the model weights of the best and most recent neural network will be saved based on the test results of the validation set. The saved function expression is as follows:

[0054] Best ckpt =Compare(New ckpt ,Best ckpt )

[0055] Last ckpt =New ckpt

[0056] The Compare function is a comparison function. By comparing the loss values ​​of two weight parameters on the validation set, the smaller one is saved as the best weight parameter. ckpt To train the latest weight parameters, Best ck pt is the best weight parameter in history, Last cpkt is the latest weight parameter.

[0057] like Figure 5 As shown, step 4 tests the algorithm detection effect process:

[0058] Step 401: Using the weight data obtained through training, complete the detection of people and isolation poles in the test set and the real scene;

[0059] Step 402: Obtain the center point position of the isolation rod detection frame through the coordinates of the isolation rod detection frame, and take the average of the coordinates of all isolation rod center points to obtain the center point position of all points, and sort all isolation rod center points in a clockwise order according to the point to form a list;

[0060] Step 403: The position of the center point of the person is obtained through the coordinates of the person detection frame, and the PointPolygonTest algorithm is used to determine whether the center point of the person is in the danger zone surrounded by the list of isolation rod center points. If it is a positive value or 0, the worker is in the danger zone. If it is a negative value, the worker is far away from the danger zone. The specific function expression is as follows:

[0061] Safecolumn_List=Sort(points)

[0062] Safeornot=PointPolygonTest(Person,Safecolumn_List)

[0063] The Sort function arranges the points of the isolation pole into a list in clockwise order, the PointPolygonTest function determines whether the coordinates of a person's point are within the area enclosed by the points in the isolation pole list, points refers to the unprocessed coordinates of the isolation pole points, Safecolumn_List represents the points of the isolation pole that have been arranged in a clockwise order, Person represents the coordinates of the person, and Safeornot represents whether the person is safe.

[0064] Figure 7 This is the detection result of factory workers entering the isolated dangerous area. All isolation poles and personnel are detected, and if a person is in the dangerous area, Danger is displayed. If not, it is displayed normally as a detected Person. The processing time of one frame of image is 9ms, which meets the real-time requirements.

[0065] The above-described embodiments are only for illustrating the preferred implementation modes of the present invention, and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solutions in accordance with the technical ideas proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A rapid detection method for dangerous area intrusion based on multi-scale feature fusion, characterized in that: The following steps are involved: Step 1: Image collection process of intrusion into dangerous areas: First, place isolation poles around the determined simulated dangerous area to form a circle, and record the staff's simulated behavior of entering and leaving the dangerous area, forming a video and image of normal behavior and illegal behavior; Step 2: Dataset preparation process: The collected videos and images are screened, preprocessed, and labeled according to the categories of people and isolation poles, and data augmentation is performed, and the data is divided into training set, validation set, and test set; Step 3: Algorithm training and debugging process: Input the training set into the multi-scale feature fusion convolutional neural network for training, and use the validation set to verify the effectiveness of the algorithm during the iteration process until the network converges and the weight file is obtained; Step 4: Test the algorithm detection effect process: Use the trained weight file to detect the behavior target of workers entering the dangerous area on the test set and in the real-time scenario, select the detection results, and transmit the results in real time to the front-end interface for display.

2. The method for rapid detection of dangerous area intrusion based on multi-scale feature fusion according to claim 1 is characterized in that: Step 1 specifically includes the following steps: Step 1-1: Install industrial cameras around the factory to fully cover the dangerous areas; Step 1-2: Multiple workers simulate entering and leaving dangerous areas in the factory. To improve the training effect of the model, normal and illegal behaviors need to appear in the same camera image at the same time. Step 1-3: Collect images of factory violations at multiple time periods, including morning, noon, afternoon, and evening, to improve the model's adaptability to various lighting conditions.

3. The method for rapid detection of dangerous area intrusion based on multi-scale feature fusion according to claim 1 is characterized in that: Step 2 specifically includes the following steps: Step 2-1: Pre-process the captured video and convert it into a picture set by taking one frame out of every five frames, from which clearer pictures are selected; Step 2-2: Use image enhancement algorithm to process the data set; Step 2-3: Label the processed dataset images according to the two categories of people and isolation rods, and use the random division method to allocate them to the training set, validation set and test set, with the allocation ratios of 70%, 10% and 20%.

4. The method for rapid detection of dangerous area intrusion based on multi-scale feature fusion according to claim 3 is characterized in that: The image enhancement algorithm includes random illumination enhancement, random rotation, random color jitter and random noise.

5. The method for rapid detection of dangerous area intrusion based on multi-scale feature fusion according to claim 1 is characterized in that: Step 3 specifically includes the following steps: Step 3-1: Input the divided training set and validation set images into the multi-scale feature fusion neural network for training; Step 3-2: During training, periodically tune the hyperparameters based on the test results of the validation set; Step 3-3: After the neural network has fully converged, obtain the model weights of the neural network and save the function expression as follows: Best ckpt =Compare(New ckpt ,Best ckpt ) Last ckpt =New ckpt The Compare function is a comparison function. By comparing the loss values ​​of two weight parameters on the validation set, the smaller one is saved as the best weight parameter. ckpt To train the latest weight parameters, Best ckpt is the best weight parameter in history, Last cpkt is the latest weight parameter.

6. The method for rapid detection of dangerous area intrusion based on multi-scale feature fusion according to claim 5 is characterized in that: The multi-scale of the multi-scale feature fusion convolutional network is achieved through convolution kernels of different sizes, including 1x1, 5x5 and 9x9.

7. The method for rapid detection of dangerous area intrusion based on multi-scale feature fusion according to claim 5 or 6, characterized in that: During the training process in step 3-1, the original image will be subjected to multiple stacked modules as shown in Figure 6 for feature extraction, and then sent to the loss function for calculation and label comparison to calculate the loss value. The loss function is: LOSS=cls_BCE(y,y(x))+obj_BCE(y,y(x))+loc_CIoU(y,y(x)) Among them, cls_BCE is the binary cross entropy loss for classification, obj_BCE is the binary cross entropy loss for target confidence, loc_CIoU is the intersection-over-union loss for the positive sample bounding box, y is the true value, and y(x) is the predicted value.

8. The method for rapid detection of dangerous area intrusion based on multi-scale feature fusion according to claim 1 is characterized in that: Step 4 specifically includes the following steps: Step 4-1: Use the weight data obtained through training to complete the detection of people and isolation poles in the test set and real scenes; Step 4-2: Obtain the center point of the isolation rod detection frame through the coordinates of the isolation rod detection frame, and take the average of the coordinates of the center points of all isolation rods to obtain the center point position of all points. Based on this point, sort all the isolation rod center points in a clockwise order to form a list; Step 4-3: Get the position of the center point of the person through the coordinates of the person detection frame, and use the PointPolygonTest algorithm to determine whether the center point of the person is in the danger zone surrounded by the center point list of the isolation rods. If it is a positive value or 0, the worker is in the danger zone; if it is a negative value, the worker is away from the danger zone.

9. The method for rapid detection of dangerous area intrusion based on multi-scale feature fusion according to claim 8 is characterized in that: Step 4-2 includes detailed processing for errors in determining dangerous areas caused by barrier poles being blocked by pedestrians: by memorizing the number and position of barrier poles in 20 consecutive frames of the video, if the barrier pole status in the current frame is different from the barrier pole status in most of the 20 consecutive frames, the barrier pole status in most of the 20 consecutive frames is still used as the determination of the dangerous area. At the same time, the memorized status of the first frame of the 20 consecutive frames is removed, and the barrier pole status in the current frame is added to the end of the 20 consecutive frames, so as to prevent the barrier poles from being actually moved or evacuated, thereby completing the update of the barrier pole status.

10. The method for rapid detection of dangerous area intrusion based on multi-scale feature fusion according to claim 8 is characterized in that: In step 4-3, the PointPolygonTest algorithm is used to determine whether the center point of the person is within the danger zone surrounded by the isolation pole center point list. The specific function expression is as follows: Safecolumn_List=Sort(points) Safeornot=PointPolygonTest(Person,Safecolumn_List) The Sort function arranges the points of the isolation pole into a list in clockwise order, the PointPolygonTest function determines whether the coordinates of a person's point are within the area enclosed by the points in the isolation pole list, points refers to the unprocessed coordinates of the isolation pole points, Safecolumn_List represents the points of the isolation pole that have been arranged in a clockwise order, Person represents the coordinates of the person, and Safeornot represents whether the person is safe.

Images