Methods, systems, and equipment for detecting abnormal coal transportation scenarios using vehicle-mounted cameras

Abstract

This invention provides a method, system, and device for detecting abnormal coal transportation scenes using a vehicle-mounted camera. The method includes the following steps: Step 0, establishing an abnormal scene detection network model, which includes CBR module I and CBR module II, etc.; Step 1, reading the image to be tested, which is a coal transportation scene image captured in real time by the vehicle-mounted camera; inputting the image to be tested into CBR module I for downsampling to extract and transform the information to the channel dimension to obtain the original feature map; Step 2, inputting the original feature map into CBR module II for downsampling to increase the number of channels to 64, and then using CSP module I to extract the ultra-large feature map γ0, etc. Therefore, this invention can effectively remove redundant features such as distant personnel and enhance the features of target personnel or target excavators that occupy a certain pixel proportion, thereby improving the accuracy of preprocessing and reducing the amount of output feature data.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and more specifically, to a method, system, and device for detecting abnormal coal transportation scenarios using a vehicle-mounted camera. Background Technology

[0002] With the development of deep learning, object detection has made significant progress. Deep learning-based object detection algorithms are mainly divided into two types: two-stage and one-stage algorithms. Typical two-stage algorithms include R-CNN, Fast R-CNN, Faster R-CNN, and Mask R-CNN. Two-stage algorithms divide object detection into two stages: detection and recognition. First, the algorithm or network finds the region of interest in the image; then, the target in the region is recognized. This type of algorithm has high detection accuracy but is relatively slow. One-stage algorithms, such as the YOLO series, SSD, and CenterNet series, abandon region selection algorithms and use bounding box regression to simultaneously complete detection and recognition, achieving end-to-end detection and recognition. This type of algorithm has lower detection accuracy but is faster.

[0003] It should be noted that coal mines generally use large trailer trucks for long-distance transportation, and coal theft often occurs during transportation. Therefore, the safety of long-distance truck transportation is a thorny issue in the industry, but traditional manual monitoring has the problems of being difficult to implement and having high maintenance costs.

[0004] However, existing object detection algorithms are mainly based on manually designed feature extractors to extract features, and are used in conjunction with classifier algorithms such as SVM. They are not good at handling occlusion and scale changes. In addition, in order to detect all targets and reduce the false negative rate, general object detection algorithms usually extract features for large, medium and small targets and use classifiers, which is not suitable for detecting abnormal scenes in the process of coal transportation by trucks.

[0005] Furthermore, considering that abnormal coal transportation scenario detection algorithms need to be deployed on edge devices, but many algorithms with the above performance use complex network structures, it is difficult to deploy them on edge devices.

[0006] In order to solve the above problems, people have been seeking an ideal technological solution. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies by providing a method, system, and device for detecting abnormal coal transportation scenarios using vehicle-mounted cameras.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] The first aspect of this invention provides a method for detecting abnormal coal transportation scenarios using a vehicle-mounted camera, comprising the following steps:

[0010] Step 0: Establish an abnormal scene detection network model, which includes CBR module I, CBR module II, CBR module III, CBR module IV, CBR module V, CSP module I, CSP module II, CSP module III, CSP module IV, CSP module V, CSP module VI, an upsampling module, and an SPPF module.

[0011] Step 1: Read the image to be tested, which is a coal transportation scene image captured in real time by the vehicle-mounted camera.

[0012] The image to be tested is input into the CBR module I for downsampling, so as to extract and transform the information to the channel dimension to obtain an original feature map of size 320×320×32;

[0013] Step 2: Input the original feature map into the CBR module II for downsampling, increase the number of channels to 64, and then use the CSP module I to extract a super-large feature map γ0 of size 160×160×64;

[0014] Step 3: Use the CBR module Ⅲ to downsample the ultra-large feature map γ0, increase the number of channels to 128, and then use the CSP module Ⅱ to extract the large target feature map γ1 with a size of 80×80×128;

[0015] Step 4: Use the CBR module IV to downsample the large target feature map γ1, increasing the number of channels to 256, and then use the CSP module III to extract the medium target feature map γ2 with a size of 40×40×256.

[0016] Step 5: Optimize the target feature map γ2 using the SPPF module to obtain the target feature map γ3;

[0017] Step 6: Use the CSP module IV to map the target feature map γ3 to obtain a mirror target feature map γ2' with a size of 40×40×256; After obtaining the mirror target feature map γ2', use the CSP module V to fuse the target feature map γ2 from step 4 and the mirror target feature map γ2' to obtain a target feature map α1 with a size of 40×40×256;

[0018] The channel is compressed using the CBR module V, and then the upsampling module is used to upsample the target feature map α1 by 2 times to obtain the mirror large target feature map γ1'. After obtaining the mirror large target feature map γ1', the large target feature map γ1 from step 3 and the mirror large target feature map γ1' are fused using the CSP module VI to obtain a large target feature map α0 with a size of 80×80×128.

[0019] A second aspect of the present invention provides an abnormal coal transportation scene detection system for vehicle-mounted cameras, comprising an initialization module and a scene detection module. The initialization module is used to pre-establish an abnormal scene detection network model, which includes CBR module I, CBR module II, CBR module III, CBR module IV, CBR module V, CSP module I, CSP module II, CSP module III, CSP module IV, CSP module V, CSP module VI, an upsampling module, and an SPPF module.

[0020] The scene detection module includes an original feature generation unit, a super-large feature generation unit, a large target feature generation unit, a medium target feature generation unit, a feature optimization unit, a first mirror feature generation unit, a first target feature fusion unit, a second mirror feature generation unit, and a second target feature fusion unit.

[0021] The original feature generation unit is used to read a test image of size 640×640×3, which is a coal transportation scene image captured in real time by a vehicle-mounted camera; it is also used to input the test image into the CBR module I for downsampling, so as to extract and transform the information to the channel dimension to obtain an original feature map of size 320×320×32.

[0022] The ultra-large feature generation unit is used to input the original feature map into the CBR module II for downsampling, increase the number of channels to 64, and then use the CSP module I to extract an ultra-large feature map γ0 of size 160×160×64.

[0023] The large target feature generation unit is used to downsample the ultra-large feature map γ0 using the CBR module Ⅲ, increasing the number of channels to 128, and then using the CSP module Ⅱ to extract a large target feature map γ1 of size 80×80×128.

[0024] The medium target feature generation unit is used to downsample the large target feature map γ1 using the CBR module IV to increase the number of channels to 256, and then use the CSP module III to extract the medium target feature map γ2 with a size of 40×40×256.

[0025] The feature optimization unit is used to optimize the target feature map γ2 using the SPPF module to obtain the target feature map γ3;

[0026] The first mirror feature generation unit is used to map the target feature map γ3 using the CSP module Ⅳ to obtain a mirror target feature map γ2' with a size of 40×40×256;

[0027] The first target feature fusion unit is used to fuse the central target feature map γ2 and the mirrored central target feature map γ2' in step 4 using the CSP module V to obtain a central target feature map α1 with a size of 40×40×256;

[0028] The second mirror feature generation unit is used to compress the channel using the CBR module V, and then use the upsampling module to upsample the middle target feature map α1 by 2 times to obtain the mirror large target feature map γ1'.

[0029] The second target feature fusion unit is used to fuse the large target feature map γ1 and the mirrored large target feature map γ1' in step 3 using the CSP module VI after obtaining the mirrored large target feature map γ1', to obtain a large target feature map α0 with a size of 80×80×128.

[0030] A third aspect of the present invention provides an abnormal coal transportation scene detection device for a vehicle-mounted camera, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the abnormal coal transportation scene detection method for a vehicle-mounted camera as described above.

[0031] A fourth aspect of the present invention provides a readable storage medium having instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method for detecting abnormal coal transportation scenarios using a vehicle-mounted camera.

[0032] This invention has outstanding substantive features and significant progress compared to the prior art, specifically:

[0033] 1) This invention provides an abnormal coal transportation scene detection for vehicle-mounted cameras. By establishing an abnormal scene detection network model, the test image is preprocessed to obtain a medium target feature map α1 with a size of 40×40×256 and a large target feature map α0 with a size of 80×80×128, which prepares for the next step of abnormal coal transportation scene detection.

[0034] This abnormal scene detection network model is a network structure that strengthens large and medium targets and suppresses small targets. It can effectively remove redundant features such as people in the distance and enhance the features of target people or target excavators that occupy a certain percentage of pixels, thereby improving the accuracy of preprocessing and reducing the amount of output feature data.

[0035] 2) Based on the generated target feature map α1 and large target feature map α0, this invention uses a detection module to detect both suspicious personnel and excavators, thereby improving the detection accuracy of abnormal coal transportation scenarios, reducing false alarm rate, and effectively preventing coal theft.

[0036] 3) This invention is an intelligent monitoring algorithm for detecting abnormal coal transportation scenarios. The network layers of the abnormal scenario detection network model all adopt general network layers. While maintaining basically the same performance, the amount of computation is greatly reduced, thereby shortening the detection time of abnormal coal transportation scenarios and improving the timeliness of abnormal coal transportation scenario detection.

[0037] 4) This invention also utilizes a queue caching mechanism to provide a reliable alarm algorithm for abnormal coal transportation scenarios, which can provide real-time alarms after detecting abnormal coal mine scenarios, ensuring the reliability of alarms and avoiding the problem of continuous alarms. Attached Figure Description

[0038] Figure 1 This is a flowchart illustrating the abnormal coal transportation scene detection method for vehicle-mounted cameras according to the present invention.

[0039] Figure 2 This is a schematic diagram of the abnormal scene detection network model of the present invention;

[0040] Figure 3 This is a schematic diagram of the abnormal coal transportation scene detection system for vehicle-mounted cameras according to the present invention;

[0041] Figure 4 This is a schematic diagram of the CSP module of the present invention;

[0042] Figure 5 This is a schematic diagram of the detection module of the present invention;

[0043] Figure 6 This is a schematic diagram of a confusion matrix in a specific embodiment of the present invention. Detailed Implementation

[0044] The technical solution of the present invention will be further described in detail below through specific embodiments.

[0045] CSP (Conv-BN-ReLU) is typically a module obtained by stacking convolution, batch normalization, and ReLU activation functions, which will not be elaborated on here.

[0046] CSP (Cross-Stage-Partial-connections) refers to cross-stage partial connection modules.

[0047] SPPF (Spatial Pyramid Pooling – Fast) refers to the fast spatial pyramid pooling structure, which is an improved version of the SPP (Spatial Pyramid Pooling) module. The SPPF module borrows the idea of ​​spatial pyramids to achieve the fusion of local and global features, enriching the expressive power of feature maps. It is beneficial for situations where the target size in the image to be detected varies greatly, thus greatly improving the detection accuracy.

[0048] Example 1

[0049] As attached Figure 1 and 2 As shown, a method for detecting abnormal coal transportation scenarios using a vehicle-mounted camera includes the following steps:

[0050] Step 0: Establish an abnormal scene detection network model, which includes CBR module I, CBR module II, CBR module III, CBR module IV, CBR module V, CSP module I, CSP module II, CSP module III, CSP module IV, CSP module V, CSP module VI, an upsampling module, and an SPPF module.

[0051] Among them, the CBR module I, CBR module II, CBR module III, CBR module IV and CBR module V have the same structure, but different numbers of output channels. Their main functions are to adjust the number of feature channels and extract features. Specifically, the stride of CBR module I is 2, and it includes a convolutional layer, a batch normalization layer and a ReLU activation layer.

[0052] The CSP modules I, II, III, IV, V, and VI have the same structure, but differ in the number of channels. Their specific structures are as follows: Figure 4 As shown, the value of N is different in each CSP module: N is 1 in CSP module I, 2 in CSP module II, 3 in CSP module III, 1 in CSP module IV, 1 in CSP module V, and 1 in CSP module VI. The five CBR modules in each CSP module include a convolutional layer, a batch normalization layer, and a ReLU activation layer. The concat module is used to perform the operation of stacking two features.

[0053] Step 1: Read the image to be tested, which is a coal transportation scene image captured in real time by the vehicle-mounted camera.

[0054] The image to be tested is input into the CBR module I for downsampling. The CBR module I uses 32 convolutional kernels to extract information and transform it to the channel dimension to obtain an original feature map of size 320×320×32.

[0055] Step 2: Input the original feature map into the CBR module II for downsampling, increase the number of channels to 64, and then use the CSP module I to extract a super-large feature map γ0 of size 160×160×64;

[0056] Step 3: Use the CBR module Ⅲ to downsample the ultra-large feature map γ0, increase the number of channels to 128, and then use the CSP module Ⅱ to extract the large target feature map γ1 with a size of 80×80×128;

[0057] Step 4: Use the CBR module IV to downsample the large target feature map γ1, increasing the number of channels to 256, and then use the CSP module III to extract the medium target feature map γ2 with a size of 40×40×256.

[0058] Step 5: Optimize the target feature map γ2 using the SPPF module to obtain the target feature map γ3;

[0059] Step 6: Use the CSP module IV to map the target feature map γ3 to obtain a mirror target feature map γ2' with a size of 40×40×256; After obtaining the mirror target feature map γ2', use the CSP module V to fuse the target feature map γ2 from step 4 and the mirror target feature map γ2' to obtain a target feature map α1 with a size of 40×40×256;

[0060] The channel is compressed using the CBR module V, and then the upsampling module is used to upsample the target feature map α1 by 2 times to obtain the mirror large target feature map γ1'. After obtaining the mirror large target feature map γ1', the large target feature map γ1 from step 3 and the mirror large target feature map γ1' are fused using the CSP module VI to obtain a large target feature map α0 with a size of 80×80×128.

[0061] It is understandable that, in order to ensure the consistency between the features and the original image, the target feature map γ3 was reversed in step 6. For details, please refer to step 6, which will not be repeated here.

[0062] Furthermore, the method for detecting abnormal coal transportation scenes using vehicle-mounted cameras also includes step 7:

[0063] The medium target feature map α1 and the large target feature map α0 are sent to the detection module to obtain the detection results of the abnormal scene; wherein, the detection results include suspicious person alarm events or excavator alarm events.

[0064] The detection module includes two convolutional modules, two reshape modules (changing feature dimensions), and an NMS (Non-maximum suppression) module. (The structure is attached.) Figure 5 As shown; wherein, the intermediate target feature map α1 and the large target feature map α0 are respectively adjusted to channel dimensions through a convolution Conv to obtain the intermediate feature α 1mid and intermediate features α 0mid ;

[0065] Intermediate feature α 1mid and intermediate features α 0mid The intermediate feature α is obtained by flattening the dimensions using a reshape module. 1out and intermediate features α 0out Specifically, see attached. Figure 5 In the figure, 4800 and 19200 represent the number of candidate bounding boxes, respectively.

[0066] Intermediate feature α 1out and intermediate features α 0out After passing through the NMS module (non-maximum suppression module), the NMS module outputs [x,y,w,h,obj,cls], where (x,y) are the coordinates of the target center point, (w,h) are the width and height of the target bounding box, obj is the confidence score, and cls is the category (including target personnel and target excavator, represented by different values);

[0067] Understandably, for each image to be tested, the detection module selects M target boxes, each corresponding to a target person or a target excavator, and an array [x,y,w,h,obj,cls], where cls indicates whether the target box contains a target person or a target excavator. The number M is determined by preset IOU thresholds and confidence thresholds, such as an IOU threshold of 0.4 and a confidence threshold of 0.6. If the detection module outputs M equal to 0, it means that there are no target people or target excavators in the image to be tested; if the detection module outputs M > 0, it means that there are target people or target excavators in the image to be tested.

[0068] It should be noted that the NMS algorithm in the NMS module is a general object detection post-processing algorithm, which will not be elaborated on here.

[0069] Furthermore, in step 5, when optimizing the target feature map γ2 using the SPPF module to obtain the target feature map γ3, the following steps are performed:

[0070] Perform 1×1 max pooling on the target feature map γ2 to generate target feature I;

[0071] Perform 5×5 max pooling on the target feature map γ2 to generate target feature II;

[0072] Perform 9×9 max pooling on the target feature map γ2 to generate target feature Ⅲ;

[0073] Perform 13×13 max pooling on the target feature map γ2 to generate target feature N;

[0074] The four pooling results of target feature I, target feature II, target feature III and target feature IV are stacked and then fused by convolution to obtain target feature map γ3.

[0075] It is understandable that after obtaining the mid-target feature map γ2 with a size of 40×40×128, the SPPF module is used to optimize the mid-target feature map γ2 to obtain the target feature map γ3. The SPPF module includes four pooling sizes: 1×1, 5×5, 9×9, and 13×13. The four pooling results are combined together through standard convolution fusion, which increases the feature resolution and thus improves the accuracy of abnormal coal mine scene detection.

[0076] It should be noted that the abnormal coal transportation scenario in this embodiment refers to the coal theft scenario during the coal transportation process by trucks, which is usually carried out by personnel operating excavators. Regarding the coal theft problem during the coal transportation process by trucks, the personnel who can be alerted are those who are close to the cargo bin, but personnel at a distance should not be mistakenly identified as suspicious persons.

[0077] In the acquired image information, the target personnel or target excavators occupy a relatively large proportion of pixels, belonging to medium or large targets. The target scene uses vehicle-mounted cameras to detect personnel or excavators around the vehicle. Therefore, there are no small targets in the abnormal coal transportation scene. In order to reduce the false alarm rate, personnel who are far away and occupy a small proportion in the image should not be alarmed.

[0078] To address the issue that existing general-purpose target detection algorithms typically extract features for large, medium, and small targets and use them as classifiers, which are not suitable for detecting abnormal coal transportation scenarios in this invention, this invention is based on YOLOv5 and has been pruned and optimized to make it suitable for detecting abnormal coal transportation scenarios. Furthermore, an abnormal scene detection network model that strengthens large and medium targets and suppresses small targets is proposed.

[0079] In addition, considering that the abnormal scene detection network model needs to be deployed on edge devices, the network layer in the abnormal scene detection network model proposed in this invention adopts a general network layer, which greatly reduces the amount of computation while maintaining basically the same performance.

[0080] Example 2

[0081] Based on Example 1, in order to ensure the reliability of the alarm and avoid the problem of continuous alarms, this example adds a queue caching mechanism to maintain an alarm queue of length N, and then provides another specific implementation method for detecting abnormal coal transportation scenarios using vehicle-mounted cameras.

[0082] Specifically, the method for detecting abnormal coal transportation scenarios using vehicle-mounted cameras further includes step 8:

[0083] When the detection results include a suspicious person alarm event or an excavator alarm event, push 1 into the pre-set alarm queue; when the detection results do not include a person alarm event or an excavator alarm event, push 0 into the pre-set alarm queue.

[0084] Calculate the proportion θ of 1s in the alarm queue, where the proportion θ = number of alarms ÷ N; where the number of alarms is dynamically changing and equal to the total number of 1s in the alarm queue, and N is the preset alarm queue length.

[0085] Determine whether the percentage θ is greater than the threshold thr.

[0086] If so, an abnormal scenario alarm command is generated, and the alarm queue is cleared;

[0087] Otherwise, proceed to step 7.

[0088] Specifically, the threshold thr includes, but is not limited to, 0.6, 0.7, 0.8, 0.9, etc., and can be adjusted adaptively according to the actual situation in practical applications.

[0089] It should be noted that the number of images to be tested is greater than or equal to 1. The vehicle-mounted camera collects a series of coal transportation scene images in real time, sorted by the collection time. One image to be tested corresponds to one detection result. A detection result of 0 indicates that there is no alarm event, and a detection result of 1 indicates that there is a suspicious person alarm event or an excavator alarm event.

[0090] For example, N=10, threshold thr=0.6; alarm queue is [0,0,0,1,1,1,1,1,1,1], at this time the proportion θ=number of alarms÷N=7÷10=0.7>th threshold thr, an abnormal scene alarm command is generated, and the alarm queue is cleared.

[0091] It should be noted that the alarm queue length N is a constant and can be preset as needed, such as 10. The longer the alarm queue length, the less sensitive the algorithm will be, and the shorter the alarm queue length, the more sensitive the algorithm will be. It can be set according to actual needs.

[0092] Example 3

[0093] Based on the above embodiments, this embodiment provides a specific implementation of an abnormal coal transportation scene detection system for vehicle-mounted cameras, as shown in the attached figure. Figure 3 As shown;

[0094] Specifically, the abnormal coal transportation scene detection system for vehicle-mounted cameras includes an initialization module and a scene detection module. The initialization module is used to pre-establish an abnormal scene detection network model, which includes CBR module I, CBR module II, CBR module III, CBR module IV, CBR module V, CSP module I, CSP module II, CSP module III, CSP module IV, CSP module V, CSP module VI, an upsampling module, and an SPPF module.

[0095] The scene detection module includes an original feature generation unit, a super-large feature generation unit, a large target feature generation unit, a medium target feature generation unit, a feature optimization unit, a first mirror feature generation unit, a first target feature fusion unit, a second mirror feature generation unit, and a second target feature fusion unit.

[0096] The original feature generation unit is used to read a test image of size 640×640×3, which is a coal transportation scene image captured in real time by a vehicle-mounted camera; it is also used to input the test image into the CBR module I for downsampling, so as to extract and transform the information to the channel dimension to obtain an original feature map of size 320×320×32.

[0097] The ultra-large feature generation unit is used to input the original feature map into the CBR module II for downsampling, increase the number of channels to 64, and then use the CSP module I to extract an ultra-large feature map γ0 of size 160×160×64.

[0098] The large target feature generation unit is used to downsample the ultra-large feature map γ0 using the CBR module Ⅲ, increasing the number of channels to 128, and then using the CSP module Ⅱ to extract a large target feature map γ1 of size 80×80×128.

[0099] The medium target feature generation unit is used to downsample the large target feature map γ1 using the CBR module IV to increase the number of channels to 256, and then use the CSP module III to extract the medium target feature map γ2 with a size of 40×40×256.

[0100] The feature optimization unit is used to optimize the target feature map γ2 using the SPPF module to obtain the target feature map γ3;

[0101] The first mirror feature generation unit is used to map the target feature map γ3 using the CSP module Ⅳ to obtain a mirror target feature map γ2' with a size of 40×40×256;

[0102] The first target feature fusion unit is used to fuse the central target feature map γ2 and the mirrored central target feature map γ2' in step 4 using the CSP module V to obtain a central target feature map α1 with a size of 40×40×256;

[0103] The second mirror feature generation unit is used to compress the channel using the CBR module V, and then use the upsampling module to upsample the middle target feature map α1 by 2 times to obtain the mirror large target feature map γ1'.

[0104] The second target feature fusion unit is used to fuse the large target feature map γ1 and the mirrored large target feature map γ1' in step 3 using the CSP module VI after obtaining the mirrored large target feature map γ1', to obtain a large target feature map α0 with a size of 80×80×128.

[0105] Furthermore, the abnormal coal transportation scene detection system for vehicle-mounted cameras also includes a detection result generation unit, which is used for:

[0106] The medium target feature map α1 and the large target feature map α0 are sent to the detection module to obtain the detection results of the abnormal scene; wherein, the detection results include suspicious person alarm events or excavator alarm events.

[0107] Furthermore, the abnormal coal transportation scene detection system for vehicle-mounted cameras also includes an alarm module, which is used for:

[0108] When the detection results include a suspicious person alarm event or an excavator alarm event, push 1 into the pre-set alarm queue; when the detection results do not include a person alarm event or an excavator alarm event, push 0 into the pre-set alarm queue.

[0109] Calculate the proportion θ of 1s in the alarm queue, where the proportion θ = number of alarms ÷ N; where the number of alarms is dynamically changing and equal to the total number of 1s in the alarm queue, and N is the preset alarm queue length.

[0110] Determine whether the percentage θ is greater than the threshold thr.

[0111] If so, an abnormal scenario alarm command is generated, and the alarm queue is cleared.

[0112] Specifically, when the feature optimization unit uses the SPPF module to optimize the target feature map γ2 to obtain the target feature map γ3, it performs the following:

[0113] Perform 1×1 max pooling on the target feature map γ2 to generate target feature I;

[0114] Perform 5×5 max pooling on the target feature map γ2 to generate target feature II;

[0115] Perform 9×9 max pooling on the target feature map γ2 to generate target feature Ⅲ;

[0116] Perform 13×13 max pooling on the target feature map γ2 to generate target feature N;

[0117] The four pooling results of target feature I, target feature II, target feature III and target feature IV are stacked and then fused by convolution to obtain target feature map γ3.

[0118] It should be noted that this embodiment was conducted on the open-source dataset MOCS, with a total of 23,405 images, including 13 categories of samples such as people and excavators. The ratio of the training set to the test set was 8:2. The confusion matrix obtained from the test is shown in the attached figure. Figure 6 As shown; by appendix Figure 6 Yes, and it can be seen that the detection accuracy of excavator and suspicious person reached 0.99 and 0.91 respectively. Therefore, the accuracy of the abnormal scene detection network model established by this invention reaches about 90%, which can meet the actual needs. Moreover, it is simple to implement and very convenient to deploy on embedded devices.

[0119] This embodiment also conducted experiments on the RKNN3399pro deep learning board. The input network image size was 640×640×3. The running speed of the abnormal coal transportation scene detection method for vehicle-mounted cameras was tested, and the result was "once runuse 61.996000 ms". It can be seen that the event of detecting one image is about 62ms, or about 16fps, which meets the real-time requirements and can be applied in practice.

[0120] Example 4

[0121] Based on the above embodiments, this embodiment provides a specific implementation of an abnormal coal transportation scene detection device for a vehicle-mounted camera, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the abnormal coal transportation scene detection method for a vehicle-mounted camera as described in Embodiment 1 or Embodiment 2.

[0122] This embodiment also provides a readable storage medium storing instructions that, when executed by a processor, implement the steps of the abnormal coal transportation scene detection device for a vehicle-mounted camera as described in Embodiment 1 or Embodiment 2.

[0123] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0124] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of the modules described above is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0125] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0126] If the integrated modules described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them; although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications can still be made to the specific implementation of the present invention or equivalent substitutions can be made to some technical features without departing from the spirit of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the scope of the technical solutions claimed in the present invention.

Claims

1. A method for detecting abnormal coal transportation scenes using a vehicle-mounted camera, characterized in that, Includes the following steps: Step 0: Establish an abnormal scene detection network model, which includes CBR module I, CBR module II, CBR module III, CBR module IV, CBR module V, CSP module I, CSP module II, CSP module III, CSP module IV, CSP module V, CSP module VI, an upsampling module, and an SPPF module. Step 1: Read the image to be tested, which is a coal transportation scene image captured in real time by the vehicle-mounted camera. The image to be tested is input into the CBR module I for downsampling, so as to extract and transform the information to the channel dimension to obtain an original feature map of size 320×320×32; Step 2: Input the original feature map into the CBR module II for downsampling, increase the number of channels to 64, and then use the CSP module I to extract a super-large feature map γ0 of size 160×160×64; Step 3: Use the CBR module Ⅲ to downsample the ultra-large feature map γ0, increase the number of channels to 128, and then use the CSP module Ⅱ to extract the large target feature map γ1 with a size of 80×80×128; Step 4: Use the CBR module IV to downsample the large target feature map γ1, increasing the number of channels to 256, and then use the CSP module III to extract the medium target feature map γ2 with a size of 40×40×256. Step 5: Optimize the target feature map γ2 using the SPPF module to obtain the target feature map γ3; Step 6: Use the CSP module IV to map the target feature map γ3 to obtain a mirror target feature map γ2' with a size of 40×40×256; After obtaining the mirror target feature map γ2', use the CSP module V to fuse the target feature map γ2 from step 4 and the mirror target feature map γ2' to obtain a target feature map α1 with a size of 40×40×256; The channel is compressed using the CBR module V, and then the upsampling module is used to upsample the target feature map α1 by 2 times to obtain the mirror large target feature map γ1'. After obtaining the mirror large target feature map γ1', the large target feature map γ1 from step 3 and the mirror large target feature map γ1' are fused using the CSP module VI to obtain a large target feature map α0 with a size of 80×80×128.

2. The method for detecting abnormal coal transportation scenes using a vehicle-mounted camera according to claim 1, characterized in that, It also includes step 7: The medium target feature map α1 and the large target feature map α0 are sent to the detection module to obtain the detection results of the abnormal scene; wherein, the detection results include suspicious person alarm events or excavator alarm events.

3. The method for detecting abnormal coal transportation scenes using a vehicle-mounted camera according to claim 2, characterized in that, It also includes step 8: When the detection results include a suspicious person alarm event or an excavator alarm event, push 1 into the pre-set alarm queue; when the detection results do not include a person alarm event or an excavator alarm event, push 0 into the pre-set alarm queue. Calculate the proportion θ of 1s in the alarm queue, where the proportion θ = number of alarms ÷ N; where the number of alarms is dynamically changing and equal to the total number of 1s in the alarm queue, and N is the preset alarm queue length. Determine whether the percentage θ is greater than the threshold thr. If so, an abnormal scenario alarm command is generated, and the alarm queue is cleared; Otherwise, proceed to step 7.

4. The method for detecting abnormal coal transportation scenes using a vehicle-mounted camera according to claim 1, characterized in that, In step 5, when optimizing the target feature map γ2 using the SPPF module to obtain the target feature map γ3, the following steps are performed: Perform 1×1 max pooling on the target feature map γ2 to generate target feature I; Perform 5×5 max pooling on the target feature map γ2 to generate target feature II; Perform 9×9 max pooling on the target feature map γ2 to generate target feature Ⅲ; Perform 13×13 max pooling on the target feature map γ2 to generate target feature N; The four pooling results of target feature I, target feature II, target feature III and target feature IV are stacked and then fused by convolution to obtain target feature map γ3.

5. An abnormal coal transportation scene detection system for vehicle-mounted cameras, characterized in that: It includes an initialization module and a scene detection module. The initialization module is used to pre-establish an abnormal scene detection network model. The abnormal scene detection network model includes CBR module I, CBR module II, CBR module III, CBR module IV, CBR module V, CSP module I, CSP module II, CSP module III, CSP module IV, CSP module V, CSP module VI, an upsampling module, and an SPPF module. The scene detection module includes an original feature generation unit, a super-large feature generation unit, a large target feature generation unit, a medium target feature generation unit, a feature optimization unit, a first mirror feature generation unit, a first target feature fusion unit, a second mirror feature generation unit, and a second target feature fusion unit. The original feature generation unit is used to read a test image of size 640×640×3, which is a coal transportation scene image captured in real time by a vehicle-mounted camera; it is also used to input the test image into the CBR module I for downsampling, so as to extract and transform the information to the channel dimension to obtain an original feature map of size 320×320×32. The ultra-large feature generation unit is used to input the original feature map into the CBR module II for downsampling, increase the number of channels to 64, and then use the CSP module I to extract an ultra-large feature map γ0 of size 160×160×64. The large target feature generation unit is used to downsample the ultra-large feature map γ0 using the CBR module Ⅲ, increasing the number of channels to 128, and then using the CSP module Ⅱ to extract a large target feature map γ1 of size 80×80×128. The medium target feature generation unit is used to downsample the large target feature map γ1 using the CBR module IV to increase the number of channels to 256, and then use the CSP module III to extract the medium target feature map γ2 with a size of 40×40×256. The feature optimization unit is used to optimize the target feature map γ2 using the SPPF module to obtain the target feature map γ3; The first mirror feature generation unit is used to map the target feature map γ3 using the CSP module Ⅳ to obtain a mirror target feature map γ2' with a size of 40×40×256; The first target feature fusion unit is used to fuse the central target feature map γ2 and the mirrored central target feature map γ2' in step 4 using the CSP module V to obtain a central target feature map α1 with a size of 40×40×256; The second mirror feature generation unit is used to compress the channel using the CBR module V, and then use the upsampling module to upsample the middle target feature map α1 by 2 times to obtain the mirror large target feature map γ1'. The second target feature fusion unit is used to fuse the large target feature map γ1 and the mirrored large target feature map γ1' in step 3 using the CSP module VI after obtaining the mirrored large target feature map γ1', to obtain a large target feature map α0 with a size of 80×80×128.

6. The abnormal coal transportation scene detection system for vehicle-mounted cameras according to claim 5, characterized in that, It also includes a detection result generation unit, which is used for: The medium target feature map α1 and the large target feature map α0 are sent to the detection module to obtain the detection results of the abnormal scene; wherein, the detection results include suspicious person alarm events or excavator alarm events.

7. The abnormal coal transportation scene detection system for vehicle-mounted cameras according to claim 6, characterized in that, It also includes an alarm module, which is used for: When the detection results include a suspicious person alarm event or an excavator alarm event, push 1 into the pre-set alarm queue; when the detection results do not include a person alarm event or an excavator alarm event, push 0 into the pre-set alarm queue. Calculate the proportion θ of 1s in the alarm queue, where the proportion θ = number of alarms ÷ N; where the number of alarms is dynamically changing and equal to the total number of 1s in the alarm queue, and N is the preset alarm queue length. Determine whether the percentage θ is greater than the threshold thr. If so, an abnormal scenario alarm command is generated, and the alarm queue is cleared.

8. The abnormal coal transportation scene detection system for vehicle-mounted cameras according to claim 5, characterized in that, When the feature optimization unit optimizes the target feature map γ2 using the SPPF module to obtain the target feature map γ3, it performs the following: Perform 1×1 max pooling on the target feature map γ2 to generate target feature I; Perform 5×5 max pooling on the target feature map γ2 to generate target feature II; Perform 9×9 max pooling on the target feature map γ2 to generate target feature Ⅲ; Perform 13×13 max pooling on the target feature map γ2 to generate target feature N; The four pooling results of target feature I, target feature II, target feature III and target feature IV are stacked and then fused by convolution to obtain target feature map γ3.

9. An abnormal coal transportation scene detection device for vehicle-mounted cameras, characterized in that: The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the abnormal coal transportation scene detection method for a vehicle-mounted camera as described in any one of claims 1 to 4.

10. A readable storage medium having instructions stored thereon, characterized in that: When executed by the processor, this instruction implements the steps of the abnormal coal transportation scene detection method for vehicle-mounted cameras as described in any one of claims 1 to 4.

Images