Dust detection method and device, electronic equipment and storage medium

Abstract

The invention provides a dust detection method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring point cloud data of at least two continuous frames of a target area; extracting point cloud features of the point cloud data, and carrying out deformable convolution processing on the point cloud features to obtain a processed feature map; weighting different positions of the processed feature maps based on an attention mechanism, and fusing the weighted feature maps to obtain a fused feature map; and performing dust detection on the target area according to the fused feature map. According to the invention, the complex shape and multi-scale characteristics of the dust can be captured, so that the dust detection precision is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and particularly to a dust detection method, apparatus, electronic device, and storage medium. Background Art

[0002] Dust is suspended in the air in many workplaces (such as mining areas). If the dust concentration exceeds a certain standard, it will cause harm to the human body. Therefore, air dust detection in occupational health assessment is particularly important. With the development of three-dimensional scanning and lidar technology, point cloud processing has become a common data processing method in mine scenes. Point cloud processing involves steps such as point cloud data acquisition, filtering, registration, segmentation, and feature extraction, providing a basis for subsequent dust detection.

[0003] However, the dust in mines may have complex shapes and multi-scale characteristics, ranging from tiny particles to large-scale dust clouds. Current dust detection schemes are difficult to capture this complexity, thus affecting the subsequent dust detection accuracy. Summary of the Invention

[0004] This application provides a dust detection method, apparatus, electronic device, and storage medium. This method can capture the complex shapes and multi-scale characteristics of dust, thereby improving the accuracy of dust detection.

[0005] In a first aspect, a dust detection method is provided, including:

[0006] Obtaining at least two consecutive frames of point cloud data of a target area;

[0007] Extracting point cloud features of the point cloud data, and performing deformable convolution processing on the point cloud features to obtain a processed feature map;

[0008] Based on an attention mechanism, weighting different positions of the processed feature map, and fusing the weighted feature maps to obtain a fused feature map;

[0009] Performing dust detection on the target area according to the fused feature map.

[0010] In a second aspect, a dust detection apparatus is provided, including:

[0011] An obtaining module, configured to obtain at least two consecutive frames of point cloud data of a target area;

[0012] An extracting module, configured to extract point cloud features of the point cloud data;

[0013] A processing module, configured to perform deformable convolution processing on the point cloud features to obtain a processed feature map;

[0014] A fusion module, configured to weight different positions of the processed feature map based on an attention mechanism, and fuse the weighted feature maps to obtain a fused feature map;

[0015] A detection module, configured to perform dust detection on the target area according to the fused feature map.

[0016] In a third aspect, an electronic device is provided, where the electronic device includes:

[0017] A memory, configured to store executable program code;

[0018] A processor, configured to call and run the executable program code from the memory, so that the electronic device executes the dust detection method described in any one of the above.

[0019] In a fourth aspect, a computer-readable storage medium is provided, where the computer-readable storage medium stores a computer program, and when the computer program is executed, the dust detection method described in any one of the above is implemented.

[0020] The beneficial effects brought by the technical solutions provided in some embodiments of the present application at least include: In the present application, deformable convolution processing is performed on the point cloud features corresponding to at least two consecutive frames of point cloud data, and then, based on the attention mechanism, different positions of the processed feature map are weighted, and the weighted feature maps are fused, so that the fused feature map has a more refined feature representation of the target area. In subsequent dust detection of the target area according to the fused feature map, the complex shape and multi-scale features of the dust can be captured, thereby improving the accuracy of dust detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative efforts.

[0022] Figure 1 It is the first flowchart of the dust detection method provided by the embodiment of the present application.

[0023] Figure 2 It is the second flowchart of the dust detection method provided by the embodiment of the present application.

[0024] Figure 3 It is a schematic diagram of processing point cloud data in the dust detection method provided by the embodiment of the present application.

[0025] Figure 4It is another schematic diagram of point cloud data processing in the dust detection method provided by the embodiments of the present application.

[0026] Figure 5 It is a schematic structural diagram of the dust detection device provided by the embodiments of the present application.

[0027] Figure 6 It is the first schematic structural diagram of the electronic device provided by the embodiments of the present application.

[0028] Figure 7 It is the second schematic structural diagram of the electronic device provided by the embodiments of the present application. Detailed implementation manners

[0029] To make the features and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0030] When the following description involves the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all the implementation manners consistent with the present application. On the contrary, they are only examples of the devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0031] The dust in mines may have complex shapes and multi-scale characteristics, from tiny particles to large-scale dust clouds. The current dust detection solutions are difficult to capture this complexity, thus affecting the subsequent dust detection accuracy.

[0032] To solve the technical problems existing in the related art, the embodiments of the present application provide a dust detection method. The following will be described in detail respectively. It should be noted that the description order of the following embodiments does not limit the preferred order of the embodiments.

[0033] Please refer to Figure 1 , Figure 1 It is the first flow schematic diagram of the dust detection method provided by the embodiments of the present application. The specific process of this dust detection method can be as follows:

[0034] S101. Obtain at least two consecutive frames of point cloud data of the target area.

[0035] In this embodiment, a point cloud is a set composed of several discrete, unordered, and topologically unstructured three-dimensional points. It is usually the initial form of data obtained by a three-dimensional sensing system and has advantages such as resistance to illumination and scale changes.

[0036] In the embodiment of the present application, the target area is the area to be dust-detected, such as a certain area in a mine. The point cloud data can be collected in real time or downloaded from the cloud through the network.

[0037] S102. Extract the point cloud features of the point cloud data, and perform deformable convolution processing on the point cloud features to obtain a processed feature map.

[0038] Optionally, the point cloud feature extraction algorithms for extracting the point cloud features of the point cloud data may include methods based on covariance matrices, methods based on neighborhoods, methods based on deep learning, etc. The method based on covariance matrices extracts features by calculating the neighborhood covariance matrix of each point, where the eigenvectors and eigenvalues reflect the curvature and normal vectors of the point cloud surface. The method based on neighborhoods extracts features by analyzing the neighborhoods of the point cloud, such as calculating the distances and angles of the points within the neighborhood. The method based on deep learning extracts the features of the point cloud by training a deep neural network. For example, specifically, a downsampling method can be adopted to convert the point cloud to the key points obtained by downsampling, so as to achieve the purpose of reducing the calculation amount.

[0039] S103. Weight different positions of the processed feature map based on the attention mechanism, and fuse the weighted feature maps to obtain a fused feature map.

[0040] In the present application, an attention mechanism is added so that during subsequent processing, the relationship between each detection position and adjacent positions can be focused on.

[0041] For each point in the processed feature map, deformable convolution is used to generate attention weights. This can be achieved by comparing the relationship between the current point and other points within the neighborhood to determine the weights of the local context. Then, the features within the local neighborhood are weighted and aggregated using the attention weights. This helps to capture the features of the targets and dust points in the point cloud. Specifically, the sampled feature values are multiplied element-wise with the weights of the convolution kernel, and then the results of all multiplications are added together to obtain an output value. Among them, the above operation process is performed at each position of the output feature map.

[0042] S104. Perform dust detection on the target area according to the fused feature map.

[0043] The dust detection method adopted in this application is based on anchor boxes, that is, a plurality of prior boxes with different aspect ratios predefined by the algorithm. For example, specifically, a set of candidate boxes is generated at each position of the fused feature map. These candidate boxes are used to define the positions and sizes of possible objects containing. The sizes and aspect ratios of the candidate boxes are usually set according to the task. Then, for each candidate box, it is determined whether it overlaps with the actual object box and which category it is associated with. Finally, based on the above information and combined with the fused feature map, dust detection is performed on the target area.

[0044] As can be seen from the above, after obtaining the point cloud data of at least two consecutive frames of the target area in the embodiment of this application, the point cloud features of the point cloud data are extracted, and the deformable convolution processing is performed on the point cloud features to obtain the processed feature map. Then, based on the attention mechanism, different positions of the processed feature map are weighted, and the weighted feature maps are fused to obtain the fused feature map. Finally, dust detection is performed on the target area according to the fused feature map. In the dust detection scheme of this application, the deformable convolution processing is performed on the point cloud features corresponding to the point cloud data of at least two consecutive frames, and then, based on the attention mechanism, different positions of the processed feature map are weighted, and the weighted feature maps are fused, so that the fused feature map has a more refined feature representation of the target area. In the subsequent dust detection of the target area according to the fused feature map, the complex shapes and multi-scale features of the dust can be captured, thereby improving the accuracy of dust detection.

[0045] Please refer to Figure 2 , Figure 2 which is the second process schematic diagram of the dust detection method provided by the embodiment of this application. This dust detection method is applied to an electronic device. The specific process of this dust detection method can be as follows:

[0046] S201. Obtain the point cloud data of at least two consecutive frames of the target area.

[0047] In this embodiment, the point cloud data sensors are divided into two categories: active and passive. The active sensors can be further divided into two types based on the TOF (Time of Flight) system and the triangulation system. Among them, the TOF system determines the true distance from the sensor to the object surface by measuring the time interval between the emitted signal reaching the object surface and returning to the receiver; the triangulation system calculates the spatial position of the point through the measurement relationship of the same point of the object by two sensors at different locations. The passive sensors rely on image pairs or image sequences and restore three-dimensional data from two-dimensional image data according to camera parameters. Typical active sensors include LiDAR (Light Detection And Ranging), TOF cameras, structured light sensors, etc.; typical passive sensors include stereo cameras, SFM (structure from motion) systems, SFS (shape from shading) systems, etc.

[0048] In the embodiment of the present application, the target area is the area to be subjected to dust detection, such as a certain area of a mine. The point cloud data can be collected in real time or downloaded from the cloud through the network.

[0049] S202. Extract the point cloud features of the point cloud data, and perform deformable convolution processing on the point cloud features to obtain a processed feature map.

[0050] For large-scale point cloud processing, directly extracting features from the point cloud can better retain the three-dimensional structure information. However, due to the disorder of the point cloud, the direct processing method requires a high computational cost when searching for neighborhoods.

[0051] In some embodiments, a common solution to the above problem is to downsample the point cloud, convert the operation on the point cloud to the key points obtained by downsampling, so as to achieve the purpose of reducing the computational amount. Correspondingly, if the number of point clouds obtained during point cloud surface reconstruction is scarce, an upsampling operation needs to be performed on the point cloud to increase the number of point clouds for better calculation of surface features.

[0052] Optionally, the voxelization method can be used to extract the features of the point cloud data. First, the point cloud data is meshed, also known as voxelization. Each grid after meshing is called a voxel. There are some points in these extremely small grids divided. Then, an average or weighted average is taken for these points to obtain a point to replace all the points in the original grid. The point cloud contained in the voxel is aggregated into a voxel feature representation.

[0053] Specifically, point cloud data is usually represented by three-dimensional coordinates and other attributes (such as reflection intensity (intensity), normal, etc.). The point cloud data representation format of this application is (x, y, z, intensity). Further, the point cloud data is divided into voxels, and each point is assigned to the corresponding voxel. Since the visible range of the point cloud data is large, a certain range of point cloud data can be selected for processing. For example, the point cloud range [0, -40, -3, 70.4, 40, 1] is selected, corresponding to xmin, ymin, zmin, xmax, ymax, zmax respectively, that is, the maximum and minimum values corresponding to x, y, and z. Meshing is mainly applied to the point cloud within the above range, and grid division is carried out according to the grid size. Here, the grid size is [0.16, 0.16, 4], as Figure 3 shown.

[0054] The point cloud data is segmented into multiple pillar units. Each pillar unit is a three-dimensional small cell (H = 440, W = 500) obtained by dividing the point cloud data on the XY plane with a certain step size [0.16, 0.16].

[0055] Further, the ball query method is used to select some point cloud data around the current pillar unit for feature extraction and splice it to the pillar feature part, so that not only each pillar unit has the features of the current position and the local information around the target, but also the rich context information contained when the spliced pillar unit is sent into the backbone network can effectively improve the accuracy of identifying the associated part of the target. Dust may be quite different from the surrounding environment in a local area. Ball query allows the model to aggregate local context features, enabling the model to better distinguish between target and non-target areas. By learning the feature differences between dust points and surrounding points, the subsequent dust detection model can more easily detect dust and improve the dust detection accuracy.

[0056] Specifically, each point cloud in the pillar unit can be encoded into a 9-dimensional vector, such as D: (x, y, z, r, xc, yc, zc, xp, yp).

[0057] Among them, x, y, and z are three-dimensional coordinates, r is the reflection intensity, xc, yc, and zc are the geometric centers of all points in the pillar unit where the point cloud is located, xp = x - xc, yp = y - yc, that is, (xp, yp) represents the relative position of the point to the geometric center. Sampling is performed for each column with more than N = 32 midpoints, and 0 is filled for those with less than N.

[0058] In addition, centering on the current pillar unit position, MLP (Multi-Layer Perceptron) feature extraction is performed on the point cloud data within a certain radius range, and the extracted features are mapped to the same dimension as the current pillar unit features for splicing, as Figure 4 shown.

[0059] In some embodiments, feature extraction can be performed on the point cloud data after tensorization. For example, the original point cloud dimension is D = 9, and the dimension of the point cloud data after tensorization is C = 64, that is, a tensor of (C, P, N) is obtained, where C is the dimension of the point cloud data after tensorization, P is the number of pillar units, and N is the number of points in the pillar unit. Then, a max pooling operation is performed according to the dimension where the pillar unit is located, that is, a feature map of (C, P) dimension is obtained. Finally, a pseudo-image is generated through a preset operator such as a scatter operator. Specifically, the generated (C, P) tensor is converted back to its original pillar coordinates through the index value of the pillar unit of each point to create a pseudo-image with a size of (C = 64, H = 440, W = 500), so as to obtain the dimension after pillar unit feature extraction. Above, the feature extraction of the point cloud data is completed. In the above process, the x and y coordinates corresponding to each pillar unit are recorded, and its dimension is P * 2, where P is the number of pillar units and 2 is the coordinates corresponding to x and y.

[0060] Optionally, after obtaining the point cloud features of the point cloud data, deformable convolution (Deformable ConvNets, DCN) can be performed on the point cloud features to obtain a processed feature map. The traditional convolution operation divides the feature map into parts of the same size as the convolution kernel and then performs convolution operations. The position of each part on the feature map is fixed. For an object with a complex shape change such as dust, the effect of using this convolution is not good. Therefore, in this application, the DCN method is used to process the point cloud features.

[0061] DCN introduces an offset in the receptive field, and this offset is learnable. That is, the receptive field of DCN is not a fixed shape, but is close to the actual shape of the object. In this way, the convolution area always covers around the object shape. When the object deforms, the convolution area can be brought close to the deformed object through the offset. That is, optionally, the step "performing deformable convolution processing on the point cloud features to obtain a processed feature map" may specifically include:

[0062] Generating an offset vector at each position of the point cloud features;

[0063] Calculate the offset position corresponding to each position according to the offset vector;

[0064] Based on the offset position and the point cloud features, obtain the processed feature map.

[0065] It should be noted that DCN learns the offset based on a parallel network, enabling the convolutional kernel to be offset at the sampling points of the feature map, concentrating on the regions or targets of interest. Therefore, an offset vector can be generated for each position of the input point cloud features. The dimension of the offset vector is usually related to the size of the convolutional kernel. For example, if the convolutional kernel is 3x3, a 3-dimensional offset vector is generated for each position. For the offset vector at each position, calculate the corresponding offset position. This is accomplished by adding the offset vector to the coordinates of the original position. When calculating the convolution, use the offset position to sample on the input feature map. For the center position of each convolutional kernel, sample the corresponding feature values according to the offset position, and finally output the processed feature map. That is, optionally, in some embodiments, the step of "based on the offset position and the point cloud features, obtain the processed feature map" may specifically include:

[0066] Sample the point cloud features using the offset position;

[0067] Based on the sampled feature values, output the processed feature map.

[0068] Use bilinear interpolation to calculate the feature information near each sampled point for the offset sampled points. For example, calculate the distance weights between it and its four nearest neighboring pixels. Perform a weighted average on the pixel values of the four nearest neighboring pixels to obtain the pixel value at the offset position. Finally, based on the pixel value at the offset position, output the processed feature map.

[0069] S203. Weight different positions of the processed feature map based on the attention mechanism, and fuse the weighted feature maps to obtain a fused feature map.

[0070] The attention mechanism is the focus on the input weight distribution. The attention mechanism was first used in the encoder-decoder. The attention mechanism obtains the input variable of the next layer by performing a weighted average on the hidden states of all time steps of the encoder. In this application, adding the attention mechanism enables subsequent processing to focus on the relationship between each detection position and its adjacent positions.

[0071] For each point in the processed feature map, deformable convolution is used to generate attention weights. This can be achieved by comparing the relationship between the current point and other points in the neighborhood to determine the weights of the local context. Then, the features within the local neighborhood are weighted and aggregated using the attention weights. This helps capture the features of the targets and dust points in the point cloud. Specifically, the sampled feature values are element-wise multiplied with the weights of the convolution kernel, and then the results of all the multiplications are summed to obtain an output value. This process is performed at each position in the output feature map.

[0072] It should be noted that each point in the output feature obtains a different weight during the weighted aggregation process. The weights can be interpreted as the importance of the points, that is, their contributions in distinguishing targets, noise, dust, etc.

[0073] The physical meaning of the fused feature map is that it adaptively weights the features of the points within the local area to capture more important feature information, and it has the following characteristics:

[0074] Local feature weighting: The output feature is the result of weighted aggregation within the local area. The weighting process takes into account the relationships between points and the local context information, so the output can better reflect the local features of the targets or dust points.

[0075] Adaptive receptive field: The adaptive receptive field of the output feature indicates that the module adaptively adjusts the size and shape of the local receptive field. This helps the module better adapt to targets or dust points of different scales, shapes, and poses.

[0076] Context information: The output feature contains the context information of the points within the local area, enabling the model to better understand the relationship between the local features and the overall point cloud.

[0077] Through DCN, local features, context information, and adaptive receptive fields in the point cloud can be better captured, thereby improving the performance of point cloud object detection and being more robust in identifying abnormal points such as dust.

[0078] S204. Perform dust detection on the target area according to the fused feature map.

[0079] Dust detection is actually a type of object detection. And object detection tasks include those based on anchor boxes and anchor-free boxes. The so-called anchor boxes are multiple prior boxes with different aspect ratios predefined by the algorithm centered on the anchor points. The dust detection method adopted in this application is based on anchor boxes. That is, optionally, in some embodiments, the step of "performing dust detection on the target area according to the fused feature map" may specifically include:

[0080] Generate multiple candidate boxes on the fused feature map;

[0081] Perform dust detection on the target area based on the generated candidate boxes.

[0082] For example, specifically, a set of candidate boxes is generated at each position of the fused feature map. These candidate boxes are used to define the positions and sizes where objects may be contained. The sizes and aspect ratios of the candidate boxes are usually set according to the task. Then, for each candidate box, it is determined whether it overlaps with the actual object box and which category it is associated with. This usually involves calculating the Intersection over Union (IoU) between the anchor box and the object box, as well as the matching of category information.

[0083] It should be noted that when performing the dust detection task, a target may be detected multiple times. Therefore, it is necessary to screen the candidate boxes. Specifically, the non-maximum suppression (NMS) algorithm can be used to remove the candidate boxes with overlapping scales. Different from the traditional NMS that only uses the IOU overlap degree as the threshold for screening overlapping candidate boxes, this solution introduces scale information and can be adaptively adjusted according to the target boxes of different scales. It is possible that the overlap degree between dust and pedestrians is greater than the threshold. Based on the traditional NMS, a redundant box may be removed, but in fact, it should be detected. Therefore, the scale information of different categories is introduced to perform NMS, which can better handle multi-scale targets and retain the candidate boxes with different scales but still effective. That is, optionally, in some embodiments, the step of "performing dust detection on the target area based on the generated candidate boxes" may specifically include:

[0084] Obtain the confidence and scale corresponding to each generated candidate box;

[0085] Perform dust detection on the target area according to the IoU, confidence, and scale between each candidate box and other candidate boxes.

[0086] It should be noted that each candidate box has category prediction, confidence score, and position information. After determining the candidate box, the currently processed candidate box can be determined. Then, calculate the IoU between this candidate box and other candidate boxes, and based on the calculation results of confidence, scale, and position information, determine the retained target boxes among the candidate boxes and other candidate boxes, and return to execute the above steps until all candidate boxes have been traversed. That is, optionally, in some embodiments, the step of "performing dust detection on the target area according to the IoU, confidence, and scale between each candidate box and other candidate boxes" may specifically include:

[0087] Determine the currently processed candidate box as the current processing object;

[0088] Calculate the IoU between the current processing object and each candidate box;

[0089] Candidate boxes with an intersection over union greater than a preset value are determined as overlapping boxes, and based on the confidence and scale of the current processing object, as well as the confidence and scale of the overlapping boxes, the retained target boxes are determined among the current processing object and the overlapping boxes;

[0090] Return to execute the determination of the current processing object until all candidate boxes are traversed;

[0091] Dust detection is performed on the target area according to all the target boxes.

[0092] The specific process is as follows:

[0093] For the currently traversed box, calculate its overlap with all subsequent unprocessed boxes. A overlap threshold can be preset in advance, for example, 0.5. If the overlap between two boxes is greater than or equal to the threshold, then they are considered overlapping. When calculating the overlap, in addition to considering the IoU, the scale difference between the detection boxes also needs to be considered. Usually, a scale difference threshold is introduced. For example, when the scale difference between two boxes exceeds a certain threshold, they are considered boxes of different scales and do not participate in the overlap calculation. If the currently traversed box has an overlap greater than or equal to the threshold with any subsequent unprocessed box and the scale difference is less than the threshold, then the box with the lower confidence is removed from the list to retain the box with the higher confidence.

[0094] Optionally, in some embodiments, the step of "determining the retained target boxes among the current processing object and the overlapping boxes based on the confidence and scale of the current processing object, as well as the confidence and scale of the overlapping boxes" may specifically include:

[0095] Determine the scale difference between the current processing object and the overlapping box according to the scale of the current processing object and the scale of the overlapping box;

[0096] When the scale difference meets the preset conditions, then determine the candidate box with the higher confidence among the current processing object and the overlapping box as the retained target box.

[0097] Finally, the output is the final object detection result, including information such as the object category and the bounding box.

[0098] By comparing the object detection results between the point cloud data of consecutive frames, object tracking and dynamic monitoring can be achieved, and a better understanding of the spread and evolution of dust can be obtained. Thereby improving the dust detection rate in the mining area, and further helping the autonomous driving electronic device to drive normally in harsh environments

[0099] It should be noted that in object detection, the loss function plays a crucial role. The loss function is used to measure the difference between the prediction result and the true label, and provides a gradient signal for the model to update and optimize the parameters. This application utilizes the conventional regression loss Loss_cls, classification loss Loss_b, and confidence loss Loss_obj, and also adds a temporal consistency loss Loss.

[0100] Data representation: Assume that each time step contains a frame of image and the target box information in that frame of image.

[0101] First, use the object matching algorithm to match the objects in consecutive frames to form object correspondence.

[0102] Object box representation: For each frame, the object box can be represented as (x, y, z, l, w, h), where (x, y, z) are the center coordinates of the object box, and l, w, and h are the length, width, and height.

[0103] Calculation of temporal consistency loss: To calculate the temporal consistency loss between consecutive frames, the IoU between object boxes can be used as a metric. The temporal consistency loss can be:

[0104] Temporal consistency loss Loss_t = scaling factor * ∑(1 - IoU(Box_t, Box_{t - 1}))

[0105] Where:

[0106] Box_t is the object box of the current frame.

[0107] Box_{t - 1} is the object box of the previous frame.

[0108] IoU(Box_t, Box_{t - 1}) is to calculate the intersection over union between object boxes.

[0109] The goal of this loss function is to minimize the intersection over union of object boxes between consecutive frames to ensure the consistency and coherence of object boxes over time. The scaling factor can be used to adjust the weight of the loss.

[0110] Add the temporal consistency loss function to the total loss function, calculate the gradient of the model during the model backpropagation process and update it, that is, the total loss function L = Loss_cls + Loss_b + Loss_obj + Loss_t.

[0111] In a mine scenario, the distribution of dust may change over time. To better capture temporal features and maintain consistency, this application designs a temporal consistency loss function. Utilize the point cloud change information between consecutive frames, and encourage the model to better track and identify dust by minimizing the movement or deformation of the point cloud.

[0112] The implementation process of the temporal consistency loss function can be divided into the following steps:

[0113] (1) Point cloud feature map samples of consecutive frames;

[0114] (2) Feature comparison: For the point cloud feature map samples between consecutive frames, compare them through cosine distance. A smaller difference indicates that the point cloud features are more consistent temporally.

[0115] (3) Loss calculation: Use the result of the difference metric as the input of the loss function. Use the mean squared error (MSE), that is, calculate the average of the squares of the feature differences

[0116] (4) Backpropagation and optimization: Take the loss function as the objective function, use the backpropagation algorithm to calculate the gradients and update the model parameters to minimize the temporal consistency loss.

[0117] The dust in the mine may have different scales and sizes, uneven density and the presence of noise, which may cause the model to detect multiple detection boxes overlapping and covering the same dust target. To deal with such different scales, the scale-based non-maximum suppression method (ScaleNMS) of object detection is adopted. ScaleNMS can group and independently process detection boxes of different scales, retain the target information at different scales, and effectively suppress overlapping detection boxes to obtain more accurate and stable object detection results. This method can adapt to the characteristics of different scales and size changes of dust in point cloud data and improve the detection performance of the model.

[0118] The specific steps are as follows:

[0119] (1) Detection box generation: Use the point cloud object detection algorithm to generate a series of candidate detection boxes in the point cloud data. These detection boxes represent the areas that may contain dust, and each detection box has different scales and sizes.

[0120] (2) Grouping by scale: Group the generated detection boxes according to their scales. They can be divided into different scale groups according to features such as the size, volume or density of the detection boxes.

[0121] (3) Perform NMS within each scale group: For each scale group, execute the traditional non-maximum suppression (NMS) algorithm. During the NMS process, for the detection boxes within the current scale group, calculate the IoU (intersection over union) between them, and select the detection box with a higher confidence as the representative target box, suppressing other overlapping detection boxes.

[0122] (4) Combine the results of different scale groups: Combine the detection box results of different scale groups after NMS processing to obtain the final dust detection result.

[0123] The above is the dust detection process of this application.

[0124] As can be seen from the above, after obtaining the point cloud data of at least two consecutive frames in the target area in the embodiment of this application, the point cloud features of the point cloud data are extracted, and the deformable convolution processing is performed on the point cloud features to obtain the processed feature map. Then, based on the attention mechanism, different positions of the processed feature map are weighted, and the weighted feature maps are fused to obtain the fused feature map. Finally, dust detection is performed on the target area according to the fused feature map. In the dust detection scheme of this application, the deformable convolution processing is performed on the point cloud features corresponding to the point cloud data of at least two consecutive frames, and then, based on the attention mechanism, different positions of the processed feature map are weighted, and the weighted feature maps are fused, so that the fused feature map has a more refined feature representation of the target area. In the subsequent dust detection of the target area according to the fused feature map, the complex shape and multi-scale features of the dust can be captured, thereby improving the accuracy of dust detection.

[0125] In addition, the embodiment of this application also provides a dust detection device. Please refer to Figure 5 , Figure 5 which is a schematic structural diagram of the dust detection device provided by the embodiment of this application. Among them, the dust detection device 300 can be applied to an electronic device. Specifically, the dust detection device 300 can include an acquisition module 301, an extraction module 302, a processing module 303, a fusion module 304, and a detection module 305, as follows:

[0126] The acquisition module 301 is used to acquire the point cloud data of at least two consecutive frames in the target area.

[0127] The extraction module 302 is used to extract the point cloud features of the point cloud data.

[0128] The processing module 303 is used to perform deformable convolution processing on the point cloud features to obtain the processed feature map.

[0129] The fusion module 304 is used to weight different positions of the processed feature map based on the attention mechanism, and fuse the weighted feature maps to obtain the fused feature map.

[0130] The detection module 305 is used to perform dust detection on the target area according to the fused feature map.

[0131] Optionally, in some embodiments, the processing module 303 may specifically include:

[0132] A generation unit for generating an offset vector at each position of the point cloud feature;

[0133] A calculation unit for calculating the offset position corresponding to each position according to the offset vector;

[0134] An output unit for obtaining a processed feature map based on the offset position and the point cloud feature.

[0135] Optionally, in some embodiments, the output unit may specifically be configured to: sample the point cloud feature using the offset position; and output a processed feature map based on the sampled feature values.

[0136] Optionally, in some embodiments, the detection module 305 may specifically include:

[0137] A generation sub-module for generating a plurality of candidate boxes on the fused feature map;

[0138] A detection sub-module for performing dust detection on the target area according to the generated candidate boxes.

[0139] Optionally, in some embodiments, the detection sub-module may specifically include:

[0140] An acquisition unit for acquiring the confidence level and scale corresponding to each generated candidate box;

[0141] A detection unit for performing dust detection on the target area according to the intersection over union (IoU), confidence level, and scale between each candidate box and other candidate boxes.

[0142] Optionally, in some embodiments, the detection unit may specifically include:

[0143] A first determination sub-unit for determining the currently processed candidate box as the currently processed object;

[0144] A calculation sub-unit for calculating the intersection over union (IoU) between the currently processed object and each candidate box;

[0145] A second determination sub-unit for determining the candidate boxes with an intersection over union (IoU) greater than a preset value as overlapping boxes, and determining the retained target boxes among the currently processed object and the overlapping boxes based on the confidence level and scale of the currently processed object and the confidence level and scale of the overlapping boxes;

[0146] A return unit for returning to execute the determination of the currently processed object until all candidate boxes are traversed;

[0147] A detection unit for performing dust detection on the target area according to all the target boxes.

[0148] Optionally, in some embodiments, the detection unit may specifically be configured to: determine the scale difference between the current processing object and the overlapping box according to the scale of the current processing object and the scale of the overlapping box; when the scale difference meets a preset condition, determine the candidate box with a high confidence level in the current processing object and the overlapping box as the retained target box.

[0149] It should be noted that the dust detection device provided in the embodiments of the present application and the dust detection method in the above embodiments belong to the same concept. Any method provided in the embodiments of the dust detection method can be run on the dust detection device. The specific implementation process can be found in the embodiments of the dust detection method and will not be elaborated here.

[0150] After the acquisition module 301 of the embodiment of the present application acquires the point cloud data of at least two consecutive frames of the target area, the extraction module 302 extracts the point cloud features of the point cloud data, and the processing module 303 performs deformable convolution processing on the point cloud features to obtain a processed feature map. Then, the fusion module 304 weights different positions of the processed feature map based on the attention mechanism and fuses the weighted feature maps to obtain a fused feature map. Finally, the detection module 305 performs dust detection on the target area according to the fused feature map. In the dust detection solution of the present application, deformable convolution processing is performed on the point cloud features corresponding to the point cloud data of at least two consecutive frames, and then different positions of the processed feature map are weighted based on the attention mechanism and the weighted feature maps are fused, so that the fused feature map has a more refined feature representation of the target area. When performing dust detection on the target area according to the fused feature map subsequently, the complex shape and multi-scale features of the dust can be captured, thereby improving the accuracy of dust detection.

[0151] This embodiment also provides a computer-readable storage medium, in which computer program code is stored. When the computer program code runs on a computer, the computer is caused to execute the above-related method steps to implement a dust detection method provided in the above embodiments.

[0152] Among them, the storage medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, etc.

[0153] Since the instructions stored in the storage medium can execute the steps in any of the dust detection methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any of the dust detection methods provided in the embodiments of the present application can be realized. For details, please refer to the previous embodiments and will not be elaborated here.

[0154] Correspondingly, an embodiment of the present application further provides an electronic device 400, which may include devices such as an in-vehicle communication box. Please refer to Figure 6 , Figure 6 FIG. Figure 6 is a schematic diagram of the first structure of the electronic device provided by an embodiment of the present application. The electronic device 400 includes a processor 401 and a memory 402. Among them, the processor 401 is electrically connected to the memory 402.

[0155] The processor 401 is the control center of the electronic device 400, connecting various parts of the entire electronic device through various interfaces and lines, and by running or calling computer programs stored in the memory 402, as well as calling data stored in the memory 402, it executes various functions of the electronic device and processes data, thereby monitoring the entire electronic device.

[0156] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the computer programs and modules stored in the memory 402. The memory 402 mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, computer programs required for at least one function, etc.; the data storage area can store data created according to the use of the electronic device, etc.

[0157] In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices. Correspondingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0158] In this embodiment, the processor 401 in the electronic device 400 will load instructions corresponding to the processes of one or more computer programs into the memory 402 according to the following steps, and the processor 401 will run the computer programs stored in the memory 402 to implement various functions, as follows:

[0159] Obtain at least two consecutive frames of point cloud data of the target area;

[0160] Extract the point cloud features of the point cloud data, and perform deformable convolution processing on the point cloud features to obtain a processed feature map;

[0161] Based on the attention mechanism, weight different positions of the processed feature map, and fuse the weighted feature maps to obtain a fused feature map;

[0162] Perform dust detection on the target area according to the fused feature map.

[0163] In some embodiments, please refer toFigure 7 , Figure 7 is the second schematic structural diagram of the electronic device provided by the embodiments of the present application. The electronic device 400 may include: a processor 401, a memory 402, a display screen 403, a camera component 404, an audio circuit 405, a sensor 406, and a power supply 407. Among them, the processor 401 is electrically connected to the display 403, the camera component 404, the audio circuit 405, the sensor 406, and the power supply 407 respectively.

[0164] The display screen 403 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the electronic device. These graphical user interfaces can be composed of images, texts, icons, videos, and any combination thereof.

[0165] The camera component 404 may include an image processing circuit. The image processing circuit can be implemented by using hardware and / or software components, and may include various processing units that define an Image Signal Processing pipeline. The image processing circuit can at least include: multiple cameras, an Image Signal Processor (ISP processor), a control logic, and an image memory, etc. Each of the cameras can at least include one or more lenses and an image sensor. The image sensor may include a color filter array (such as a Bayer filter). The image sensor can acquire the light intensity and wavelength information captured by each imaging pixel of the image sensor, and provide a set of raw image data that can be processed by the image signal processor.

[0166] The audio circuit 405 can be used to provide an audio interface between the user and the electronic device through a speaker and a microphone. Among them, the audio circuit 405 includes a microphone. The microphone is electrically connected to the processor 401. The microphone is used to receive voice information input by the user.

[0167] The sensor 406 is used to collect information of the electronic device itself, or information of the user, or information of the external environment. For example, the sensor 406 may include one or more of sensors such as a vibration sensor, a temperature sensor, a distance sensor, a magnetic field sensor, a light sensor, an acceleration sensor, a fingerprint sensor, a Hall sensor, a position sensor, a gyroscope, an inertial sensor, an attitude sensor, a barometer, and a heart rate sensor.

[0168] The power supply 407 is used to supply power to each component of the electronic device 400. In some embodiments, the power supply 407 may be logically connected to the processor 401 through a power management system, so as to implement functions such as management of charging, discharging, and power consumption management through the power management system.

[0169] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of devices or units can be in electrical, mechanical or other forms.

[0170] For the dust detection device of the embodiments of the present application, its various functional modules can be integrated in a processing chip, or each module can exist physically alone, or two or more modules can be integrated in one module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0171] The above has introduced in detail the dust detection method, device, electronic device and storage medium provided by the embodiments of the present application. Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The above embodiment descriptions are only used to help understand the method and its core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A dust detection method, characterized in that, Including: Obtain point cloud data of at least two consecutive frames of the target area; Extract the point cloud features of the point cloud data, and perform deformable convolution processing on the point cloud features to obtain a processed feature map; Based on the attention mechanism, weight different positions of the processed feature map, and fuse the weighted feature maps to obtain a fused feature map; Perform dust detection on the target area according to the fused feature map.

2. The dust detection method according to claim 1, characterized in that, The performing deformable convolution processing on the point cloud features to obtain a processed feature map includes: Generate offset vectors at each position of the point cloud features; Calculate the offset positions corresponding to each position according to the offset vectors; Based on the offset positions and the point cloud features, obtain the processed feature map.

3. The dust detection method according to claim 2, wherein The obtaining the processed feature map based on the offset positions and the point cloud features includes: Sample the point cloud features using the offset positions; Based on the sampled feature values, output the processed feature map.

4. The dust detection method according to claim 1, wherein The performing dust detection on the target area according to the fused feature map includes: Generate multiple candidate boxes on the fused feature map; Perform dust detection on the target area according to the generated candidate boxes.

5. The dust detection method according to claim 4, wherein The performing dust detection on the target area according to the generated candidate boxes includes: Obtain the confidence and scale corresponding to each generated candidate box; Perform dust detection on the target area according to the intersection over union between each candidate box and other candidate boxes, the confidence, and the scale.

6. The dust detection method according to claim 5, characterized in that, The performing dust detection on the target area according to the intersection over union between each candidate box and other candidate boxes, the confidence, and the scale includes: Determine the currently processed candidate box as the currently processed object; Calculate the intersection over union between the currently processed object and each candidate box; Determine the candidate boxes with an intersection over union greater than the preset value as overlapping boxes, and based on the confidence and scale of the currently processed object, as well as the confidence and scale of the overlapping boxes, determine the retained target boxes among the currently processed object and the overlapping boxes; Return to execute determining the currently processed object until all candidate boxes are traversed; Perform dust detection on the target area according to all the target boxes.

7. The dust detection method according to claim 6, characterized in that, The determining the retained target boxes among the currently processed object and the overlapping boxes based on the confidence and scale of the currently processed object, as well as the confidence and scale of the overlapping boxes includes: Determine the scale difference between the currently processed object and the overlapping box according to the scale of the currently processed object and the scale of the overlapping box; When the scale difference meets the preset condition, then determine the candidate box with a higher confidence among the currently processed object and the overlapping box as the retained target box.

8. A dust detection device, characterized in that, Including: An acquisition module for obtaining point cloud data of at least two consecutive frames of the target area; An extraction module for extracting the point cloud features of the point cloud data; A processing module for performing deformable convolution processing on the point cloud features to obtain a processed feature map; A fusion module for weighting different positions of the processed feature map based on the attention mechanism and fusing the weighted feature maps to obtain a fused feature map; A detection module for performing dust detection on the target area according to the fused feature map.

9. An electronic device, characterized in that, Including: A memory for storing executable program code; A processor for calling and running the executable program code from the memory, such that the electronic device executes the dust detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which when executed, implements the dust detection method according to any one of claims 1 to 7.

Images