Intelligent identification method of explosion-proof intelligent terminal using industrial 3D vision sensor

Abstract

The present invention relates to an intelligent recognition method for explosion-proof smart terminals using industrial 3D vision sensors. The method: collect the first depth image of the target scene through the industrial 3D vision sensor and perform point cloud conversion and preprocessing to obtain preprocessed point cloud data; calculate the object shape descriptor; perform data enhancement on the preprocessed point cloud data, and construct a multi-scale feature fusion network including global branches, local coarse-grained branches and local fine-grained branches, and train to obtain a first object recognition model; perform transfer learning on the first object recognition model to obtain a second object recognition model; deploy the second object recognition model to the edge computing unit of the explosion-proof smart terminal, combine the second depth image collected in real time to perform online recognition and environmental perception, and output object recognition results and environmental safety assessment reports. The implementation of the present invention improves the recognition ability and safety performance of explosion-proof smart terminals in industrial environments.

Description

Technical Field

[0001] The present invention relates to the technical field of visual sensors, and in particular to an intelligent recognition method for explosion-proof intelligent terminals using industrial 3D visual sensors. Background Art

[0002] Safety monitoring and intelligent identification in industrial environments have always been important topics in the field of industrial automation and intelligent manufacturing. Traditional 2D vision technology has been unable to meet the needs of accurate identification and environmental perception in complex industrial scenarios. The introduction of industrial 3D vision sensors provides a new possibility to solve this problem. It can obtain richer depth information, which helps to improve the accuracy of object recognition and the precision of environmental perception. However, the application of 3D vision technology in explosion-proof environments faces many challenges.

[0003] Explosion-proof environments have extremely high requirements for the safety of equipment, and traditional visual equipment can hardly meet these strict standards. Secondly, the complexity and variability of industrial sites make object recognition and environmental perception difficult, requiring more robust and adaptive algorithms. Furthermore, the cost of data acquisition and annotation at industrial sites is high, and how to achieve high-precision recognition under limited sample conditions is also a problem that needs to be solved urgently. In addition, industrial sites have high real-time requirements, and how to achieve fast and accurate recognition and decision-making on edge devices is also a challenge. At the same time, there are many types of objects in industrial scenes with complex shapes. How to effectively extract and represent 3D features, and how to design recognition models that can adapt to different scenes are all problems that need to be solved urgently. Summary of the invention

[0004] The main purpose of the present invention is to provide an explosion-proof intelligent terminal intelligent identification method using an industrial 3D vision sensor to improve the identification ability and safety performance of the explosion-proof intelligent terminal in an industrial environment.

[0005] To achieve the above object, the present invention provides an explosion-proof intelligent terminal intelligent identification method using an industrial 3D visual sensor, comprising the following steps:

[0006] The first depth image of the target scene is collected by the industrial 3D vision sensor in the explosion-proof intelligent terminal, and point cloud conversion and preprocessing are performed to obtain preprocessed point cloud data;

[0007] Based on the preprocessed point cloud data, extracting comprehensive 3D features of local curvature, normal vector distribution and shape index, and performing feature projection and coefficient calculation through modified spherical harmonics to obtain an object shape descriptor;

[0008] Based on the random rectangular occlusion algorithm, data enhancement is performed on the pre-processed point cloud data, and a multi-scale feature fusion network including a global branch, a local coarse-grained branch, and a local fine-grained branch is constructed, and a first object recognition model is obtained through joint loss function training;

[0009] Using a prototype network to perform transfer learning on the first object recognition model, and obtaining a second object recognition model through task sampling, internal and external loop training, and meta-learning optimization;

[0010] The second object recognition model is deployed to the edge computing unit of the explosion-proof intelligent terminal, and online recognition and environmental perception are performed in combination with the second depth image collected in real time, and the object recognition results and environmental safety assessment report are output.

[0011] The present invention also provides an explosion-proof intelligent terminal intelligent identification device using an industrial 3D visual sensor, comprising:

[0012] An acquisition module is used to acquire a first depth image of a target scene through an industrial 3D vision sensor in an explosion-proof intelligent terminal and perform point cloud conversion and preprocessing to obtain preprocessed point cloud data;

[0013] A calculation module, for extracting comprehensive 3D features of local curvature, normal vector distribution and shape index based on the preprocessed point cloud data, and performing feature projection and coefficient calculation through modified spherical harmonic functions to obtain an object shape descriptor;

[0014] A construction module is used to perform data enhancement on the pre-processed point cloud data based on a random rectangular occlusion algorithm, and to construct a multi-scale feature fusion network including a global branch, a local coarse-grained branch, and a local fine-grained branch, and to obtain a first object recognition model through joint loss function training;

[0015] An optimization module, configured to perform transfer learning on the first object recognition model using a prototype network, and obtain a second object recognition model through task sampling, internal and external loop training, and meta-learning optimization;

[0016] The output module is used to deploy the second object recognition model to the edge computing unit of the explosion-proof intelligent terminal, perform online recognition and environmental perception in combination with the second depth image collected in real time, and output the object recognition result and environmental safety assessment report.

[0017] The present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0018] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods are implemented.

[0019] In summary, the technical solution provided by the present invention obtains a more accurate and robust object shape descriptor by extracting comprehensive 3D features of local curvature, normal vector distribution and shape index, and uses modified spherical harmonics for feature projection and coefficient calculation, thereby improving the accuracy of object recognition. A multi-scale feature fusion network containing global branches, local coarse-grained branches and local fine-grained branches is constructed, which can simultaneously capture the global structure and local detail information of the object, and enhance the model's recognition ability for objects of complex shapes. The random rectangular occlusion algorithm is used for data enhancement, and the prototype network is combined for transfer learning, which effectively solves the problem of insufficient samples in industrial scenes and improves the generalization ability of the model in new scenes. The trained model is deployed to the edge computing unit of the explosion-proof intelligent terminal, realizing real-time object recognition and environmental perception, and improving the response speed and decision-making efficiency of the system. Combining object recognition results and scene segmentation information, an environmental semantic map is constructed and a safety level assessment is performed. Through voxelization and density encoding, as well as adaptive sampling and principal axis alignment, the model's recognition ability for objects of different scales and postures is improved. The introduction of self-attention and channel attention mechanisms in fine-grained feature extraction improves the model's ability to capture key features and enhances recognition accuracy. By generating reports containing 3D scene reconstruction, object annotation, and safety risk heat maps, the interpretability and visualization of recognition results are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a schematic diagram of the steps of an explosion-proof intelligent terminal intelligent identification method using an industrial 3D visual sensor in one embodiment of the present invention;

[0021] Figure 2 This is a structural block diagram of an explosion-proof intelligent terminal intelligent identification device using an industrial 3D visual sensor in one embodiment of the present invention;

[0022] Figure 3 It is a schematic block diagram of the structure of a computer device according to an embodiment of the present invention.

[0023] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0025] Reference Figure 1 This embodiment provides an explosion-proof intelligent terminal intelligent identification method using an industrial 3D vision sensor, comprising the following steps:

[0026] S1, collecting the first depth image of the target scene through the industrial 3D vision sensor in the explosion-proof intelligent terminal and performing point cloud conversion and preprocessing to obtain preprocessed point cloud data;

[0027] Specifically, the first depth image of the target scene is collected by the industrial 3D vision sensor in the explosion-proof intelligent terminal, and the depth value mapping is performed for each pixel in the depth image, and the two-dimensional image coordinates and their corresponding depth values ​​are converted into three-dimensional space coordinates to generate original point cloud data. A spatial index structure is constructed based on the original point cloud data, the local statistical characteristics of each point are calculated, and abnormal points are eliminated according to the preset threshold standard to obtain preliminary filtered point cloud data. The initial filtered point cloud is subjected to noise suppression processing, and the depth value of each point is locally weighted updated to reduce the influence of environmental noise to obtain noise point cloud data. Based on the denoised point cloud data, the multi-frame point cloud is registered, and the spatial transformation relationship between the point pairs is iteratively optimized, and the corresponding relationship between the point clouds is gradually adjusted until globally consistent point cloud data is formed, ensuring that the spatial position and shape information of point clouds in different frames can be accurately matched. The globally consistent point cloud data is spatially divided into several small unit areas, and the point cloud data is aggregated in each divided unit. By performing aggregation operations on multiple points in the same area, all points in the unit are replaced with the aggregation results to form downsampled point cloud data. The downsampling process effectively reduces the amount of data and improves the efficiency of subsequent processing. The principal component analysis normal vector calculation is performed on the downsampled point cloud data to obtain the normal vector of each point and describe the geometric characteristics of the local area of ​​the point cloud. Based on the calculated normal vector field, the local surface fitting operation is performed on the downsampled point cloud data to generate a more accurate description of the geometric surface shape, and finally the preprocessed point cloud data is obtained.

[0028] S2, based on the preprocessed point cloud data, extracts the comprehensive 3D features of local curvature, normal vector distribution and shape index, and performs feature projection and coefficient calculation through modified spherical harmonics to obtain the object shape descriptor;

[0029] Specifically, local surface fitting is performed on each point in the preprocessed point cloud data, and the average curvature and Gaussian curvature of each point are calculated to describe the local geometric characteristics of the point, thereby generating the curvature characteristics of the point cloud. Based on the curvature characteristics, the shape index and curvature index are calculated for each point. The shape index is used to characterize the geometric shape characteristics of the local surface, while the curvature index reflects the details of the local curvature change. The initial shape characteristics are obtained by constructing the local shape descriptor of the point. A normal vector histogram is constructed for the preprocessed point cloud data, and the normal vector direction distribution of each point in its neighborhood is statistically analyzed to obtain the normal vector distribution characteristics. The normal vector distribution can reflect the neighborhood surface morphology of the point, and the complexity of the local surface can be revealed by statistically analyzing its direction distribution characteristics. The initial shape features are combined with the normal vector distribution features to construct a multi-scale feature descriptor. By cascading local features of different scales, the descriptor can capture both local geometric characteristics and global structural features, forming a more comprehensive comprehensive 3D feature. A spherical projection operation is performed on the comprehensive 3D feature, and the feature vector is mapped onto the unit sphere to obtain the spherical feature distribution. Spherical projection can effectively map the point cloud features in three-dimensional space into a distribution form on a sphere, which is convenient for subsequent spherical harmonic expansion processing. Based on the spherical feature distribution, a set of basis functions of the modified spherical harmonics is constructed. By adjusting the order and frequency of the spherical harmonics, it is ensured that the generated basis functions can accurately describe the different levels of details of the point cloud features and adapt to the complexity of the point cloud features. The spherical feature distribution is expanded by the modified spherical harmonics, and the spherical features are represented as a series of spherical harmonic coefficients. For the spherical harmonic coefficients obtained after expansion, the expansion coefficients are calculated to obtain the spherical harmonic coefficient vector. Based on the spherical harmonic coefficient vector, low-order coefficients are selected to construct a more compact feature representation. By selecting low-order coefficients, high-frequency noise can be effectively removed, data redundancy can be reduced, and computational efficiency can be improved. The spherical harmonic coefficient vector is subjected to feature normalization and dimensionality reduction operations to ensure that the numerical range and dimension of the feature vector meet the requirements of downstream tasks and generate the final object shape descriptor.

[0030] S3, based on the random rectangular occlusion algorithm, the pre-processed point cloud data is enhanced, and a multi-scale feature fusion network including a global branch, a local coarse-grained branch, and a local fine-grained branch is constructed, and the first object recognition model is obtained through joint loss function training;

[0031] Specifically, a random rectangular occlusion algorithm is applied to the preprocessed point cloud data. By randomly generating multiple rectangular occlusion areas in the point cloud and removing the points in these areas, an occlusion-enhanced point cloud dataset is obtained. Through the occlusion operation, the diversity of the point cloud data is enhanced and the robustness of the model to missing information is improved. The occlusion-enhanced point cloud dataset is input into the multi-scale feature fusion network. The network consists of five parts: a global branch network, a coarse-grained branch network, a fine-grained branch network, a feature fusion module, and a classification head. The global branch network extracts the global features of the point cloud. The network includes 5 3D convolutional layers and 2 fully connected layers. Each 3D convolutional layer is followed by a batch normalization layer and a ReLU activation function. The global features of the occlusion-enhanced point cloud data are extracted layer by layer to obtain the first global feature representation. At the same time, the occlusion-enhanced point cloud data is divided into 27 local regions by 3×3×3 voxel partitioning. Each local region is input into the coarse-grained branch network, which consists of 3 3D convolutional layers and 1 fully connected layer. The first coarse-grained local feature representation is obtained by extracting and fusing features from each local region. The occlusion-enhanced point cloud data is divided into 5×5×5 voxels to generate 125 fine local regions, which are input into the fine-grained branch network. The network consists of 2 3D convolutional layers and 1 fully connected layer. The features of each fine local region are extracted and fused to obtain the first fine-grained local feature representation. The first global feature representation, the first coarse-grained local feature representation and the first fine-grained local feature representation are input into the feature fusion module. The module includes 2 fully connected layers and 1 self-attention layer. The features of different scales are fused using an adaptive weighting method to generate the first multi-scale fused feature. The fusion step can capture the comprehensive features of the object by effectively combining global and local information. The first multi-scale fused feature is input into the classification head for classification. The classification head consists of two fully connected layers and one Softmax layer. By mapping the fused features to the category probability distribution space, the category prediction results of the object are generated. In order to optimize the model, a joint loss function is designed, including cross entropy loss, center loss and triple loss. Cross entropy loss is used to optimize the classification accuracy, center loss improves the recognition ability of the model by enhancing the compactness of intra-class features, and triple loss improves the model's ability to distinguish different categories by increasing the distance between classes. These three losses are weighted and combined to form an overall loss function. The loss value of the object category prediction result is calculated based on the overall loss function, and the multi-scale feature fusion network is end-to-end trained using the Adam optimizer. Through the back-propagation algorithm, the gradient is calculated step by step and the parameters in the network are updated until the training converges, and finally the first object recognition model is generated. Through the deep fusion of multi-scale features and multiple optimizations of the loss function, the model can achieve efficient and accurate object recognition in the complex environment of explosion-proof intelligent terminals.

[0032] S4, using the prototype network to perform transfer learning on the first object recognition model, and obtaining the second object recognition model through task sampling, internal and external loop training and meta-learning optimization;

[0033] Specifically, the first object recognition model is structurally decomposed, and its feature extraction layer is used as an embedding function to extract feature representation from the input data; the classification layer is used as a prototype calculation layer, which is specifically used to calculate prototype vectors of different categories. Through structural decomposition, the initial prototype network structure is obtained. A task sampler is constructed to generate tasks on the new scene data set. The function of the task sampler is to randomly generate multiple learning tasks based on the new data set to form a task pool. Each task consists of two parts: a support set and a query set. The support set is used to train the model, and the query set is used to evaluate the performance of the model. After randomly sampling a task from the task pool, the support set of the task is input into the embedding function of the initial prototype network, and the feature vector of each sample in the support set is extracted through the embedding function. Based on the feature vector, the category prototype is calculated, and the sample features in each category are averaged to obtain the prototype vector of each category. The prototype vector can be regarded as the feature representative of each category for subsequent classification operations. The query set of the task is input into the embedding function, and the features of the query set samples are extracted through the embedding function. The Euclidean distance between each query sample and the prototype vector of each category is calculated, and the similarity between the query sample and each category is judged by the distance metric. In order to convert the Euclidean distance into a usable category probability distribution, the Softmax function is applied to generate the classification results of the query set by exponential operation and normalization of the distance. The cross entropy loss is calculated based on the classification results of the query set and the true label. In order to prevent the model from overfitting, the L2 regularization term is added to the loss function. Combining the cross entropy loss and the regularization term, the optimization objective function of meta-learning is constructed. The parameters of the initial prototype network structure are updated to complete the inner loop training. This stage is mainly used to adapt the model to the data distribution and characteristics of a specific task, so as to obtain a task-adaptive prototype network. In order to improve the generalization ability of the model, the training of multiple tasks is repeated. During the training process of each task, the system accumulates the gradient of each task, processes the accumulated gradient through the meta-optimizer, and updates the parameters of the embedding function. The outer loop training step in the meta-learning framework aims to enhance the learning ability of the model through cross-task training, so that the model can quickly adapt when facing new tasks. Through the joint optimization of inner and outer loop training, a second object recognition model with stronger generalization ability is obtained. The model can adapt to object recognition tasks in different scenarios and provide more accurate and robust recognition results in complex explosion-proof environments.

[0034] S5, deploy the second object recognition model to the edge computing unit of the explosion-proof intelligent terminal, combine the second depth image collected in real time to perform online recognition and environmental perception, and output the object recognition result and environmental safety assessment report.

[0035] Specifically, the second object recognition model is quantized and compressed to reduce the computational complexity and storage requirements of the model, so that it can run more efficiently on the edge computing unit of the explosion-proof intelligent terminal. After the model is compressed, it is deployed to the edge computing unit of the terminal to ensure that the model can perform real-time object recognition tasks in a resource-constrained environment. The second depth image is collected in real time by the industrial 3D vision sensor in the explosion-proof intelligent terminal, and point cloud conversion and preprocessing are performed to generate real-time point cloud data. The real-time point cloud data is input into the global branch network of the second object recognition model, which consists of 5 3D convolutional layers and 2 fully connected layers. Each convolutional layer is followed by batch normalization and ReLU activation functions to ensure that the network has better numerical stability and nonlinear expression ability when extracting features. Through the global branch network, the second global feature representation is extracted layer by layer. At the same time, the real-time point cloud data is divided into 3×3×3 voxels to obtain 27 local regions. The local regions are input into the coarse-grained branch network of the second object recognition model. The coarse-grained branch network contains three 3D convolutional layers and one fully connected layer. The features of each local region are extracted and fused through the maximum pooling operation to obtain the second coarse-grained local feature representation. Similarly, the real-time point cloud data is divided into 5×5×5 voxels to generate 125 finer local regions. The fine local regions are input into the fine-grained branch network, which consists of two 3D convolutional layers and one fully connected layer. The features of these fine regions are weighted and fused through the attention mechanism to obtain the second fine-grained local feature representation. The second global feature representation, the second coarse-grained local feature representation, and the second fine-grained local feature representation are input into the feature fusion module. The features are transformed through two fully connected layers, and the self-attention mechanism is used to adaptively weight the features of different scales to generate the second multi-scale fused features. Based on the multi-scale fused features, the category probability distribution of the object is calculated through the classification head, and the category with the highest probability is selected as the final object recognition result. Scene segmentation and target detection are performed on the real-time point cloud data to identify key elements and potential risk factors in the environment. Combined with the results of object recognition, a semantic map of the environment is constructed to analyze the objects, obstacles, and possible dangerous elements in the scene. Through a rule-based reasoning system, based on this recognition information, the system can evaluate the safety level of the current scene and generate an environmental safety assessment result. The object recognition results and environmental safety assessment results are integrated into structured data, and a report containing 3D scene reconstruction, object annotation, and safety risk heat map is generated through the visualization module. The report shows the object recognition information in the current environment and also reflects the potential safety hazards in the environment.

[0036] The real-time point cloud data is voxelized to convert the 3D point cloud data into a fixed-size 3D grid structure, where the number of points in each voxel is calculated to obtain a density value, thereby forming a density voxel grid. The irregular point cloud data is mapped to regular 3D grid data to adapt to the 3D convolution operation. Based on the generated density voxel grid, a four-dimensional input tensor is constructed, where the first three dimensions represent the spatial position of the point cloud and the fourth dimension represents the point cloud density in each voxel. The four-dimensional tensor is the input data, which can effectively express the geometric and density information of the point cloud. The input data is input to the first 3D convolution layer, and 32 3×3×3 convolution kernels are used to perform preliminary feature extraction on the point cloud data to generate a primary feature map. The feature map is standardized by batch normalization to ensure numerical stability during network training, and the ReLU activation function is applied after normalization to enhance the nonlinear expression ability of the primary feature map, thereby obtaining the first layer activation feature map. The first layer activation feature map is input to the second to fifth 3D convolution layers in sequence. Each convolution layer uses a different number of convolution kernels and strides to extract deeper features layer by layer. After each convolution operation, batch normalization and ReLU activation processing are also required to ensure the numerical stability and nonlinear enhancement effect of feature extraction. Through layer-by-layer convolution operations, the multi-level features of point cloud data are captured, which can not only extract the overall shape of the object from the global structure, but also identify the tiny features of the object from the local details. After five layers of convolution, a multi-level feature map is generated, which represents the different scale information of the point cloud data. The feature map output by the fifth 3D convolution layer is subjected to global average pooling operation, which condenses the global spatial information by compressing the three-dimensional feature map into a one-dimensional feature vector. This process reduces the dimension of the feature data while retaining the global information to generate a global pooling feature. The global pooling feature is input into the first fully connected layer, and the pooling feature is linearly transformed using a weight matrix, and the ReLU activation function is applied again to generate the intermediate layer feature representation. The intermediate layer feature representation is input into the second fully connected layer, and the feature is linearly transformed and dimension reduction is performed to finally obtain the second global feature representation. The global feature representation contains multi-level global features extracted from the original point cloud and can be effectively used for object recognition tasks.

[0037] The real-time point cloud data is spatially uniformly sampled to ensure that the sampling points of the point cloud are uniformly distributed in space, so as to improve the stability of subsequent feature extraction. Based on the uniformly distributed sampling point cloud, the overall spatial range of the point cloud is calculated and divided into a 3×3×3 grid structure to obtain 27 local regions. Each local region represents a sub-region of the entire point cloud space and can capture local geometric information respectively. The points in each local region are density encoded. The local density feature map is obtained by calculating the number of points in each voxel and normalizing it. The density feature map can reflect the distribution of points in the local region. The local density feature map is input into the first 3D convolution layer, and 16 3×3×3 convolution kernels are used for preliminary feature extraction. After each convolution operation, batch normalization is applied to maintain the stability of the feature value and avoid too large or too small values ​​during training. The ReLU activation function is applied to enhance the nonlinear expression ability of the network to obtain the first layer of local feature maps. The first layer of local feature maps is input into the second 3D convolution layer. 32 3×3×3 convolution kernels are used for deeper feature extraction, batch normalization and ReLU activation function are applied to obtain the second layer of local feature maps. By increasing the number of convolution kernels and extracting features layer by layer, the network is able to capture more local geometric details and structural features. The second layer of local feature maps are input into the third 3D convolution layer, and 64 3×3×3 convolution kernels are used for feature extraction to enhance the depth and precision of local features, and batch normalization and ReLU activation function are applied to generate the third layer of local feature maps. The third layer of local feature maps are subjected to global maximum pooling operation to compress the feature map of each channel into a single feature value to obtain a local feature vector of fixed dimension. Pooling operation effectively reduces the dimensionality of the data, retains the most significant feature information, and forms a simplified representation of local features. The local feature vectors of 27 local regions are concatenated to form a complete local feature representation. The concatenated local feature vector is input into a fully connected layer, the features are linearly transformed by the weight matrix, and the ReLU activation function is applied to further condense and enhance the feature information. Generate the second coarse-grained local feature representation, integrate the information of 27 local areas, and provide coarse-grained spatial features in the object recognition process. Through multi-level convolution and pooling operations, it can effectively capture the local geometric structure in the point cloud data and provide key coarse-grained feature support for object recognition.

[0038] Adaptively sample the real-time point cloud data, and obtain the optimized sampled point cloud by retaining the detailed features, ensuring high accuracy and no loss of local geometric information. The main axis direction of the point cloud is calculated based on the sampled point cloud with retained details, and the point cloud data is processed in a unified coordinate system by rotating and aligning the space. It is then divided into a 5×5×5 grid structure to generate 125 fine local regions. Each fine local region contains the local geometric details in the point cloud. Feature encoding is performed on the points in each fine local region, and a multi-channel feature map is generated by calculating the spatial distribution and local geometric features of the points in each voxel. This feature map can characterize the geometric structure in the local region and provide rich input data for the convolutional network. The multi-channel feature map is input into the first 3D convolutional layer. The first convolutional layer uses 32 3×3×3 convolution kernels to perform preliminary extraction of the features of the fine local region. The LeakyReLU activation function is used to ensure that inputs less than zero also have non-zero outputs, so that the network has better stability when processing sparse point clouds. At the same time, instance normalization is used to standardize the convolution results to ensure that the numerical range of features in different batches of input data remains consistent, and the first layer of fine-grained feature maps is obtained. The first layer of fine-grained feature maps are input into the second 3D convolution layer, and 64 3×3×3 convolution kernels are used for deeper feature extraction. The second layer of convolution operation uses the LeakyReLU activation function and instance normalization operation to enhance the network's ability to capture fine-grained geometric features. After this layer of convolution, the second layer of fine-grained feature maps are generated to more accurately express the geometric structure and distribution characteristics of the local area. Self-attention calculation is performed on the second layer of fine-grained feature maps. By calculating the correlation matrix between different positions in the feature map, the relationship between local features is captured, and the attention-enhanced feature map is generated by weighted summation. Through the self-attention mechanism, the network can automatically identify and weight important local features to enhance its ability to express key geometric information. The attention-enhanced feature map is input into the fully connected layer, linearly transformed by a weight matrix, and processed using the LeakyReLU activation function to obtain a local feature vector of fixed dimension. The local feature vector condenses the important information in each fine local region. Channel attention weighting is performed on the local feature vectors of 125 fine local regions. By learning the importance weight of each channel, the contribution of different channels to the final feature representation is effectively distinguished, and the final second fine-grained local feature representation is generated by weighted summation.

[0039] In one example, an industrial 3D vision sensor in an explosion-proof intelligent terminal collects a first depth image of a target scene and performs point cloud conversion and preprocessing to obtain preprocessed point cloud data, including:

[0040] Performing depth value mapping on each pixel in the first depth image, converting the two-dimensional image coordinates and the corresponding depth values ​​into three-dimensional space coordinates to obtain original point cloud data;

[0041] Based on the original point cloud data, a spatial index structure is constructed, local statistical features of the points are calculated, and abnormal points are removed according to the preset threshold to obtain the initial filtered point cloud;

[0042] Perform noise suppression on the initial filtered point cloud and perform local weighted update on the depth value of each point to obtain denoised point cloud data;

[0043] Based on the denoised point cloud data, the multi-frame point cloud is registered, and the globally consistent point cloud data is obtained by iteratively optimizing the spatial transformation relationship between point pairs.

[0044] The globally consistent point cloud data is spatially divided, and the point aggregation operation is performed in each divided unit. The aggregation result is used to replace all points in the unit to obtain the downsampled point cloud data.

[0045] The principal component analysis normal vector calculation is performed on the downsampled point cloud data to obtain the normal vector field of the point cloud, and based on the normal vector field, local surface fitting is performed on the downsampled point cloud data to obtain preprocessed point cloud data.

[0046] In this example, depth value mapping is performed based on the two-dimensional image coordinates of each pixel point in the first depth image and the corresponding depth value. ) is converted into three-dimensional space coordinates. The specific conversion process depends on the camera's intrinsic parameter matrix, which describes the camera's focal length, optical center and other parameters. The three-dimensional space coordinates can be calculated using the following formula:

[0047] ;

[0048] in, is the point cloud coordinate in three-dimensional space, are the pixel coordinates in the 2D image, Indicates the depth value of the pixel. and are the horizontal and vertical focal lengths of the camera, respectively. and is the optical center position of the camera (i.e., the center point of the image). Through this formula, each pixel is mapped from the image space to the three-dimensional space to generate the initial raw point cloud data. A spatial index structure is constructed based on the raw point cloud data to calculate the local statistical features of the points. Commonly used index structures include KD trees or octrees, which can effectively accelerate subsequent point cloud processing operations. Through the spatial index structure, the neighborhood point set around a point is quickly found, and the local statistical features of the point are calculated, such as the local point density, the curvature change in the neighborhood, etc. The abnormal points in the point cloud are removed based on the preset threshold. Abnormal points are usually caused by sensor noise, reflectivity differences, or surface characteristics, which will have a negative impact on the overall quality of the point cloud. A threshold is set according to the local statistical features (such as local density or curvature), and those points that are too different from the surrounding points are removed to obtain the point cloud data after preliminary filtering. The initial filtered point cloud is subjected to noise suppression processing to smooth the noise in the point cloud data while retaining the geometric details of the data. Common methods include neighborhood-based weighted average filtering, and the formula is as follows:

[0049] ;

[0050] in, Yes The updated depth value of Indicate point The surrounding neighborhood point set, It is a neighbor point The depth value of is the weight, usually based on the point With point The distance between them is defined. Through local weighted updates, the noise is effectively reduced while preserving the surface shape to generate denoised point cloud data. Based on the denoised point cloud data, multi-frame point cloud registration operations are performed. By iteratively optimizing the spatial transformation relationship between point pairs, point cloud data from different perspectives are aligned to generate globally consistent point cloud data. For example, using the ICP (Iterative Closest Point) algorithm, the distance between corresponding point pairs of two frames of point clouds is minimized by iteratively adjusting the rigid body transformation parameters (translation and rotation) between point clouds. Assume that point cloud A and point cloud B contain The goal of ICP is to find the best transformation matrix by minimizing the following cost function: :

[0051] ;

[0052] in, Contains translation and rotation matrices, and are the corresponding points in point clouds A and B respectively. Through multiple iterations, the system can find the best spatial transformation relationship so that the two frames of point clouds are aligned and globally consistent point cloud data is obtained. The globally consistent point cloud data is spatially divided, and the entire point cloud space is divided into multiple units. In each divided unit, the points are aggregated, and all the points in a unit are replaced by a certain aggregation strategy (such as taking the mean or median) to generate downsampled point cloud data. The purpose of downsampling is to reduce the redundancy and storage requirements of point cloud data while retaining global geometric features, thereby improving the efficiency of subsequent processing. Principal component analysis is performed on the downsampled point cloud data to calculate the normal vector field of the point cloud. Principal component analysis is a statistical method used to extract the most important directions from multidimensional data. For the neighborhood of each point, the normal vector of the surface on which the point is located can be obtained by principal component analysis, and the formula is:

[0053] ;

[0054] in, is the normal vector, is a point in the point cloud, Represents the neighborhood of the point. By calculating each point in the point cloud, the normal vector field of the point cloud is generated. Based on the obtained normal vector field, local surface fitting is performed on the downsampled point cloud data. The normal vector is used to describe the local geometric morphology of the point cloud, and a more accurate surface description is obtained through surface fitting. In the fitting process, a local geometric surface model is generated according to the normal vector and spatial position of the point, and finally the preprocessed point cloud data is obtained, which effectively reflects the three-dimensional morphology of the object.

[0055] In one example, based on the preprocessed point cloud data, the comprehensive 3D features of local curvature, normal vector distribution and shape index are extracted, and the feature projection and coefficient calculation are performed through the modified spherical harmonic function to obtain the object shape descriptor, including:

[0056] Perform local surface fitting on each point in the preprocessed point cloud data, and calculate the average curvature and Gaussian curvature of the point to obtain the curvature characteristics of the point cloud;

[0057] Based on the curvature features, the shape index and curvature index are calculated for each point, the local shape descriptor of the point is constructed, and the initial shape features are obtained;

[0058] Construct a normal vector histogram for the preprocessed point cloud data, count the direction distribution of the normal vector in the neighborhood of each point, and obtain the normal vector distribution characteristics;

[0059] Based on the initial shape features and normal vector distribution features, a multi-scale feature descriptor is constructed. By cascading local features of different scales, a comprehensive 3D feature is obtained. The comprehensive 3D feature is spherically projected, and the feature vector is mapped onto the unit sphere to obtain the spherical feature distribution.

[0060] Based on the spherical feature distribution, a modified spherical harmonics basis function set is constructed, and the basis function adapted to the point cloud features is obtained by adjusting the order and frequency of the spherical harmonics.

[0061] The spherical feature distribution is expanded by modified spherical harmonics, and the expansion coefficients are calculated to obtain the spherical harmonic coefficient vector. Based on the spherical harmonic coefficient vector, low-order coefficients are selected to construct a compact feature representation. Through feature normalization and dimensionality reduction operations, the object shape descriptor is obtained.

[0062] In this example, a local surface fitting is performed on each point in the preprocessed point cloud data and the geometric structure of the surface is estimated by the neighborhood of the point. The neighborhood point set , describe the local geometry of the point by fitting a quadratic surface. The form of fitting a quadratic surface is usually:

[0063] ;

[0064] in, and is the horizontal and vertical coordinates in the local coordinate system, is the height of the surface, and are the coefficients of the fitted surface. These coefficients are obtained by fitting the neighborhood points through the least squares method, which effectively describes the shape of the surface. The curvature characteristics are calculated based on the geometric information on the surface. Mean curvature and Gaussian curvature are two important indicators to describe the surface shape, and the formulas are:

[0065] ;

[0066] in, and are the principal curvatures, which represent the maximum and minimum curvatures of the surface in two orthogonal directions. describes the curvature of the local surface, while Gaussian curvature It reflects the overall shape of the surface (such as positive Gaussian curvature represents local convexity, negative Gaussian curvature represents local concavity). By calculating the average curvature and Gaussian curvature of each point, the local geometric description of the point is generated, and the curvature characteristics of the point cloud are obtained. Based on the curvature characteristics, the shape index and curvature index are calculated to construct the local shape descriptor of the point. Shape index It is a dimensionless geometric feature that describes the surface shape by the curvature ratio. The formula is:

[0067] ;

[0068] Curvature Index It is used to describe the curvature of the surface, and its calculation formula is:

[0069] ;

[0070] By shape index and curvature index , the local geometry of the point cloud surface is expressed as a quantitative feature value, and the local shape descriptor of each point is constructed. Normal vector analysis is performed on the preprocessed point cloud data. A normal vector histogram is constructed to count the direction distribution of the normal vector in the neighborhood of each point. The normal vector is an important feature for describing the local geometry of the surface, and is defined as a unit vector perpendicular to the surface. Given a point The neighborhood point set , the normal vector of the point is estimated by the PCA (principal component analysis) method. After calculating the normal vector of each point, the directional distribution of the normal vector is statistically calculated as a histogram to describe the directional changes in the local neighborhood of the point. This distribution can effectively reflect the roughness and anisotropy of the surface. Based on the initial shape features and normal vector distribution features, a multi-scale feature descriptor is constructed. The multi-scale feature descriptor can comprehensively capture the global shape and local details of the object by extracting local features of points at different scales. By cascading local features of different scales, a comprehensive 3D feature is constructed. The comprehensive 3D feature is spherically projected, and the feature vector is mapped onto the unit sphere to generate a spherical feature distribution. Spherical projection describes the features through a unified coordinate system, so that the features are rotationally invariant and adapt to different viewing angles and rotation operations. Based on the generated spherical feature distribution, a modified spherical harmonic basis function set is constructed. Spherical harmonics is a type of orthogonal basis function defined on a sphere, which is often used to process frequency analysis of three-dimensional data. By adjusting the order of the spherical harmonic function and frequency , generate basis functions that adapt to different point cloud features. The definition of spherical harmonics is:

[0071] ;

[0072] in, and is the angle in spherical coordinates, are the associated Legendre polynomials, represents the order, Represents frequency. By performing modified spherical harmonic expansion on the spherical feature distribution, the complex three-dimensional shape is described as a set of spherical harmonic coefficient vectors. By performing modified spherical harmonic expansion on the spherical feature distribution, the expansion coefficients are calculated to obtain a set of spherical harmonic coefficient vectors. Each dimension of the spherical harmonic coefficient vector corresponds to a spherical harmonic basis of different orders and frequencies. These coefficients are used to describe the shape characteristics of the object surface. In order to simplify the description and improve computational efficiency, low-order spherical harmonic coefficients are selected to construct a compact feature representation. Low-order coefficients can retain the global characteristics of the shape, while discarding high-order coefficients helps to reduce noise and unnecessary details. The shape descriptor of the object is obtained by performing feature normalization and dimensionality reduction operations on the spherical harmonic coefficient vector. The normalization operation ensures the scale consistency of the feature values ​​and avoids the influence of large differences in the magnitude of different features on the recognition results. The dimensionality reduction operation removes redundant features and reduces computational complexity through principal component analysis or other dimensionality reduction methods to obtain an efficient and compact object shape descriptor.

[0073] In one example, data enhancement is performed on preprocessed point cloud data based on a random rectangular occlusion algorithm, and a multi-scale feature fusion network including a global branch, a local coarse-grained branch, and a local fine-grained branch is constructed. The first object recognition model is obtained through joint loss function training, including:

[0074] A random rectangular occlusion algorithm is applied to the preprocessed point cloud data to obtain an occlusion-enhanced point cloud dataset by randomly generating multiple rectangular occlusion regions in the point cloud and removing the points in the regions.

[0075] Input the occlusion enhanced point cloud dataset into a multi-scale feature fusion network, which includes a global branch network, a coarse-grained branch network, a fine-grained branch network, a feature fusion module and a classification head;

[0076] The global branch network contains 5 3D convolutional layers and 2 fully connected layers. Each 3D convolutional layer is followed by a batch normalization layer and a ReLU activation function. The global features of the occlusion-enhanced point cloud dataset are extracted layer by layer through the global branch network to obtain the first global feature representation.

[0077] The occlusion-enhanced point cloud dataset is divided into 3×3×3 voxels to obtain 27 local regions, and each local region is input into a coarse-grained branch network. The coarse-grained branch network contains 3 3D convolutional layers and 1 fully connected layer. By extracting features from each local region and performing feature fusion, the first coarse-grained local feature representation is obtained.

[0078] The occlusion-enhanced point cloud dataset is divided into 5×5×5 voxels to obtain 125 fine local regions, and each fine local region is input into a fine-grained branch network. The fine-grained branch network contains 2 3D convolutional layers and 1 fully connected layer. By extracting features from each fine local region and performing feature fusion, the first fine-grained local feature representation is obtained.

[0079] Inputting the first global feature representation, the first coarse-grained local feature representation, and the first fine-grained local feature representation into a feature fusion module, the feature fusion module includes two fully connected layers and one self-attention layer, and obtains a first multi-scale fusion feature by adaptively weighting and fusing features of different scales;

[0080] The first multi-scale fusion feature is input into the classification head, which contains two fully connected layers and one Softmax layer. The object category prediction result is obtained by mapping the fusion feature to the category probability distribution space.

[0081] Design a joint loss function, including cross entropy loss, center loss and triplet loss. Cross entropy loss is used to optimize classification accuracy, center loss is used to increase intra-class compactness, and triplet loss is used to increase inter-class distance. By weighted combination of these three losses, the overall loss function is obtained.

[0082] The loss value of the object category prediction result is calculated based on the overall loss function, and the multi-scale feature fusion network is trained end-to-end using the Adam optimizer. The gradient is calculated and the network parameters are updated through the back propagation algorithm to obtain the first object recognition model.

[0083] In this example, multiple rectangular occlusion regions are randomly generated in the point cloud, and the points in these regions are removed to simulate the occlusion that may occur in real scenes, and an occlusion-enhanced point cloud dataset is generated. The occlusion enhancement process helps the model adapt to different types of noise and missing data during training, and improves the model's ability to recognize partially occluded objects. The generated occlusion-enhanced point cloud dataset is input into the multi-scale feature fusion network for processing. The network consists of multiple branches, including a global branch network, a coarse-grained branch network, a fine-grained branch network, a feature fusion module, and a classification head. Each branch network extracts features for different scales of point cloud data, thereby improving the model's ability to perceive different details and structures. The global branch network contains 5 3D convolutional layers and 2 fully connected layers. In the global branch, each 3D convolutional layer is followed by a batch normalization layer and a ReLU activation function. The batch normalization layer can ensure the stable distribution of data during training and avoid numerical explosion or gradient disappearance, while the ReLU activation function provides nonlinear expression capabilities for the model. Through the global branch network, the global features in the occlusion-enhanced point cloud dataset are extracted layer by layer to generate the first global feature representation. At the same time, the occlusion-enhanced point cloud dataset is divided into 3×3×3 voxels to generate 27 local regions. Each local region represents a subspace of the point cloud and retains local geometric information. These local regions are input into the coarse-grained branch network. The coarse-grained branch network consists of 3 3D convolutional layers and 1 fully connected layer. Through the operation of these convolutional layers, the features of each local region can be extracted layer by layer. After the extraction is completed, these features are fused to obtain the first coarse-grained local feature representation, which can effectively characterize the local geometric structure of the object. In order to capture more detailed geometric information, the occlusion-enhanced point cloud dataset is divided into 5×5×5 voxels to generate 125 fine local regions. These fine local regions are input into the fine-grained branch network, which consists of 2 3D convolutional layers and 1 fully connected layer. By extracting and fusing the geometric structure of each region, the first fine-grained local feature representation is finally generated. Compared with coarse-grained features, fine-grained features can better characterize the tiny details and local changes of objects, and improve the model's ability to recognize complex geometric shapes. The first global feature representation, the first coarse-grained local feature representation, and the first fine-grained local feature representation are input together into the feature fusion module for comprehensive processing. The feature fusion module contains 2 fully connected layers and 1 self-attention layer. The fully connected layer further integrates features of different scales through linear transformation, while the self-attention layer is responsible for adaptively weighting features of different scales according to the importance of the features to ensure that key information can be retained during fusion. Through this module, the first multi-scale fusion feature is generated. The multi-scale fusion feature is input into the classification head. The classification head contains 2 fully connected layers and 1 Softmax layer.The fully connected layer maps the features to the classification space, while the Softmax layer converts the generated features into a category probability distribution. In this way, the objects are classified and the object category prediction results are generated. In order to ensure the classification accuracy of the model, a joint loss function is designed, which combines the cross entropy loss, the center loss and the triple loss. Cross entropy loss. Used to optimize classification accuracy, the formula is:

[0084] ;

[0085] in, It is The true labels of samples, is the class probability predicted by the model. The cross entropy loss improves classification accuracy by minimizing the difference between the true label and the predicted probability. By reducing the feature differences of samples in the same category and increasing the intra-class closeness, samples in the same category are made closer. By increasing the distance between different categories, the inter-class distinction is improved, making the sample features of different categories more separated. By weighted combination of these three losses, the overall loss function is formed:

[0086] ;

[0087] in, , and are weighted coefficients that control the impact of the three losses on the final loss value. Based on the overall loss function, the loss value of the object category prediction result is calculated, and the Adam optimizer is used to perform end-to-end training on the entire multi-scale feature fusion network. The Adam optimizer can accelerate the convergence process of the network by adaptively adjusting the learning rate, while avoiding problems such as overfitting and gradient explosion. During the training process, the gradient of the loss function is calculated through the back-propagation algorithm, and the parameters in the network are updated based on these gradients to gradually improve the recognition ability of the model.

[0088] In one example, a prototype network is used to perform transfer learning on a first object recognition model, and a second object recognition model is obtained through task sampling, internal and external loop training, and meta-learning optimization, including:

[0089] The first object recognition model is structurally decomposed, the feature extraction layer is used as the embedding function, and the classification layer is used as the prototype calculation layer to obtain the initial prototype network structure;

[0090] Build a task sampler, randomly generate a task pool based on the new scene data set;

[0091] Sample a task from the task pool, input the support set of the task into the embedding function of the initial prototype network structure, perform category prototype calculation on the obtained feature vector, and obtain the prototype vector of each category by calculating the mean of the sample features within the class;

[0092] Input the query set of the task into the embedding function, calculate the Euclidean distance between the query sample and the prototype vector of each category, convert the distance into category probability distribution through the Softmax function, and obtain the classification result of the query set;

[0093] The cross entropy loss is calculated based on the classification results of the query set and the true label, and the meta-learning objective function is constructed in combination with the L2 regularization term. The parameters of the initial prototype network structure are updated, and the inner loop training is completed to obtain the task-adaptive prototype network.

[0094] Repeat the training of multiple tasks, accumulate the gradients of each task, process the accumulated gradients through the meta-optimizer, update the parameters of the embedding function, complete the outer loop training, and obtain the second object recognition model.

[0095] In this example, the original first object recognition model is structurally disassembled. The feature extraction layer of the model is used as an embedding function, which is used to extract high-dimensional feature representations of input samples, while the classification layer is reconstructed as a prototype calculation layer, which calculates the prototype vector of each category based on the sample features of different categories. The purpose of structural decomposition is to convert the model into a prototype network structure so that it can better adapt to the needs of completing tasks with a small number of samples. A task sampler is constructed, which generates a series of different tasks based on a new scene dataset to form a task pool. Each task in the task pool contains a support set and a query set. The support set is used to train the model, while the query set is used to evaluate the performance of the model in a specific task. The task sampler randomly samples a task from the task pool and inputs the support set into the embedding function of the initial prototype network structure. The embedding function extracts the high-dimensional feature vector of each sample in the support set through a convolutional or fully connected layer. The category prototype is calculated for the feature vector. The category prototype represents the typical features of a category and can be calculated by the mean of the features of samples of the same category in the support set. Given a support set ,in It is a sample. is the category label of the sample, category The prototype vector It can be expressed as:

[0096] ;

[0097] in, Indicates that the support set belongs to the category The set of all samples of is the feature vector output by the embedding function, Yes Category The number of samples. Through this process, the prototype vector of each category is calculated. The query set of the task is input into the same embedding function to extract the feature representation of each sample in the query set. The Euclidean distance between the query sample and the prototype vector of each category is calculated. Assume that the feature vector of the query sample is ,category The prototype vector is , then the Euclidean distance between them It can be calculated by the following formula:

[0098] ;

[0099] The Euclidean distance describes the similarity between the query sample and the prototype of each category. The smaller the distance, the more likely the sample belongs to that category. In order to convert the Euclidean distance into a category probability distribution, the Softmax function is used for normalization. Given the distance between the query sample and all category prototypes, the Softmax function maps each distance to a category probability value. , the formula is:

[0100] ;

[0101] Through this process, a category probability distribution is generated for each sample in the query set, and the classification result of the query set is obtained. After the classification is completed, the cross entropy loss is calculated based on the classification result of the query set and the true label. The cross entropy loss is used to measure the difference between the category distribution predicted by the model and the true category distribution. Its formula is:

[0102] ;

[0103] in, It is The true category of the query samples, is the class probability predicted by the model for this sample. In order to prevent the model from overfitting, the L2 regularization term is introduced. L2 regularization constrains the model complexity by imposing a penalty on the size of the model parameters. Combining the cross entropy loss and the L2 regularization term, the system constructs the objective function of meta-learning:

[0104] ;

[0105] in, are the parameters of the embedding function, is the regularization coefficient. By minimizing the objective function, the parameters of the prototype network are optimized to make it have better performance on the current task. After completing the training of a task, the parameters of the initial prototype network structure are updated to generate a task-adaptive prototype network. In order to improve the generalization ability of the model, multiple different tasks are repeatedly performed. After each task training, the gradient of the task is accumulated, and the accumulated gradient is processed by a meta-optimizer (such as the Adam optimizer). The meta-optimizer updates the parameters of the embedding function on multiple tasks so that the model can adapt to more diverse task scenarios. Specifically, after each task training, the meta-optimizer uses the accumulated gradients to update the model parameters and gradually optimizes the embedding function. By repeatedly executing the internal and external loop training process, the system gradually improves the generalization ability and adaptability of the embedding function, and finally obtains the second object recognition model. The model can adapt quickly when facing new tasks, and can accurately classify by relying on the prototype vector even if the number of samples in the task is small.

[0106] In one example, the second object recognition model is deployed to the edge computing unit of the explosion-proof intelligent terminal, and combined with the second depth image collected in real time for online recognition and environmental perception, the object recognition result and environmental safety assessment report are output, including:

[0107] Quantizing and compressing the second object recognition model, and deploying the second object recognition model to an edge computing unit of the explosion-proof intelligent terminal;

[0108] The second depth image is collected in real time by the industrial 3D vision sensor in the explosion-proof intelligent terminal, and the second depth image is converted into a point cloud and preprocessed to obtain real-time point cloud data;

[0109] The real-time point cloud data is input into the global branch network of the second object recognition model, and the global features are extracted layer by layer through 5 3D convolutional layers and 2 fully connected layers. Batch normalization and ReLU activation function are applied after each 3D convolutional layer to obtain the second global feature representation;

[0110] The real-time point cloud data is divided into 3×3×3 voxels to obtain 27 local regions. Each local region is input into the coarse-grained branch network of the second object recognition model. Features are extracted through three 3D convolutional layers and one fully connected layer. The features of the 27 local regions are subjected to maximum pooling operations to obtain the second coarse-grained local feature representation.

[0111] The real-time point cloud data is divided into 5×5×5 voxels to obtain 125 fine local regions. Each fine local region is input into the fine-grained branch network of the second object recognition model. Features are extracted through two 3D convolutional layers and one fully connected layer. The features of the 125 fine local regions are weightedly fused to obtain the second fine-grained local feature representation.

[0112] The second global feature representation, the second coarse-grained local feature representation, and the second fine-grained local feature representation are input into the feature fusion module, feature transformation is performed through two fully connected layers, and the self-attention mechanism is used to adaptively weight features of different scales to obtain a second multi-scale fusion feature;

[0113] Based on the second multi-scale fusion features, the probability distribution of object categories is calculated through the classification head, and the category with the highest probability is selected as the object recognition result;

[0114] Perform scene segmentation and target detection on real-time point cloud data, identify key elements and potential risk factors in the environment, build an environmental semantic map based on object recognition results, evaluate the safety level of the current scene through a rule-based reasoning system, and obtain environmental safety assessment results;

[0115] The object recognition results and environmental safety assessment results are integrated into structured data, and a report including 3D scene reconstruction, object annotation and safety risk heat map is generated through the visualization module to output the object recognition results and environmental safety assessment report.

[0116] In this example, the second object recognition model is quantized and compressed to reduce the computational complexity and memory usage of the model, so that it can run on edge devices with limited resources. The quantization method includes converting floating-point parameters in the model into low-precision integer parameters, such as quantizing 32-bit floating-point numbers into 8-bit integers. This process can significantly reduce computing costs and storage requirements without significantly affecting the recognition accuracy of the model. The quantized second object recognition model is deployed to the edge computing unit of the explosion-proof smart terminal. The edge computing unit in the explosion-proof smart terminal processes the data collected from the 3D vision sensor in real time, and performs object recognition and environmental perception locally to avoid the delay problem caused by cloud computing. After the second depth image is collected in real time by the industrial 3D vision sensor, the depth image is converted and preprocessed into a point cloud to obtain real-time point cloud data. Point cloud data is generated by mapping each pixel of the depth image to three-dimensional space. The specific mapping formula is:

[0117] ;

[0118] in, is the three-dimensional space coordinate, is the two-dimensional pixel coordinate, is the depth value in the depth map, and is the focal length of the camera, and is the optical center position of the camera. Through mapping, the real-time point cloud data can reflect the three-dimensional structure and spatial position of the object. The real-time point cloud data is input into the global branch network of the second object recognition model. The global branch network consists of 5 3D convolutional layers and 2 fully connected layers. Each 3D convolutional layer is followed by a batch normalization layer and a ReLU activation function. The batch normalization layer is used to make each batch of data have a more stable distribution and avoid the problem of gradient disappearance or explosion, while the ReLU activation function introduces nonlinearity to enhance the network's ability to express complex features. After being processed by the global branch network, the global features of the point cloud data are extracted layer by layer, and the second global feature representation is generated. This representation can capture the overall morphology and spatial distribution of the object and is suitable for global object recognition tasks. The real-time point cloud data is divided into 3×3×3 voxels to generate 27 local regions. Each local region represents a subspace of the point cloud and retains local geometric information. The local region is input into the coarse-grained branch network. The coarse-grained branch network contains three 3D convolutional layers and one fully connected layer. These convolutional layers extract the features of each local area, and the maximum pooling operation compresses these features into a smaller feature representation to generate the second coarse-grained local feature representation. The role of the pooling operation is to reduce the dimension of the features and retain the most significant feature information, so that the coarse-grained features can accurately describe the local structure of the object. The real-time point cloud data is divided into 5×5×5 voxels to generate 125 fine local regions. These regions are input into the fine-grained branch network, which consists of two 3D convolutional layers and one fully connected layer. After extracting the features of the fine regions through the convolutional layers, the attention mechanism is used to perform weighted fusion of these features to generate the second fine-grained local feature representation. The attention mechanism can adaptively assign weights according to the importance of the features and enhance the expression of key information. The second global feature representation, the second coarse-grained local feature representation, and the second fine-grained local feature representation are input into the feature fusion module. This module contains 2 fully connected layers, which integrate feature information of different scales through feature transformation operations, and use the self-attention mechanism to perform weighted fusion of global, coarse-grained and fine-grained features. Through adaptive weighting, the feature information at different levels is comprehensively utilized to generate the second multi-scale fusion feature. The second multi-scale fusion feature is input into the classification head for object recognition. The classification head contains 2 fully connected layers and 1 Softmax layer. The first two fully connected layers are responsible for mapping the multi-scale fusion features to the category space, while the Softmax layer maps these features to the category probability distribution. The formula of the Softmax function is:

[0119] ;

[0120] in, is the input sample Belongs to category The probability of is the classifier for the category The prediction score of is the number of all categories. Through the Softmax layer, the probability distribution of object categories is obtained, and the category with the highest probability is selected as the object recognition result. In order to enhance the environmental perception ability, scene segmentation and target detection are performed on the real-time point cloud data. Through the segmentation algorithm, the key elements in the environment are separated from the background, and potential risk factors such as obstacles or dangerous areas are identified. Based on the object recognition results and scene segmentation information, a semantic map of the current environment is constructed, which describes the spatial relationship between different objects and scene elements. Combining the object recognition results and the environmental semantic map, the safety level of the current scene is evaluated through the rule reasoning system. For example, if an object is located in a dangerous area, the risk level of the area is increased according to the preset rules, and finally the environmental safety assessment result is generated.

[0121] In one example, real-time point cloud data is input into the global branch network of the second object recognition model, and global features are extracted layer by layer through 5 3D convolutional layers and 2 fully connected layers. Batch normalization and ReLU activation function are applied after each 3D convolutional layer to obtain the second global feature representation, including:

[0122] voxelize the real-time point cloud data, convert the real-time point cloud data into a three-dimensional grid of fixed size, fill the density value of the point in each voxel, and obtain a density voxel grid;

[0123] Construct a four-dimensional input tensor based on the density voxel grid, where the first three dimensions represent the spatial dimensions and the fourth dimension represents the point cloud density, and obtain the network input data;

[0124] The network input data is input into the first 3D convolution layer, and 32 3×3×3 convolution kernels are used for feature extraction to obtain the primary feature map;

[0125] Perform batch normalization on the primary feature map and apply the ReLU activation function to obtain the first layer activation feature map;

[0126] The activated feature map of the first layer is input into the second to fifth 3D convolutional layers in sequence. Each layer uses a different number of convolution kernels and strides to extract more advanced features layer by layer. Batch normalization and ReLU activation function are applied after each convolution layer to obtain multi-level feature maps.

[0127] Perform a global average pooling operation on the feature map output by the last 3D convolutional layer, compress the three-dimensional feature map into a one-dimensional feature vector, and obtain the global pooling feature;

[0128] The global pooling features are input into the first fully connected layer, linearly transformed using the weight matrix, and the ReLU activation function is applied to obtain the intermediate layer feature representation, and the intermediate layer feature representation is input into the second fully connected layer for feature transformation and dimensionality reduction to obtain the second global feature representation.

[0129] In this example, point cloud data is voxelized. Continuous 3D point cloud data is mapped to a fixed-size 3D grid structure. Each grid unit is called a voxel and represents a spatial region in the point cloud data. The spatial extent of the entire point cloud data is determined and the space is divided into multiple 3D voxel units of uniform size. For each voxel unit, the number of points contained in it is calculated, which is the density value of the voxel. Based on the density voxel grid, a 4D input tensor is constructed. The first three dimensions represent the spatial coordinates of the voxel grid (i.e., the x, y, and z positions of the voxels), and the fourth dimension represents the density value of the point cloud. The 4D input tensor is input to the first 3D convolutional layer for feature extraction. The 3D convolutional layer extracts local spatial features voxel by voxel by applying 32 3×3×3 convolution kernels. Each convolution kernel extracts important geometric patterns from the local spatial region of the input data. Through the convolution operation, a primary feature map is generated, which retains the local geometric structure information of the point cloud data. Perform batch normalization on the primary feature map to standardize the data of each small batch so that its mean is 0 and its variance is 1, which effectively alleviates the problem of gradient vanishing or gradient exploding and improves the training efficiency and stability of the model. Apply the ReLU (Rectified Linear Unit) activation function. The formula of ReLU is:

[0130] ;

[0131] The role of ReLU is to return all eigenvalues ​​less than zero to zero, keep the positive values ​​unchanged, thereby introducing nonlinearity and enabling the neural network to learn more complex features. Through batch normalization and ReLU processing, the activation feature map of the first layer is obtained. The activation feature map of the first layer is input into the second to fifth 3D convolution layers in sequence. These convolution layers use different numbers of convolution kernels and step sizes to extract more advanced features layer by layer. After each convolution operation, batch normalization and ReLU activation functions are applied to ensure the numerical stability and nonlinear expression ability of feature extraction. The setting of the second to fifth convolution layers enables the network to extract features of point clouds from multiple global and local scales, and gradually capture the spatial distribution and geometric shape of point clouds. In the process of convolution, the step size determines the distance that the convolution kernel slides each time, and the number of convolution kernels determines the depth of the feature map. Through multi-level convolution operations, multi-level feature maps are generated. The feature map output by the last 3D convolution layer is subjected to global average pooling operation to compress the entire three-dimensional feature map into a one-dimensional feature vector. Global pooling can retain the global information of feature maps while reducing the dimension of data by calculating the average value of each channel feature map. Its formula is:

[0132] ;

[0133] in, is the output after pooling, is each value in the input feature map, is the total number of values ​​in the feature map. In this way, a global pooling feature is generated to summarize the global geometric information of the entire point cloud data. The global pooling feature is input into the first fully connected layer, and a weight matrix is ​​used to linearly transform the input feature. The formula is:

[0134] ;

[0135] in, is the weight matrix, is the input feature vector, is the bias term, is the output feature. The features are recombined and mapped through the linear transformation system of the fully connected layer. The ReLU activation function is applied to generate the feature representation of the intermediate layer. The intermediate layer feature representation is input into the second fully connected layer, and the features are transformed and reduced to obtain the second global feature representation.

[0136] In one example, the real-time point cloud data is divided into 3×3×3 voxels to obtain 27 local regions, each of which is input into the coarse-grained branch network of the second object recognition model, and features are extracted through three 3D convolutional layers and one fully connected layer. The features of the 27 local regions are subjected to a maximum pooling operation to obtain a second coarse-grained local feature representation, including:

[0137] The real-time point cloud data is spatially uniformly sampled to obtain a uniformly distributed sampling point cloud. Based on the uniformly distributed sampling point cloud, the spatial range of the point cloud is calculated, and the point cloud space is divided into a 3×3×3 grid structure to obtain 27 local areas.

[0138] Density encoding is performed on the points in each local area, and the number of points in each voxel is calculated and normalized to obtain a local density feature map;

[0139] The local density feature map is input into the first 3D convolution layer, 16 3×3×3 convolution kernels are used for feature extraction, and batch normalization and ReLU activation function are applied to obtain the first layer of local feature map;

[0140] The first layer of local feature maps is input into the second 3D convolution layer, 32 3×3×3 convolution kernels are used for feature extraction, and batch normalization and ReLU activation function are applied to obtain the second layer of local feature maps;

[0141] The second-layer local feature map is input into the third 3D convolution layer, 64 3×3×3 convolution kernels are used for feature extraction, and batch normalization and ReLU activation function are applied to obtain the third-layer local feature map;

[0142] Perform a global maximum pooling operation on the local feature map of the third layer to compress the feature map of each channel into a single feature value to obtain a local feature vector of fixed dimension;

[0143] The local feature vectors of 27 local regions are concatenated and input into a fully connected layer, which is linearly transformed and ReLU activated through the weight matrix to obtain the second coarse-grained local feature representation.

[0144] In this example, the real-time point cloud data is spatially uniformly sampled to obtain a sampled point cloud with uniform distribution from the original data. By calculating the density of points in each local area, representative points are selected randomly or based on distance to ensure that the point cloud is evenly distributed in the entire three-dimensional space, avoiding the situation where points are dense or sparse in some areas. The spatial range of the point cloud is calculated based on the uniformly distributed sampled point cloud, and the minimum bounding box is defined to determine the boundary of the point cloud in the three-dimensional space to ensure that the spatial division of the subsequent processing steps is reasonable. By determining the minimum bounding box of the point cloud, the entire point cloud space is divided into a 3×3×3 grid structure, and the point cloud space is divided into 27 uniform local areas. Each local area represents a cubic subspace in the point cloud. The points in each local area are density encoded, and the point cloud density of the area is reflected by calculating the number of points in each voxel. For example, given a voxel grid in a local area, the number of points in each voxel is counted, and then these points are mapped to the interval of 0 to 1 through normalization to obtain a local density feature map, which describes the density distribution of the point cloud in different areas. The local density feature map is input into the first 3D convolutional layer for feature extraction. The first 3D convolution layer uses 16 3×3×3 convolution kernels to extract geometric patterns and spatial features from the local voxel grid. The 3D convolution kernel slides in three-dimensional space and gradually extracts features by capturing the three-dimensional structural information of the point cloud. The results are batch normalized during the convolution operation to ensure that the data remains numerically stable during the convolution process and to avoid the phenomenon of gradient vanishing or exploding. The ReLU activation function is applied, and the formula is:

[0145] ;

[0146] The ReLU function sets all negative values ​​to 0 and keeps positive values ​​unchanged, introducing nonlinearity into the neural network. Nonlinear transformation enables the network to capture more complex feature patterns and improve the recognition ability of point cloud data, obtaining the local feature map of the first layer. The local feature map of the first layer is input into the second 3D convolution layer. The second convolution layer uses 32 The convolution kernel is used to extract deeper features. The second convolution layer also applies batch normalization and ReLU activation function. Through layer-by-layer extraction, more representative geometric features and spatial patterns are gradually mined from the point cloud data. The dimension of the feature map gradually increases with the convolution operation, and the depth of each feature map reflects the number of different features extracted by the network at this layer. After the second layer of convolution processing, a more refined second-layer local feature map is obtained. The second-layer local feature map is input into the third 3D convolution layer. The third convolution layer uses 64 Convolution kernels are used to extract higher-level features. Each convolution operation captures features of different scales and depths, and the global and local geometric patterns are extracted from the spatial structure of the point cloud data by gradually sliding the convolution kernel. Batch normalization and ReLU activation functions are applied to ensure the numerical stability of the feature map and the nonlinear expression ability of the model to obtain the third-layer local feature map. In order to reduce the dimension of the data and retain the most significant features, a global maximum pooling operation is performed on the third-layer local feature map. The feature map of each channel is compressed into a single eigenvalue. The maximum pooling operation selects the maximum value in each feature map as the representative eigenvalue, and the formula is:

[0147] ;

[0148] in, is the output after pooling, is each value in the feature map. Through this process, the entire feature map is compressed into a local feature vector of fixed dimension, retaining the most representative geometric information and reducing feature redundancy. The local feature vectors of 27 local areas are spliced ​​to form a comprehensive feature representation. By splicing local features, the global and local information of the entire point cloud space is captured. The spliced ​​features are input into a fully connected layer. The fully connected layer linearly transforms the input features through a weight matrix, and the formula is:

[0149] ;

[0150] in, is the weight matrix, is the input feature vector, is the bias term, is the output feature. The fully connected layer recombines and maps the local features to the new feature space, and introduces the ReLU activation function to generate the second coarse-grained local feature representation.

[0151] In one example, the real-time point cloud data is divided into 5×5×5 voxels to obtain 125 fine local regions, each of which is input into the fine-grained branch network of the second object recognition model, and features are extracted through 2 3D convolutional layers and 1 fully connected layer. The features of the 125 fine local regions are weightedly fused to obtain the second fine-grained local feature representation, including:

[0152] Adaptively sample the real-time point cloud data to obtain a sampled point cloud with detailed features preserved. Compute the principal axis direction of the point cloud based on the sampled point cloud with detailed features preserved. Rotate and align the point cloud space and divide it into a 5×5×5 grid structure to obtain 125 fine local areas.

[0153] Feature encoding is performed on the points in each fine local area, and a multi-channel feature map is obtained by calculating the spatial distribution and local geometric features of the points in each voxel;

[0154] The multi-channel feature map is input into the first 3D convolution layer, 32 3×3×3 convolution kernels are used for feature extraction, and the LeakyReLU activation function and instance normalization are applied to obtain the first layer of fine-grained feature map;

[0155] The first layer of fine-grained feature maps are input into the second 3D convolution layer, 64 3×3×3 convolution kernels are used for feature extraction, and the LeakyReLU activation function and instance normalization are applied to obtain the second layer of fine-grained feature maps;

[0156] Perform self-attention calculation on the second-layer fine-grained feature map, and obtain the attention-enhanced feature map by calculating the correlation matrix inside the feature map and performing weighted summation;

[0157] The attention-enhanced feature map is input into the fully connected layer, and linearly transformed and LeakyReLU activated by the weight matrix to obtain a local feature vector of fixed dimension;

[0158] The local feature vectors of 125 fine local regions are weighted by channel attention, and the second fine-grained local feature representation is obtained by learning the importance weight of each channel and performing weighted summation.

[0159] In this example, the detail features in the point cloud are retained through adaptive sampling technology. Adaptive sampling is a dynamic sampling method that can adjust according to the geometric complexity of different areas in the point cloud data. In areas with complex geometric structures, such as edges or curved areas, the system will retain more points to ensure that detailed features are not lost. In flat or regular areas, the system will reduce the sampling points to avoid the influence of redundant data. Through the adaptive sampling method, a sampled point cloud that retains detailed features is obtained. Based on the sampled point cloud, the principal axis direction of the point cloud is calculated. The principal axis direction is calculated by principal component analysis (PCA), which reflects the main distribution direction of the point cloud data in space. The PCA algorithm will perform eigendecomposition on the covariance matrix of the point cloud data and obtain the eigenvector corresponding to the maximum eigenvalue as the principal axis direction. The formula of the covariance matrix is:

[0160] ;

[0161] in, is the covariance matrix of the point cloud data, It is The coordinates of the points, is the mean of the point cloud, is the total number of points. By performing eigenvalue decomposition on the covariance matrix, the main axis direction of the point cloud is determined to help rotate and align the point cloud. After obtaining the main axis direction of the point cloud, the point cloud space is rotated and aligned so that the main distribution direction of the point cloud is aligned with the coordinate axis. The point cloud space after rotation and alignment is divided into A grid structure is formed to form 125 fine local regions. Each local region represents a small cube in the point cloud, which can capture local geometric details. The points in each fine local region are feature encoded, and the spatial distribution and local geometric features of the points in each voxel are calculated. The distribution of points within a voxel can reflect the geometric complexity of the local region. For example, some areas may have a higher density, while other areas may be sparse. By quantizing the distribution of these points, a multi-channel feature map is generated, in which each channel corresponds to a different geometric feature, which can express the complexity of the area. The multi-channel feature map is input into the first 3D convolutional layer. This convolutional layer uses 32 The convolution kernel is used for feature extraction. The 3D convolution kernel can capture the spatial structure pattern in the feature map and gradually extract geometric features. In order to maintain numerical stability and improve the nonlinear expression ability of the network, LeakyReLU activation function and instance normalization are applied. LeakyReLU is an improved ReLU activation function that allows negative values ​​to pass through, but multiplies them by a small slope parameter , the formula is:

[0162] ;

[0163] in, is a very small constant, usually set to 0.01. Compared with traditional ReLU, LeakyReLU can prevent the "death" of neurons, allowing the network to learn more negative features. Instance normalization eliminates the impact of batch size on training by normalizing each sample independently, thereby improving the model's adaptability to different data distributions. Through these processes, the first layer of fine-grained feature maps is obtained. The first layer of fine-grained feature maps is input into the second 3D convolutional layer. The second convolutional layer uses 64 The convolution kernel is used for deeper feature extraction. Similar to the first convolution layer, the LeakyReLU activation function and instance normalization are applied to ensure the stability and robustness of feature extraction. Through the convolution operation, more detailed features are gradually extracted to obtain the second-layer fine-grained feature map. After the second layer of convolution is completed, self-attention calculation is performed on the second-layer fine-grained feature map. The self-attention mechanism captures global dependencies by calculating the correlation between different positions within the feature map. The self-attention mechanism calculates the correlation matrix of the feature map and then performs a weighted summation of the correlation. The formula is:

[0164] ;

[0165] in, is the query matrix, is the key matrix, is the value matrix, is the scaling factor. Through this formula, the system can adaptively assign weights according to the similarity of different positions within the feature map to obtain the feature map with enhanced attention. The feature map with enhanced attention is input to the fully connected layer. The fully connected layer linearly transforms the input features through a weight matrix, and the formula is:

[0166] ;

[0167] in, is the weight matrix, is the input feature vector, is the bias term, is the output feature. The function of the fully connected layer is to further combine and map the features to generate a local feature vector of fixed dimension. In order to improve the nonlinear expression ability, the LeakyReLU activation function is applied. The local feature vectors of 125 fine local areas are weighted by channel attention. By learning the importance weight of each channel, the channels that are more critical to the recognition results are emphasized. The channel features are combined through the weighted sum operation, and the formula is:

[0168] ;

[0169] in, It is a channel The weight of It is a channel Through the weighting mechanism, the expressiveness of the features is improved, and finally the second fine-grained local feature representation is obtained.

[0170] Reference Figure 2 This embodiment provides an explosion-proof intelligent terminal intelligent identification device using an industrial 3D vision sensor, including:

[0171] Acquisition module 1, used to acquire a first depth image of the target scene through an industrial 3D vision sensor in the explosion-proof intelligent terminal and perform point cloud conversion and preprocessing to obtain preprocessed point cloud data;

[0172] The calculation module 2 is used to extract the comprehensive 3D features of local curvature, normal vector distribution and shape index based on the preprocessed point cloud data, and perform feature projection and coefficient calculation through the modified spherical harmonic function to obtain the object shape descriptor;

[0173] Building module 3, used to perform data enhancement on the pre-processed point cloud data based on the random rectangular occlusion algorithm, and construct a multi-scale feature fusion network including a global branch, a local coarse-grained branch, and a local fine-grained branch, and obtain a first object recognition model through joint loss function training;

[0174] Optimization module 4, used to perform transfer learning on the first object recognition model using a prototype network, and obtain a second object recognition model through task sampling, internal and external loop training and meta-learning optimization;

[0175] The output module 5 is used to deploy the second object recognition model to the edge computing unit of the explosion-proof intelligent terminal, perform online recognition and environmental perception in combination with the second depth image collected in real time, and output the object recognition result and the environmental safety assessment report.

[0176] In this embodiment, for the specific implementation of each unit in the above device embodiment, please refer to the above method embodiment, which will not be repeated here.

[0177] Reference Figure 3 In an embodiment of the present invention, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a display screen, an input device, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above method is implemented.

[0178] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied.

[0179] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0180] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided by the present invention and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM.

[0181] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0182] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An explosion-proof intelligent terminal intelligent identification method using an industrial 3D visual sensor, characterized in that: The following steps are involved: The first depth image of the target scene is collected by the industrial 3D vision sensor in the explosion-proof intelligent terminal, and point cloud conversion and preprocessing are performed to obtain preprocessed point cloud data; Based on the preprocessed point cloud data, extracting comprehensive 3D features of local curvature, normal vector distribution and shape index, and performing feature projection and coefficient calculation through modified spherical harmonics to obtain an object shape descriptor; The pre-processed point cloud data is data enhanced based on a random rectangular occlusion algorithm, and a multi-scale feature fusion network including a global branch, a local coarse-grained branch and a local fine-grained branch is constructed, and a first object recognition model is obtained by training with a joint loss function; specifically, the method comprises: applying a random rectangular occlusion algorithm to the pre-processed point cloud data, randomly generating a plurality of rectangular occlusion regions in the point cloud and removing the points in the regions to obtain an occlusion-enhanced point cloud data set; inputting the occlusion-enhanced point cloud data set into a multi-scale feature fusion network, wherein the multi-scale feature fusion network comprises a global branch network, a coarse-grained branch network, a fine-grained branch network, a feature fusion module and a classification head; the global classification The branch network includes 5 3D convolutional layers and 2 fully connected layers, each 3D convolutional layer is followed by a batch normalization layer and a ReLU activation function, and the global features of the occlusion-enhanced point cloud dataset are extracted layer by layer through the global branch network to obtain a first global feature representation; the occlusion-enhanced point cloud dataset is divided into 3×3×3 voxels to obtain 27 local regions, and each local region is input into the coarse-grained branch network, and the coarse-grained branch network includes 3 3D convolutional layers and 1 fully connected layer, and the first coarse-grained local feature representation is obtained by extracting features from each local region and performing feature fusion; the occlusion-enhanced point cloud dataset is divided into 5×5×5 voxels, 125 fine local regions are obtained, and each fine local region is input into the fine-grained branch network, the fine-grained branch network includes 2 3D convolutional layers and 1 fully connected layer, and a first fine-grained local feature representation is obtained by extracting features from each fine local region and performing feature fusion; the first global feature representation, the first coarse-grained local feature representation and the first fine-grained local feature representation are input into a feature fusion module, the feature fusion module includes 2 fully connected layers and 1 self-attention layer, and features of different scales are fused by adaptive weighting to obtain a first multi-scale fusion feature; the first multi-scale fusion feature is input into a classification head, and the classification head includes 2 A fully connected layer and a Softmax layer are used to map the fused features to the category probability distribution space to obtain the object category prediction result; a joint loss function is designed, including cross entropy loss, center loss and triple loss, wherein the cross entropy loss is used to optimize the classification accuracy, the center loss is used to increase the intra-class compactness, and the triple loss is used to increase the inter-class distance. The overall loss function is obtained by weighted combination of these three losses; the loss value of the object category prediction result is calculated based on the overall loss function, and the multi-scale feature fusion network is end-to-end trained using the Adam optimizer, and the gradient is calculated and the network parameters are updated through the back propagation algorithm to obtain the first object recognition model; Using a prototype network to perform transfer learning on the first object recognition model, and obtaining a second object recognition model through task sampling, internal and external loop training, and meta-learning optimization; The second object recognition model is deployed to the edge computing unit of the explosion-proof intelligent terminal, and online recognition and environmental perception are performed in combination with the second depth image collected in real time, and the object recognition results and environmental safety assessment report are output.

2. The method for intelligent identification of explosion-proof intelligent terminals using industrial 3D visual sensors according to claim 1 is characterized in that: The method of collecting a first depth image of the target scene by an industrial 3D vision sensor in the explosion-proof intelligent terminal and performing point cloud conversion and preprocessing to obtain preprocessed point cloud data includes: Performing depth value mapping on each pixel in the first depth image, converting the two-dimensional image coordinates and the corresponding depth values ​​into three-dimensional space coordinates to obtain original point cloud data; Building a spatial index structure based on the original point cloud data, calculating local statistical features of the points, removing abnormal points according to a preset threshold, and obtaining an initial filtered point cloud; Performing noise suppression processing on the initial filtered point cloud, and performing local weighted update on the depth value of each point to obtain denoised point cloud data; Based on the denoised point cloud data, a registration operation is performed on the multi-frame point clouds, and globally consistent point cloud data is obtained by iteratively optimizing the spatial transformation relationship between point pairs; The globally consistent point cloud data is spatially divided, a point aggregation operation is performed in each divided unit, and all points in the unit are replaced with the aggregation result to obtain downsampled point cloud data; The downsampled point cloud data is subjected to principal component analysis and normal vector calculation to obtain a normal vector field of the point cloud, and based on the normal vector field, a local surface fitting is performed on the downsampled point cloud data to obtain preprocessed point cloud data.

3. The method for intelligent identification of explosion-proof intelligent terminals using industrial 3D visual sensors according to claim 1 is characterized in that: The method extracts the comprehensive 3D features of local curvature, normal vector distribution and shape index based on the pre-processed point cloud data, and performs feature projection and coefficient calculation through modified spherical harmonic functions to obtain the object shape descriptor, including: Performing local surface fitting on each point in the preprocessed point cloud data, and calculating the average curvature and Gaussian curvature of the point to obtain the curvature characteristics of the point cloud; Based on the curvature feature, a shape index and a curvature index are calculated for each point, a local shape descriptor of the point is constructed, and an initial shape feature is obtained; Constructing a normal vector histogram for the preprocessed point cloud data, counting the direction distribution of the normal vectors in the neighborhood of each point, and obtaining normal vector distribution characteristics; Based on the initial shape feature and the normal vector distribution feature, a multi-scale feature descriptor is constructed, and a comprehensive 3D feature is obtained by cascading local features of different scales, and a spherical projection is performed on the comprehensive 3D feature, and the feature vector is mapped onto a unit sphere to obtain a spherical feature distribution; Based on the spherical feature distribution, a modified spherical harmonic function basis function set is constructed, and a basis function adapted to the point cloud feature is obtained by adjusting the order and frequency of the spherical harmonic function; The spherical feature distribution is subjected to modified spherical harmonic function expansion, and the expansion coefficients are calculated to obtain a spherical harmonic coefficient vector. Based on the spherical harmonic coefficient vector, low-order coefficients are selected to construct a compact feature representation, and the object shape descriptor is obtained through feature normalization and dimensionality reduction operations.

4. The method for intelligent identification of explosion-proof intelligent terminals using industrial 3D visual sensors according to claim 1 is characterized in that: The method of using a prototype network to perform transfer learning on the first object recognition model, and obtaining a second object recognition model through task sampling, internal and external loop training, and meta-learning optimization, includes: Performing structural decomposition on the first object recognition model, using the feature extraction layer as an embedding function and the classification layer as a prototype calculation layer to obtain an initial prototype network structure; Build a task sampler, randomly generate a task pool based on the new scene data set; Sampling a task from the task pool, inputting the support set of the task into the embedding function of the initial prototype network structure, performing category prototype calculation on the obtained feature vector, and obtaining the prototype vector of each category by calculating the mean of the sample features within the category; Inputting the query set of the task into the embedding function, calculating the Euclidean distance between the query sample and the prototype vector of each category, converting the distance into category probability distribution through the Softmax function, and obtaining the classification result of the query set; Calculating the cross entropy loss based on the classification results and true labels of the query set, building a meta-learning objective function in combination with an L2 regularization term, updating the parameters of the initial prototype network structure, completing inner loop training, and obtaining a task-adaptive prototype network; The training of multiple tasks is repeatedly performed, the gradient of each task is accumulated, the accumulated gradient is processed by a meta-optimizer, the parameters of the embedding function are updated, the outer loop training is completed, and the second object recognition model is obtained.

5. The explosion-proof intelligent terminal intelligent identification method using an industrial 3D visual sensor according to claim 1 is characterized in that: The second object recognition model is deployed to the edge computing unit of the explosion-proof intelligent terminal, and online recognition and environmental perception are performed in combination with the second depth image collected in real time, and the object recognition result and environmental safety assessment report are output, including: Quantizing and compressing the second object recognition model, and deploying the second object recognition model to an edge computing unit of an explosion-proof intelligent terminal; The second depth image is collected in real time by the industrial 3D vision sensor in the explosion-proof intelligent terminal, and the second depth image is converted into a point cloud and preprocessed to obtain real-time point cloud data; Input the real-time point cloud data into the global branch network of the second object recognition model, extract global features layer by layer through 5 3D convolutional layers and 2 fully connected layers, apply batch normalization and ReLU activation function after each 3D convolutional layer, and obtain a second global feature representation; Divide the real-time point cloud data into 3×3×3 voxels to obtain 27 local regions, input each local region into the coarse-grained branch network of the second object recognition model, extract features through three 3D convolutional layers and one fully connected layer, perform a maximum pooling operation on the features of the 27 local regions, and obtain a second coarse-grained local feature representation; Divide the real-time point cloud data into 5×5×5 voxels to obtain 125 fine local regions, input each fine local region into the fine-grained branch network of the second object recognition model, extract features through 2 3D convolutional layers and 1 fully connected layer, perform attention weighted fusion on the features of the 125 fine local regions, and obtain a second fine-grained local feature representation; Input the second global feature representation, the second coarse-grained local feature representation, and the second fine-grained local feature representation into a feature fusion module, perform feature transformation through two fully connected layers, and use a self-attention mechanism to adaptively weight features of different scales to obtain a second multi-scale fusion feature; Based on the second multi-scale fusion feature, the classification head calculates the probability distribution of object categories, and selects the category with the highest probability as the object recognition result; Perform scene segmentation and target detection on the real-time point cloud data, identify key elements and potential risk factors in the environment, build an environmental semantic map based on the object recognition results, evaluate the safety level of the current scene through a rule reasoning system, and obtain an environmental safety assessment result; The object recognition results and the environmental safety assessment results are integrated into structured data, and a report including 3D scene reconstruction, object annotation and safety risk heat map is generated through a visualization module, and the object recognition results and environmental safety assessment report are output.

6. The method for intelligent identification of explosion-proof intelligent terminals using industrial 3D visual sensors according to claim 5 is characterized in that: The real-time point cloud data is input into the global branch network of the second object recognition model, and global features are extracted layer by layer through 5 3D convolutional layers and 2 fully connected layers. Batch normalization and ReLU activation function are applied after each 3D convolutional layer to obtain a second global feature representation, including: voxelize the real-time point cloud data, convert the real-time point cloud data into a three-dimensional grid of a fixed size, fill each voxel with a density value of a point, and obtain a density voxel grid; Based on the density voxel grid, a four-dimensional input tensor is constructed, wherein the first three dimensions represent the spatial dimensions and the fourth dimension represents the point cloud density, to obtain network input data; Input the network input data into the first 3D convolution layer, use 32 3×3×3 convolution kernels to extract features, and obtain a primary feature map; Performing batch normalization on the primary feature map and applying a ReLU activation function to obtain a first layer activation feature map; The activated feature map of the first layer is input into the second to fifth 3D convolutional layers in sequence. Each layer uses a different number of convolution kernels and strides to extract higher-level features layer by layer. Batch normalization and ReLU activation function are applied after each convolution layer to obtain multi-level feature maps. Perform a global average pooling operation on the feature map output by the last 3D convolutional layer, compress the three-dimensional feature map into a one-dimensional feature vector, and obtain the global pooling feature; The global pooling feature is input into the first fully connected layer, a weight matrix is ​​used for linear transformation, and a ReLU activation function is applied to obtain an intermediate layer feature representation, and the intermediate layer feature representation is input into the second fully connected layer, feature transformation and dimensionality reduction operations are performed to obtain a second global feature representation.

7. The method for intelligent identification of explosion-proof intelligent terminals using industrial 3D visual sensors according to claim 5 is characterized in that: The real-time point cloud data is divided into 3×3×3 voxels to obtain 27 local regions, each local region is input into the coarse-grained branch network of the second object recognition model, features are extracted through three 3D convolutional layers and one fully connected layer, and the features of the 27 local regions are subjected to maximum pooling operation to obtain a second coarse-grained local feature representation, including: Performing spatial uniform sampling on the real-time point cloud data to obtain a uniformly distributed sampling point cloud, and calculating the spatial range of the point cloud based on the uniformly distributed sampling point cloud, dividing the point cloud space into a 3×3×3 grid structure to obtain 27 local areas; Density encoding is performed on the points in each local area, and the number of points in each voxel is calculated and normalized to obtain a local density feature map; Input the local density feature map into the first 3D convolution layer, use 16 3×3×3 convolution kernels to extract features, and apply batch normalization and ReLU activation function to obtain the first layer of local feature map; The first layer of local feature maps is input into the second 3D convolution layer, 32 3×3×3 convolution kernels are used for feature extraction, and batch normalization and ReLU activation function are applied to obtain the second layer of local feature maps; The second layer of local feature maps is input into the third 3D convolution layer, 64 3×3×3 convolution kernels are used for feature extraction, and batch normalization and ReLU activation function are applied to obtain the third layer of local feature maps; Performing a global maximum pooling operation on the local feature map of the third layer, compressing the feature map of each channel into a single feature value, and obtaining a local feature vector of fixed dimension; The local feature vectors of 27 local regions are concatenated and input into a fully connected layer, which is linearly transformed and ReLU activated through the weight matrix to obtain the second coarse-grained local feature representation.

8. The method for intelligent identification of explosion-proof intelligent terminals using industrial 3D visual sensors according to claim 5 is characterized in that: The real-time point cloud data is divided into 5×5×5 voxels to obtain 125 fine local regions, each fine local region is input into the fine-grained branch network of the second object recognition model, features are extracted through two 3D convolutional layers and one fully connected layer, and the features of the 125 fine local regions are weightedly fused to obtain a second fine-grained local feature representation, including: Adaptively sampling the real-time point cloud data to obtain a sampled point cloud retaining detail features, and computing the main axis direction of the point cloud based on the sampled point cloud retaining detail features, rotating and aligning the point cloud space, and dividing it into a 5×5×5 grid structure to obtain 125 fine local areas; Feature encoding is performed on the points in each fine local area, and a multi-channel feature map is obtained by calculating the spatial distribution and local geometric features of the points in each voxel; Input the multi-channel feature map into the first 3D convolution layer, use 32 3×3×3 convolution kernels to extract features, and apply LeakyReLU activation function and instance normalization to obtain the first layer of fine-grained feature map; The first layer of fine-grained feature maps are input into the second 3D convolution layer, 64 3×3×3 convolution kernels are used for feature extraction, and the LeakyReLU activation function and instance normalization are applied to obtain the second layer of fine-grained feature maps; Performing self-attention calculation on the second-layer fine-grained feature map, calculating the correlation matrix inside the feature map and performing weighted summation to obtain an attention-enhanced feature map; Input the attention-enhanced feature map into a fully connected layer, perform linear transformation and LeakyReLU activation through a weight matrix, and obtain a local feature vector of fixed dimension; The local feature vectors of 125 fine local regions are weighted by channel attention, and the second fine-grained local feature representation is obtained by learning the importance weight of each channel and performing weighted summation.

9. An explosion-proof intelligent terminal intelligent identification device using an industrial 3D visual sensor, characterized in that: For implementing the steps of the method according to any one of claims 1 to 8, the device comprises: An acquisition module is used to acquire a first depth image of a target scene through an industrial 3D vision sensor in an explosion-proof intelligent terminal and perform point cloud conversion and preprocessing to obtain preprocessed point cloud data; A calculation module, for extracting comprehensive 3D features of local curvature, normal vector distribution and shape index based on the preprocessed point cloud data, and performing feature projection and coefficient calculation through modified spherical harmonics to obtain an object shape descriptor; A construction module is used to perform data enhancement on the pre-processed point cloud data based on a random rectangular occlusion algorithm, and to construct a multi-scale feature fusion network including a global branch, a local coarse-grained branch, and a local fine-grained branch, and to obtain a first object recognition model through joint loss function training; specifically comprising: applying a random rectangular occlusion algorithm to the pre-processed point cloud data, and obtaining an occlusion-enhanced point cloud data set by randomly generating a plurality of rectangular occlusion regions in the point cloud and removing the points in the regions; and inputting the occlusion-enhanced point cloud data set into a multi-scale feature fusion network, wherein the multi-scale feature fusion network includes a global branch network, a coarse-grained branch network, a fine-grained branch network, a feature fusion module, and a classification head; The global branch network includes 5 3D convolutional layers and 2 fully connected layers, each 3D convolutional layer is followed by a batch normalization layer and a ReLU activation function, and the global features of the occlusion-enhanced point cloud dataset are extracted layer by layer through the global branch network to obtain a first global feature representation; the occlusion-enhanced point cloud dataset is divided into 3×3×3 voxels to obtain 27 local regions, and each local region is input into the coarse-grained branch network, and the coarse-grained branch network includes 3 3D convolutional layers and 1 fully connected layer. The features of each local region are extracted and feature fused to obtain the first coarse-grained local feature representation; the occlusion-enhanced point cloud dataset is divided into 5×5×5 voxels The method comprises the following steps: dividing the first global feature representation, the first coarse-grained local feature representation and the first fine-grained local feature representation into a feature fusion module, wherein the feature fusion module comprises two fully connected layers and one self-attention layer, and adaptively weighting and fusing features of different scales to obtain a first multi-scale fusion feature; inputting the first multi-scale fusion feature into a classification head, wherein the classification head comprises 2 fully connected layers and 1 Softmax layer, by mapping the fusion features to the category probability distribution space, the object category prediction result is obtained; a joint loss function is designed, including cross entropy loss, center loss and triple loss, wherein the cross entropy loss is used to optimize the classification accuracy, the center loss is used to increase the intra-class compactness, and the triple loss is used to increase the inter-class distance, and the overall loss function is obtained by weighted combination of these three losses; based on the overall loss function, the loss value of the object category prediction result is calculated, and the multi-scale feature fusion network is end-to-end trained using the Adam optimizer, and the gradient is calculated and the network parameters are updated through the back propagation algorithm to obtain the first object recognition model; An optimization module, configured to perform transfer learning on the first object recognition model using a prototype network, and obtain a second object recognition model through task sampling, internal and external loop training, and meta-learning optimization; The output module is used to deploy the second object recognition model to the edge computing unit of the explosion-proof intelligent terminal, perform online recognition and environmental perception in combination with the second depth image collected in real time, and output the object recognition result and environmental safety assessment report.

Images