Dynamic feature fusion and anomaly detection method for multi-modal data in industrial environment

Abstract

The invention discloses a dynamic feature fusion and anomaly detection method for multi-modal data in an industrial environment, and belongs to the technical field of computers, and the method comprises the steps: carrying out the preprocessing, background interference removal, noise cleaning and data format adjustment of collected industrial data, and generating cleaner and normative point clouds and images; secondly, sampling and querying of different scales are carried out on the preprocessed point cloud data, the calculation complexity is reduced, local geometric structure information is reserved, and the robustness of the data is enhanced; thirdly, inputting the processed point cloud data into a feature extractor, extracting point cloud features and storing the point cloud features; meanwhile, image feature extraction and storage are carried out on the RGB image data; secondly, inputting the point cloud features and the RGB image features into a multi-modal feature dynamic fusion module, performing fusion by fully utilizing information of two modals, and storing fusion features; and finally, performing anomaly detection by combining the point cloud features, the image features and the fusion features, and outputting a feature map of an abnormal region.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a dynamic feature fusion and anomaly detection method for multimodal data in an industrial environment. Background Art

[0002] With the development of industrial automation and intelligent manufacturing, equipment and production lines in industrial environments are becoming more and more complex, and the demand for equipment status monitoring and fault diagnosis is increasing. Traditional single-modal detection methods, such as those based on single sensor data, are simple and easy to use, but in complex industrial environments, single-modal data often cannot provide enough information to accurately identify and locate anomalies. For example, temperature sensors may not be able to detect wear on mechanical parts, and vibration sensors may not capture faults in electrical systems. Therefore, multimodal data fusion technology came into being, which can more comprehensively reflect the operating status of equipment by integrating visual images (RGB images) and point cloud data.

[0003] However, most of the existing multimodal fusion methods adopt a static fusion strategy, that is, the fusion method is fixed during the training phase, and the fusion weights cannot be dynamically adjusted according to the changes in the actual scene, resulting in poor fusion effect when facing different working conditions and environmental changes. In addition, the existing methods have deficiencies in multi-scale feature processing, making it difficult to fully capture abnormal information at different scales. Multimodal data fusion usually requires processing a large amount of high-dimensional data, which consumes a lot of computing resources, especially in real-time detection scenarios, which require higher computing efficiency. Summary of the invention

[0004] In order to solve the shortcomings of the prior art, improve the robustness and accuracy of the fusion effect, reduce the consumption of computing resources, and improve the feasibility of real-time detection, the present invention adopts the following technical solutions:

[0005] The dynamic feature fusion and anomaly detection method of multimodal data in industrial environment includes the following steps:

[0006] Step 1: Construct an industrial environment dataset, including point cloud data and image data;

[0007] Step 2: Sampling the preprocessed point cloud data at different scales to make the sampling points as evenly distributed as possible in space, and querying the point cloud data to find the neighborhood point set within the query range, thereby reducing the computational complexity while retaining the local geometric structure information;

[0008] Step 3: Neighborhood point set obtained by ball query , extract the corresponding features from the feature matrix of the original point cloud so that each sampling point can retain the feature information of its neighborhood points, and perform multi-scale feature extraction and aggregation on the point cloud data to capture information at different levels from local details to overall structure;

[0009] Step 4: Extract image features from RGB image data;

[0010] Step 5: The extracted point cloud features and RGB image features are dynamically fused to obtain fused features, making full use of the information of the two modalities, and the fused features are stored in the fused feature library for subsequent anomaly detection;

[0011] Step 6: Combine point cloud features, image features and fusion features to perform anomaly detection and output the feature map of the abnormal area.

[0012] Furthermore, the step 1 comprises the following steps:

[0013] Step 1.1: Dataset division: Divide the collected data into training sets and test sets for training and testing multi-scale feature extraction and aggregation models and multi-modal feature dynamic fusion models;

[0014] Step 1.2: Remove plane interference; This is achieved by extracting edge points of the point cloud, fitting the plane using the RANSAC algorithm, calculating the distance from the point to the plane, and removing the points in the plane; First, an unorganized edge point cloud is constructed by extracting the front, back, left, and right edge points to reduce the amount of calculation while improving the accuracy of plane fitting; Then, the RANSAC algorithm is used to randomly sample points in the point cloud to fit the plane model, represented by the plane equation Ax+By+Cz+D=0; Next, the distance from each point to the plane is calculated, and the points in the plane whose distance is less than the set threshold are removed; Finally, a point cloud and RGB image with the plane removed are generated, providing cleaner data for subsequent cleaning and filling;

[0015] Step 1.3: Noise cleaning: Density-based clustering is performed on the point cloud data to identify the main target and irrelevant noise areas. The clustering algorithm groups the points into different clusters according to the neighborhood density of each point, and marks a small number of isolated points as noise. After that, the sizes of all clusters are counted and the largest cluster is selected, and the points of other small clusters are removed to ensure that the final data only contains the main target area. This process effectively removes irrelevant noise and makes the data more focused on the main object.

[0016] Step 1.4: Data filling; adjust the preprocessed point cloud and RGB image to a standard square size to meet the needs of subsequent processing; determine the target size and calculate the filling amount of each dimension, and ensure that the point cloud and image are displayed in the center by evenly adding zero values ​​at the boundaries; the filled data retains the original content and has a unified standard shape, which is convenient for subsequent batch processing and model input, and improves the consistency and simplicity of the processing flow.

[0017] Furthermore, the step 2 comprises the following steps:

[0018] Step 2.1: Sampling; Use the farthest point sampling to select representative points from the original point cloud at different scales to ensure that the sampling points evenly cover the entire space and avoid being concentrated in local areas; the core idea of ​​FPS is to select the point farthest from the current sampled point set each time to make the sampling points as evenly distributed as possible in space;

[0019] Step 2.2: Sphere query: Sphere query is a method for constructing a neighborhood of sampling points by setting a radius around each representative point to find a set of neighborhood points within the range.

[0020] Furthermore, in step 2.1, an initial point is randomly selected as the first sampling point, the shortest distance from each remaining point to the sampled point set is calculated, and then the point with the farthest distance is selected to join the sampling set, and the process is repeated until the required number of sampling points is reached. This method helps to evenly distribute the sampling points and avoid concentration in local areas, thereby improving the effect of point cloud feature extraction. The formula is:

[0021]

[0022] Among them, P represents the set of all points in the point cloud, Indicated in scale The representative point set selected below, Represent two points in the set P, respectively. Indicate point and The Euclidean distance between them and the uniform distribution enable the model to fully cover the entire point cloud at different scales, reducing the risk of oversampling in specific areas, thereby improving the ability to capture spatial geometric features.

[0023] Furthermore, the formula corresponding to step 2.2 is as follows:

[0024]

[0025] in, Representation scale and radius The neighborhood point set under , controlling the size of the neighborhood, obtaining fine local information at small scales and capturing broader spatial structures at large scales.

[0026] Furthermore, in step 3, the training of multi-scale feature extraction and aggregation includes the following steps:

[0027] Step 3.1: Feature index extraction: Get the initial features of the neighborhood points by extracting the given index; the formula is:

[0028]

[0029] in, Represents the neighborhood set according to the query The extracted feature matrix, Represents the feature matrix of the original point cloud, where each point corresponds to a feature vector;

[0030] Step 3.2: Local feature processing: Local feature processing is implemented by multi-layer perceptron. MLP is a neural network composed of multiple fully connected layers, which is often used to extract nonlinear features. Here, MLP processes the features of neighborhood points and maps geometric information into an expression of high-dimensional feature space. The formula for local feature processing is:

[0031]

[0032] in, Representation scale The local feature vector extracted under The feature matrix representing the neighborhood set is transformed nonlinearly by the multi-layer perceptron MLP. The model can effectively learn the complex patterns in the local geometric structure of the point cloud, such as curvature, boundaries, etc., to obtain more expressive features;

[0033] Step 3.3: Multi-scale feature aggregation: Aggregate multi-scale voxel features through the gating mechanism and aggregate s gated maps The corresponding feature channels are normalized and the weights are calculated through softmax normalization. Then the corresponding channel weights are multiplied by the original features and aggregated to obtain multi-scale fusion features. Finally, the extracted point cloud features are stored in the point cloud feature library for subsequent anomaly detection.

[0034] Furthermore, the calculation formula of step 3.3 is as follows:

[0035]

[0036] in, represents the learning weight matrix calculated by the linear function, Represents the gating map, which is used to accept or suppress the transmission of each scale feature. Indicates Point cloud features at different scales;

[0037]

[0038] in, represents the final multi-scale fusion feature, a represents the number of scales involved in the fusion process, and softmax represents the Perform normalization operation to obtain the weight distribution of different positions, split[·] s Indicates the extraction of the sth scale from the softmax result The corresponding weight part.

[0039] Furthermore, in step 5, a multimodal feature dynamic fusion model is constructed, and its training process includes the following steps:

[0040] Step 5.1: Multimodal feature extraction: The RGB image and point cloud are passed to the pre-trained feature extractors respectively. When extracting features, the RGB image feature extractor will extract features such as edges and textures in the image layer by layer; the point cloud feature extractor captures the geometric information of the point cloud by processing the spatial structural features of the point cloud data; the features of these two modalities can complement each other in the next fusion process, thereby enhancing the model's comprehensive understanding of the target;

[0041] Step 5.2: Cross-modal attention mechanism: The initial features of the RGB image and point cloud are processed by cross-modal attention, and then the features of each modality are processed separately through a multi-head self-attention layer, and the similarity between them is calculated to dynamically adjust the weights between the modalities, so that the RGB image features and point cloud features are effectively integrated and supplemented in the interaction process;

[0042] Step 5.3: Dynamic Weight Generator: After completing the cross-modal attention mechanism, the feature map obtained contains the interaction information between RGB and point cloud. Next, the fused features are further weighted by the dynamic weight generator. The generated adaptive weight matrix determines the proportion of different modalities in the fusion process, ensuring that important information is highlighted and avoiding interference from irrelevant or redundant information.

[0043] Step 5.4: Multi-scale feature downsampling: A downsampling module similar to the feature pyramid network (FPN) structure is used to perform multi-scale convolution downsampling on the RGB image and SDF point cloud features to gradually reduce the resolution of the feature map and extract high-level semantic information. Each level of downsampling uses a convolutional layer to reduce the resolution and retain key information. This process can generate multi-scale features containing details and global structures and maintain the interactivity of cross-modal features.

[0044] Step 5.5: Multi-scale cross-modal feature fusion; At each scale, use the cross-modal attention mechanism to process the RGB image and SDF point cloud features to ensure that the interactive information between the modalities can be effectively fused at different scales. In particular, after multi-scale downsampling, the structure and importance of the features may change. Therefore, the attention mechanism dynamically adjusts the weights according to the feature structure of each scale to achieve more effective cross-modal information fusion at each scale.

[0045] Step 5.6: Feature upsampling: Upsample the features obtained by downsampling at each scale to restore them to the original resolution. The upsampling operation restores the low-resolution multi-scale feature map to a higher resolution and integrates the multi-scale fusion information into each scale, so that the model can extract finer detail features from the multi-scale global information.

[0046] Step 5.7: Dynamic weighted fusion; Use the dynamic weight generator to perform weighted fusion on the upsampled multi-scale features to obtain fused features; at this time, the weight generator will dynamically adjust the weights according to the importance of each scale feature in the current scene, so that the features at different scales are appropriately weighted, and finally generate a multimodal fusion feature.

[0047] Furthermore, in step 5.2, the similarity matrix between the RGB image features and the point cloud features is calculated based on the feature tensors of the RGB image and the point cloud through the self-attention mechanism; the calculation method is:

[0048]

[0049] in, and Represent the feature tensors of RGB image and point cloud respectively, Represents the self-attention mechanism;

[0050] Use the similarity matrix to update the RGB image features and point cloud features to obtain the feature tensor processed by the cross-modal attention mechanism:

[0051]

[0052]

[0053] Updated feature tensors for images and point clouds and Fusion of interactive information between RGB image and point cloud modalities;

[0054] In step 5.3, the adaptive weight matrix generated , is the feature tensor based on the updated image and point cloud, calculated through a fully connected network FC, and the formula is as follows:

[0055]

[0056] The weight matrix generated Multiply the features of the RGB image and point cloud element by element, and sum them up to get the final weighted fusion feature:

[0057]

[0058] Weighted fusion features Contains the most valuable information in RGB image and point cloud modalities, weight matrix It is adaptive and dynamically adjusted according to the relative importance of input features among different modalities;

[0059] The formula of step 5.4 is:

[0060]

[0061] in, Represents the RGB image and point cloud features after downsampling at the i-th layer.

[0062] Furthermore, the formula of step 5.5 is as follows:

[0063]

[0064]

[0065]

[0066] in, and They represent the feature tensors of the RGB image and point cloud at the i-th scale respectively, Attention represents the self-attention mechanism, Represents the similarity matrix of RGB image and point cloud features at the i-th scale, and They respectively represent the feature tensors of the fused RGB image and point cloud at the i+1th scale.

[0067] The advantages and beneficial effects of the present invention are:

[0068] By introducing a cross-modal attention mechanism and a dynamic weight generator, the present invention can dynamically adjust the fusion weights of different modal features according to the actual scenario, thereby improving the robustness and accuracy of the fusion effect; by adopting multi-scale feature downsampling and upsampling technology, combined with the cross-modal attention mechanism, the abnormal information of different scales is fully captured, thereby improving the comprehensiveness and meticulousness of anomaly detection; by optimizing the feature extraction and fusion algorithms, the consumption of computing resources is reduced, and the feasibility of detection is improved; by combining the multi-modal features of RGB images and point cloud data, the quality status of the product can be more comprehensively reflected, and the accuracy and reliability of anomaly detection can be improved; the present invention has strong adaptability, can adapt to different working conditions and environmental changes, is suitable for various complex industrial application scenarios, and improves the efficiency of product quality control and management. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 is a flow chart of a method in an embodiment of the present invention.

[0070] Figure 2 Schematic diagram of the structure of the multi-scale feature extraction and aggregation module in an embodiment of the present invention.

[0071] Figure 3 It is a framework diagram of dynamic fusion of multimodal features in an embodiment of the present invention. DETAILED DESCRIPTION

[0072] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0073] like Figure 1 As shown in FIG. 1 , the dynamic feature fusion and anomaly detection method of multimodal data in an industrial environment includes the following steps:

[0074] Step 1: First, collect 10 categories of data in the industrial field, and then preprocess them to improve the quality of point clouds and RGB images and the effectiveness of subsequent analysis. Preprocessing includes removing background interference, cleaning noise, and adjusting data formats to generate cleaner and more standardized point clouds and images, ready for subsequent model training. Data preprocessing includes the following steps:

[0075] Step 1.1: Dataset division: Divide the collected data into training set and test set in a ratio of 8:2 for each category, which are used to train and test the multi-scale feature extraction and aggregation model and the multi-modal feature dynamic fusion model.

[0076] Step 1.2: Remove plane interference; The deplaning operation is achieved by extracting edge points of the point cloud, fitting the plane using the RANSAC algorithm, calculating the distance from the point to the plane, and removing the points in the plane. First, an unorganized edge point cloud is constructed by extracting the front, back, left, and right edge points to reduce the amount of calculation while improving the accuracy of plane fitting. Then the RANSAC algorithm is used to randomly sample points in the point cloud to fit the plane model, represented by the plane equation Ax+By+Cz+D=0. Next, the distance from each point to the plane is calculated, and the points in the plane whose distance is less than the set threshold are removed. Finally, a point cloud and RGB image with the plane removed are generated, providing cleaner data for subsequent cleaning and filling.

[0077] Step 1.3: Noise cleaning: Perform density-based clustering on the point cloud data to identify the main target and irrelevant noise areas. The clustering algorithm groups points into different clusters based on the neighborhood density of each point, and marks a small number of isolated points as noise. After that, the size of all clusters is counted and the largest cluster is selected, and the points of other small clusters are removed to ensure that the final data only contains the main target area. This process effectively removes irrelevant noise and makes the data more focused on the main object.

[0078] Step 1.4: Data filling: adjust the preprocessed point cloud and RGB image to a standard square size to meet the needs of subsequent processing. Determine the target size and calculate the filling amount of each dimension, and ensure that the point cloud and image are displayed in the center by evenly adding zero values ​​at the boundaries. The filled data retains the original content and has a unified standard shape, which is convenient for subsequent batch processing and model input, and improves the consistency and simplicity of the processing process.

[0079] Step 2: Sampling and querying the preprocessed point cloud data at different scales to reduce computational complexity while retaining local geometric structure information. Sampling and grouping operations at different scales include the following steps:

[0080] Step 2.1: Sampling; The present invention uses the Farthest Point Sampling (FPS) method to select representative points from the original point cloud at different scales to ensure that the sampling points evenly cover the entire space and avoid concentration in local areas. The core idea of ​​FPS is to select the point farthest from the current sampled point set each time so that the sampling points are distributed as evenly as possible in space. The specific steps include: randomly selecting an initial point as the first sampling point; calculating the shortest distance from each remaining point to the sampled point set; and then selecting the farthest point to add to the sampling set. This process is repeated until the required number is reached. This method helps to evenly distribute the sampling points and avoid concentration in local areas, thereby improving the effect of point cloud feature extraction. The formula is:

[0081]

[0082] Among them, P is the set of all points in the point cloud, is in scale The representative point set selected below, are points in the set P respectively. Indicate point and The uniform distribution enables the model to fully cover the entire point cloud at different scales, reducing the risk of oversampling in specific areas, thereby improving the ability to capture spatial geometric features.

[0083] Step 2.2: Ball Query: Ball Query is a method for constructing a neighborhood of sampling points by setting a radius around each representative point. , thereby finding the set of neighborhood points within the range . Its formula is:

[0084]

[0085] in, Representation scale and radius The neighborhood point set under , the size of the neighborhood can be controlled to obtain fine local information at a small scale and capture a wider range of spatial structures at a large scale.

[0086] Step 3: Input the processed point cloud data into the multi-scale feature extraction and aggregation module to capture information at different levels from local details to overall structure, and store these features in the point cloud feature library. Figure 2 As shown in Figure 1, the training of the multi-scale feature extraction and aggregation model includes the following steps:

[0087] Step 3.1: Feature index extraction: First, the initial features of the neighborhood points are obtained by extracting the given index. This step is based on the neighborhood set obtained by the ball query. , extract the corresponding features from the feature matrix of the original point cloud, so that each sampling point can retain the feature information of its neighboring points. The formula for this step is expressed as:

[0088]

[0089] in, Represents the neighborhood set according to the query The extracted feature matrix, Represents the feature matrix of the original point cloud, where each point corresponds to a feature vector.

[0090] Step 3.2: Local feature processing; Local feature processing is usually implemented through a multi-layer perceptron (MLP). MLP is a neural network composed of multiple fully connected layers, which is often used to extract nonlinear features. Here, MLP processes the features of neighborhood points and maps the geometric information into an expression in a high-dimensional feature space. The formula for local feature processing is:

[0091]

[0092] in, It is a scale The local feature vector extracted under The feature matrix representing the neighborhood set. Through the nonlinear transformation of MLP, the model can effectively learn the complex patterns in the local geometric structure of the point cloud, such as curvature, boundaries, etc., to obtain more expressive features.

[0093] Step 3.3: Multi-scale feature aggregation: Through step 3.2, local features are obtained, and then multi-scale voxel features are aggregated through the gating mechanism. The calculation formula is as follows:

[0094]

[0095] in, represents the learning weight matrix calculated by the linear function, Represents a gating map used to accept or suppress the transmission of features at each scale.

[0096] The present invention aggregates s gated graphs The corresponding feature channels are normalized by softmax to calculate the weights, and then the weights of the corresponding channels are multiplied by the original features and aggregated to obtain the multi-scale fusion features. The calculation formula is as follows:

[0097]

[0098] in, represents the final multi-scale fusion feature, and Respectively represent Gated graphs and point cloud features at different scales.

[0099] Finally, the extracted point cloud features are stored in the point cloud feature library for subsequent anomaly detection.

[0100] Step 4: Extract image features from the RGB image data and store the extracted features in the image feature library for subsequent dynamic fusion and anomaly detection of multimodal data.

[0101] Step 5: Input the extracted point cloud features and RGB image features into the multimodal feature dynamic fusion module for modal fusion, making full use of the information of the two modalities, and store the fused features in the fusion feature library for subsequent anomaly detection. Figure 3 As shown in Figure 1, the training of the multimodal feature dynamic fusion model includes the following steps:

[0102] Step 5.1: Multimodal feature extraction: First, the input RGB image and point cloud are passed to the pre-trained feature extractor respectively. When extracting features, the RGB extractor extracts features such as edges and textures in the image layer by layer; the point cloud feature extractor captures the geometric information of the point cloud by processing the spatial structural features of the point cloud data. The features of these two modalities can complement each other in the next fusion process, thereby enhancing the model's comprehensive understanding of the target.

[0103] Step 5.2: Cross-modal attention mechanism: Input the initial features of RGB and point cloud into the cross-modal attention module for processing. Then process the features of each modality separately through the multi-head self-attention layer, calculate the similarity between them, and dynamically adjust the weights between the modalities, so that RGB and point cloud features can be effectively integrated and supplemented during the interaction process. The similarity matrix between RGB and point cloud features is calculated as follows:

[0104]

[0105] in, and Represent the feature tensors of RGB and point cloud respectively, Represents the self-attention mechanism.

[0106] Use the similarity matrix to update the RGB and point cloud features to obtain the feature tensor processed by the cross-modal attention mechanism:

[0107]

[0108]

[0109] These updated feature tensors and The interactive information between RGB and point cloud modalities is integrated.

[0110] Step 5.3: Dynamic Weight Generator: After completing the cross-modal attention mechanism, the feature map obtained contains the interaction information between RGB and point cloud. Next, these fused features are further weighted by the dynamic weight generator. The generated adaptive weight matrix determines the proportion of different modalities in the fusion process, ensuring that important information is highlighted and avoiding interference from irrelevant or redundant information. Generated adaptive weight matrix It is calculated through a fully connected network, and the formula is as follows:

[0111]

[0112] The weight matrix generated Multiply the RGB and point cloud features element by element and sum them up to get the final weighted fusion features:

[0113]

[0114] The weighted fusion feature here Contains the most valuable information in RGB and point cloud modalities, weight matrix It is adaptive and dynamically adjusted according to the relative importance of input features between different modalities.

[0115] Step 5.4: Multi-scale feature downsampling; In this step, a downsampling module similar to the feature pyramid network (FPN) structure is used to perform multi-scale convolution downsampling on the RGB and SDF features to gradually reduce the resolution of the feature map and extract high-level semantic information. Each level of downsampling uses a convolutional layer to reduce the resolution and retain key information. This process can generate multi-scale features that contain details and global structures while maintaining the interactivity of cross-modal features. The relevant formula is:

[0116]

[0117] in, Represents the RGB image and point cloud features after downsampling at the i-th layer;

[0118] Step 5.5: Multi-scale cross-modal feature fusion; At each scale, use the cross-modal attention mechanism to process RGB and SDF features to ensure that the interactive information between modalities can be effectively fused at different scales. Especially after multi-scale downsampling, the structure and importance of features may change, so the attention mechanism needs to dynamically adjust the weights according to the feature structure of each scale in order to achieve more effective cross-modal information fusion at each scale. The formula is as follows:

[0119]

[0120]

[0121]

[0122] in, is the similarity matrix of RGB and point cloud features at the i-th scale.

[0123] Step 5.6: Feature upsampling: The features obtained by downsampling at each scale need to be upsampled to restore to the original resolution. The upsampling operation restores the low-resolution multi-scale feature map to a higher resolution and integrates the multi-scale fusion information into each scale, so that the model can extract finer detail features from the multi-scale global information.

[0124] Step 5.7: Final dynamic weighted fusion: Use the dynamic weight generator to perform weighted fusion on the upsampled multi-scale features. At this point, the weight generator will dynamically adjust the weights according to the importance of each scale feature in the current scene, so that the features at different scales are appropriately weighted, and finally generate a multi-modal fusion feature.

[0125] Step 6: Combine the point cloud feature library, fusion feature library and image feature library to perform anomaly detection and output the feature map of the abnormal area.

[0126] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. Dynamic feature fusion and anomaly detection method for multimodal data in industrial environments, characterized by The steps include: Step 1: Construct an industrial environment dataset, including point cloud data and image data; Step 2: Sampling the point cloud data at different scales to make the sampling points evenly distributed in space, and querying the point cloud data to find the neighborhood point set within the query range; Step 3: According to the neighborhood point set obtained by the query, the corresponding features are extracted from the features of the original point cloud so that the sampling point retains the feature information of its neighborhood points. Multi-scale feature extraction and aggregation are performed on the point cloud data to capture information at different levels from local details to overall structure. Step 4: Extract image features from image data; Step 5: Perform multi-modal feature dynamic fusion on the extracted point cloud features and image features to obtain fused features; Step 6: Combine point cloud features, image features and fusion features to perform anomaly detection and output the feature map of the abnormal area.

2. The method for dynamic feature fusion and anomaly detection of multimodal data in an industrial environment according to claim 1, characterized in that: The step 1 comprises the following steps: Step 1.1: Dataset division: Divide the collected data into training set and test set; Step 1.2: Remove plane interference; This is achieved by extracting edge points of the point cloud, fitting a plane, calculating the distance from the point to the plane, and removing points in the plane. First, an unorganized edge point cloud is constructed by extracting the front, back, left, and right edge points. Then, points are randomly sampled in the point cloud to fit the plane model. Next, the distance from each point to the plane is calculated, and points in the plane whose distance is less than a set threshold are removed. Finally, a point cloud and RGB image without the plane are generated. Step 1.3: Noise cleaning: Density-based clustering is performed on the point cloud data to identify the main targets and irrelevant noise areas. The clustering algorithm groups the points into different clusters according to the neighborhood density of each point, and marks a small number of isolated points as noise. After that, the sizes of all clusters are counted and the largest cluster is selected, and the points of other small clusters are eliminated. Step 1.4: Data padding; adjust the preprocessed point cloud and RGB image to a standard square size; determine the target size and calculate the padding amount for each dimension, ensuring that the point cloud and image are displayed in the center by evenly adding zero values ​​at the boundaries.

3. The method for dynamic feature fusion and anomaly detection of multimodal data in an industrial environment according to claim 1, characterized in that: The step 2 comprises the following steps: Step 2.1: Sampling: Use the farthest point sampling to select representative points from the original point cloud at different scales, and each time select the point farthest from the current sampled point set to make the sampling points evenly distributed in space; Step 2.2: Ball query; By setting a radius around each representative point, a set of neighborhood points within the range is found.

4. The method for dynamic feature fusion and anomaly detection of multimodal data in an industrial environment according to claim 3, characterized in that: In step 2.1, an initial point is randomly selected as the first sampling point, the shortest distance from each remaining point to the sampled point set is calculated, and then the point with the farthest distance is selected to be added to the sampling set. The process is repeated until the required number of sampling points is reached. The formula is: , Among them, P represents the set of all points in the point cloud, Indicated in scale The representative point set selected below, Represent two points in the set P, respectively. Indicate point and The Euclidean distance between .

5. The method for dynamic feature fusion and anomaly detection of multimodal data in an industrial environment according to claim 3, characterized in that: The corresponding formula of step 2.2 is as follows: , in, Representation scale and radius The neighborhood point set under , controlling the size of the neighborhood, obtaining fine local information at small scales and capturing broader spatial structures at large scales.

6. The method for dynamic feature fusion and anomaly detection of multimodal data in an industrial environment according to claim 1, characterized in that: In step 3, the training of multi-scale feature extraction and aggregation includes the following steps: Step 3.1: Feature index extraction: Obtain the initial features of the neighborhood points through given index extraction; Step 3.2: Local feature processing: Local feature processing processes the features of neighborhood points through a multi-layer perceptron and maps the geometric information into an expression in a high-dimensional feature space. Step 3.3: Multi-scale feature aggregation; Multi-scale voxel features are aggregated through a gating mechanism, feature channels corresponding to the gating map are aggregated, and weights are calculated through normalization. The corresponding channel weights are then multiplied by the original features and aggregated to obtain multi-scale fusion features. Finally, the extracted point cloud features are stored in a point cloud feature library.

7. The method for dynamic feature fusion and anomaly detection of multimodal data in an industrial environment according to claim 6, characterized in that: The calculation formula of step 3.3 is as follows: , in, represents the learning weight matrix calculated by the linear function, Represents the gating map, which is used to accept or suppress the transmission of each scale feature. Indicates Point cloud features at different scales; , in, represents the final multi-scale fusion feature, a represents the number of scales involved in the fusion process, softmax represents the normalization operation, split[·] s Indicates the extraction of the sth scale from the softmax result The corresponding weight part.

8. The method for dynamic feature fusion and anomaly detection of multimodal data in an industrial environment according to claim 1, characterized in that: In step 5, a multimodal feature dynamic fusion model is constructed, and its training process includes the following steps: Step 5.1: Multimodal feature extraction: The image and point cloud are respectively passed into the pre-trained feature extractor. When extracting features, the image feature extractor will extract features such as edges and textures in the image layer by layer; The point cloud feature extractor captures the geometric information of the point cloud by processing the spatial structural features of the point cloud data; Step 5.2: Cross-modal attention mechanism: The initial features of the image and point cloud are processed by cross-modal attention, and then the features of each modality are processed separately through the multi-head self-attention layer, and the similarity between them is calculated to dynamically adjust the weights between the modalities, so that the image features and point cloud features are integrated and supplemented in the interaction process; Step 5.3: Dynamic weight generator; The fused features are further weighted through a dynamic weight generator; the generated adaptive weight matrix determines the proportion of different modalities in the fusion process; Step 5.4: Multi-scale feature downsampling; Perform multi-scale convolution downsampling on image and point cloud features to gradually reduce the resolution of feature maps and extract high-level semantic information. Each level of downsampling reduces the resolution and retains key information, generating multi-scale features that contain details and global structures, and maintaining the interactivity of cross-modal features. Step 5.5: Multi-scale cross-modal feature fusion; At each scale, a cross-modal attention mechanism is used to process image and point cloud features. The attention mechanism dynamically adjusts the weights according to the feature structure of each scale. Step 5.6: Feature upsampling; The features obtained by downsampling at each scale are then upsampled to restore to the original resolution; Step 5.7: Dynamic weighted fusion; The dynamic weight generator is used to perform weighted fusion on the upsampled multi-scale features to obtain the fused features.

9. The method for dynamic feature fusion and anomaly detection of multimodal data in an industrial environment according to claim 8, characterized in that: In the step 5.2, the similarity matrix between the image features and the point cloud features is calculated based on the feature tensors of the image and the point cloud through the self-attention mechanism; the image features and the point cloud features are updated using the similarity matrix to obtain the feature tensors processed by the cross-modal attention mechanism; the updated feature tensors of the image and the point cloud incorporate the interactive information between the image and the point cloud modalities; In step 5.3, the generated adaptive weight matrix is ​​calculated based on the updated feature tensors of the image and point cloud through a fully connected network; the generated weight matrix is ​​element-by-element multiplied with the features of the image and point cloud, and the sum is performed to obtain the final weighted fusion feature; The weighted fusion features contain the most valuable information in the image and point cloud modalities, and the weight matrix is ​​adaptive and dynamically adjusted according to the relative importance of the input features between different modalities.

10. The method for dynamic feature fusion and anomaly detection of multimodal data in an industrial environment according to claim 8, characterized in that: The formula of step 5.5 is as follows: , , , in, and They represent the feature tensors of the image and point cloud at the i-th scale respectively, Attention represents the self-attention mechanism, Represents the similarity matrix of image and point cloud features at the i-th scale, and They respectively represent the feature tensors of the fused image and point cloud at the i+1th scale.

Images