Multi-modal data driven precision shaft appearance detection system and method

Abstract

The invention provides a multi-modal data-driven precision shaft appearance detection system and method, relates to the technical field of appearance detection, and aims to perform multi-modal detection on the appearance of a precision shaft, extract appearance damage characteristics of different modals and improve the detection accuracy of the appearance of the precision shaft. And determining the recognition attention of each mode in the appearance detection of the precision shaft based on the information variance of the appearance damage characteristics in each mode and the distribution difference of the appearance damage among the modes. Extracting collaborative information of appearance damage feature interactive learning among different modes, determining fusion weights of detection data of different modes according to the recognition attention of each mode, and further constructing an appearance detection model through the collaborative information and the fusion weights of the detection data of different modes; and inputting the multi-modal detection data detected in real time into the appearance detection model, identifying the defect characteristics of the appearance of the precision shaft, and visually displaying the defect characteristics, so that the multi-modal cooperative detection of the appearance defects of the precision shaft can be realized, and the robustness of the appearance detection of the precision shaft is improved.

Description

Technical Field

[0001] The present application relates to the technical field of appearance inspection, and more specifically, to a multimodal data-driven precision shaft appearance inspection system and method. Background Art

[0002] Appearance inspection technology plays a vital role in the appearance inspection of precision shafts. Through high-precision visual, geometric and thermal imaging information collection and analysis, it can comprehensively and accurately identify various defects on the surface and structure of precision shafts, such as scratches, cracks, dimensional deviations and material unevenness, to ensure that the quality and performance of the shaft meet production standards.

[0003] In the appearance inspection of precision shafts, the traditional single-modal data detection method has certain limitations when dealing with complex damage types and minor defects. For example, visible light images can only reveal the texture characteristics of the surface, and have limited recognition capabilities for minor or hidden damage such as cracks and structural deformations. Although the point cloud data provided by laser scanning equipment can accurately capture the geometric shape of the object surface, it has poor detection effects on minor damage such as surface defects and temperature changes. Thermal imaging technology can effectively detect thermal anomalies and reveal thermal differences caused by structural defects or material inhomogeneities, but its capabilities in high resolution and surface detail presentation are relatively weak. Therefore, a single modality often cannot fully and accurately detect all defects on the surface and structure of precision shafts. However, by fusing multimodal data obtained from multimodal equipment (such as cameras, laser scanning equipment, and thermal imagers), data from different modalities can be combined to achieve accurate identification and positioning of different types of damage, thereby making up for the shortcomings of a single modality in dealing with different types of damage. Therefore, how to achieve multimodal collaborative detection of appearance defects of precision shafts and thus improve the robustness of appearance inspection of precision shafts has become a difficult problem faced by the industry. Summary of the invention

[0004] The present application provides a multimodal data-driven precision shaft appearance inspection system and method, which can realize multimodal collaborative detection of precision shaft appearance defects, thereby improving the robustness of precision shaft appearance inspection.

[0005] In a first aspect, the present application provides a multimodal data-driven precision shaft appearance detection method, the detection method comprising the following steps: Based on the configured camera equipment, laser scanning equipment and thermal imager, the appearance of the precision shaft is inspected to obtain multi-modal inspection data; Performing calibration and alignment processing on the multimodal detection data to obtain aligned multimodal detection data, and then mapping the aligned multimodal detection data to a multimodal space of a unified coordinate system; Extracting appearance damage features of different modes from the multimodal space, and then determining the information variance of the appearance damage features in each mode, and determining the recognition attention of each mode in the precision shaft appearance inspection based on the information variance of the appearance damage features in each mode and the distribution difference of the appearance damage between each mode; Based on the deep learning network, the collaborative information of interactive learning of appearance damage features between different modalities is extracted, and the fusion weights of different modal detection data are determined according to the recognition attention of each modality, and then a multimodal data-driven appearance detection model is constructed through the collaborative information and the fusion weights of different modal detection data; The multimodal detection data detected in real time is input into the appearance detection model, thereby identifying the defect features of the appearance of the precision shaft and visually displaying the defect features.

[0006] In this embodiment, the multimodal detection data includes visible light image data, point cloud data and thermal imaging data.

[0007] In this embodiment, the calibration and alignment processing is performed on the multimodal detection data to obtain the aligned multimodal detection data, which specifically includes: The multimodal detection data is calibrated using a preset multimodal calibration plate to obtain calibrated multimodal detection data; The calibrated multimodal detection data is spatially aligned based on the spatial feature point matching method to obtain aligned multimodal detection data.

[0008] In this embodiment, mapping the aligned multimodal detection data to the multimodal space of the unified coordinate system specifically includes: Construct a unified three-dimensional coordinate system based on the position information of the precision axis; The aligned multimodal detection data is mapped into the three-dimensional coordinate system using a spatial reprojection method, thereby constructing a multimodal space.

[0009] In this embodiment, based on the information variance of the appearance damage features in each modality and the distribution difference of the appearance damage between each modality, the recognition attention of each modality in the precision shaft appearance detection is determined specifically including: Determine the distribution difference of appearance damage between each mode based on all appearance damage characteristics; Determining the feature relative entropy of each modality through the information variance of the appearance damage feature in each modality and the distribution difference; The recognition attention of each modality in precision shaft appearance detection is determined based on the proportion of the relative entropy of each modal feature.

[0010] In this embodiment, constructing a multimodal data-driven appearance detection model by using the fusion weights of the collaborative information and different modal detection data specifically includes: Construct an initial detection model based on a multimodal feature fusion network; Encode each modal detection data to generate feature vectors of different modalities; Using the collaborative information as the interactive features of the interactive layer in the initial detection model; The fusion weights of different modal detection data are set as the modal weights of the weighted fusion layer in the initial detection model, and the feature vectors of different modalities are used as input features of the initial detection model, thereby obtaining a multimodal data-driven appearance detection model.

[0011] In this embodiment, the imaging device is an industrial camera.

[0012] In this embodiment, the laser scanning device is a three-dimensional laser scanner.

[0013] In this embodiment, the thermal imager is an infrared thermal imaging camera.

[0014] In a second aspect, the present application provides a multimodal data driven precision shaft appearance detection system, which is used to perform a multimodal data driven precision shaft appearance detection method, and the multimodal data driven precision shaft appearance detection system includes: The data acquisition module is used to detect the appearance of the precision shaft based on the configured camera equipment, laser scanning equipment and thermal imager to obtain multi-modal detection data; A spatial mapping module, used to perform calibration and alignment processing on the multimodal detection data to obtain aligned multimodal detection data, and then map the aligned multimodal detection data to a multimodal space of a unified coordinate system; An attention determination module is used to extract appearance damage features of different modes from the multimodal space, and then determine the information variance of the appearance damage features in each mode, and determine the recognition attention of each mode in the precision shaft appearance detection based on the information variance of the appearance damage features in each mode and the distribution difference of the appearance damage between each mode; A detection model construction module is used to extract collaborative information of interactive learning of appearance damage features between different modalities based on a deep learning network, determine the fusion weights of different modal detection data according to the recognition attention of each modality, and then construct a multimodal data-driven appearance detection model through the collaborative information and the fusion weights of different modal detection data; The defect detection module is used to input the real-time multimodal detection data into the appearance detection model, thereby identifying the defect features of the appearance of the precision shaft and visually displaying the defect features.

[0015] The technical solution provided by the embodiments disclosed in this application has the following beneficial effects: First, the appearance of the precision shaft is inspected based on the configured camera equipment, laser scanning equipment and thermal imager to obtain multimodal inspection data; the multimodal inspection data is calibrated and aligned to obtain aligned multimodal inspection data, and then the aligned multimodal inspection data is mapped to the multimodal space of a unified coordinate system; the appearance damage features of different modes are extracted from the multimodal space, and then the information variance of the appearance damage features in each mode is determined, and the recognition attention of each mode in the appearance inspection of the precision shaft is determined based on the information variance of the appearance damage features in each mode and the distribution difference of the appearance damage between each mode; the collaborative information of the interactive learning of the appearance damage features between different modes is extracted based on the deep learning network, and the fusion weight of the different modal inspection data is determined according to the recognition attention of each modality, and then the multimodal data-driven appearance inspection model is constructed through the collaborative information and the fusion weight of the different modal inspection data; the real-time multimodal inspection data is input into the appearance inspection model, and then the defect features of the appearance of the precision shaft are identified, and the defect features are visualized.

[0016] It can be seen that the present application scheme constructs a multimodal data-driven appearance detection model through the collaborative information of interactive learning of appearance damage features between different modalities and the fusion weights of different modal detection data, and then performs appearance defect detection on precision shafts through the appearance detection model; firstly, the scheme calibrates, aligns and spatially maps the multimodal detection data to ensure that data from different devices can be fused in a unified three-dimensional space, thereby providing an accurate spatial reference for subsequent analysis; secondly, the scheme extracts the appearance damage features of each modality, and based on the information variance and the distribution differences between the modalities, determines the recognition attention of each modality, and assigns reasonable weights to different modalities, so as to more accurately capture the effective information of each modality during feature fusion. Information, through the above method, not only can the detection of damage features be strengthened, but also their importance in detection can be dynamically adjusted according to the contribution of the mode, and the overall performance of the model can be optimized; then, through the cross-modal feature interactive learning of the deep learning network, the scheme further improves the coordination between modalities and strengthens the information complementarity and interaction between the features of each modality, thereby maximizing the use of the advantages of each modality and ensuring that more accurate and comprehensive detection results can be provided when dealing with complex defect types; finally, through the input of real-time detection data and the visual display of detection results, a complete detection closed loop is provided, from data collection, feature extraction, modal fusion to final defect identification and feedback, forming an efficient, intelligent and real-time detection system. In summary, the present application scheme can realize multi-modal collaborative detection of precision shaft appearance defects, thereby improving the robustness of precision shaft appearance detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0018] Figure 1 is a flow chart of a multimodal data-driven precision shaft appearance inspection method provided by the present application; Figure 2 is an exemplary flow chart for determining aligned multimodal detection data provided by the present application; Figure 3 is an exemplary flow chart for determining recognition attention provided by the present application; Figure 4 This is a module structure diagram of the multimodal data-driven precision shaft appearance inspection system provided in this application. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] The embodiment of the present application provides a multimodal data-driven precision shaft appearance detection system and method, the core of which is to perform multimodal detection on the appearance of the precision shaft, extract appearance damage features of different modalities, and then determine the recognition attention of each modality in the precision shaft appearance detection based on the information variance of the appearance damage features in each modality and the distribution difference of the appearance damage between each modality; extract the collaborative information of the interactive learning of the appearance damage features between different modalities, determine the fusion weight of different modal detection data according to the recognition attention of each modality, and then construct an appearance detection model through the collaborative information and the fusion weight of different modal detection data; input the real-time multimodal detection data into the appearance detection model, identify the defect features of the precision shaft appearance, and visualize the defect features. The present application can realize multimodal collaborative detection of precision shaft appearance defects, thereby improving the robustness of precision shaft appearance detection. Example

[0021] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods. Figure 1As shown in FIG. 1 , this figure is an exemplary flow chart of a multimodal data-driven precision shaft appearance detection method according to this embodiment of the present application, and the detection method includes the following steps: In step S1, the appearance of the precision shaft is inspected based on the configured camera equipment, laser scanning equipment and thermal imager to obtain multimodal inspection data.

[0022] In the specific implementation, first, a high-resolution camera device is installed at a predetermined position. The camera device can be an industrial camera. In the present application, the industrial camera can be used to collect visible light image data of the appearance of the precision shaft in real time; then, a laser scanning device is installed at a predetermined position. The laser scanning device can be a three-dimensional laser scanner. In the present application, the three-dimensional laser scanner can be used to perform a three-dimensional scan on the surface of the precision shaft to obtain point cloud data of the precision shaft, which is three-dimensional data; finally, a thermal imager is installed at a predetermined position. The thermal imager can be an infrared thermal imaging camera. The infrared thermal imaging camera detects infrared radiation emitted by the precision shaft through an infrared sensor (such as a focal plane array), converts the infrared radiation into an electronic signal and generates corresponding thermal imaging data.

[0023] It should be noted that the multimodal detection data in the present application includes visible light image data, point cloud data and thermal imaging data.

[0024] In step S2, the multimodal detection data is calibrated and aligned to obtain aligned multimodal detection data, and then the aligned multimodal detection data is mapped to a multimodal space of a unified coordinate system.

[0025] In this embodiment, reference Figure 2 As shown in FIG. 1 , this figure is an exemplary flow chart of determining the aligned multimodal detection data in an embodiment of the present application. In this embodiment, the multimodal detection data is calibrated and aligned to obtain the aligned multimodal detection data, which can be achieved by the following steps: In step S21, the multimodal detection data is calibrated using a preset multimodal calibration plate to obtain calibrated multimodal detection data; In step S22, the calibrated multimodal detection data is spatially aligned based on a spatial feature point matching method to obtain aligned multimodal detection data.

[0026] In the specific implementation, first, the preset multimodal calibration plate contains known spatial feature points of the precision axis, and all known spatial feature points are converted into coordinate information of the precision axis, and then the existing calibration algorithm (such as Zhang Zhengyou calibration method) is used to correct the detection data collected by the industrial camera, three-dimensional laser scanner and thermal imager to obtain the corrected multimodal detection data, and the corrected multimodal detection data is used as the calibrated multimodal detection data. It should be noted that in the present application, the imaging error of each modal device (i.e., industrial camera, three-dimensional laser scanner and thermal imager) is minimized by calculating the internal parameters, external parameters and distortion parameters of the industrial camera, three-dimensional laser scanner and thermal imager, thereby improving the reliability of the multimodal detection data; then, based on the spatial feature point matching method (such as the iterative nearest point algorithm), the corresponding feature points are found between the different modal detection data, and their relative positions are iteratively optimized, and finally the alignment of the different modal detection data in the same coordinate system is realized, and the aligned multimodal detection data is obtained. It should be noted that the aligned multimodal detection data finally obtained in the present application can provide an accurate spatial data basis for subsequent feature extraction and fusion.

[0027] In this embodiment, mapping the aligned multimodal detection data to the multimodal space of the unified coordinate system can be achieved by using the following steps: Construct a unified three-dimensional coordinate system based on the position information of the precision axis; The aligned multimodal detection data is mapped into the three-dimensional coordinate system using a spatial reprojection method, thereby constructing a multimodal space.

[0028] In the specific implementation, first, the spatial position and posture information of the precision axis is obtained, the origin and axial direction of the coordinate system are determined, and the three-dimensional coordinate system is obtained; then, the calibrated and aligned multimodal detection data (including visible light image data, point cloud data and thermal imaging data) is mapped to the three-dimensional coordinate system using the spatial reprojection method (such as homogeneous coordinate transformation).

[0029] In step S3, appearance damage features of different modalities are extracted from the multimodal space, and then the information variance of the appearance damage features in each modality is determined. Based on the information variance of the appearance damage features in each modality and the distribution differences of the appearance damage between each modality, the recognition attention of each modality in precision shaft appearance detection is determined.

[0030] In this embodiment, the extraction of appearance damage features of different modes from the multimodal space can be achieved by using the following steps: Extracting appearance damage features of the visible light image modality from the multimodal space based on a convolutional neural network; Performing curvature recognition on the point cloud data in the multimodal space to obtain appearance damage features of the laser scanning mode; The appearance damage features of the thermal imaging modality are extracted from the multi-modal space by using a gradient inversion algorithm.

[0031] In the specific implementation, first, the existing deep learning model (such as VGG or ResNet) can be used to extract texture features in the visible light image through multi-layer convolution operations, such as scratches, pits and other surface damage, and the trained deep learning model can be used for image classification and segmentation to obtain the accurate position and type of the damaged area, and the accurate position and type of the damaged area are feature vectorized to obtain the appearance damage characteristics of the visible light image modality; then, for the point cloud data of the laser scanning modality, the curvature recognition algorithm (commonly used techniques include Gaussian curvature and principal curvature calculation methods) is used to extract the geometric damage characteristics of the precision shaft surface, and the concave, convex or deformed areas on the precision shaft surface are identified by calculating the local curvature of the point cloud data, and the concave, convex or deformed areas on the precision shaft surface are feature vectorized to obtain the appearance damage characteristics of the laser scanning modality; finally, the gradient of the temperature change in the thermal imaging image is analyzed by the gradient inversion algorithm, and the temperature abnormality area caused by material inhomogeneity or cracks is inferred, and the temperature abnormality area caused by material inhomogeneity or cracks is feature vectorized to obtain the appearance damage characteristics of the thermal imaging modality.

[0032] In this embodiment, determining the information variance of the appearance damage features in each mode can be achieved by using the following steps: For the appearance damage features of each mode, determine the information density of the appearance damage features; The information variance of the appearance damage feature is determined based on the information density, and then the information variance of the appearance damage feature in each mode is determined.

[0033] In specific implementation, for the appearance damage features of each modality, first, local statistical methods can be used, such as measuring the concentration of information by analyzing the spatial distribution or gradient changes of the damage features. For example, in the visible light image modality, the information density is estimated by calculating the texture complexity and edge strength of each area. In the laser scanning modality, the information density can be evaluated by calculating the standard deviation of the geometric change or local curvature of each point. In the thermal imaging modality, the information density can be measured by the temperature gradient distribution of the thermal map. Then, the characteristic volatility of the appearance damage features can be determined by statistics. Common methods include variance analysis (such as ANOVA) or entropy-based calculations (such as Shannon entropy). By determining the information variance of the appearance damage features in each modality, the dispersion of the appearance damage features in the entire modality can be quantified. A modality with a larger information variance means that the modality contains more variable information and is generally considered to be more important in damage identification.

[0034] It should be noted that the information variance in the present application measures the degree of fluctuation of the appearance damage characteristics in the spatial dimension. The larger the information variance, the more significant the change in the appearance damage characteristics and the richer the information it carries. Conversely, the smaller the change in the appearance damage characteristics and the more concentrated the information.

[0035] In this embodiment, reference Figure 3 As shown in FIG. 1 , this figure is an exemplary flow chart for determining recognition attention in an embodiment of the present application. In this embodiment, the recognition attention of each modality in the precision shaft appearance inspection is determined based on the information variance of the appearance damage features in each modality and the distribution difference of the appearance damage between each modality. The following steps can be used to implement it: In step S31, the distribution difference of appearance damage between each mode is determined based on all appearance damage features; In step S32, the feature relative entropy of each modality is determined by the information variance of the appearance damage feature in each modality and the distribution difference; In step S33, the recognition attention of each modality in the precision shaft appearance detection is determined based on the proportion of the relative entropy of each modal feature.

[0036] In specific implementation, first, the appearance damage characteristics of each modality can be modeled with probability distribution, and then the KL (Kullback-Leibler) divergence between the modes can be calculated. The KL divergence can be used to evaluate the differences in the spatial distribution of appearance damage between different modes; then, the ratio of the KL divergence between the modes to the information variance of the appearance damage characteristics in each mode plus 1 can be used as the characteristic relative entropy of each modality; finally, based on the proportion of the relative entropy of each modality feature, a weighting mechanism is used to determine the recognition attention of each modality. The weighting mechanism automatically adjusts the importance of the modality by analyzing the relative amount of information of each modality in the recognition, ensuring that the contribution of each modality can be maximized in the appearance inspection of precision shafts.

[0037] It should be noted that the recognition attention in this application measures the relative importance of each modal inspection data in contributing to defect recognition in the precision shaft appearance inspection task.

[0038] In step S4, collaborative information of interactive learning of appearance damage features between different modalities is extracted based on the deep learning network, and the fusion weights of different modal detection data are determined according to the recognition attention of each modality. Then, a multimodal data-driven appearance detection model is constructed through the collaborative information and the fusion weights of different modal detection data.

[0039] In this embodiment, the collaborative information of interactive learning of appearance damage features between different modalities extracted based on the deep learning network can be implemented by the following steps: For the appearance damage features of each two modalities, the appearance damage features of the two modalities are mapped to each other based on the cross-modal fusion mechanism in the deep learning network, and the synergy degree of the interactive learning of the appearance damage features between the two modalities is extracted, and then the synergy degree of the interactive learning of the appearance damage features between each two modalities is obtained; All the synergies are used as collaborative information for interactive learning of appearance damage features between different modalities.

[0040] In specific implementation, first, a cross-modal attention network model can be used to interactively learn the appearance damage features of the two modalities through an attention mechanism (such as Cross-modal Attention) to capture the correlation and complementary information between them. Through the cross-modal fusion mechanism, the appearance damage features of the two modalities are mapped and fused in a high-dimensional space, which can ensure that the information between different modalities can effectively interact. The collaborative features output by the cross-modal attention network model are then used to calculate the degree of coordination of the interactive learning of appearance damage features between each two modalities. It should be noted that the degree of coordination in this application can be quantified by cosine similarity to measure the degree of information sharing and complementarity between modalities. In other embodiments, other methods can also be used to quantify the degree of coordination of interactive learning of appearance damage features between modalities, which is not limited here; then, the degree of coordination between all modal pairs is used as collaborative information for subsequent multimodal fusion and decision optimization to help improve the accuracy and robustness of appearance damage detection.

[0041] It should be noted that the degree of synergy in this application is used to measure the degree of interactive learning of appearance damage features among multiple modalities, which reflects the degree of information sharing and complementarity among different modalities.

[0042] It should also be noted that in this application, determining the fusion weight of different modal detection data based on the recognition attention of each modality means dynamically allocating the fusion weight of each modal detection data in the fusion process by evaluating the importance of each modality in appearance damage detection; in specific implementation, the recognition attention of the modality reflects the amount of information and effectiveness provided by the modality in the damage detection task. The modality with higher recognition attention means that it has a greater information contribution in the detection and should be given a higher fusion weight, while the modality with lower recognition attention should be given a smaller fusion weight.

[0043] In this embodiment, the following steps may be used to construct a multimodal data driven appearance detection model by using the fusion weights of the collaborative information and different modal detection data: Construct an initial detection model based on a multimodal feature fusion network; Encode each modal detection data to generate feature vectors of different modalities; Using the collaborative information as the interactive features of the interactive layer in the initial detection model; The fusion weights of different modal detection data are set as the modal weights of the weighted fusion layer in the initial detection model, and the feature vectors of different modalities are used as input features of the initial detection model, thereby obtaining a multimodal data-driven appearance detection model.

[0044] In the specific implementation, first, the existing technical architecture of the multimodal Transformer is used as the basic framework for preliminary modeling to obtain the initial detection model; secondly, for the detection data of each modality, the feature vectors of each modality are generated through an independent encoder (for example, a pre-trained CNN model is used for visible light images, PointNet is used for point cloud data, and an improved CNN is used for thermal imaging images); then, the collaborative information (that is, the degree of collaboration of feature interaction learning between modalities) is used as the input feature of the interaction layer, and the cross-modal attention mechanism or self-attention mechanism (such as Multi-HeadAttention) is used to strengthen the interaction behavior between modalities, and further improve the feature fusion effect; finally, the fusion weights of different modal detection data are allocated to the weighted fusion layer according to the recognition attention calculated above to achieve weighted feature fusion; finally, the feature vectors of each modality are used as the input of the initial detection model, and the weighted fused features are passed into the multimodal feature fusion network for subsequent damage detection tasks to obtain a multimodal data-driven appearance detection model.

[0045] In step S5, the multimodal detection data detected in real time is input into the appearance detection model, so as to identify the defect features of the appearance of the precision shaft, and visualize the defect features.

[0046] In this embodiment, the real-time multi-modal detection data is input into the appearance detection model, thereby identifying the defect features of the appearance of the precision shaft, and visually displaying the defect features can be achieved by the following steps: Using the multimodal detection data detected in real time as input parameters of the appearance detection model; Outputting the detection result of the appearance defects of the precision shaft through the appearance detection model; The detection results are visualized in three-dimensional panoramic defect distribution, and a defect detection report is generated.

[0047] In the specific implementation, first, the collected real-time multimodal detection data (including visible light images, point cloud data and thermal imaging images) are passed as input parameters to the appearance detection model. These data generate their own feature vectors through the aforementioned encoding network (such as convolutional neural network (CNN) for image data, PointNet for point cloud data, and improved CNN for thermal imaging images), and feature fusion is performed through the interaction layer in the model (such as multimodal Transformer) to extract and enhance the collaborative information between multimodalities. The weighted fused features enter the decision layer of the model, and finally the detection results of the appearance defects of the precision shaft are output, including the type, location, size and severity of the defects; then, the three-dimensional visualization technology is used to visualize the three-dimensional panoramic defect distribution of the detection results, for example, the defect location is marked in the three-dimensional model of the precision shaft using a point cloud visualization tool (such as PCL or VTK), and the detailed information of the defect is displayed in combination with a heat map and an annotation box.

[0048] It can be seen that the present application scheme constructs a multimodal data-driven appearance detection model through the collaborative information of interactive learning of appearance damage features between different modalities and the fusion weights of different modal detection data, and then performs appearance defect detection on precision shafts through the appearance detection model; firstly, the scheme calibrates, aligns and spatially maps the multimodal detection data to ensure that data from different devices can be fused in a unified three-dimensional space, thereby providing an accurate spatial reference for subsequent analysis; secondly, the scheme extracts the appearance damage features of each modality, and based on the information variance and the distribution differences between the modalities, determines the recognition attention of each modality, and assigns reasonable weights to different modalities, so as to more accurately capture the effective information of each modality during feature fusion. Information, through the above method, not only can the detection of damage features be strengthened, but also their importance in detection can be dynamically adjusted according to the contribution of the mode, and the overall performance of the model can be optimized; then, through the cross-modal feature interactive learning of the deep learning network, the scheme further improves the coordination between modalities and strengthens the information complementarity and interaction between the features of each modality, thereby maximizing the use of the advantages of each modality and ensuring that more accurate and comprehensive detection results can be provided when dealing with complex defect types; finally, through the input of real-time detection data and the visual display of detection results, a complete detection closed loop is provided, from data collection, feature extraction, modal fusion to final defect identification and feedback, forming an efficient, intelligent and real-time detection system. In summary, the present application scheme can realize multi-modal collaborative detection of precision shaft appearance defects, thereby improving the robustness of precision shaft appearance detection. Example

[0049] This application provides a multi-modal data driven precision shaft appearance inspection system, referring to Figure 4As shown, this figure is a schematic diagram of a multimodal data driven precision shaft appearance detection system according to this embodiment of the present application, and the multimodal data driven precision shaft appearance detection system includes: The data acquisition module 100 is used to detect the appearance of the precision shaft based on the configured camera equipment, laser scanning equipment and thermal imager to obtain multi-modal detection data; The spatial mapping module 200 is used to perform calibration and alignment processing on the multimodal detection data to obtain aligned multimodal detection data, and then map the aligned multimodal detection data to a multimodal space of a unified coordinate system; An attention determination module 300 is used to extract appearance damage features of different modes from the multimodal space, and then determine the information variance of the appearance damage features in each mode, and determine the recognition attention of each mode in the precision shaft appearance detection based on the information variance of the appearance damage features in each mode and the distribution difference of the appearance damage between each mode; The detection model construction module 400 is used to extract the collaborative information of interactive learning of appearance damage features between different modalities based on the deep learning network, determine the fusion weight of different modal detection data according to the recognition attention of each modality, and then construct a multimodal data-driven appearance detection model through the collaborative information and the fusion weight of different modal detection data; The defect detection module 500 is used to input the multi-modal detection data detected in real time into the appearance detection model, thereby identifying the defect features of the appearance of the precision shaft and visually displaying the defect features.

[0050] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0051] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, the storage medium including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically-erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0052] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

Claims

1. A multimodal data driven precision shaft appearance inspection method, characterized in that: The detection method comprises the following steps: Based on the configured camera equipment, laser scanning equipment and thermal imager, the appearance of the precision shaft is inspected to obtain multi-modal inspection data; Performing calibration and alignment processing on the multimodal detection data to obtain aligned multimodal detection data, and then mapping the aligned multimodal detection data to a multimodal space of a unified coordinate system; Extracting appearance damage features of different modes from the multimodal space, and then determining the information variance of the appearance damage features in each mode, and determining the recognition attention of each mode in the precision shaft appearance inspection based on the information variance of the appearance damage features in each mode and the distribution difference of the appearance damage between each mode; Based on the deep learning network, the collaborative information of interactive learning of appearance damage features between different modalities is extracted, and the fusion weights of different modal detection data are determined according to the recognition attention of each modality, and then a multimodal data-driven appearance detection model is constructed through the collaborative information and the fusion weights of different modal detection data; The multimodal detection data detected in real time is input into the appearance detection model, thereby identifying the defect features of the appearance of the precision shaft and visually displaying the defect features.

2. A multi-modal data driven precision shaft appearance detection method as claimed in claim 1, characterized in that: The multimodal detection data includes visible light image data, point cloud data and thermal imaging data.

3. A multi-modal data driven precision shaft appearance inspection method as claimed in claim 1, characterized in that: The multimodal detection data is calibrated and aligned to obtain aligned multimodal detection data, which specifically includes: The multimodal detection data is calibrated using a preset multimodal calibration plate to obtain calibrated multimodal detection data; The calibrated multimodal detection data is spatially aligned based on the spatial feature point matching method to obtain aligned multimodal detection data.

4. A multi-modal data driven precision shaft appearance detection method as claimed in claim 1, characterized in that: Mapping the aligned multimodal detection data to the multimodal space of the unified coordinate system specifically includes: Construct a unified three-dimensional coordinate system based on the position information of the precision axis; The aligned multimodal detection data is mapped into the three-dimensional coordinate system using a spatial reprojection method, thereby constructing a multimodal space.

5. The multi-modal data driven precision shaft appearance detection method according to claim 1, characterized in that: Based on the information variance of the appearance damage features in each mode and the distribution difference of the appearance damage between each mode, the recognition attention of each mode in the precision shaft appearance inspection is determined to include: Determine the distribution difference of appearance damage between each mode based on all appearance damage characteristics; Determining the feature relative entropy of each modality through the information variance of the appearance damage feature in each modality and the distribution difference; The recognition attention of each modality in precision shaft appearance detection is determined based on the proportion of the relative entropy of each modal feature.

6. A multi-modal data driven precision shaft appearance inspection method as claimed in claim 1, characterized in that: Constructing a multimodal data driven appearance detection model through the fusion weights of the collaborative information and different modal detection data specifically includes: Construct an initial detection model based on a multimodal feature fusion network; Encode each modal detection data to generate feature vectors of different modalities; Using the collaborative information as the interactive features of the interactive layer in the initial detection model; The fusion weights of different modal detection data are set as the modal weights of the weighted fusion layer in the initial detection model, and the feature vectors of different modalities are used as input features of the initial detection model, thereby obtaining a multimodal data-driven appearance detection model.

7. A multi-modal data driven precision shaft appearance inspection method as claimed in claim 1, characterized in that: The imaging device is an industrial camera.

8. The multi-modal data driven precision shaft appearance inspection method according to claim 1, characterized in that: The laser scanning device is a three-dimensional laser scanner.

9. A multi-modal data driven precision shaft appearance inspection method as claimed in claim 1, characterized in that: The thermal imager is an infrared thermal imaging camera.

10. A multimodal data driven precision shaft appearance inspection system, used to execute a multimodal data driven precision shaft appearance inspection method as claimed in any one of claims 1 to 9, characterized in that: The multimodal data-driven precision shaft appearance inspection system comprises: The data acquisition module is used to detect the appearance of the precision shaft based on the configured camera equipment, laser scanning equipment and thermal imager to obtain multi-modal detection data; A spatial mapping module, used to perform calibration and alignment processing on the multimodal detection data to obtain aligned multimodal detection data, and then map the aligned multimodal detection data to a multimodal space of a unified coordinate system; An attention determination module is used to extract appearance damage features of different modes from the multimodal space, and then determine the information variance of the appearance damage features in each mode, and determine the recognition attention of each mode in the precision shaft appearance detection based on the information variance of the appearance damage features in each mode and the distribution difference of the appearance damage between each mode; A detection model construction module is used to extract collaborative information of interactive learning of appearance damage features between different modalities based on a deep learning network, determine the fusion weights of different modal detection data according to the recognition attention of each modality, and then construct a multimodal data-driven appearance detection model through the collaborative information and the fusion weights of different modal detection data; The defect detection module is used to input the real-time multimodal detection data into the appearance detection model, thereby identifying the defect features of the appearance of the precision shaft and visually displaying the defect features.

Images