Automobile part defect intelligent detection method and system based on multi-modal data fusion

Abstract

The invention provides an automobile part defect intelligent detection method and system based on multi-modal data fusion, and the method comprises the steps: collecting image data and three-dimensional point cloud data of an automobile part, and carrying out the space alignment processing, and obtaining a registration modal set; constructing a structure diagram, extracting modal features of a corresponding region from the registration modal set, and generating a structure diagram node embedding vector through a cross-modal fusion strategy; based on the structure diagram and the structure diagram node embedding vector, node-level reasoning is carried out by utilizing a pre-constructed diagram neural network, and a node state vector and a structure diagram node defect state sequence are output; and inputting the structure diagram node defect state sequence of the same automobile part at the plurality of detection time points into a time sequence modeling network guided by a diagram structure, outputting a defect risk score, and outputting a corresponding quality control strategy based on the current defect evolution risk. According to the invention, accurate identification, positioning, grading and trend prediction of automobile part defects can be realized in a complex industrial environment.

Description

Technical Field

[0001] This invention belongs to the field of intelligent detection of defects in automotive parts, and particularly relates to an intelligent detection method and system for automotive parts defects based on multimodal data fusion. Background Technology

[0002] As the automotive manufacturing industry continues to evolve towards intelligence and automation, the quality inspection of automotive components before they leave the factory has become a crucial link in ensuring the performance, safety, and consistency of the entire vehicle. Traditional inspection systems mainly rely on human experience or inspection equipment based on a single visual modality. While they are effective in handling issues such as surface scratches, coating defects, and localized dents, they still face significant limitations in real-world industrial environments. Visual inspection is highly susceptible to factors such as uneven lighting, strong reflections, and occlusions. In particular, it often results in false positives or false negatives when dealing with micro-cracks on metal surfaces, low-contrast scratches, or complex assembly shadows, making it impossible to guarantee the stability and repeatability of the inspection. At the same time, single-modal visual signals are insufficient to express the three-dimensional geometry of components and their assembly structural relationships, making it impossible to accurately analyze deep-seated defects such as structural misalignments, uneven welding, and loose fasteners. Although the introduction of multimodal sensing technologies in recent years, such as point cloud scanning, structured light ranging, and ultrasonic testing, has enriched the inspection information of components, most existing methods remain at the stage of simple splicing or data-level superposition of modal features. They fail to achieve collaborative modeling between modalities at the level of structural semantics and physical constraints, resulting in semantic misalignment and information redundancy in the fused feature representation. Furthermore, in actual production processes, automotive components undergo multiple station inspections. Inspection data at different time points contain the evolutionary patterns of defects from their initial emergence to their gradual expansion. However, most existing inspection systems only focus on the results of a single inspection, ignoring the correlation of defect changes over time, and lacking the ability to predict potential risks and learn from process trends. Existing methods cannot form a unified perception of component structure, surface, and temporal evolution; the inspection system remains at the passive identification stage and cannot meet the requirements of modern automotive manufacturing for accurate inspection, dynamic traceability, and proactive early warning.

[0003] Therefore, there is an urgent need for an intelligent detection system with spatial structure understanding, modal fusion and adaptive capabilities, and time series evolution analysis capabilities, which can achieve high-precision detection and trend prediction of multimodal defects in automotive parts under complex industrial conditions. Summary of the Invention

[0004] The purpose of this invention is to propose an intelligent detection method and system for automotive component defects based on multimodal data fusion, thereby solving the aforementioned problems.

[0005] To achieve the above objectives, a first aspect of the present invention provides an intelligent detection method for defects in automotive components based on multimodal data fusion, the method comprising the following steps: S1. Acquire image data and 3D point cloud data of automotive parts, and perform spatial alignment processing to map them to a unified structural coordinate system to obtain a registration mode set; wherein, the registration mode set includes tensors after transforming the image data to the structural coordinate system and the 3D position expression of the 3D point cloud data in the workpiece coordinate system; S2. Construct a structural graph based on the structural design model of the automotive component, extract modal features of the corresponding region from the registration modality set, and generate structural graph node embedding vectors with structural semantics through a cross-modal fusion strategy; S3. Based on the structure graph and the structure graph node embedding vector, a pre-built graph neural network is used to perform node-level reasoning, outputting a node state vector and a current defect structure graph node defect state sequence; the structure graph node defect state sequence includes type identification, location determination, and severity classification; wherein, the pre-built graph neural network is a three-layer graph convolutional neural network, each layer using an improved variable receptive field convolution; S4. Input the structural graph node defect state sequence of the same automotive component at multiple detection time points into a time-series modeling network guided by graph structure, output a defect risk score to represent the current defect evolution risk, and output a corresponding quality control strategy based on the current defect evolution risk.

[0006] Furthermore, the image data is acquired by an industrial area array camera fixedly installed above the inspection station on the production line; the three-dimensional point cloud data is acquired using a high-precision structured light 3D scanner, and the three-dimensional point cloud data under the sensor coordinates is reconstructed through triangulation. Specifically, the spatial alignment process, which maps the model to a unified structural coordinate system to obtain a registration mode set, involves: Acquire the image data and 3D point cloud data; A fixed pose matrix is ​​obtained using production line calibration, and this spatial mapping is achieved by combining it with 3D point cloud data to obtain the first... The position of each point in the workpiece coordinate system; Each pixel of the image data is associated with a 3D position representation in the 3D point cloud data through camera intrinsic parameters and alignment with the corresponding 3D point cloud data; By combining the three-dimensional position representation and the tensor of the image data after transformation to the structural coordinate system, the registration mode set is obtained.

[0007] Furthermore, the construction of a structural diagram based on the structural design model of the automotive component, and the extraction of modal features of the corresponding region from the registration modality set, specifically includes: Each node in the structure diagram represents an actual functional unit, and each node is associated with a structural region. The structural region is used to extract image sub-tensors and point cloud subsets from the registration modality set within that region. The image region is processed by extracting texture features from the first three Bottleneck layers of the lightweight convolutional network MobileNetV3. The point cloud subset is processed using a three-layer EdgeConv module, which first considers the dynamics in three-dimensional space. The nearest neighbor constructs a local geometry graph, and then the edge feature encoding method is applied to each neighborhood point pair to extract the local rate of change, outputting point cloud features.

[0008] Furthermore, the generation of structural graph node embedding vectors with structural semantics through a cross-modal fusion strategy specifically involves: Obtain the texture features and point cloud features; Based on the texture features of the image data, residual signals of point cloud modal alignment are constructed, and fusion features are formed through a residual enhancement mechanism; and based on the fusion features of any two nodes, a structural consistency regularization is constructed to suppress feature shift problems caused by modal mismatch during the training phase. The fusion features corresponding to each structural graph node used for embedding are preserved to form a structural graph node embedding vector.

[0009] Furthermore, the residual enhancement mechanism is implemented through a modal alignment operation based on channel attention, specifically as follows: First, the point cloud features are reduced in dimensionality to [value] through two layers of channel-wise convolution. The dimensions are then used to form a weight vector through Sigmoid activation, which is used to dynamically adjust the responsiveness of the point cloud channels to the image channels.

[0010] Furthermore, the pre-built graph neural network also includes a residual tension offset factor, used to model the perception of stretching or misalignment caused by defects in the structural connection; the residual tension offset factor is the aggregation weight of each graph convolution layer, and the aggregation weight is an adaptive coefficient that fuses the feature differences between nodes and the structural residual tension, calculated by combining the spatial coordinates of any two graph nodes, the expected spatial displacement difference, and the tension residual in the assembly structure.

[0011] The tension residual is used to reflect the degree of offset between the point cloud structure and the CAD model on the connecting edge during actual acquisition.

[0012] Furthermore, the node state vector is ultimately set to three dimensions, corresponding to the probability of defect existence, the defect level index, and the local inconsistency index in the feature space, respectively. The type identification is obtained by performing Softmax classification on the probability of the defect's existence; the location is determined as the center coordinates of the corresponding node; and the severity level is determined as a defect level index.

[0013] Furthermore, the graph-guided temporal modeling network uses a single-layer bidirectional gated recurrent unit in the time dimension to capture the nonlinear development trend of the defect state; in the structural graph dimension, a static adjacency graph is used to expand the temporal node state into local region state blocks to guide the network to learn the state linkage between adjacent nodes. Specifically, the step of inputting the structural graph node defect state sequence of the same automotive component at multiple inspection time points into a graph-guided time-series modeling network and outputting a defect risk score includes: The defect state sequence of each node in the structure graph is input into the single-layer bidirectional gated recurrent unit to form an evolution path representation. Combined with the adjacency structure information of the node's neighbors in the structure graph, the difference in the latest state change of the node's neighbors is used as an auxiliary factor for evolution trend modeling. Through linear layer mapping, a trend focusing regularization term is introduced for constraint, and the defect risk score for the next time step is output. The trend focusing regularization term is derived from the node's own historical state changes to generate a pseudo-supervision signal, which is used to assist the temporal modeling network in learning nodes that are more sensitive to changes.

[0014] Furthermore, the corresponding quality control strategy specifically includes: If the defect risk score is less than the first preset threshold, it means that the defect risk is acceptable and production can continue. If the defect risk score is greater than or equal to the first preset threshold and less than the second preset threshold, a manual review process is triggered. If the defect risk score is greater than or equal to the second preset threshold, a rework signal is output to the production line control system.

[0015] A second aspect of the invention provides an intelligent defect detection system for automotive components based on multimodal data fusion, the system comprising: The data acquisition and alignment module is used to acquire image data and 3D point cloud data of automotive parts, and perform spatial alignment processing to map them to a unified structural coordinate system to obtain a registration mode set; wherein, the registration mode set includes tensors after transforming the image data to the structural coordinate system and the 3D position expression of the 3D point cloud data in the workpiece coordinate system; The structure graph construction and feature fusion module is used to construct a structure graph based on the structural design model of the automotive component, extract modal features of the corresponding region from the registration modality set, and generate structure graph node embedding vectors with structural semantics through a cross-modal fusion strategy. The defect identification and classification module is used to perform node-level reasoning based on the structure graph and the structure graph node embedding vector, using a pre-built graph neural network, and output a node state vector and a current defect structure graph node defect state sequence; the structure graph node defect state sequence includes type identification, location determination and severity classification; wherein, the pre-built graph neural network is a three-layer graph convolutional neural network, each layer using an improved variable receptive field convolution; The temporal evolution prediction module is used to input the structural graph node defect state sequence of the same automotive component at multiple detection time points into a temporal modeling network guided by graph structure, output a defect risk score to represent the current defect evolution risk, and output a corresponding quality control strategy based on the current defect evolution risk.

[0016] The beneficial technical effects of the present invention are at least as follows: This invention establishes a structural diagram centered on the component's CAD model and employs a spatial alignment mechanism to project images acquired by industrial cameras and point cloud data obtained from structured light scanning onto a unified coordinate system, ensuring spatial consistency of multimodal data at the physical level. Building upon this, a structure-guided cross-modal residual fusion strategy is introduced, using image texture features as guidance and point cloud geometric features as supplementary features to construct a node-level fusion representation with structural semantics, achieving highly sensitive expression of complex defects such as subtle dents, minor weld misalignments, and assembly errors. Subsequently, an adaptive graph convolutional neural network with structural tension offset modeling is introduced to fuse local modal features and global topological relationships at the node level, enabling the identification of defect types, location determination, and hierarchical reasoning of severity, particularly suitable for identifying complex defects such as uneven welding stress, geometric warping, and multi-scale scratches. Furthermore, the system constructs a temporal defect evolution model based on the node states of the structural graph. Through temporal recursive units under graph structure constraints, the model learns the changing patterns of defect states at multiple detection times and introduces a trend-focusing regularization term to identify nodes with potential risk expansion trends, achieving proactive prediction of defect development risks. The system ultimately outputs comprehensive detection results, including defect type, spatial location, severity level, and future risk probability. It also integrates with the production line quality control system to generate automated rework or manual review decision signals. Through this technical solution, the present invention achieves an integrated design of structural modeling, modal fusion, and temporal prediction, significantly improving the system's robustness and intelligence under complex lighting, occlusion, and multi-stage operating conditions, providing reliable technical support for digital quality control in the automotive manufacturing industry. Attached Figure Description

[0017] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0018] Figure 1 This is a flowchart of the intelligent detection method for automotive component defects based on multimodal data fusion according to the present invention. Detailed Implementation

[0019] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0020] like Figure 1 As shown in the figure, the intelligent detection method for automotive component defects based on multimodal data fusion provided by the embodiments of the present invention includes: S1. Acquire image data and 3D point cloud data of automotive components, and perform spatial alignment processing to map them to a unified structural coordinate system to obtain a registration mode set; wherein, the registration mode set includes tensors after transforming the image data to the structural coordinate system and the 3D position expression of the 3D point cloud data in the workpiece coordinate system.

[0021] Specifically, this step aims to complete the standardized acquisition and physical spatial alignment of multimodal detection data for automotive components, and output structurally consistent images and point cloud data, providing a highly consistent foundation for subsequent structural semantic mapping and feature fusion.

[0022] Furthermore, image data Data is acquired by an industrial area scan camera fixedly installed above the inspection station on the production line. It uses a Sony IMX253 sensor or equivalent, with a resolution of 1920×1200 and a sampling rate of 30 frames per second. Data is transmitted to the industrial control host via a GigEVision interface. Camera calibration is performed offline using a standard checkerboard pattern and OpenCV to obtain the camera intrinsic parameter matrix. With distortion coefficient. During the shooting process, LED shadowless lights are used to eliminate areas of strong reflection, especially for surface scratch detection of metal parts. The light source is arranged in an "oblique upward + surrounding" manner to ensure clear texture boundaries.

[0023] 3D point cloud The data was acquired using a high-precision structured light 3D scanner (such as the Keyence LJ-X8000 series), with a scanning frequency of 1000Hz. The axial accuracy is better than 10μm. The scanner is installed to the side of the same workstation, and its angle is adjusted by a robotic arm so that its scanning direction forms an angle of approximately 30° with the image's viewing direction, facilitating the acquisition of edge variations on local tilted surfaces. The acquired data is the original depth map. Reconstructing 3D point clouds in sensor coordinates using triangulation The reconstruction process is as follows:

[0024] in, This represents the coordinates of a 3D point in the depth camera coordinate system. These are the pixel coordinates on the image plane. The depth value for each pixel. This is the intrinsic parameter matrix of the depth camera, provided by the equipment manufacturer or obtained through self-calibration. This transformation is used to restore a 2D depth map to a 3D spatial point cloud.

[0025] Furthermore, the acquired images And point cloud Each component belongs to its own independent coordinate system and needs to be transformed to a unified component coordinate system. We use the fixed pose matrix obtained from production line calibration. To achieve this space mapping:

[0026] in, Indicates the first A point in the workpiece coordinate system The lower position, It is an extrinsic transformation matrix that includes rotation and translation, obtained by calibration using the hand-eye calibration board and the robotic arm coordinate system. All transformations are completed during system initialization and are not adjusted during runtime.

[0027] image Each pixel in the image is obtained through camera intrinsics. The alignment relationship with the point cloud can be associated with the 3D coordinates in the point cloud. Because images and point clouds are acquired using different methods, imperfect overlap of boundary regions may occur during the actual matching process. Therefore, in the data post-processing stage, a spatial neighborhood-based interpolation and masking method is used to retain only the registration data of the structural body region. Specifically, in the engine bracket inspection, only the central connecting support and corner reinforcing rib areas are retained, while background occlusion or reflective interference is filtered out.

[0028] The final output is a registration mode set in a unified structured coordinate system:

[0029] in, This indicates that the image data is transformed to the structure coordinate system. The resulting tensor retains the color and texture information of each pixel; This represents the three-dimensional position of point cloud data in the workpiece coordinate system, preserving the geometric shape of each structural node. The two are spatially aligned, forming the basis for constructing the modal representation of structural graph nodes.

[0030] S2. Construct a structural graph based on the structural design model of the automotive component, extract modal features of the corresponding region from the registration modality set, and generate structural graph node embedding vectors with structural semantics through a cross-modal fusion strategy.

[0031] Specifically, this step follows the registration mode set output from the previous step. The core objective is to construct a multimodal fusion representation for automotive component structures based on existing spatially consistent image and point cloud data. This representation must not only fully preserve the key information describing defects in each modality, but also achieve semantic partitioning and node alignment of functional regions of the automotive component through a structural graph. Unlike general graph construction methods, defects in automotive components are mostly concentrated at structural deformation points (such as the edges of reinforcing ribs, connection holes, and weld roots). This requires that the partitioning of the structural graph closely matches the functional units defined in the CAD component drawings, and that the aggregation of modal information can reflect the coupling anomalies between texture and geometry, rather than their independent changes. Therefore, this step designs a region-level modal coding method based on structural graphs and introduces a structure-guided multimodal residual fusion mechanism to improve the node embedding's ability to perceive structural mismatches, subtle geometric deformations, and texture abrupt changes.

[0032] Furthermore, structural diagram Constructed from the topology in the CAD model, where each node This represents an actual functional unit, such as a screw connection, edge guard plate, or inner reinforcing rib. Each node is associated with a structural region. Used from Extract the image sub-tensor within this region With Point Cloud Subset The node region can be obtained from the predefined part boundaries in the CAD drawing and aligned with the current workpiece using station calibration parameters. (Image region) Texture features are extracted using the first three Bottleneck layers of the lightweight convolutional network MobileNetV3. Point Cloud Subset This is handled using a three-layer EdgeConv module, which is based first on dynamics in three-dimensional space. Nearest neighbors construct local geometry ( Then, for each neighboring point pair, the local rate of change is extracted using edge feature encoding, and the point cloud features are output. .

[0033] Considering that defects in automotive parts often exhibit asymmetrical visibility in texture and geometry (e.g., shallow scratches are obvious in images but invisible in point clouds, and local deformations are significant in point clouds but almost imperceptible visually), we designed a cross-modal residual-aware fusion structure to suppress redundant information and emphasize the mutual reinforcement of defects between modes. Specifically, the image modality is used as a baseline to construct the residual signal of the point cloud modality, and a fusion feature is formed through a residual enhancement mechanism. :

[0034] in, and Image and point cloud modalities at nodes, respectively. Feature vectors extracted from regions It is a linear mapping matrix. For activation functions (such as GELU). This is the residual reinforcement coefficient, which is 1.2 by default. This represents a modality alignment operation based on channel attention, with the following structure: First, ... Dimensionality reduction is achieved through two layers of channel-wise convolution. The dimensions are then used to form a weight vector through Sigmoid activation, which is used to dynamically adjust the response of the point cloud channels to the image channels. In this way, the differences between the point cloud and the image are emphasized and added as an anomaly enhancement channel to the fusion representation.

[0035] Furthermore, to better address potential issues such as localized glare, occlusion, and sparse point clouds that may arise during data acquisition of automotive components in complex industrial environments, this step introduces a structural consistency regularization term to suppress feature shift caused by modal mismatch during the training phase. This regularization term does not rely on additional supervision signals but only on the connection edges within the graph structure. The consistency constraint on the embedding vectors of adjacent nodes is designed as follows: ;

[0036] in Representing nodes in a structure graph and The spatial proximity is obtained by normalizing the Euclidean distance between the center points of the node regions. This encourages structurally adjacent nodes to have continuous fusion representations, while weighting according to spatial proximity to avoid over-smoothing all node embeddings.

[0037] Ultimately, each node in the structure graph corresponding embedding vector Retained, forming an embedded set This set will serve as the input to the structural graph neural network to achieve subsequent defect identification and classification tasks. Through a combination of residual enhancement and structural regularization, this step not only constructs a semantically clear structural representation but also significantly improves the system's robust representation capability of defect features under multimodal inconsistency conditions, enabling the system to more stably handle complex component surface inspection tasks under different working conditions.

[0038] S3. Based on the structure graph and the structure graph node embedding vector, a pre-built graph neural network is used to perform node-level reasoning, outputting a node state vector and a current defect structure graph node defect state sequence; the structure graph node defect state sequence includes type identification, location determination, and severity classification; wherein, the pre-built graph neural network is a three-layer graph convolutional neural network, each layer using an improved variable receptive field convolution.

[0039] Specifically, this step embeds the set of nodes from the structure graph output in the previous stage. As input, combined with the structure diagram Based on the topological relationships, this step identifies, precisely locates, and assesses the severity of defects in automotive components at the structural level. This step is the first in the entire system to explicitly determine a "defect," directly undertaking the core task of converting the intermediate features obtained from the first two steps, "modal space alignment" and "structural guided feature fusion," into a structural-level output representing the actual physical defect state. Unlike traditional pixel classification or image-level discrimination methods, this step models defects as "structural anomaly signals" propagating in the structural graph. Its judgment not only relies on the local modal features of the current node but also considers its adjacency relationships at the structural level and its deviation patterns from the prior CAD structure, thus possessing greater industrial semantics and being more suitable for handling complex defects such as structural assembly deviations, minor dents, and boundary cracks.

[0040] Furthermore, the input is the structure graph node embedding vector. ,in In step two, the image modal features are used to... Point cloud modal features The fusion method incorporates cross-modal residual signals to enhance structure-aware representation, thereby enabling... It already possesses rich encoding capabilities for local textures, geometric perturbations, and multimodal heterogeneity. Each node Each represents a structural region Its spatial coordinates are determined in step one. Already in structural coordinate system Alignment is complete, so each structural node is embedded. They all possess clear physical semantics, structural positioning, and modal information sources.

[0041] Furthermore, in order to achieve defect identification and classification, we use structural diagrams... A graph neural network structure is deployed to perform node state reasoning. The base model is a three-layer graph convolutional neural network, with each layer using an improved adaptive variable receptive field convolution (Adaptive GCN) and incorporating a specially designed residual tension offset factor. (ResidualTensionFactor) is used to model the perception of "stretching" or "misalignment" caused by defects in structural connections. This model is not a linear superposition of standard GCNs, but rather dynamically adjusts weights during information aggregation to enhance the ability to identify "structural tension imbalances" in defect regions. Each layer of graph convolution is represented as follows: ;

[0042] in, For nodes The state vector (representing the potential defect response level of the node). It is a learnable weight matrix. For bias terms, It uses the LeakyReLU activation function. The most crucial innovation lies in the residual tension offset factor. The definition method To aggregate weights, traditional GCNs use degree-normalized adjacency matrices; here we design... An adaptive coefficient that integrates inter-node feature differences and structural residual tension:

[0043] in, For nodes spatial coordinates, This refers to the theoretical connection vector of the same connecting edge in the CAD drawing, i.e., in structural design. and The expected spatial displacement difference In practice, it represents the tension residual in the assembly structure, used to reflect the degree of offset between the point cloud structure and the CAD model at the connection edges during actual data acquisition. This is a tension adjustment factor. This design enables the model to automatically focus on edges where the actual structure deviates from the design structure when judging the propagation of structural diagram information, thereby generating a high response to defects that cannot be judged solely by modal characteristics, such as weld misalignment, misinstalled bolt holes, and geometric skew.

[0044] Among them, the node state vector The final dimensions are set to three dimensions, corresponding to:

[0045] The first dimension represents the probability of the defect existing. , used for Softmax classification;

[0046] The second dimension represents the defect level index. ,Depend on The amplitude is mapped by Sigmoid and then thresholded.

[0047] The third dimension is an indicator of the degree of local inconsistency in the feature space. This is for use only in subsequent evolutionary modeling.

[0048] Furthermore, to improve the system's separability for multiple defect types, a structural misalignment focusing loss function is introduced during the training phase. Its form is: ;

[0049] in, For nodes The structural local inconsistency index is extracted from the third dimension of the aforementioned state vector. The class labels predicted by the model. The labels are manually annotated, and CrossEntropy represents the cross-entropy loss. This is a structural anomaly focusing factor. This loss term enhances the model's accuracy in identifying structural anomalies and improves its robustness to environments with overlapping connections, occluded regions, and lighting interference. The final output is a sequence of defect states for the structural graph nodes. ,in These are the types of defects (scratches, dents, misalignment, etc.). The severity level is indicated by a classification (e.g., mild / moderate / severe). The center coordinates of the corresponding node are determined by the CAD structural drawing in step one.

[0050] S4. Input the structural graph node defect state sequence of the same automotive component at multiple detection time points into a time-series modeling network guided by graph structure, output a defect risk score to represent the current defect evolution risk, and output a corresponding quality control strategy based on the current defect evolution risk.

[0051] Specifically, the goal of this step is to perform time-series modeling of the defect state of each node in the structure diagram at different inspection times, thereby predicting future defect evolution trends and outputting executable quality control strategies accordingly. Unlike the traditional single-inspection-judgment approach, this step starts from the actual setup of multi-stage inspection stations on industrial production lines. It utilizes images and point cloud data collected multiple times at different inspection nodes (such as before welding, after welding, after painting, before assembly, etc.) of the same automotive component. After processing in the previous steps, a set of defect states of structure diagram nodes at multiple time points can be obtained, thereby constructing a time-series structure diagram.

[0052] Furthermore, the input data originates from the current structural graph node defect state sequence output from the previous stage. ,in Indicates the detection time step. Represents a node Defect types, Defect level, This represents the spatial location of nodes in the structural coordinate system. All nodes maintain the same structural topology at all time steps; only the node states are updated over time, forming a multi-time-slice structural diagram with a fixed structural topology and dynamically changing states. Taking the previous example, if the goal is to detect the deformation trend of the door reinforcement beam at multiple workstations, images and point cloud data can be acquired after welding, after painting, and before final assembly to obtain the defect state at multiple time points. The nodes of the middle section, joint section and edge section of the reinforced beam are modeled separately based on the structural positioning of the diagram.

[0053] Furthermore, to model the evolution of node states over time, this step employs a graph-guided temporal modeling network. Specifically, a single-layer bidirectional gated recurrent unit (Bi-GRU) is used in the time dimension to capture the nonlinear development trend of defect states; in the structural graph dimension, a static adjacency graph is used to expand the temporal node states into local region state blocks to guide the network in learning the state linkages between adjacent nodes. In actual engineering, each node state... Constructed as a three-dimensional vector, containing: , representing the confidence probability of the defect occurring, is derived from the output of the previous classification header; The value indicates the defect level, with values ​​such as 1 for mild, 2 for moderate, and 3 for severe. The amplitude is obtained through calculation; , represents the node local modal consistency perturbation index, which measures the local structural volatility and is derived from local gradient statistics in the feature space.

[0054] Furthermore, the multi-time-step states of each node are input into the Bi-GRU encoder to form an evolutionary path representation. Combined with the adjacency structure information in the structural graph, the difference in the latest state changes of a node's neighbors is used as an auxiliary factor for modeling the evolutionary trend, and this is further processed through a linear layer. Mapping outputs the defect risk score for the next time step. The expression is as follows:

[0055] in, The time gradient represents the defect state, reflecting the rate at which the defect changes at that node; For nodes in the structure graph The adjacency set, It is the latest state of the neighboring nodes; This is the structural disturbance amplification factor; This is a linear layer with Dropout, used to stabilize the output and avoid overfitting.

[0056] Considering that some defect evolution paths are phased and localized, such as slow initial changes but rapid deterioration after a critical point, this step introduces a trend-focusing regularization term. This step does not rely on manual labels, but rather derives pseudo-supervision signals from the historical state changes of the nodes themselves, helping the model learn nodes that are "more sensitive to change." The formula is as follows: ;

[0057] in, The variance representing the historical variation in defect severity. The regularization term amplifies nodes with "large level changes" or "frequent level fluctuations," thereby focusing on potential risk points in the changing stages of the manufacturing process and achieving the goal of "early detection and early handling" in industrial quality control.

[0058] The final output is the set of defect risk scores for the next time step. The risk value can be mapped to three types of quality strategies based on a set threshold: like This indicates that the defect risk is acceptable and production can continue. like This triggers the manual review process; like If so, a rework signal will be output to the production line control system.

[0059] During implementation, all risk scores will be coded into the production line quality inspection system, corresponding one-to-one with the number of each workpiece, forming a digital inspection file.

[0060] This invention also provides an intelligent detection system for automotive component defects based on multimodal data fusion, the system comprising: The data acquisition and alignment module is used to acquire image data and 3D point cloud data of automotive parts, and perform spatial alignment processing to map them to a unified structural coordinate system to obtain a registration mode set; wherein, the registration mode set includes tensors after transforming the image data to the structural coordinate system and the 3D position expression of the 3D point cloud data in the workpiece coordinate system; The structure graph construction and feature fusion module is used to construct a structure graph based on the structural design model of the automotive component, extract modal features of the corresponding region from the registration modality set, and generate structure graph node embedding vectors with structural semantics through a cross-modal fusion strategy. The defect identification and classification module is used to perform node-level reasoning based on the structure graph and the structure graph node embedding vector, using a pre-built graph neural network, and output a node state vector and a current defect structure graph node defect state sequence; the structure graph node defect state sequence includes type identification, location determination and severity classification; wherein, the pre-built graph neural network is a three-layer graph convolutional neural network, each layer using an improved variable receptive field convolution; The temporal evolution prediction module is used to input the structural graph node defect state sequence of the same automotive component at multiple detection time points into a temporal modeling network guided by graph structure, output a defect risk score to represent the current defect evolution risk, and output a corresponding quality control strategy based on the current defect evolution risk.

[0061] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0062] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or units may be electrical, mechanical, or other forms.

[0063] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0064] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for intelligent detection of defects in automobile parts based on multi-modal data fusion, characterized in that, The method comprises the following steps: S1, collecting image data and three-dimensional point cloud data of an automobile part, and performing spatial alignment processing to map them to a unified structure coordinate system to obtain a registration modal set; wherein the registration modal set comprises a tensor obtained by transforming the image data to the structure coordinate system and a three-dimensional position expression of the three-dimensional point cloud data in the workpiece coordinate system; S2, constructing a structure graph based on a structure design model of the automobile part, extracting modal features of a corresponding region from the registration modal set, and generating a structure graph node embedding vector with structure semantics through a cross-modal fusion strategy; S3, based on the structure graph and the structure graph node embedding vector, using a pre-constructed graph neural network to perform node-level reasoning, outputting a node state vector and a structure graph node defect state sequence of the current defect; the structure graph node defect state sequence comprises type identification, position determination and severity classification; wherein the pre-constructed graph neural network is a three-layer graph convolutional neural network, and each layer uses an improved variable receptive field convolution; S4, inputting the structure graph node defect state sequence of the same automobile part at multiple detection time points into a graph structure guided time series modeling network to output a defect risk score for representing the current defect evolution risk, and outputting a corresponding quality control strategy based on the current defect evolution risk.

2. The intelligent detection method of automobile component defects based on multi-modal data fusion according to claim 1, characterized in that, The image data is collected by an industrial area array camera fixedly installed directly above a production line detection station; the three-dimensional point cloud data is obtained by a high-precision structured light three-dimensional scanner, and the three-dimensional point cloud data in the sensor coordinate system is reconstructed by triangulation; Wherein, the spatial alignment processing is performed to map them to a unified structure coordinate system to obtain a registration modal set, specifically as follows: Obtain the image data and the three-dimensional point cloud data; The fixed pose matrix is obtained by using line calibration, and the space mapping is realized by combining the three-dimensional point cloud data, so as to obtain the position of the first point in the workpiece coordinate system. ​ Each pixel of the image data is associated with a three-dimensional position expression in the three-dimensional point cloud data through the alignment relationship between the camera intrinsic parameter and the corresponding three-dimensional point cloud data; Combine the three-dimensional position expression and the tensor obtained by transforming the image data to the structure coordinate system to obtain the registration modal set.

3. The intelligent detection method of automobile component defects based on multi-modal data fusion according to claim 1, characterized in that, The structure graph is constructed based on the structure design model of the automobile part, and the modal features of the corresponding region are extracted from the registration modal set, specifically as follows: Each node of the structure graph represents an actual functional unit, and each node is associated with a structure region, which is used to extract an image sub-tensor and a point cloud subset in the region from the registration modal set; The image region is extracted by the first three Bottleneck layers of a lightweight convolutional network MobileNetV3 to obtain texture features, and the point cloud subset is processed by a three-layer EdgeConv module, which first constructs a local geometry based on dynamic The local geometry is constructed based on the dynamic 4. The intelligent detection method of automobile component defects based on multi-modal data fusion according to claim 3, characterized in that, The structure graph node embedding vector with structure semantics is generated through a cross-modal fusion strategy, specifically as follows: Obtain the texture features and point cloud features; Construct a residual signal of the point cloud modal based on the texture features of the image data, and form a fusion feature through a residual enhancement mechanism; Based on the fusion feature of any two nodes, construct a structure consistency regularization for suppressing the feature deviation problem caused by modal mismatch in the training stage; The fusion feature corresponding to each structure graph node for embedding is reserved to form a structure graph node embedding vector.

5. The intelligent detection method of automobile component defects based on multi-modal data fusion according to claim 4, characterized in that, The residual enhancement mechanism is realized through a modal alignment operation based on channel attention, specifically as follows: The point cloud features are first reduced in dimension by two layers of channel-wise convolution to dimension, and then a sigmoid activation is used to form a weight vector for dynamically adjusting the response degree of the point cloud channels to the image channels.

6. The intelligent detection method of automobile component defects based on multi-modal data fusion according to claim 1, characterized in that, The pre-constructed graph neural network further comprises a residual tension offset factor for modeling stretch or misalignment perception caused by defects in structural connection; the residual tension offset factor is an aggregated weight of each layer of graph convolution, and the aggregated weight is an adaptive coefficient fusing feature difference between nodes and structural residual tension, and is calculated by combining spatial coordinates of any two graph nodes, expected spatial displacement difference, and tension residual in the assembled structure; The tension residual is used to reflect the offset degree of the point cloud structure in the actual sampling and the connection edge of the CAD model.

7. The intelligent detection method of automobile component defects based on multi-modal data fusion according to claim 1, characterized in that, The final dimension of the node state vector is set to three dimensions, respectively corresponding to the defect existence probability, the defect level index, and the local inconsistency degree index in the feature space; The type identification is obtained by Softmax classification of the defect existence probability; the position determination is the center coordinates of the corresponding node; and the severity grading is the defect level index.

8. The intelligent detection method of automobile component defects based on multi-modal data fusion according to claim 1, characterized in that, The time sequence modeling network guided by the graph structure uses a single-layer bidirectional gated recurrent unit in the time dimension to capture the nonlinear development trend of the defect state; in the structural graph dimension, the time sequence node state is expanded into a local area state block using a static adjacency graph to guide the network to learn the state linkage between adjacent nodes; The input of the same structural graph node defect state sequence of the automobile part at multiple detection time points into the time sequence modeling network guided by the graph structure outputs a defect risk score, and specifically includes: The structural graph node defect state sequence of each node is input into the single-layer bidirectional gated recurrent unit to form an evolution path representation, and the latest state change difference of the node neighbors is combined with the adjacency structure information of the node neighbors in the structural graph to serve as an auxiliary factor for evolution trend modeling, and a linear layer is mapped to output the defect risk score at the next time point; wherein the trend focusing regularization term generates a pseudo-supervision signal from the historical state change of the node itself, which is used to assist the time sequence modeling network to learn the more sensitive nodes in the change.

9. The intelligent detection method of automobile component defects based on multi-modal data fusion according to claim 1, characterized in that, The corresponding quality control strategy specifically includes: If the defect risk score is less than a first preset threshold, it indicates that the defect risk is acceptable, and the production continues; If the defect risk score is greater than or equal to the first preset threshold and less than a second preset threshold, an artificial review process is triggered; If the defect risk score is greater than or equal to the second preset threshold, a rework signal is output to the production line control system.

10. An intelligent detection system for defects in automobile parts based on multi-modal data fusion, characterized by, The system includes: A data acquisition and alignment module for acquiring image data and three-dimensional point cloud data of an automobile part and performing spatial alignment processing to map them to a unified structural coordinate system to obtain a registration modal set; wherein the registration modal set includes a tensor obtained by transforming the image data to the structural coordinate system and a three-dimensional position expression of the three-dimensional point cloud data in the workpiece coordinate system; A structural graph construction and feature fusion module for constructing a structural graph based on a structural design model of the automobile part, extracting modal features of the corresponding region from the registration modal set, and generating a structural graph node embedding vector with structural semantics through a cross-modal fusion strategy; The defect identification and grading module is configured to perform node-level reasoning by using a pre-constructed graph neural network based on the structure graph and the structure graph node embedding vectors, output a node state vector, and output a structure graph node defect state sequence of a current defect. The structure graph node defect state sequence includes type identification, position determination, and severity grading. The pre-constructed graph neural network is a three-layer graph convolutional neural network, and each layer uses an improved variable receptive field convolution. The time evolution prediction module is configured to input the structure graph node defect state sequences of the same automobile component at multiple detection time points into a graph structure guided time series modeling network, output a defect risk score, and output a corresponding quality control strategy based on the current defect evolution risk.

Images