Data analysis method and device based on multi-modal data fusion, equipment and medium

Abstract

The invention relates to the technical field of data analysis, and discloses a data analysis method and device based on multi-modal data fusion, equipment and a medium, and the method comprises the steps: collecting visual data, audio data and process text data, carrying out the preprocessing and standardization processing of different types of data, constructing a multi-modal data set with aligned timestamps, and storing the multi-modal data set in a database; the method comprises the following steps: extracting a time-frequency dynamic feature and a semantic vector feature, extracting a map structure feature, a time-frequency dynamic feature and a semantic vector feature, fusing the features by using a cross-modal attention mechanism to generate a fused feature vector, finally performing analysis processing based on the fused feature vector, and outputting an analysis result. According to the method, the multi-modal data set with consistent time is constructed, the structural features of various modals are extracted, and the cross-modal attention mechanism is introduced to realize deep fusion of the feature level, so that the problems of single information utilization and weak feature relevance of the existing detection means are solved, and the comprehensiveness of defect detection and the accuracy of fault diagnosis are improved.

Description

Technical Field

[0001] This invention relates to the field of data analysis technology, and in particular to a data analysis method, apparatus, device, and storage medium based on multimodal data fusion. Background Technology

[0002] Against the backdrop of the rapid development of the new energy vehicle industry, quality inspection and fault diagnosis in the production process are facing higher requirements for accuracy and efficiency. To meet the demands of high-quality production, the industry has gradually introduced various automated inspection technologies, including image recognition, acoustic analysis, and production data analysis. However, existing inspection systems still have many shortcomings in complex manufacturing environments, especially in handling and deeply integrating multi-source heterogeneous information, making it difficult to effectively support the identification and localization of highly complex defect patterns.

[0003] Current mainstream quality inspection methods are mostly based on manual vision or single-modal sensors. For example, in the inspection of vehicle body surface defects, some production lines have deployed vision inspection systems based on industrial cameras, which use image processing algorithms to identify surface defects such as scratches and dents. These systems are effective in handling targets with regular textures and clear boundaries, but they are prone to problems such as decreased detection accuracy, false alarms, or missed detections when faced with surface reflection interference, complex background textures, or micro-cracks. At the same time, for defects hidden inside the structure or hidden errors in the assembly process, these purely visual methods often cannot detect and locate them, thus limiting their coverage.

[0004] In terms of equipment operation status monitoring, some production systems employ vibration or acoustic signal monitoring methods, such as using spectrum analysis to determine abnormal operation of motors or drive systems. Although these methods possess a certain degree of real-time performance and sensitivity, they are highly susceptible to interference from strong background noise in the production environment. Furthermore, when different fault types occur concurrently, or when fault signals are highly coupled with normal operating condition signals, traditional acoustic analysis methods struggle to accurately distinguish the fault source, resulting in low diagnostic specificity and a high false alarm rate.

[0005] Another common problem lies in the efficiency of utilizing process data and maintenance documents. Although modern production systems can accumulate a large amount of process parameters, alarm records, and historical maintenance documents, this data is mostly stored in structured and unstructured forms, lacking a unified processing mechanism. Currently, systems generally lack the ability to understand the semantics and perform correlation analysis on textual information, making it difficult to establish an effective connection between process fluctuations, operational behaviors, and actual faults. This results in potential risks not being detected in a timely manner, and experiential knowledge cannot be accumulated and reused at the system level.

[0006] Overall, current technologies are unable to fully integrate data resources from multiple modalities, such as vision, audio, and text, and struggle to uncover deep correlations between various types of information. This results in insufficient capabilities for defect detection to handle complex scenarios and for fault diagnosis to address multi-factor coupling. For quality control scenarios in the manufacturing process of new energy vehicles, there is an urgent need for an intelligent processing solution that can integrate multimodal data and improve the comprehensiveness of detection and the accuracy of diagnosis to overcome the limitations of current technologies. Summary of the Invention

[0007] The main objective of this invention is to provide a data analysis method, apparatus, device, and storage medium based on multimodal data fusion, aiming to solve the technical problem that traditional single-modal or shallow multimodal fusion methods cannot fully explore the deep cross-modal correlations between visual, audio, and process text data, resulting in low defect detection accuracy in complex industrial scenarios.

[0008] To achieve the above objectives, the present invention provides a data analysis method based on multimodal data fusion, comprising: Collect visual data, audio data, and process text data; The visual data is subjected to geometric correction and illumination compensation processing, the audio data is subjected to bandpass filtering processing, and the process text data is subjected to format standardization processing to generate a timestamp-aligned multimodal dataset. Feature extraction is performed on each type of data in the multimodal dataset to generate spectral structure features of visual data, time-frequency dynamic features of audio data, and semantic vector features of process text data. A fused feature vector is generated by fusing the graph structure features, time-frequency dynamic features, and semantic vector features through a cross-modal attention mechanism. The analysis results are generated based on the fused feature vectors.

[0009] Furthermore, to achieve the above objectives, the present invention provides a data analysis device based on multimodal data fusion, comprising: The multimodal data acquisition module is used to acquire visual data, audio data, and process text data. The data preprocessing and alignment module is used to perform geometric correction and illumination compensation processing on the visual data, bandpass filtering processing on the audio data, format standardization processing on the process text data, and generate a timestamp-aligned multimodal dataset. The feature extraction module is used to extract features from each type of data in the multimodal dataset to generate spectral structure features of visual data, time-frequency dynamic features of audio data, and semantic vector features of process text data. The feature fusion module is used to fuse the graph structure features, time-frequency dynamic features and semantic vector features through a cross-modal attention mechanism to generate a fused feature vector; The intelligent analysis module is used to perform analysis and processing based on the fused feature vector to generate analysis results.

[0010] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a data analysis program based on multimodal data fusion stored in the memory and executable on the processor, wherein when the data analysis program based on multimodal data fusion is executed by the processor, it implements the steps of the data analysis method based on multimodal data fusion as described above.

[0011] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a data analysis program based on multimodal data fusion, wherein the data analysis program based on multimodal data fusion, when executed by a processor, implements the steps of the data analysis method based on multimodal data fusion as described above.

[0012] Beneficial Effects: This invention relates to the field of data analysis technology and discloses a data analysis method, apparatus, device, and medium based on multimodal data fusion. The method includes: collecting visual data, audio data, and process text data; preprocessing and standardizing different types of data separately; constructing a timestamp-aligned multimodal dataset; extracting spectral structure features, time-frequency dynamic features, and semantic vector features; fusing these features using a cross-modal attention mechanism to generate a fused feature vector; and finally, analyzing and processing based on the fused feature vector to output the analysis results. This invention constructs a time-consistent multimodal dataset, extracts structured features from various modalities, and introduces a cross-modal attention mechanism to achieve deep feature fusion. This solves the problems of limited information utilization and weak feature correlation in existing detection methods, thereby improving the comprehensiveness of defect detection and the accuracy of fault diagnosis. Attached Figure Description

[0013] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a schematic diagram of an application environment for a data analysis method based on multimodal data fusion according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating an embodiment of the data analysis method based on multimodal data fusion according to the present invention; Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the data analysis device based on multimodal data fusion of the present invention; Figure 4This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0014] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0015] The data analysis method based on multimodal data fusion provided in this invention can be applied to, for example... Figure 1 In this application environment, the user terminal communicates with the server via a network. The server can collect visual data, audio data, and process text data from the user terminal, preprocess and standardize different types of data respectively, construct a timestamp-aligned multimodal dataset, and then extract graph structure features, time-frequency dynamic features, and semantic vector features. A cross-modal attention mechanism is then used to fuse these features, generating a fused feature vector. Finally, analysis is performed based on the fused feature vector, and the analysis results are output. This invention constructs a time-consistent multimodal dataset, extracts structured features from various modalities, and introduces a cross-modal attention mechanism to achieve deep feature fusion. This solves the problems of single information utilization and weak feature correlation in existing detection methods, thereby improving the comprehensiveness of defect detection and the accuracy of fault diagnosis. The user terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0016] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the data analysis method based on multimodal data fusion provided by the present invention. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0017] like Figure 2 As shown, the data analysis method based on multimodal data fusion proposed in this invention includes the following steps: S10 collects visual data, audio data, and process text data; In this embodiment, the acquisition of visual data, audio data, and process text data involves the parallel acquisition and unified organization of heterogeneous data sources. The aim is to build a comprehensive perception data foundation covering equipment status, process flow, and production line environment. Visual data refers to two-dimensional images or video sequences generated by image sensing devices, typically used to represent visible morphological features such as welding surfaces, assembly interfaces, and part outlines. This type of data primarily originates from industrial cameras installed at key production stations, preferably multispectral imaging devices, to compensate for recognition uncertainties caused by differences in reflection under different materials or operating conditions. By integrating infrared, visible light, or near-ultraviolet sensors, stable image samples can be obtained even under conditions of surface reflection interference, oil contamination, or heat source disturbance.

[0018] Audio data is derived from acoustic information such as structural sounds, airborne sounds, and electromagnetic noise at the work site. High-sensitivity microphone arrays are deployed to capture high-frequency howling, low-frequency vibration sounds, or abnormal collision sounds generated by equipment during operation. These microphones typically possess good frequency response characteristics, signal-to-noise ratio, and electromagnetic interference immunity. Their deployment locations should avoid strong background noise sources but be close to the monitoring target device. To achieve spatial positioning and direction awareness, the microphone array can be configured as a linear, ring, or area array structure, and combined with beamforming algorithms to enhance signal quality and suppress noise interference.

[0019] Process text data originates from text resources such as parameter logs, quality records, abnormal alarm information, and maintenance instructions automatically recorded or manually entered during the manufacturing process. These resources reside in digital platforms such as Manufacturing Execution Systems (MES), Equipment Management Systems (EMS), and Quality Management Systems (QMS). Text content types include structured form fields (such as temperature, pressure, and current), semi-structured time-event pairs (such as equipment alarm records), and unstructured process descriptions and maintenance logs (such as operator notes or maintenance engineer reports). This data is typically stored in different databases in various formats (such as JSON, CSV, and SQL records), requiring access through a unified data extraction mechanism and standardized processing for field mapping, time alignment, and encoding.

[0020] During the acquisition of multimodal data, visual, audio, and text data require a unified time control mechanism to coordinate the acquisition timing. Typically, a hardware synchronization signal or a synchronization pulse signal output by a time server is used as a unified time reference to trigger the industrial camera to start exposure, the microphone to start recording, and the process system to generate timestamp records, ensuring that the data has temporal comparability and a basis for subsequent fusion analysis.

[0021] In a practical implementation, a row of industrial cameras with visible and infrared band channels can be deployed above the welding assembly line to capture 30 frames per second of 1920x1080 resolution image sequences. Six microphones, forming a 120-degree arc-shaped linear array, are deployed on a bracket beside the welding torch, with a sampling frequency set to 48kHz, for real-time monitoring of arc sound changes. An API-based log push mechanism is configured in the manufacturing execution system, with each data record containing an automatically generated UTC timestamp field indicating welding current, movement speed, and temperature control parameters. All three types of equipment are connected to the master clock system via a time synchronization module. The master clock outputs a 10Hz synchronization pulse per second, which is transmitted in parallel to each acquisition submodule via low-latency cables as the sampling start signal.

[0022] Example Description: During the inspection of drive motor assemblies in new energy vehicles, visual data can record whether the end caps are misaligned, audio data can monitor the whistling sound during bearing startup, and process text data records operation time, current setpoints, and actual temperature. In one data acquisition session, an industrial camera recorded image frames showing a slight displacement of the motor housing, an abnormal increase in the 2500Hz frequency band was detected in the audio, and the text record showed that the welding current deviated from the setpoint by more than 10%. These three types of information are integrated and used with consistent timestamps to provide early diagnostic support for equipment anomalies.

[0023] This embodiment achieves precise synchronization of visual images, sound samples, and text logs on the timeline by uniformly scheduling the data acquisition behavior of industrial cameras, microphone arrays, and information systems, thus constructing a cross-modal, time-consistent basic dataset. This operation effectively avoids information fusion errors caused by asynchronous data sources, improving the temporal resolution and accuracy of subsequent feature extraction and model inference.

[0024] S20, perform geometric correction and illumination compensation processing on the visual data, perform bandpass filtering processing on the audio data, perform format standardization processing on the process text data, and generate a timestamp-aligned multimodal dataset; In this embodiment, geometric correction and illumination compensation are performed on the visual data to address common problems of viewpoint distortion and uneven illumination during industrial image acquisition. Geometric correction refers to pixel-level remapping adjustment of image distortions caused by lens distortion, viewpoint shift, tilted mounting, etc. This is typically achieved by constructing a back-projection mapping using distortion model parameters (such as radial and tangential distortion coefficients) and the camera intrinsic matrix to restore the image's geometric proportions. In practical deployments, camera distortion parameters can be pre-obtained using a calibration board, and each frame can be corrected using bilinear interpolation or inverse mapping geometric transformation algorithms to eliminate edge stretching and distortion, thereby improving the positioning accuracy of subsequent visual region recognition.

[0025] Illumination compensation primarily addresses uneven brightness caused by unstable light sources, metal reflection interference, or localized occlusion in industrial environments. Common methods include global histogram equalization, adaptive histogram equalization (CLAHE), and local dynamic range adjustment based on the image's Laplacian pyramid. By enhancing the contrast of low-brightness areas and suppressing overexposure in high-brightness areas, the recognizability of details such as weld seams and edge contours can be effectively improved, providing a balanced image foundation for subsequent feature extraction.

[0026] Bandpass filtering of audio data is used to extract key frequency band signals of equipment operation in noisy manufacturing environments. A bandpass filter defines a frequency window that preserves the target frequency band signal (e.g., 30Hz to 18kHz) while suppressing low-frequency mechanical vibrations and high-frequency electromagnetic noise. FIR or IIR filter designs are commonly used, with filter coefficients constructed using Butterworth, Chebyshev, or Kaiser window functions. Implementation methods include performing time-domain convolution or frequency-domain multiplication on the audio signal stream after digital filter design to remove background noise irrelevant to detection, thereby improving the ability of subsequent feature analysis to identify weak abnormal sounds.

[0027] Standardizing the format of process text data involves transforming process information extracted from multi-source heterogeneous systems into a unified encoding structure to achieve structured field extraction and cross-system time alignment. This operation includes field naming mapping (e.g., unifying "welding current" and "welding current" into "welding_current"), encoding method conversion (e.g., from GBK to UTF-8), and unit normalization (e.g., unifying °C and K into Celsius). Simultaneously, to support accurate fusion of text timestamps, the time field in the original records needs to be parsed and converted into a unified UTC time format. This standardization process facilitates the integration of record information from MES, QMS, and PLC systems and improves the consistency of semantic analysis.

[0028] Generating timestamp-aligned multimodal datasets involves aligning visual, audio, and process text data according to timestamps after each preprocessing step, forming a unified set of data segments for subsequent joint modeling or feature extraction. The key to timestamp alignment lies in the different sampling frequencies, durations, and event trigger boundaries of the multimodal data. A unified time window needs to be defined to slice each data type according to its timestamp, ensuring they correspond to the same physical operating cycle. Alignment methods can employ fixed windows (e.g., 1 second), sliding windows (e.g., 0.2-second steps), or event-driven windows (e.g., a sudden current change triggers the process). Simultaneously, keyframe extraction is performed on visual frames within the window, energy compression is applied to audio signals, and field filling is performed on text data. The final result is a data consortium where each time slice contains one image frame, one audio feature segment, and one set of text parameters.

[0029] On a new energy vehicle welding production line, cameras can be deployed to capture video frames with a resolution of 1920×1080 and a frame rate of 25fps. Using pre-calibrated intrinsic parameter matrices and distortion coefficients, distortion correction is performed on the images, followed by adaptive histogram equalization to enhance the contrast characteristics of different welding positions. Raw audio data acquired by a microphone array is sampled at 48kHz and noise suppression is achieved using an IIR bandpass filter with a designed bandwidth of 30Hz to 18kHz. Process text data is extracted from the MES in CSV format, with fields including current, voltage, displacement, and time. After parsing, it is uniformly encoded in UTF-8 and the field names are standardized. To complete data alignment, every second is used as a time window. Images, audio, and text records appearing within this time period are fused to generate a joint data unit containing three modal data segments for subsequent feature extraction.

[0030] This embodiment effectively eliminates noise interference and structural ambiguity within a modality by uniformly preprocessing and aligning image, audio, and text data with timestamps. At the same time, it establishes a consistent time reference point among multiple modalities, enabling visual anomalies, acoustic anomalies, and process parameter deviations at the same production moment to be jointly captured and located, thereby providing a unified and high-quality input data structure for subsequent joint analysis.

[0031] S30, respectively extract features from each type of data in the multimodal dataset to generate the spectral structure features of visual data, the time-frequency dynamic features of audio data, and the semantic vector features of process text data; In this embodiment, targeted and fusion-compatible representations need to be constructed for different types of information in the multimodal dataset. For visual data, the ability to model spatial relationships and defect distribution can be improved by extracting structured region representations from images and introducing graph structure modeling methods. Specifically, semantically meaningful regions, including weld seams, edge contours, and component boundaries, are identified in the image after illumination compensation. Target region boundaries are obtained by combining pixel segmentation, edge detection, and heatmap response methods. These regions are mapped as graph nodes, and the spatial distance, directional relationship, and topological structure between nodes are used to define the edges of the graph. The graph structure can be constructed as an undirected graph or a weighted graph, and node attributes can include multi-dimensional vectors such as grayscale texture, edge intensity, and region size. Furthermore, a graph neural network is used to perform multi-layer message passing on the graph structure, dynamically updating the embedding representation of each graph node based on neighbor node information, and outputting the feature representation of the entire graph as graph structure features for subsequent cross-modal fusion.

[0032] For audio data, a time-frequency transformation is used to map the original time series to a two-dimensional spectrogram space, from which key dynamic features are extracted. First, a short-time Fourier transform is applied to the bandpass-filtered audio signal, followed by frame processing using a sliding window. The window length and step size are selected based on the temporal granularity of the target event, with a typical configuration of a 25-millisecond window and a 10-millisecond step size. The frequency distribution in the spectrogram reflects the structural patterns and short-term anomalies during equipment movement. Mel-frequency cepstral coefficients can be further extracted to represent a audibly consistent energy distribution. Simultaneously, first-order and second-order differences are added to supplement the dynamic change trend, thus constructing a time-frequency dynamic feature. This feature representation can be used to detect problems such as equipment status fluctuations, abnormal welding sounds, and friction signal imbalances.

[0033] For process text data, semantic modeling is needed to map unstructured or semi-structured process parameters and log descriptions into computable vector representations. First, semantic segmentation is performed on the standardized text to extract keywords, units, and symbols, followed by named entity recognition to construct entity-attribute pairs, such as "welding current = 250A" and "gun tip temperature = 320℃". Next, a combination of word segmentation tools and a domain-defined dictionary is used to process process terms, vectorizing entity phrases. Static word vector models (such as Word2Vec and FastText) or context-pre-trained models (such as BERT and RoBERTa) can be used to map text fragments into fixed-dimensional semantic vectors, preserving their contextual semantic relationships, syntactic structure, and terminology combination characteristics. The resulting text semantic vectors serve as the expression form of the process text modality, residing in the same feature space that can be fused with graph structure features and time-frequency dynamic features.

[0034] Visual data processing can employ residual network structures for image semantic segmentation, acquiring the target weld seam region and contour region and using them as graph nodes. Node edge weights are constructed based on Euclidean distance and edge gradient differences. The graph structure is represented by graph processing frameworks such as DGL, and the graph neural network uses GraphSAGE or GCN structures. After three layers of message propagation, it outputs the high-dimensional embedding of each node, and generates the overall graph structure features through graph readout layers (such as average pooling). Audio feature extraction uses the librosa toolkit to perform STFT calculation, extracting 13-dimensional MFCC and its first-order difference features, forming a 39-dimensional time-frequency vector representation. Text data is cleaned using regular expressions and then input into a BERT pre-trained model, extracting 768-dimensional semantic vectors to represent process description paragraphs.

[0035] Example Explanation: When processing a data segment from the welding process of a new energy vehicle battery pack, six weld point regions were extracted as graph nodes from the illumination-compensated image. After calculating their features using a graph neural network, it was found that the texture feature distribution of two nodes was significantly different from other regions. The corresponding audio signal spectrum showed continuous high-amplitude perturbations in the 2000-4000Hz range, which differed from the historical equipment operating noise distribution, and the MFCC feature showed continuous jumps. The parameter extracted from the text record was "welding current stability not up to standard," which was identified as a negative-tense sentence by the semantic model. These three types of features provided consistent abnormal feedback for the same physical welding action within the same time slice, providing structured support for subsequent determination of insufficient welding strength at this location.

[0036] This embodiment extracts targeted features from different types of modal data, capturing spatial structures and inter-regional relationships in visual information, identifying dynamic frequency change trends in audio information, and constructing semantic-level explanations of process background and equipment behavior in text information. All three types of features possess high-dimensional representation capabilities and maintain intra-modal consistency, providing a structured and learnable foundational expression for the cross-modal fusion stage, thereby improving the accuracy and generalization ability of overall fault diagnosis and behavior recognition.

[0037] S40, the graph structure features, time-frequency dynamic features and semantic vector features are fused through a cross-modal attention mechanism to generate a fused feature vector; In this embodiment, the key to fusing graph structural features, time-frequency dynamic features, and semantic vector features lies in establishing alignment relationships among the multiple modalities, enabling various features to reference and complement each other. Graph structural features originate from the embedding results of structural regions in visual images processed by graph neural networks, possessing spatial topological representation capabilities and information on interactions between regions; time-frequency dynamic features are time-series spectral representations extracted based on short-time Fourier transform and Mel-frequency cepstral analysis, reflecting local dynamic signal changes during equipment operation; semantic vector features are derived from language modeling of standardized process text, embodying contextual information such as current process status, equipment parameters, and fault descriptions.

[0038] To unify the three types of features into a shared representation space, a cross-modal attention mechanism is needed to establish computational relationships between queries, keys, and values. Graph structural features are mapped to query vectors through a learnable linear transformation, representing the starting point for visual information's attention to other modalities. Time-frequency dynamic features are mapped to key and value vectors; the key vector is used to calculate a relevance score with the query vector, and the value vector is used to map this relevance to a representation contribution. The dot product between the query and key is scaled and normalized to obtain attention weights, which are used to weighted sum the value vectors, generating an interactive representation between the visual and audio modalities and capturing the collaborative information between the welded structure and the acoustic signal.

[0039] After acquiring the interaction features between vision and audio, the semantic vector features are concatenated with this interaction representation or fused through a gating mechanism to introduce process state information or prior logical relationships expressed by the text modality. The fused vector undergoes a nonlinear transformation by a multilayer perceptron to compress redundancy and enhance discriminative ability in a continuous mapping space, ultimately forming an integrated fused feature vector with unified dimensions and multi-source semantic information, which is used for downstream fault diagnosis and analysis tasks.

[0040] The spectral structure features are set to a 128-dimensional embedding representation. The time-frequency dynamic features are 39-dimensional MFCC+Δ+ΔΔ vectors. "MFCC+Δ+ΔΔ" corresponds to the aforementioned feature of "further extracting Mel-frequency cepstral coefficients to represent a perceptually consistent energy distribution, while adding first-order and second-order differences to supplement the dynamic change trend, thus forming time-frequency dynamic features." Here, "MFCC" corresponds to "Mel-frequency cepstral coefficients," "Δ" corresponds to "first-order difference," and "ΔΔ" corresponds to "second-order difference." The semantic vector features are 768-dimensional BERT encoded outputs. Query, key, and value vectors are all transformed to a unified 64-dimensional representation space using independent linear layers. Dot-product attention calculation uses the standard Scaled Dot-Product formula, obtaining normalized weights through softmax operation, and then performing matrix multiplication on the value vector to obtain visual-audio fusion features. The semantic vector is compressed to 128 dimensions using a fully connected layer and then concatenated with the aforementioned fusion features to form the final multimodal joint vector. The multilayer perceptron is configured with a two-layer structure. The first layer has the ReLU activation function, and the second layer has an output dimension of 256, which is used for subsequent classification models or sequence modeling modules.

[0041] Example Description: In the data processing of a motor rotor assembly task, a graph neural network extracts multiple bearing mounting point regions from camera images and generates a structure map embedding. After short-time Fourier transform, a set of high-frequency noise is detected to persist for a specific time period, with significant changes in the corresponding MFCC features. The text information indicates that the current operation is "high-speed rotation section" and there is a historical maintenance record of "rotor eccentricity". Visual features are used as queries, and audio features are used as keys and values. An attention mechanism is used to obtain a joint visual-audio representation, which is then concatenated with the text semantic vector. After the fused vector is input into the perceptron, it is accurately identified by the classification model as "mild eccentric bearing anomaly", verifying the effectiveness of the fusion mechanism in the joint modeling of multimodal fault features.

[0042] This embodiment utilizes a cross-modal attention mechanism to achieve weighted combination of multi-source features. This not only explicitly models the interaction relationships between modalities but also dynamically adjusts the contribution of different modalities in the fusion process. The visual modality focuses on structural morphology and spatial regions, the audio modality reflects the dynamic state of the device, and the text modality provides semantic background and process parameters. The attention mechanism enables complementary fusion of information, avoiding modal redundancy and semantic conflicts during feature concatenation, and improving the effectiveness and discriminative power of the final feature representation.

[0043] S50, perform analysis and processing based on the fused feature vector to generate analysis results.

[0044] In this embodiment, the fused feature vector is a high-order representation that integrates modal alignment and information integration of spectral structural features, time-frequency dynamic features, and semantic vector features. It possesses the comprehensive ability to express the welding image structure, acoustic variations, and process semantic logic. This fused vector needs to undergo further analysis to extract results that can be used for defect identification and localization. The analysis and processing flow needs to implement a closed-loop logic of feature classification, association matching, localization mapping, and semantic interpretation.

[0045] First, the fused feature vector is fed as input to a pre-trained classification model to identify whether the current workpiece has defects, and outputs the corresponding defect type label and classification probability value as confidence. The classification model can be a multilayer perceptron, a convolutional neural network followed by a fully connected classifier, or a graph attention model followed by a softmax decision head. Its function is to classify and determine the fault mode based on the fused representation.

[0046] Subsequently, a historical fault case database is introduced, and the similarity between the current fused feature vector and the fused features already stored in the database is measured. The measurement method can use cosine similarity, Euclidean distance, or a similarity score calculated via a Siamese network. The matched high-similarity cases form a candidate set, providing support for subsequent fault visualization, localization, and causal tracing.

[0047] Next, using the image coordinate information annotated in the candidate cases, the possible defect areas are located in the current raw visual data, typically expressed in the form of the minimum bounding rectangle coordinates, and mapped back to keyframes or real-time images for highlighting the defect areas. Simultaneously, based on the extracted time-frequency dynamic features and combined with the attention focus area, the time intervals corresponding to the abnormal frequency bands are deduced to locate key anomalies during device operation.

[0048] Finally, by using structured text comparison, the process parameters related to the current operating conditions in the standardized text data are compared with the cause descriptions in the candidate cases. The parameter deviation descriptions with relevance are extracted to construct structured cause analysis statements.

[0049] All output information will be integrated in a structured format, including defect type labels, visual location coordinates, start and end points of abnormal frequency bands, and textual explanations of causes, forming analysis results supported by multi-source evidence to serve subsequent decision-making or maintenance recommendation systems.

[0050] The fusion feature vector dimension is uniformly set to 256 dimensions. The classification model adopts a three-layer perceptron structure, and the output is 11 defect labels and a softmax probability distribution. The similarity matching process uses a vector index library built based on Faiss, and filters the top 5 similar cases by L2 distance and returns their annotation information. Image localization uses the coordinate mapping mechanism in OpenCV to mark rectangles in the original image frames and output the coordinates of the four corners. Anomaly localization of time-frequency dynamic features is based on the energy mutation point of the first-order difference Δ of MFCC, and the start and end frames of the anomaly are located by threshold difference. Textual process parameters and case descriptions are generated into causal statements by using Jaccard similarity and keyword alignment rules. All information is organized into a structured dictionary format and output as a JSON object.

[0051] Example Description: In a welding process on a new energy vehicle frame assembly production line, the collected fused feature vector, after being input into the model, was identified as "unstable spot welding energy" with a confidence level of 93.6%. Simultaneously, three fault samples with a similarity exceeding 0.92 were matched from the historical case database. The system read the annotation information of one of these samples, circled two welding points near the lower right front door in the camera image, and generated a minimum rectangular coordinate frame. Spectrum analysis revealed an abnormal high-frequency energy segment appearing in frames 124 to 137, corresponding to times 2.48 to 2.74 seconds, initially confirming abnormal pulsation in the welding current. Further comparison with the process text record revealed that the current upper limit exceeded the set value by 7.2%, inferred to be a short-term overvoltage in the equipment power module. The final output included a structured result containing the defect category, coordinate location, abnormal time period, and cause statement, providing maintenance personnel with accurate location and parameter adjustment references.

[0052] This embodiment, by fusing visual, audio, and textual features into the classification model, not only achieves more accurate defect type determination but also supplements external knowledge support through similarity matching with historical data, making the results more referential and contextually semantic. Simultaneously, by utilizing existing image annotation information and equipment operating condition records from historical cases, intuitive defect locations and time periods can be obtained without retraining the localization model, generating interpretable fault cause reports based on text parameters. This significantly improves the accuracy, operability, and diagnostic closure capability of the analysis results.

[0053] This invention relates to the field of data analysis technology and discloses a data analysis method, apparatus, device, and medium based on multimodal data fusion. The method includes: collecting visual data, audio data, and process text data; preprocessing and standardizing different types of data separately; constructing a timestamp-aligned multimodal dataset; extracting spectral structure features, time-frequency dynamic features, and semantic vector features; fusing these features using a cross-modal attention mechanism to generate a fused feature vector; and finally, analyzing and processing based on the fused feature vector to output the analysis results. This invention, by constructing a time-consistent multimodal dataset, extracting structured features from various modalities, and introducing a cross-modal attention mechanism to achieve deep feature fusion, solves the problems of limited information utilization and weak feature correlation in existing detection methods, thereby improving the comprehensiveness of defect detection and the accuracy of fault diagnosis.

[0054] In one embodiment, step S10 above includes: S101 acquires raw video stream data of the production process through a multispectral industrial camera array; S102, acquires raw audio stream data of the operating location of key equipment through a microphone array; S103, Obtain raw text stream data containing structured process parameters and unstructured maintenance logs from the production execution system database; S104, apply a hardware trigger signal to the multispectral industrial camera array, microphone array and production execution system database to synchronize the acquisition timing of the raw video stream data, raw audio stream data and raw text stream data; S105 adds millisecond-precision timestamps to the time-synchronized raw video stream data, raw audio stream data, and raw text stream data to generate time-stamped raw visual data, time-stamped raw audio data, and time-stamped raw process text data.

[0055] In this embodiment, the process of acquiring visual data is completed through a multispectral industrial camera array. Multispectral cameras can simultaneously capture image information from different wavelengths (such as visible light and near-infrared), thereby providing rich-dimensional data support for material surface conditions, temperature distribution, or coating integrity. The deployment of industrial camera arrays is typically around key processes such as welding stations and assembly sections. Using fixed-angle or adjustable-angle shooting devices, complete coverage of the target area is ensured, adapting to different types of workpieces and structural complexities.

[0056] Audio data acquisition is achieved through microphone arrays deployed at the locations of critical equipment. These arrays contain multiple high signal-to-noise ratio, low-distortion microphones, supporting simultaneous sampling and noise suppression. They are capable of capturing high-frequency howling, low-frequency vibrations, or short pulse sounds during equipment operation, making them suitable for identifying anomalous sound events such as welding arc abnormalities, assembly interference, and bearing damage. Background noise interference is considered during deployment, and directional design and spatial array layout are used to enhance reception of target sound sources.

[0057] The process text data is collected from the workshop-level production execution system database, including real-time uploaded structured process parameters (such as welding current, voltage, gas pressure, speed, etc.) and unstructured maintenance logs (such as manually recorded fault descriptions and inspection notes). This data is automatically retrieved through a database interface, obtaining complete data records for the corresponding time period according to conditions such as equipment number and time index, ensuring traceability and correlation for subsequent analysis.

[0058] To ensure precise consistency of the three types of data across the time dimension, the system applies a unified hardware trigger signal to the camera array, microphone array, and database interface. This signal is generated by the time synchronization module, typically using an external high-precision time reference (such as a GPS clock or IEEE 1588 PTP clock) as the timing control base point. This drives each acquisition device to sample from a unified starting point, avoiding data misalignment or timing drift caused by differences in device response times.

[0059] After acquisition, all raw data are appended with a uniform millisecond-level timestamp. The timestamp accuracy is controlled by the acquisition system's main control module, and the data is output in a unified format. Each frame in the video data is timestamped, audio data uses sampling points to generate a continuous timeline, and text data is mapped with time tags based on the data recording time. The final output consists of timestamped raw visual data, timestamped raw audio data, and timestamped raw process text data, providing accurate temporal references for multimodal alignment.

[0060] The multispectral industrial camera uses a 480fps frame rate, six-channel imaging system, with each camera having its exposure synchronized and driven by an FPGA synchronization signal. The microphone array is deployed around the welding host and electric drive system, employing MEMS high-sensitivity microphones with a 24-bit ADC and pre-amplifier filtering circuitry. Text data is obtained from the MES system's API interface; structured parameters are organized in JSON format, while unstructured text is extracted using a time window. The timing synchronization module integrates a high-precision crystal oscillator module and supports the external PTP protocol; trigger signals are distributed via CAN bus or industrial Ethernet. The timestamp resolution is set to 1ms, uniformly identified using ISO 8601 format and stored in association with each modal data.

[0061] This embodiment, through the synchronous acquisition of video, audio, and text data, not only allows for a complete description of the operating conditions at each moment from multiple perspectives but also ensures the data foundation required for time consistency in subsequent processing steps, avoiding analytical biases caused by inconsistent sampling times between modalities. Simultaneously, the millisecond-level unified timestamp annotation provides high-precision time-series anchors for operations such as modal alignment, feature fusion, and causal correlation, making it easier to uncover and model logical relationships between data, thereby improving the timeliness of fault detection and multimodal interpretability.

[0062] In one embodiment, step S20 above includes: S201, Perform geometric distortion correction processing on the visual data to generate geometrically corrected visual data; S202, perform histogram equalization processing on the geometrically corrected visual data to generate illumination-compensated visual data; S203, Perform bandpass filtering on the audio data to generate filtered audio data; S204, convert the unstructured text in the process text data into a preset encoding format to generate standardized text data; S205, based on the timestamps of the illumination-compensated visual data, filtered audio data, and standardized text data, align them according to a preset time window slice to generate a timestamp-aligned multimodal dataset containing illumination-compensated visual data, filtered audio data, and standardized text data.

[0063] In this embodiment, the geometric distortion correction processing of visual data mainly targets the nonlinear imaging distortion generated by multispectral industrial camera arrays during actual shooting. Common distortion types include radial distortion and tangential distortion, which are rooted in the non-ideal optical characteristics of industrial lenses and sensor surface effects. To eliminate these effects, the system uses the intrinsic parameter matrix and distortion coefficient model obtained from camera calibration to perform pixel-level reverse remapping of image coordinates, generating geometrically corrected visual data that conforms to the real structure of the physical scene, ensuring accurate restoration of the size, angle, and boundary information of target components in the image.

[0064] After geometric correction, histogram equalization is performed to compensate for image illumination and eliminate strong light differences and uneven local exposure in industrial scenes such as welding areas and vehicle body panels. This process reconstructs a balanced distribution of image brightness based on the histogram distribution of pixel intensity, improving contrast and detail, thereby enhancing the robustness of subsequent image node extraction and edge detection operations. The processing methods include global equalization and adaptive local histogram processing, allowing for the selection of the optimal brightness remapping strategy based on different image partitions.

[0065] Audio data processing uses bandpass filtering as its core step, aiming to extract the effective signals carrying operational information within the frequency band while filtering out background environmental noise and low-frequency baseline drift. The system employs finite impulse response (FIR) or infinite impulse response (IIR) filters, with the bandpass typically set from 30Hz to 18kHz, covering the characteristic frequency range of sound sources such as mechanical vibration, motor noise, and impacts. Filtering parameters are configured according to the specific equipment's noise spectrum characteristics to ensure maximum retention of useful signal components and minimize diagnostic interference.

[0066] For process text data, since some information during the acquisition process comes from manual input, system logs, or natural language descriptions, its unstructured content needs to be converted into a unified encoding format to support subsequent analysis. The conversion process includes character set standardization (such as UTF-8 encoding), text segmentation and cleaning, punctuation normalization, special symbol removal, and format alignment. The standardized text is encapsulated in key-value pairs or structured fields, facilitating multimodal fusion with image and audio data within a unified framework.

[0067] All three types of processed data are now accompanied by a unified millisecond-level timestamp. To achieve temporal consistency across modalities, a sliding time window approach is used for slice alignment. Using the keyframe time points of the illumination-compensated visual data as the primary time anchor, audio segments and text records matching their timestamps are retrieved. If temporal coverage is missing in any of the three data types, the system can employ interpolation reconstruction or truncation / cropping strategies to ultimately generate a structurally consistent and temporally synchronized multimodal dataset, providing accurate input for subsequent cross-modal feature extraction and fusion analysis.

[0068] Geometric distortion correction employs the OpenCV framework for calibration and remapping. First, a distortion model is acquired by collecting a checkerboard image sequence, then the `cv::undistort()` function is called to complete the correction. Histogram equalization is based on `cv::equalizeHist()`, and for RGB images, HSV space is used for channel-by-channel processing. In the audio filtering module, the sampling frequency is 44.1kHz, and the bandpass filter is a 6th-order filter designed and generated by Butterworth, covering a frequency band of 30Hz-18kHz, implemented using the SciPy signal library. Text normalization is based on regular expressions and word segmentation tools; unstructured content is converted into JSON structured fields using a key field extraction function. Timestamp alignment logic is completed by a multi-threaded time-series alignment engine. A time tolerance threshold of ±10ms is set between keyframes, audio frames, and text records. The time overlap ratio controls the window effectiveness, and finally, aligned data is output in groups by window.

[0069] This embodiment improves the accuracy of structural feature extraction by achieving geometric consistency and brightness uniformity in visual data through image distortion correction and illumination compensation. Audio bandpass filtering effectively removes background noise and device harmonic interference, providing a clear foundation for subsequent spectral analysis. Text format standardization transforms complex unstructured input into parsable information, enhancing the reliability of semantic modeling. By synchronizing timestamps and slicing windows for the three types of data, modal alignment with millisecond-level precision is achieved, building a stable data foundation for cross-modal analysis and effectively enhancing the system's overall fault state perception and spatiotemporal consistency analysis capabilities.

[0070] In one embodiment, step S30 above includes: S301, for the illumination-compensated visual data in the multimodal dataset, identify the welding area and component outline area in the image, and define each identified area as a graph node; S302, Based on the spatial coordinates of the graph nodes, construct an undirected graph structure representing the connection relationship between adjacent nodes; S303, Message passing and feature updating are performed on the graph nodes in the undirected graph structure through a multi-layer graph neural network to generate a graph structure feature vector containing the association information between nodes; S304, For the filtered audio data in the multimodal dataset, calculate the short-time Fourier transform to generate a spectrum; S305, Extract the Mel frequency cepstral coefficients and their first-order difference features from the spectrum to generate a time-frequency dynamic feature vector; S306, Perform word segmentation and named entity recognition processing on the standardized text data in the multimodal dataset to generate structured text fragments; S307, The structured text fragment is encoded into semantic vector features using a pre-trained language model.

[0071] In this embodiment, the illumination-compensated visual data contains a large amount of spatial information related to the process. Image processing algorithms can identify regions with geometric boundary features or texture differences. These regions typically correspond to key structures such as welds, weld points, bolts, flange faces, and sealing lines. Based on contour extraction, edge detection, and semantic segmentation models, the welding areas and component contour areas in the image are identified and marked as graph nodes. The definition of a graph node includes not only its geometric center position in the image coordinate system but also extended attributes such as the shape, area, and boundary descriptors of the region, serving as the initial feature representation of the node.

[0072] After recognition, an undirected graph structure is constructed based on the spatial coordinates of the graph nodes. Connection rules can be set according to Euclidean distance, topological contact relationships, or functional logical adjacency. Connection relationships are used to define edge sets, so that the connection between nodes is not only reflected in pixel similarity, but also in process connectivity. The undirected graph structure is organized in the form of an adjacency matrix or edge list, and serves as the data structure basis for the subsequent graph neural network input.

[0073] Undirected graph structures are input into multi-layer graph neural networks for feature learning, with typical structures such as GCN, GAT, or GraphSAGE. Each layer of the graph neural network performs message passing and feature updates based on the feature information of its neighboring nodes through a weighted aggregation mechanism. In this process, each graph node receives information from its first-order or multi-order neighboring nodes and transforms it in conjunction with its own initial features, thereby generating a graph structural feature vector that reflects the correlation and spatial arrangement patterns of the welded structure. This feature vector retains both local information and encodes global dependencies, which is beneficial for subsequent fusion analysis.

[0074] The filtered audio data is mapped to the time-frequency domain via a short-time Fourier transform, generating a spectrogram that reveals the joint distribution structure of the audio signal in the time and frequency dimensions. To capture spectral features more sensitive to fault characterization, the Mel-frequency cepstral coefficients and their first-order difference features are further extracted from the spectrogram. The Mel-frequency cepstral coefficients, by mimicking the human ear's frequency perception mechanism, construct a filter bank and nonlinearly allocate frequency band weights, exhibiting stronger frequency discrimination capabilities when dealing with mechanical vibrations and abnormal noises. The first-order difference features are used to describe the dynamic trend of feature changes over time, improving the response capability to signal abrupt changes or periodic anomalies. Finally, the set of time-frequency dynamic feature vectors characterizes the vibration modes and acoustic energy distribution of the entire audio segment.

[0075] Standardized text data is first segmented using a domain-specific vocabulary. The segmentation strategy supports vocabulary expansion, compound word recognition, and context-sensitive rules. Subsequently, named entity recognition is performed to identify device models, anomaly categories, parameter names, and units within the text fragments as structured tags, facilitating subsequent semantic modeling. This process typically utilizes structures such as CRF, BiLSTM-CRF, or Transformer-CRF to improve recognition accuracy and robustness.

[0076] Structured text fragments are further input into pre-trained language models, such as BERT, RoBERTa, or Erlangshen, to obtain deep semantic representations of each text. The pre-trained model models the dependencies between contexts through its encoder structure and generates fixed-dimensional vector representations. These semantic vector features, as embeddings of text modalities, can be used to describe the potential semantic associations between fault phenomena and process conditions in maintenance records, and play a semantic guiding role in the multimodal fusion stage.

[0077] In practical implementation, the welding area and component outline area can be segmented using YOLOv8 or SegFormer models. After outputting the mask area, OpenCV is used to calculate its minimum bounding rectangle and extract the coordinates of the region center as the graph node position. The graph composition strategy can set the edge connection between regions within 10 pixels or with an actual distance of less than 3cm. The adjacency matrix is ​​implemented using SciPy sparse matrices to adapt to large-scale graphs.

[0078] The graph neural network employs a two-layer GCN, with ReLU activation and residual connections added after each layer. During training, the loss function is jointly optimized using contrastive loss and label-supervised processing. Audio data undergoes short-time Fourier transform using a Hanning window with a window length of 2048 points, a frame shift of 512 points, and a sampling rate of 44100Hz. MFCC extraction uses a 20-dimensional Metric filter bank implemented via Librosa, with a difference calculation window size of 2 frames.

[0079] Text segmentation uses a sub-word segmenter built on the BERT vocabulary. The NER model is fine-tuned and trained based on the BERT+CRF structure, supporting the recognition of 8 entity types. Semantic vectors are output by the BERT-Base model as [CLS] vectors with a dimension of 768. Feature vector outputs from all modalities are cached in batches and dimensionally aligned to ensure they can be directly input into the fusion network later.

[0080] This embodiment constructs graph nodes and models connectivity relationships through image structural regions, which not only enhances the ability to express structural defects but also improves the model's ability to model spatial dependencies. The message passing mechanism of the graph neural network effectively learns the functional and structural associations between regions, providing a foundation for subsequent judgments on weld continuity, component misalignment, and other issues. The introduction of Mel frequency and first-order difference in the audio feature extraction process enhances the response to high-frequency sharp sounds, improving the ability to identify knocking, friction, and howling faults. The text semantic encoding process, combining named entities and a pre-trained model, effectively preserves domain knowledge and linguistic patterns from the original maintenance records, demonstrating significant effectiveness in semantically guided diagnostic reasoning. Finally, the multimodal features possess good complementarity and discriminative power, contributing to improved comprehensiveness and accuracy of the diagnostic system.

[0081] In one embodiment, step S40 above includes: S401, linearly transforms the graph structure feature vector into the first modality query vector; S402 linearly transforms the time-frequency dynamic feature vector into the second mode bond vector and the second mode value vector; S403, Calculate the dot product attention score matrix of the first modality query vector and the second modality key vector; S404, Normalize the attention score matrix to generate visual and audio cross-modal attention weights; S405, the second modality value vector is weighted and summed according to the visual and audio cross-modal attention weights to generate a visual and audio related feature vector; S406, The semantic vector features and the visual and audio related feature vectors are concatenated to generate a multimodal joint feature vector; S407, Perform a multilayer perceptron transformation on the multimodal joint feature vector to generate a fused feature vector.

[0082] In this embodiment, the graph structure feature vector, as a high-level representation of the visual modality, contains the spatial dependencies between the welding area and the component structure. To enable this feature to function as a query in cross-modal attention computation, it needs to be linearly transformed and mapped to a unified attention space. This process is typically implemented through a fully connected layer, with parameter dimensions adjusted according to the target dimension requirements. The output vector is defined as the first modality query vector, whose structure possesses sufficient expressive power to capture the response patterns of the graph structure across different modalities.

[0083] The time-frequency dynamic feature vector originates from the spectral analysis of the audio signal, reflecting the temporal evolution and frequency distribution characteristics of acoustic behavior. To support the construction of key-value pairs in the attention mechanism, these vectors need to be linearly mapped into key vectors and value vectors, respectively. The key vector measures the correlation between the query and the current modality, while the value vector participates in feature fusion during the weighted summation stage. The dimensions of these two sets of vectors must be aligned with the query vector to ensure the mathematical validity of the dot product operation.

[0084] A matrix-dot product operation is performed between the first modality query vector and the second modality key vector to obtain an attention score matrix. This matrix represents the degree of matching between each visual structural feature unit and the audio time-frequency feature unit; a higher value indicates a stronger correlation. Since the numerical range in the score matrix may be large, it needs to be normalized. Typically, the softmax function is used to calculate along the key vector dimension, ensuring that the sum of the weights in each row is 1, thereby ensuring the stability and interpretability of subsequent weighted calculations. The normalized result is the cross-modal attention weight for vision and audio.

[0085] The attention weights are used as weighting coefficients to perform a weighted summation on the value vector, forming a visual-audio related feature vector. This vector not only retains the numerical representation of the original audio modality information, but also incorporates information related to visual structure through the attention mechanism, thus possessing cross-modal semantic alignment capabilities.

[0086] Building upon the fusion of visual and audio data, semantic vector features from the text modality are introduced, typically derived from the embedding results of standardized process text in a pre-trained language model. These semantic vector features are then concatenated with the visual and audio-related feature vectors to form a unified multimodal joint feature vector. The concatenation operation is usually dimensional, and normalization or residual connections can be used after concatenation to mitigate differences in modal distributions.

[0087] Finally, the joint feature vector of the multimodal modes is input into a multilayer perceptron structure for nonlinear transformation to generate a fused feature vector. The perceptron typically consists of multiple fully connected networks, activation functions (such as ReLU and GELU), and normalization layers to enhance nonlinear expressive power, enabling effective modeling of the relationships between different modalities in a high-dimensional space. This fused feature vector serves as the input to the subsequent diagnostic model, representing defect signals while improving the discriminative power of the diagnostic task.

[0088] The linear transformation stage can employ two sets of parameter-independent fully connected layers to transform the graph structure feature vector from the input dimension (e.g., 128-dimensional) to the query vector (e.g., 64-dimensional); similarly, the time-frequency dynamic feature vector (e.g., 104-dimensional) is transformed into key vectors and value vectors (each 64-dimensional). The dot product attention score matrix is ​​implemented through matrix multiplication, with the result dimension being the number of queries × the number of keys; softmax is used for normalization along the key dimension. The attention weighting process can be completed by multiplying the matrix by the value vector matrix, resulting in an associated feature vector (e.g., 64-dimensional). After concatenating the semantic vector features (e.g., 768-dimensional), an 832-dimensional multimodal joint feature is obtained. The multilayer perceptron structure uses a two-layer MLP, each layer containing linear mapping, ReLU activation, and Dropout regularization, with the output being a fused feature vector (e.g., 256-dimensional). This implementation framework can be built on TensorFlow or PyTorch, and the training phase combines cross-entropy loss and multimodal consistency loss to improve discriminative ability and fusion quality.

[0089] This embodiment utilizes a cross-modal attention mechanism to guide the focus of visual structural information on audio signals, enabling the system to capture dynamic changes highly correlated with specific structural regions in acoustic space, effectively enhancing the alignment between structural faults and acoustic anomalies. By introducing semantic vectors and concatenating audio-visual joint features, the system incorporates causal information and rule knowledge from linguistic expression while maintaining spatial and temporal consistency, improving the interpretability and accuracy of anomaly diagnosis. The multilayer perceptron further integrates complementary features between modalities, achieving high-order semantic-level fusion representation, providing more expressive and structurally compact input representations for subsequent classification, matching, and reasoning tasks.

[0090] In one embodiment, step S50 above includes: S501, Input the fused feature vector into the pre-trained classification model and output the defect type label and confidence score; S502, Match the similarity between the fused feature vector and the feature vector in the historical fault case library, and filter candidate fault cases whose similarity exceeds a preset similarity threshold; S503, based on the defect location annotation information of the candidate fault cases, annotate the minimum bounding rectangle coordinates of the defect area in the visual data; S504, extracts the start and end points of the audio signal time corresponding to the abnormal frequency band from the time-frequency dynamic feature vector; S505, Associate the process parameter deviation values ​​in the process text data with the cause descriptions of the candidate fault cases to generate a defect cause analysis statement; S506, integrate the defect type label, defect area coordinates, abnormal frequency band time start and end points, and defect cause analysis statements to generate a diagnostic report.

[0091] In this embodiment, the fused feature vector represents a deep fusion of graph structure features, time-frequency dynamic features, and semantic vector features. As input for anomaly analysis, it incorporates the contextual relevance and structural consistency of multimodal data. Inputting this fused feature vector into the classification model allows for rapid discriminative judgment of the current detected object's state. The classification model is typically a pre-trained and optimized multi-layer neural network, and its output includes specific defect type labels and corresponding confidence scores. Labels identify the determined anomaly category, and the confidence score reflects the model's output credibility, serving as one of the quantitative indicators of report reliability in subsequent decision chains.

[0092] The fused feature vectors also need to be matched with high-dimensional feature vectors in the historical fault case database to verify the consistency of the classification model's judgment and introduce prior knowledge. The matching process is implemented using mechanisms such as cosine similarity, Euclidean distance, or clustering retrieval based on embedding space. Only cases with similarity exceeding a preset threshold are considered as candidates to enhance the interpretability and attribution ability of the discrimination results.

[0093] Candidate fault cases typically record the spatial location and region information of typical defects, which is saved in the form of image coordinates or vector annotations. The system locates the defects on the raw visual data based on the defect location annotation information in the matching cases. The localization method generally involves drawing a minimum bounding rectangle to define and highlight the abnormal area, making the visual representation more readable and providing visual basis for subsequent human-computer interaction and analytical decision-making.

[0094] The audio analysis section extracts frequency bands and time ranges strongly correlated with candidate faults based on the abnormal signal behavior revealed by the time-frequency dynamic feature vector. This process combines the spectrogram morphology and energy change trends to locate the start and end times of the abnormal frequency bands on the time axis, which is used to establish a temporal mapping relationship between sound and defective behavior.

[0095] Regarding text data, the system parses deviations in process parameters, such as temperature, voltage, and pressure, from standard values ​​from standardized text. These deviations are then semantically correlated with descriptions of the causes of candidate failure cases, using methods such as keyword matching, semantic embedding similarity, or rule mapping, to output explanatory statements and form structured defect cause analysis statements.

[0096] Finally, by integrating the above four types of information—including defect type labels, visual location coordinates, audio anomaly time points, and text-based cause analysis statements—a diagnostic report is constructed. The diagnostic report has a structured format, can be automatically parsed and archived by the system, and can also be directly used by operations and maintenance personnel for fault location, cause tracing, and handling decisions.

[0097] The classification model can employ a pre-trained multilayer perceptron or a lightweight convolutional architecture, with the final layer outputting a softmax activation vector to determine the defect label category and confidence distribution. During training, the model uses a labeled fused feature vector sample set for supervised learning to ensure strong discriminative ability against multiple anomalies. Similarity matching establishes an embedding space index for the fused feature vectors, such as using the FAISS library or high-dimensional vector retrieval tools like LSH, to perform a K-nearest neighbor query on the current input feature vector, returning the N cases with the highest similarity and filtering out cases below a threshold (e.g., 0.85). Rectangular labeling of defect regions uses the image coordinate system as a reference, extracting the four vertices of the bounding boxes in candidate cases, drawing rectangles on the image rendering layer, and setting highlight border colors to distinguish normal areas. Time-frequency anomaly extraction combines MFCC features with spectral energy analysis, establishing energy thresholds at points of drastic MFCC dynamic changes, and extracting start and end time points, which can be achieved using sliding window matching or frequency integration. The text-based causal analysis module uses a pre-trained BERT model to encode the causal description and the current text. It calculates the similarity between the process parameter deviation values ​​and the vectors of the causal sentences in the case study, extracts the most matching sentence template, and populates the current parameters to generate causal sentences. The diagnostic report is saved in JSON or XML format, with fields including: defect label, confidence level, image coordinates, spectral time period, list of process parameters, and causal sentence. The report supports system display, document export, and downstream system calls.

[0098] This embodiment improves the accuracy and robustness of diagnostic output through a dual approach of classification model and historical case matching. The diagnostic report generated after integrating visual, audio, and textual information not only has logical integrity supported by data but also enables the ability to trace the causes of abnormal events. The system's judgment results can be specified down to image location, frequency time interval, and text interpretation level, providing a clear path for actual maintenance and problem reproduction, significantly reducing fault handling time and improving processing efficiency.

[0099] In one embodiment, after step S50 above, the method further includes: S601 records the defect diagnosis results and actual cause data confirmed by manual review, and generates labeled training samples; S602, add the labeled training samples to the model training dataset; S603, Based on the training dataset of the model, update the parameters of the classification model using the mini-batch gradient descent algorithm; S604, Add the fault entity nodes and related edges corresponding to the candidate fault cases to the domain knowledge graph; S605, when the false alarm rate of the classification model exceeds the preset false alarm threshold, the full parameter retraining process of the classification model is triggered based on the model training dataset.

[0100] In this embodiment, the defect diagnosis results and actual cause data generated during the manual review process constitute the original information source for the system's closed-loop feedback. This information typically includes the defect type confirmed by operators with professional knowledge, the actual failure mechanism, and supplementary descriptions of on-site environmental conditions or interference sources. By structurally recording these confirmation results and combining them with corresponding fused feature vectors, labeled training samples are formed. These training samples not only contain multimodal feature inputs after deep fusion but also come with accurate and authoritative classification labels, constituting high-quality supervised training data.

[0101] Add the generated labeled training samples to the existing model training dataset, ensuring consistency in data structure and label standards. New samples can be directly merged with historical training data to form a dataset that supports incremental optimization, used for subsequent online model learning or periodic updates. The resulting training dataset is stored in key-value pairs of "fused feature vector - defect label" to ensure a clear model learning task.

[0102] Mini-batch gradient descent is used to achieve rapid parameter updates without compromising the stability of the original model. This algorithm uses only a subset of samples in each training iteration, offering advantages such as fast convergence and low memory consumption. This optimization method fine-tunes the parameters of the current classification model, allowing it to adapt to new defect feature distributions more quickly while maintaining its generalization ability.

[0103] The domain knowledge graph maintains entities such as equipment, parts, defect types, process parameters, and abnormal behaviors, as well as the causal, association, and co-occurrence relationships between them. Whenever a new candidate fault case is confirmed through manual review, an entity node representing that fault needs to be added to the graph, and edges connecting it to existing equipment models, process deviations, or historical cases need to be established. This operation not only increases the entity coverage of the graph but also provides data support for subsequent question answering, causal reasoning, and rule extraction.

[0104] To ensure the stability of the model during long-term operation, its inference performance also needs to be monitored. The system continuously tracks the deviation rate between the prediction results of the classification model in practical applications and those manually reviewed. If the false positive rate exceeds a preset safety threshold, a full-parameter retraining process is automatically triggered. This process uses the updated complete training dataset (including the original data and new samples) for full training, thereby reconstructing the model's representation capabilities and correcting any accumulated error biases.

[0105] The generation of labeled training samples is based on the record management module. This module accepts the judgment results input by the auditors and binds them with the fused feature vectors output in the analysis phase, automatically generating supervised sample items and storing them in the training dataset. The training dataset is constructed with a distributed storage structure, supporting dynamic addition of samples and label correction. It also maintains auxiliary information such as sample metadata, collection time, auditor ID, and sample confidence level for easy retrieval and auditing later. The classification model's parameters are updated using the Adam optimizer to perform mini-batch training tasks. A certain proportion of new samples are extracted for training in each training cycle, and the prediction error is calculated using the cross-entropy loss function and backpropagated. During training, some parameters are fixed, and only the weights of the middle and later layers are fine-tuned to avoid overfitting. The knowledge graph update module supports maintenance in Neo4j or graph embedding libraries. New nodes are generated using standard naming rules, and edge relationships are automatically matched with entity labels based on event information in the audit records. If multiple matching edges exist, a confidence evaluation model is used to select the optimal connection path. The false positive rate monitoring module continuously tracks the deviation between the model output and the verified results. It calculates the real-time false positive rate using a sliding window approach and sets up a dynamically adjusted threshold judgment mechanism to avoid erroneously triggering the retraining task due to short-term fluctuations. Once the trigger condition is met, the current model state is cleared and the full training process is restarted, using the complete training dataset to rebuild the classification model structure and weight parameters.

[0106] Example Description: At a critical welding station on the power battery packaging line for new energy vehicles, a data acquisition unit consisting of a multispectral industrial camera and a directional microphone is deployed. Simultaneously, process text data, including process parameters for the current welding task and historical maintenance logs, is acquired in real-time from the production execution system database. When the battery module enters the welding area, the camera begins acquiring high-resolution images of the weld formation process, the microphone records the sound signal generated by the welding arc or laser, and the control system synchronously sends out timestamp trigger signals. Image frames, audio clips, and process text are bound to a unified timeline, and all raw data is appended with millisecond-precision timestamps, forming a timestamped raw data stream.

[0107] After data acquisition, geometric distortion correction and histogram equalization are performed on the visual images to improve edge sharpness and contrast in the weld area. The audio signal is processed by a bandpass filter to remove low-frequency power interference and high-frequency noise while preserving the welding frequency band energy. The process text is encoded, converted, and cleaned using a format standardization module, retaining structured parameter fields and extracted maintenance description information. The three types of data are sliced ​​and aligned according to a unified timestamp window to ensure strict synchronization of image frames, audio segments, and text parameters within each time slice, generating a timestamp-aligned multimodal dataset.

[0108] Subsequently, the system extracts depth features for three modalities. For visual images, the system first identifies welding areas and component outlines in the illumination-compensated image, constructing a graph structure where nodes represent weld points or feature regions, and edges represent adjacency relationships or texture similarity. The graph structure is input into a graph neural network for multi-layer aggregation, outputting visual graph structure features. For audio segments, a short-time Fourier transform is performed to extract Mel-frequency cepstral coefficients and their first-order difference features, forming time-frequency dynamic features. The process text undergoes word segmentation and entity recognition processing to extract key equipment, parameters, and event descriptions, which are then encoded into high-dimensional semantic vectors using a pre-trained language model.

[0109] The extracted spectral structure features, time-frequency dynamic features, and semantic vector features are fed into the cross-modal attention mechanism module. Visual feature vectors are transformed linearly to generate query vectors, and audio features are used to generate key-value vectors. The system calculates and normalizes the dot product attention score to generate visual-audio attention weights, which are used to calculate audio interaction features driven by vision. Simultaneously, the semantic vectors are concatenated with the visual-audio fusion result to form a multimodal joint feature vector, which is then input into a multilayer perceptron for feature compression and nonlinear mapping, ultimately outputting a fused feature vector.

[0110] The fused feature vector is input into a pre-trained classification model, which outputs a defect type label and a confidence score. Simultaneously, it performs similarity matching with the fused features in a historical fault case database, filtering out candidate fault cases with similarity exceeding a set threshold. Based on the defect location annotation information stored in the candidate cases, the system marks the defect region boundaries in the visual image; it extracts the time intervals corresponding to abnormal frequency bands from audio features for highlighting; and it analyzes the semantic relationship between parameter deviation values ​​in the text and the cause descriptions of the candidate cases, automatically generating cause analysis statements. Finally, the system integrates classification labels, location coordinates, abnormal time periods, and cause descriptions, outputting a structured diagnostic report and pushing it to the quality management platform.

[0111] If the quality control engineer verifies the accuracy of the diagnosis, the system records the feedback and generates supervised training samples. The fused feature vectors are then bound to manually labeled data and appended to the model training dataset. The system uses a mini-batch gradient descent algorithm to fine-tune the parameters of the existing model, improving its adaptability to new samples. Simultaneously, entity nodes for the fault case are added to the knowledge graph, establishing associations with entities such as equipment models and process parameters. If the false alarm rate identified by the system continues to exceed a set threshold during subsequent detection processes, a full-parameter retraining process for the classification model will be automatically triggered to ensure the model maintains stable performance under dynamic operating conditions. This process achieves a fully closed-loop quality monitoring and fault diagnosis capability, from multimodal data fusion and feature extraction to diagnostic feedback optimization, adapting to the diverse defect types and complex production environments in new energy vehicle production.

[0112] This embodiment introduces a mechanism linking manual review and feedback with a knowledge graph, enabling the system to continuously self-correct and accumulate knowledge. The classification model can achieve rapid parameter tuning through small-batch training in the early stages of error accumulation, and automatically triggers full-parameter retraining when long-term performance degrades, effectively controlling the risk of model inaccuracy. Simultaneously, by expanding entity nodes and knowledge graph relationship edges, structured accumulation of case experience is achieved, enhancing the interpretability and evolutionary capabilities of the diagnostic system, allowing it to continuously adapt to new defect patterns and complex environmental changes during ongoing operation.

[0113] In one embodiment, a data analysis device based on multimodal data fusion is provided, which corresponds one-to-one with the data analysis method based on multimodal data fusion described in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the data analysis device based on multimodal data fusion of the present invention. The modules include a multimodal data acquisition module 10, a data preprocessing and alignment module 20, a feature extraction module 30, a feature fusion module 40, and an intelligent analysis module 50. Detailed descriptions of each functional module are as follows: The multimodal data acquisition module 10 is used to acquire visual data, audio data, and process text data; The data preprocessing and alignment module 20 is used to perform geometric correction and illumination compensation processing on the visual data, bandpass filtering processing on the audio data, format standardization processing on the process text data, and generate a timestamp-aligned multimodal dataset. Feature extraction module 30 is used to extract features from each type of data in the multimodal dataset to generate spectral structure features of visual data, time-frequency dynamic features of audio data, and semantic vector features of process text data. Feature fusion module 40 is used to fuse the graph structure features, time-frequency dynamic features and semantic vector features through a cross-modal attention mechanism to generate a fused feature vector; The intelligent analysis module 50 is used to perform analysis and processing based on the fused feature vector to generate analysis results.

[0114] In one embodiment, the multimodal data acquisition module 10 is specifically used for: Raw video stream data of the production process is acquired using a multispectral industrial camera array; Raw audio stream data of the operating location of key equipment is acquired using a microphone array; Retrieve raw text stream data containing structured process parameters and unstructured maintenance logs from the Production Execution System database; A hardware trigger signal is applied to the multispectral industrial camera array, microphone array, and production execution system database to synchronize the acquisition timing of the raw video stream data, raw audio stream data, and raw text stream data. Add millisecond-precision timestamps to the time-synchronized raw video stream data, raw audio stream data, and raw text stream data to generate time-stamped raw visual data, time-stamped raw audio data, and time-stamped raw process text data.

[0115] In one embodiment, the data preprocessing and alignment module 20 is specifically used for: Perform geometric distortion correction processing on the visual data to generate geometrically corrected visual data; Histogram equalization is performed on the geometrically corrected visual data to generate illumination-compensated visual data; The audio data is subjected to bandpass filtering to generate filtered audio data; The unstructured text in the process text data is converted into a preset encoding format to generate standardized text data; Based on the timestamps of the illumination-compensated visual data, filtered audio data, and standardized text data, the timestamp-aligned multimodal dataset containing the illumination-compensated visual data, filtered audio data, and standardized text data is generated by slicing and aligning the data according to a preset time window.

[0116] In one embodiment, the feature extraction module 30 is specifically used for: For the illumination-compensated visual data in the multimodal dataset, identify the welding area and component outline area in the image, and define each identified area as a graph node; Based on the spatial coordinates of the graph nodes, construct an undirected graph structure representing the connection relationship between adjacent nodes; By performing message passing and feature updating on the graph nodes in the undirected graph structure through a multi-layer graph neural network, a graph structure feature vector containing inter-node association information is generated. For the filtered audio data in the multimodal dataset, a short-time Fourier transform is calculated to generate a spectrogram; Extract the Mel frequency cepstral coefficients and their first-order difference features from the spectrum to generate a time-frequency dynamic feature vector; Perform word segmentation and named entity recognition processing on the standardized text data in the multimodal dataset to generate structured text fragments; The structured text fragments are encoded into semantic vector features using a pre-trained language model.

[0117] In one embodiment, the feature fusion module 40 is specifically used for: The graph structure feature vector is linearly transformed into the first modality query vector; The time-frequency dynamic feature vector is linearly transformed into the second mode key vector and the second mode value vector; Calculate the dot product attention score matrix of the first modality query vector and the second modality key vector; The attention score matrix is ​​normalized to generate visual and audio cross-modal attention weights; The second modality value vector is weighted and summed according to the visual and audio cross-modal attention weights to generate a visual and audio related feature vector; The semantic vector features and the visual-audio association feature vector are concatenated to generate a multimodal joint feature vector; A multilayer perceptron transformation is performed on the multimodal joint feature vector to generate a fused feature vector.

[0118] In one embodiment, the intelligent analysis module 50 is specifically used for: The fused feature vector is input into a pre-trained classification model, which outputs a defect type label and a confidence score. Match the similarity between the fused feature vector and the feature vector in the historical fault case library, and filter candidate fault cases whose similarity exceeds a preset similarity threshold; Based on the defect location annotation information of the candidate fault cases, the coordinates of the minimum bounding rectangle of the defect area are annotated in the visual data; Extract the start and end points of the audio signal corresponding to the abnormal frequency band from the time-frequency dynamic feature vector; By associating the process parameter deviation values ​​in the process text data with the cause descriptions of the candidate failure cases, a defect cause analysis statement is generated; The diagnostic report is generated by integrating the defect type labels, defect area coordinates, abnormal frequency band start and end points, and defect cause analysis statements.

[0119] In one embodiment, the intelligent analysis module 50 is specifically used for: Record the defect diagnosis results and actual cause data that have been manually reviewed and confirmed, and generate labeled training samples; Add the labeled training samples to the model training dataset; Based on the training dataset of the model, the parameters of the classification model are updated using the mini-batch gradient descent algorithm; Add fault entity nodes and associated edges corresponding to candidate fault cases to the domain knowledge graph; When the false positive rate of the classification model exceeds a preset false positive threshold, the full parameter retraining process of the classification model is triggered based on the model training dataset.

[0120] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used for communication with external user terminals via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a server-side data analysis method based on multimodal data fusion.

[0121] In one embodiment, a computer device is provided, which may be a user terminal, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the user-side functions or steps of a data analysis method based on multimodal data fusion.

[0122] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Collect visual data, audio data, and process text data; The visual data is subjected to geometric correction and illumination compensation processing, the audio data is subjected to bandpass filtering processing, and the process text data is subjected to format standardization processing to generate a timestamp-aligned multimodal dataset. Feature extraction is performed on each type of data in the multimodal dataset to generate spectral structure features of visual data, time-frequency dynamic features of audio data, and semantic vector features of process text data. A fused feature vector is generated by fusing the graph structure features, time-frequency dynamic features, and semantic vector features through a cross-modal attention mechanism. The analysis results are generated based on the fused feature vectors.

[0123] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Collect visual data, audio data, and process text data; The visual data is subjected to geometric correction and illumination compensation processing, the audio data is subjected to bandpass filtering processing, and the process text data is subjected to format standardization processing to generate a timestamp-aligned multimodal dataset. Feature extraction is performed on each type of data in the multimodal dataset to generate spectral structure features of visual data, time-frequency dynamic features of audio data, and semantic vector features of process text data. A fused feature vector is generated by fusing the graph structure features, time-frequency dynamic features, and semantic vector features through a cross-modal attention mechanism. The analysis results are generated based on the fused feature vectors.

[0124] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and user side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0125] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0126] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0127] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A data analysis method based on multi-modal data fusion, characterized in that, The method comprises the following steps: Collecting visual data, audio data and process text data; Performing geometric correction and illumination compensation processing on the visual data, performing band-pass filtering processing on the audio data, and performing format standardization processing on the process text data to generate a timestamp-aligned multi-modal data set; Respectively extracting features from each type of data in the multi-modal data set to generate graph structure features of visual data, time-frequency dynamic features of audio data, and semantic vector features of process text data; Fusing the graph structure features, time-frequency dynamic features and semantic vector features through a cross-modal attention mechanism to generate a fusion feature vector; Based on the fusion feature vector, an analysis result is generated.

2. The data analysis method based on multi-modal data fusion as claimed in claim 1, wherein, Collecting visual data, audio data and process text data, comprising: Collecting original video stream data of production processes through a multi-spectral industrial camera array; Collecting original audio stream data of key equipment operating positions through a microphone array; Obtaining original text stream data containing structured process parameters and unstructured maintenance logs from a production execution system database; Applying a hardware trigger signal to the multi-spectral industrial camera array, the microphone array and the production execution system database to synchronize the collection timing of the original video stream data, the original audio stream data and the original text stream data; Attaching a millisecond-precision timestamp to the time-synchronized original video stream data, the original audio stream data and the original text stream data to generate timestamped original visual data, timestamped original audio data and timestamped original process text data.

3. The data analysis method based on multi-modal data fusion as claimed in claim 1, wherein, Performing geometric correction and illumination compensation processing on the visual data, performing band-pass filtering processing on the audio data, and performing format standardization processing on the process text data to generate a timestamp-aligned multi-modal data set, comprising: Performing geometric distortion correction processing on the visual data to generate geometrically corrected visual data; Performing histogram equalization processing on the geometrically corrected visual data to generate illumination compensated visual data; Performing band-pass filtering processing on the audio data to generate filtered audio data; Converting unstructured text in the process text data into a preset encoding format to generate standardized text data; According to the timestamps of the illumination compensated visual data, the filtered audio data and the standardized text data, aligning the time window slices according to the preset time window to generate a timestamp-aligned multi-modal data set containing illumination compensated visual data, filtered audio data and standardized text data.

4. The data analysis method based on multi-modal data fusion as claimed in claim 1, wherein, Respectively extracting features from each type of data in the multi-modal data set to generate graph structure features of visual data, time-frequency dynamic features of audio data, and semantic vector features of process text data, comprising: For the illumination compensated visual data in the multi-modal data set, identifying the welding area and the part contour area in the image, and defining each identified area as a graph node; According to the spatial position coordinates of the graph nodes, constructing an undirected graph structure representing the connection relationship between adjacent nodes; Performing message passing and feature updating on the graph nodes in the undirected graph structure through a multi-layer graph neural network to generate a graph structure feature vector containing inter-node association information; For the filtered audio data in the multi-modal data set, a short-time Fourier transform is calculated to generate a spectrogram; Mel-frequency cepstral coefficients of the spectrogram and their first-order differential features are extracted to generate a time-frequency dynamic feature vector; For the normalized text data in the multi-modal data set, a word segmentation and named entity recognition process is performed to generate structured text segments; The structured text segments are encoded into semantic vector features through a pre-trained language model.

5. The data analysis method based on multi-modal data fusion as claimed in claim 1, wherein, The graph structure features, time-frequency dynamic features, and semantic vector features are fused through a cross-modal attention mechanism to generate a fusion feature vector, including: Linearly transforming the graph structure feature vector into a first modal query vector; Linearly transforming the time-frequency dynamic feature vector into a second modal key vector and a second modal value vector; Calculating the dot product attention score matrix of the first modal query vector and the second modal key vector; Normalizing the attention score matrix to generate visual and audio cross-modal attention weights; Weighted summing the second modal value vector according to the visual and audio cross-modal attention weights to generate a visual and audio associated feature vector; Concatenating the semantic vector features and the visual and audio associated feature vector to generate a multi-modal joint feature vector; Performing a multi-layer perceptron transformation on the multi-modal joint feature vector to generate a fusion feature vector.

6. The data analysis method based on multi-modal data fusion as claimed in claim 1, wherein, Based on the fusion feature vector, an analysis result is generated, including: Inputting the fusion feature vector into a pre-trained classification model to output a defect type label and a confidence score; Matching the fusion feature vector with the feature vectors in the historical fault case library to filter candidate fault cases with a similarity exceeding a preset similarity threshold; According to the defect location annotation information of the candidate fault case, annotating the minimum bounding rectangle coordinates of the defect area in the visual data; Extracting the audio signal time start and end points corresponding to the abnormal frequency band from the time-frequency dynamic feature vector; Associating the process parameter deviation values in the process text data with the cause description of the candidate fault case to generate a defect cause analysis sentence; Integrating the defect type label, defect area coordinates, abnormal frequency band time start and end points, and defect cause analysis sentence to generate a diagnosis report.

7. The data analysis method based on multi-modal data fusion as claimed in claim 1, wherein, After generating the analysis result based on the fusion feature vector, it also includes: Recording the defect diagnosis results confirmed by manual review and the true cause data to generate labeled training samples; Adding the labeled training samples to the model training data set; Updating the parameters of the classification model based on the model training data set through a small batch gradient descent algorithm; Adding the fault entity nodes and associated relationship edges corresponding to the candidate fault cases in the domain knowledge graph; When the false positive rate of the classification model exceeds a preset false positive threshold, triggering the full parameter retraining process of the classification model based on the model training data set.

8. A data analysis apparatus based on multi-modal data fusion, characterized by, The multi-modal data fusion-based data analysis apparatus includes: A multi-modal data acquisition module for acquiring visual data, audio data, and process text data; The data preprocessing and alignment module is configured to perform geometric correction and illumination compensation on the visual data, perform band-pass filtering on the audio data, and perform format standardization on the process text data, and generate a timestamp-aligned multi-modal data set. The feature extraction module is configured to extract features from each type of data in the multi-modal data set, and generate graph structure features of the visual data, time-frequency dynamic features of the audio data, and semantic vector features of the process text data. The feature fusion module is configured to fuse the graph structure features, the time-frequency dynamic features, and the semantic vector features through a cross-modal attention mechanism, and generate a fusion feature vector. The intelligent analysis module is configured to analyze the fusion feature vector and generate an analysis result.

9. A computer device, comprising: The computer device includes a memory, a processor, and a multi-modal data fusion-based data analysis program stored in the memory and executable on the processor. When the multi-modal data fusion-based data analysis program is executed by the processor, the steps of the multi-modal data fusion-based data analysis method according to any one of claims 1-7 are implemented.

10. A computer-readable storage medium, characterized in that, The storage medium stores a multi-modal data fusion-based data analysis program. When the multi-modal data fusion-based data analysis program is executed by the processor, the steps of the multi-modal data fusion-based data analysis method according to any one of claims 1-7 are implemented.

Images