An artificial intelligence-based data processing and analysis method

Abstract

The application relates to an artificial intelligence-based data processing and analysis method. The method comprises the following steps: performing low-illumination enhancement on a railway turnout on-site image to obtain enhanced image data, and converting a vibration signal into two-dimensional pseudo-color time-frequency spectrum data; extracting visual semantic features and vibration semantic features through two symmetrical structure Transform encoders; completing feature alignment and purification through an alignment network and a purification network; obtaining a deep fusion feature sequence based on a multi-head cross-attention mechanism; adjusting the feature dimension and splicing an instruction text embedding vector to obtain a combined prompt condition; inputting a generative large language model to generate a natural language diagnosis report containing fault information. The method can realize railway turnout multi-modal data fusion diagnosis, and improve fault recognition accuracy and report generation intelligence level.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence data processing and analysis, and in particular relates to a data processing and analysis method based on artificial intelligence. Background Technology

[0002] As a critical switching device in rail transit lines, the operational status of railway turnouts directly affects the safety and stability of train operations. Currently, the industry commonly employs a multi-source sensing data combined with artificial intelligence algorithms for analysis. By automating the processing of monitoring information such as images and vibrations collected on-site, abnormal turnout conditions can be identified and faults can be determined, generating analytical conclusions that can directly guide operation and maintenance work.

[0003] Existing turnout condition analysis methods mostly rely on single-mode monitoring data for judgment, making it difficult to effectively correlate and complement image information and vibration signals. This can easily lead to deviations in diagnostic results due to noise interference from single data sources. Traditional analysis models often use fixed feature processing modes, which cannot distinguish and purify effective information and interference noise from multi-source data, resulting in low feature utilization efficiency. Most existing solutions can only output simple fault judgment results and cannot automatically generate complete natural language reports containing fault locations, abnormal causes, and maintenance suggestions. On-site maintenance personnel still need to perform a lot of manual interpretation and processing work, making the diagnostic process cumbersome and impractical. Summary of the Invention

[0004] Therefore, it is necessary to provide an artificial intelligence-based data processing and analysis method to address the aforementioned technical problems.

[0005] Firstly, this application provides a data processing and analysis method based on artificial intelligence, including: S1. Perform low-light enhancement on the collected railway turnout field images based on CLAHE and Retinex algorithms to obtain enhanced image data. Perform continuous wavelet transform on the collected vibration signals to obtain continuous wavelet transform coefficients. Map the real part, imaginary part and amplitude square of the continuous wavelet transform coefficients to the RGB three channels respectively to generate two-dimensional pseudo-color time-frequency spectrum data. S2. The visual semantic features of the enhanced image data are extracted by the visual Transformer encoder. The two-dimensional pseudo-color time-frequency spectrum data are extracted by the spectral feature Transformer encoder, which is symmetrical to the structure of the visual Transformer encoder. The vibration semantic features of the two-dimensional pseudo-color time-frequency spectrum data are extracted. S3. Using an alignment network trained under supervision based on a symmetric contrast loss function, the visual semantic features and vibration semantic features are semantically aligned. Using a purification network trained under supervision based on a variational information bottleneck loss function, the aligned visual semantic features and aligned vibration semantic features are constrained and purified to obtain purified visual semantic features and purified vibration semantic features. S4. Based on the multi-head cross-attention mechanism, the purified vibration semantic features are used as the query matrix, and the purified visual semantic features are used as the key matrix and value matrix for calculation to obtain the deep fusion feature sequence. S5. Through a linear projection layer, adjust the vector dimension of the deep fusion feature sequence, and concatenate the adjusted deep fusion feature sequence and the instruction text embedding vector to obtain the combined prompt conditions; where the instruction text embedding vector is a sequence of word vectors mapped from the system instruction text, and the system instruction text is used to define the roles and analysis tasks of the generative large language model; S6. Input the combined prompt conditions into a generative large language model based on a Transformer decoder to generate a natural language diagnostic report in an autoregressive manner. The natural language diagnostic report includes the specific fault location, physical anomaly manifestations, possible causes, and maintenance suggestions.

[0006] Secondly, this application also provides an artificial intelligence-based data processing and analysis system for implementing the method described in the first aspect, comprising: The multimodal data preprocessing module is used to perform low-light enhancement on the acquired railway turnout field images based on the CLAHE and Retinex algorithms to obtain enhanced image data. It also performs continuous wavelet transform on the acquired vibration signals to obtain continuous wavelet transform coefficients. The real part, imaginary part and amplitude square of the continuous wavelet transform coefficients are mapped to the RGB three channels respectively to generate two-dimensional pseudo-color time-frequency spectrum data. The dual-modal feature extraction module is used to extract image features from the enhanced image data through a visual Transformer encoder to obtain the visual semantic features of the enhanced image data, and to extract semantic features from the two-dimensional pseudo-color time-frequency spectrum data through a spectral feature Transformer encoder that is structurally symmetrical with the visual Transformer encoder to obtain the vibration semantic features of the two-dimensional pseudo-color time-frequency spectrum data. The feature alignment and purification module is used to perform semantic alignment of visual semantic features and vibration semantic features by an alignment network trained under supervision based on the symmetric contrast loss function, and to perform constrained purification of the aligned visual semantic features and the aligned vibration semantic features by a purification network trained under supervision based on the variational information bottleneck loss function, so as to obtain purified visual semantic features and purified vibration semantic features. The cross-modal deep fusion module is used to calculate the deep fusion feature sequence by using the purified vibration semantic features as the query matrix and the purified visual semantic features as the key matrix and value matrix based on the multi-head cross-attention mechanism. The prompt condition construction module is used to adjust the vector dimension of the deep fusion feature sequence through a linear projection layer, and then concatenate the adjusted deep fusion feature sequence with the instruction text embedding vector to obtain the combined prompt condition; wherein, the instruction text embedding vector is a sequence of word vectors mapped from the system instruction text, and the system instruction text is used to define the role and analysis task of the generative large language model; The diagnostic report generation module is used to input combined prompts into a generative large language model based on a Transformer decoder to generate a natural language diagnostic report in an autoregressive manner.

[0007] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement an artificial intelligence-based data processing and analysis method as described in the first aspect.

[0008] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements an artificial intelligence-based data processing and analysis method as described in the first aspect.

[0009] The aforementioned AI-based data processing and analysis method enhances low-light performance of railway turnout images using the CLAHE and Retinex algorithms, and generates two-dimensional pseudo-color time-frequency spectrum data from collected vibration signals through continuous wavelet transform, achieving a unified representation of different physical modal data in terms of dimension and topology. By extracting visual and vibration semantic features using a visual Transformer encoder and a spectral feature Transformer encoder respectively, the ability to capture global context and long-range dependencies can be improved. Through an alignment network trained under supervision based on a symmetric contrastive loss function and a purification network trained under supervision based on a variational information bottleneck loss function, cross-modal semantic alignment and redundant noise filtering of visual and vibration semantic features can be achieved. By using a multi-head cross-attention mechanism to dynamically retrieve purified visual semantic features as a query matrix for purified vibration semantic features, the spatial positioning accuracy of fault-related physical components can be improved. By combining deep fusion feature sequences and instruction text embedding vectors into a generative large language model based on a Transformer decoder and outputting a natural language diagnostic report in an autoregressive manner, the readability of diagnostic results and the interpretability of fault analysis can be improved. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart illustrating a data processing and analysis method based on artificial intelligence in one embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of an artificial intelligence-based data processing and analysis system according to an embodiment of the present invention. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0013] In one embodiment, such as Figure 1 As shown, an artificial intelligence-based data processing and analysis method is provided. This embodiment illustrates the application of this method to a processing device. It is understood that this method can also be applied to a server, and further to a system including both a processing device and a server, and is implemented through the interaction between the processing device and the server. In this embodiment, the method may include the following steps: S1. Perform low-light enhancement on the collected railway turnout field images based on CLAHE and Retinex algorithms to obtain enhanced image data. Perform continuous wavelet transform on the collected vibration signals to obtain continuous wavelet transform coefficients. Map the real part, imaginary part and amplitude square of the continuous wavelet transform coefficients to the RGB three channels respectively to generate two-dimensional pseudo-color time-frequency spectrum data.

[0014] Specifically, the on-site images of railway turnouts can be digital image data of the turnout area acquired in real time by high-frequency intelligent cameras beside the track during train passage. These images can include key mechanical components such as switch rails, slide plates, locking rods, and fasteners. Vibration signals can be time-series acceleration data recorded by high-frequency triaxial accelerometers deployed at key connection points of the turnout switch machine. The numerical value of the vibration signal can characterize the instantaneous vibration intensity of the railway turnout structure in three-dimensional space. CLAHE, or Limit Contrast Adaptive Histogram Equalization, is an image enhancement algorithm that limits the height of local histograms to suppress excessive noise amplification. The Retinex algorithm is an image enhancement method based on color constancy theory. It can brighten low-illuminance areas and remove fogging by separating the illuminance and reflectance components of the image. Continuous wavelet transform can be a time-frequency analysis method. By scaling and time-shifting the mother wavelet, continuous wavelet transform maps a one-dimensional time-series signal to a two-dimensional time-frequency plane, simultaneously providing localized features of the signal in both the time and frequency domains.

[0015] Optionally, the processing device can use an adaptive histogram equalization algorithm and a Retinex algorithm to calculate the local contrast of the railway turnout site image, separate the illumination component, and obtain enhanced image data. The processing device can use a complex Morlet wavelet as the mother wavelet function to perform a continuous wavelet transform on the acquired vibration signal, calculating the integral over multiple scale and translation factors to obtain the continuous wavelet transform coefficients. The processing device can extract the real part, imaginary part, and amplitude squared value of the continuous wavelet transform coefficients, and normalize these three sets of values ​​respectively. The processing device can map the normalized real part, normalized imaginary part, and normalized amplitude squared value to the RGB three channels respectively, and generate two-dimensional pseudo-color time-frequency spectrum data by combining the data from the RGB three channels.

[0016] S2. Image feature extraction is performed on the enhanced image data using a visual Transformer encoder to obtain the visual semantic features of the enhanced image data. Semantic features are then extracted from the two-dimensional pseudo-color time-frequency spectrum data using a spectral feature Transformer encoder that is structurally symmetrical with the visual Transformer encoder. Vibration semantic features of the two-dimensional pseudo-color time-frequency spectrum data are then extracted.

[0017] Specifically, the visual Transformer encoder can be a neural network structure based on a self-attention mechanism. The spectral feature Transformer encoder can be an isomorphic network structure with the same number of layers, the same embedding dimension, and the same number of attention heads as the visual Transformer encoder.

[0018] Optionally, the processing device can segment the enhanced image data into multiple fixed-size image blocks, perform linear projection mapping on each block, and attach classification labels and learnable positional codes to the sequence header to form an initial visual label sequence. The processing device can then perform layer-by-layer calculations on the initial visual label sequence using the multi-head self-attention layer and feedforward network layer in the visual Transformer encoder to extract spatial geometric and texture features from the enhanced image data and output visual semantic features. Alternatively, the processing device can use the same segmentation method as for the enhanced image data to segment the two-dimensional pseudo-color time-frequency spectral data into blocks, and after linear projection and positional coding, form an initial spectral label sequence. The processing device can then perform the same nonlinear transformation calculations on the initial spectral label sequence using a spectral feature Transformer encoder with a structure symmetrical to the visual Transformer encoder, capturing high-frequency anomalous impact features and low-frequency modulation harmonic features in the two-dimensional pseudo-color time-frequency spectral data and output vibration semantic features.

[0019] S3. Using an alignment network trained under supervision based on a symmetric contrast loss function, visual semantic features and vibration semantic features are semantically aligned. Then, using a purification network trained under supervision based on a variational information bottleneck loss function, the aligned visual semantic features and aligned vibration semantic features are constrained and purified to obtain purified visual semantic features and purified vibration semantic features.

[0020] Specifically, the symmetric contrastive loss function can be an optimization objective function for cross-modal representation learning. It drives different modal encoders to learn modality-independent semantically consistent representations by maximizing the cosine similarity of paired positive samples in the joint latent space and minimizing the similarity of batches of negative samples. The alignment network can be a lightweight network composed of trainable linear projection layers and L2 normalization operations. The variational information bottleneck loss function can be a regularized objective function based on mutual information theory and variational inference. It forces feature representations to discard task-irrelevant redundant noise information by constraining the KL divergence between the feature encoding distribution and the standard normal prior distribution. The refinement network can be a feature compression network built on a variational autoencoder structure.

[0021] Optionally, the processing device can input visual semantic features and vibration semantic features into an alignment network, map the visual semantic features and vibration semantic features to a joint semantic latent space of the same dimension using a linear projection matrix, and normalize the mapped visual semantic features and vibration semantic features using the L2 norm to obtain aligned visual semantic features and aligned vibration semantic features. The processing device can then concatenate the aligned visual semantic features and aligned vibration semantic features along the feature dimension and input them into a refinement network. The processing device can predict the mean and variance of the multivariate Gaussian distribution of the input features using the variational inference layer in the refinement network, and sample from this multivariate Gaussian distribution using a reparameterization technique to output refined visual semantic features and refined vibration semantic features.

[0022] For example, the processing device can calculate the symmetric contrast loss function value during the training phase of the alignment network and update the network parameters through the backpropagation algorithm to force the cosine similarity of the paired visual semantic features and vibration semantic features to be maximized, thereby achieving semantic alignment.

[0023] For example, the processing device can calculate the variational information bottleneck loss function value during the training phase of the purification network, filter out modal private noise in the aligned visual semantic features and aligned vibration semantic features by minimizing the KL divergence of the posterior distribution and the standard normal prior, and output the purified visual semantic features and purified vibration semantic features.

[0024] S4. Based on the multi-head cross-attention mechanism, the purified vibration semantic features are used as the query matrix, and the purified visual semantic features are used as the key matrix and value matrix for calculation to obtain the deep fusion feature sequence.

[0025] Specifically, the deep fusion feature sequence refers to the multimodal fusion representation sequence output after multi-layer multi-head cross-attention calculation and residual connection. The multimodal fusion representation sequence can simultaneously encode the attention focus information of the vibration mode on the visual mode and the spatial structural semantics of the visual mode itself.

[0026] Optionally, the processing device can use the purified vibration semantic features as input to the query matrix of the multi-head cross-attention mechanism, and map the query matrix to multiple independent subspaces through different linear projection matrices. The processing device can also use the purified visual semantic features as input to the key and value matrices respectively, and map the key and value matrices to the same subspace dimension as the query matrix through the corresponding linear projection matrices. Within each subspace, the processing device can calculate the transpose matrix multiplication of the query matrix and the key matrix, divide the multiplication result by a fixed scaling factor, and normalize it using the Softmax function to obtain the attention weight distribution matrix. The processing device can perform multiplication operations on the attention weight distribution matrix and the value matrix to obtain the weighted output of each subspace. Finally, the processing device can concatenate the weighted outputs of all subspaces along the feature dimension and restore them to the original dimension through a linear transformation to obtain a deep fusion feature sequence.

[0027] S5. Through a linear projection layer, adjust the vector dimension of the deep fusion feature sequence, and concatenate the adjusted deep fusion feature sequence and the instruction text embedding vector to obtain the combined prompt conditions.

[0028] Specifically, the instruction text embedding vector can be a sequence of word vectors mapped from the system instruction text. The system instruction text can be used to define roles and analysis tasks in generative large language models.

[0029] Specifically, the linear projection layer can be a dimension-transformed network layer consisting of fully connected weight matrices. The combined cue conditions can be a set of pre-sequences that serve as input conditions for a generative large language model.

[0030] Optionally, the processing device can input the deep fusion feature sequence into a linear projection layer, and adjust the vector dimension of the deep fusion feature sequence to match the word embedding dimension of the generative large language model through weight matrix multiplication, thereby obtaining a projection fusion feature sequence with soft cueing characteristics. The processing device can read preset system instruction text and convert it into instruction text embedding vectors through the vocabulary mapping mechanism of the generative large language model. The processing device can then perform a head-to-tail concatenation operation on the instruction text embedding vectors and the projection fusion feature sequence along the temporal dimension to construct combined cueing conditions.

[0031] S6. Input the combined prompts into a generative large language model based on a Transformer decoder to generate a natural language diagnostic report in an autoregressive manner.

[0032] Specifically, natural language diagnostic reports can include the specific location of the fault, physical abnormalities, possible causes, and maintenance recommendations.

[0033] Optionally, the processing device can input the combined cue conditions into the Transformer decoder of the generative large language model. The unidirectional causal attention masking mechanism inside the Transformer decoder can restrict the current position to only observe historical sequence information before that position. The processing device can perform multi-layer self-attention calculation and feedforward network transformation on the combined cue conditions through the Transformer decoder to generate the contextual hidden state vector at the current time step. The processing device can calculate the conditional probability distribution of each candidate word in the vocabulary based on the contextual hidden state vector at the current time step through the output classification layer, and select the word with the highest probability in the conditional probability distribution as the output word at the current time step. The processing device can concatenate the output word at the current time step to the end of the input sequence to form a new input sequence, and input the updated sequence into the Transformer decoder to generate the output word at the next time step. The above steps are repeated until the Transformer decoder outputs a preset termination symbol. The processing device can concatenate and format the output word sequence during the generation process to generate a natural language diagnostic report.

[0034] This embodiment provides an AI-based data processing and analysis method. Low-light enhancement of railway turnout field images is achieved using the CLAHE and Retinex algorithms. Furthermore, the collected vibration signals are transformed into two-dimensional pseudo-color time-frequency spectrum data via continuous wavelet transform, enabling a unified representation of different physical modal data in terms of dimension and topology. Visual and vibration semantic features are extracted using a visual Transformer encoder and a spectral feature Transformer encoder, respectively, improving the ability to capture global context and long-range dependencies. Cross-modal semantic alignment and redundant noise filtering of visual and vibration semantic features are achieved through an alignment network trained under supervision based on a symmetric contrastive loss function and a purification network trained under supervision based on a variational information bottleneck loss function. Dynamic retrieval of purified visual semantic features using purified vibration semantic features as a query matrix via a multi-head cross-attention mechanism improves the spatial positioning accuracy of fault-related physical components. Finally, inputting a combination of deep fusion feature sequences and instruction text embedding vectors into a generative large language model based on a Transformer decoder and outputting a natural language diagnostic report in an autoregressive manner improves the readability of diagnostic results and the interpretability of fault analysis.

[0035] In one embodiment, a continuous wavelet transform is performed on the acquired vibration signal, and the real part, imaginary part, and squared amplitude of the transform coefficients are mapped to the RGB channels respectively to generate two-dimensional pseudo-color time-frequency spectrum data, which may include: S11. The acquired vibration signal is scaled and translated using complex Morlet wavelets. The integral transform value of the vibration signal under different scale factors and translation factors is calculated to obtain the continuous wavelet transform coefficients.

[0036] For example, the formula for calculating the continuous wavelet transform coefficients can be: in, These are continuous wavelet transform coefficients; For scaling factor, scaling factor It can correspond to the reciprocal of the frequency of the vibration signal; Translation factor It can be used for time positioning; The collected vibration signal; It is the complex conjugate of the complex Morlet wavelet.

[0037] Specifically, the complex Morlet wavelet can be a complex-form mother wavelet function composed of a complex exponential function and a Gaussian envelope. The scaling factor can be a parameter controlling the frequency resolution in the continuous wavelet transform. The translation factor can be a parameter controlling the time positioning in the continuous wavelet transform.

[0038] Optionally, the processing device can use a complex Morlet wavelet as the mother wavelet function. The processing device can set numerical sequences of multiple scale factors and multiple translation factors. For each combination of scale and translation factors, the processing device can multiply the acquired vibration signal with the complex conjugate of the scaled and translated complex Morlet wavelet, and perform integration on the time axis to obtain the integral transform value corresponding to that scale and translation factor. The processing device can arrange the integral transform values ​​under all combinations of scale and translation factors according to the scale and translation factor dimensions to construct a continuous wavelet transform coefficient matrix.

[0039] Specifically, the complex characteristics of the complex Morlet wavelet allow the continuous wavelet transform coefficients to contain both real and imaginary information, which can simultaneously characterize the amplitude and phase changes of the acquired vibration signal in the time-frequency domain.

[0040] S12. Normalize the real part, imaginary part, and amplitude square of the continuous wavelet transform coefficients to obtain the normalized real part, normalized imaginary part, and normalized amplitude square. Map the normalized real part to the R channel, the normalized imaginary part to the G channel, and the normalized amplitude square to the B channel to generate two-dimensional pseudo-color time-frequency spectrum data.

[0041] Specifically, the processing device can extract the real part values ​​of the continuous wavelet transform coefficients, calculate the minimum and maximum values ​​of the real part values, and linearly map all real part values ​​to the interval between 0 and 1 based on the minimum and maximum values ​​to obtain the normalized real part. The processing device can also extract the imaginary part values ​​of the continuous wavelet transform coefficients, calculate the minimum and maximum values ​​of the imaginary part values, and linearly map all imaginary part values ​​to the interval between 0 and 1 based on the minimum and maximum values ​​to obtain the normalized imaginary part. The processing device can calculate the sum of the squares of the real and imaginary part values ​​of the continuous wavelet transform coefficients to obtain the amplitude square, calculate the minimum and maximum values ​​of the amplitude square, and linearly map all amplitude squares to the interval between 0 and 1 based on the minimum and maximum values ​​to obtain the normalized amplitude square. The processing device can assign the normalized real part values ​​to the R channel, the normalized imaginary part values ​​to the G channel, and the normalized amplitude square values ​​to the B channel. The processing device can generate two-dimensional pseudo-color time-frequency spectrum data by combining the values ​​from the R, G, and B channels.

[0042] This embodiment provides an artificial intelligence-based data processing and analysis method. By scaling and translating the acquired vibration signal using complex Morlet wavelets and calculating the integral transform value, joint resolution analysis of the acquired vibration signal in the time and frequency domains can be achieved. By normalizing the real, imaginary, and amplitude squares of the continuous wavelet transform coefficients and mapping them to the R, G, and B channels respectively to generate two-dimensional pseudo-color time-frequency spectrum data, visualization encoding of the low-frequency macroscopic impact response and high-frequency distortion components in the acquired vibration signal can be achieved. The generation of two-dimensional pseudo-color time-frequency spectrum data improves the data structure consistency between the acquired vibration signal and the railway turnout field image in the subsequent feature extraction stage.

[0043] In one embodiment, S2 may include: S21. The enhanced image data is divided into blocks and linearly projected, and a learnable positional encoding vector is added to obtain the initial visual label sequence.

[0044] Specifically, the location encoding vector can be used to preserve spatial topological information.

[0045] Specifically, linear projection mapping can be the process of converting image patch pixel values ​​into high-dimensional feature vectors using a trainable weight matrix. The learnable location encoding vector can be a sequence of one-dimensional vectors whose values ​​are updated during network training. The initial visual label sequence can be a sequence of data concatenated from classification labels, image patch feature vectors, and location encoding vectors. Spatial topological information can be the relative positional relationships of image patches within the augmented image data.

[0046] Optionally, the processing device can uniformly segment the enhanced image data according to a preset image patch size to obtain multiple non-overlapping image patches. The processing device expands the pixel values ​​within each image patch into a one-dimensional vector and performs a multiplication operation on each one-dimensional vector and a linear projection matrix, mapping each one-dimensional vector to a preset embedding dimension space to obtain multiple image patch feature vectors. The processing device can generate a learnable positional encoding vector sequence matching the number of segments in the enhanced image data, and can element-wise add each learnable positional encoding vector in the learnable positional encoding vector sequence to its corresponding image patch feature vector. The processing device can append a classification label vector at the beginning of the sequence to gather global information, constructing an initial visual label sequence.

[0047] S22. The initial visual label sequence is nonlinearly extracted using a visual Transformer encoder to capture high-dimensional spatial geometric features and complex texture features, thereby obtaining visual semantic features.

[0048] Specifically, high-dimensional spatial geometric features can be abstract representations of the edge contours and component arrangement structures of turnout components in enhanced image data. Complex texture features can be abstract representations of surface material roughness and wear marks in enhanced image data. A visual Transformer encoder can include multiple stacked Transformer coding blocks, each consisting of a multi-head self-attention sublayer and a feedforward network sublayer.

[0049] Optionally, the processing device can input the initial visual label sequence into a visual Transformer encoder. Within each Transformer encoding block, the processing device can perform layer normalization on the input initial visual label sequence, calculate the correlation weights between vectors in the initial visual label sequence using a multi-head self-attention mechanism, and perform a weighted sum of the correlation weights and value vectors before forming a residual connection with the initial visual label sequence. The processing device can then perform layer normalization again on the residual-connected initial visual label sequence, perform a nonlinear transformation through a feedforward network sublayer, and perform a residual connection between the transformation result and the sublayer input. The processing device can use the sequence output after layer-by-layer computation through multiple Transformer encoding blocks as visual semantic features.

[0050] S23. The two-dimensional pseudo-color time-frequency spectrum data is divided into blocks and linearly projected, and a learnable position encoding vector is added to obtain the initial spectrum label sequence.

[0051] Specifically, the initial spectral label sequence can be a sequence of data composed of classification labels, spectral image patch feature vectors, and location encoding vectors.

[0052] Optionally, the processing device can uniformly segment the two-dimensional pseudo-color time-frequency spectral data according to the same image patch size as the enhanced image data, obtaining multiple non-overlapping spectral image patches. The processing device can expand the R-channel, G-channel, and B-channel pixel values ​​within each spectral image patch into a one-dimensional vector. Through multiplication of linear projection matrices, each one-dimensional vector is mapped to a preset embedding dimension space, resulting in multiple spectral image patch feature vectors. The processing device can generate a learnable positional encoding vector sequence matching the number of segments in the two-dimensional pseudo-color time-frequency spectral data, and can element-wise add each learnable positional encoding vector in the learnable positional encoding vector sequence to its corresponding spectral image patch feature vector. The processing device can then concatenate a classification label vector for aggregating global information at the beginning of this positional encoding vector sequence to construct an initial spectral label sequence.

[0053] S24. The initial spectral marker sequence is nonlinearly extracted using a spectral feature Transformer encoder to capture high-frequency anomalous impact and low-frequency modulated harmonic features, thereby obtaining vibration semantic features.

[0054] Specifically, high-frequency anomalous impact features can be local energy mutation patterns appearing in the high-frequency region of two-dimensional pseudo-color time-frequency spectral data. Low-frequency modulation harmonic features can be periodic energy distribution patterns appearing in the low-frequency region of two-dimensional pseudo-color time-frequency spectral data. The spectral feature Transformer encoder and the visual Transformer encoder have the same number of layers, hidden dimensions, and multi-head attention configuration.

[0055] Optionally, the processing device can input the initial spectral label sequence into the spectral feature Transformer encoder. Within each Transformer encoding block of the spectral feature Transformer encoder, the processing device can perform layer normalization on the initial spectral label sequence and calculate the correlation weights between vectors in the initial spectral label sequence using a multi-head self-attention mechanism. The correlation weights and value vectors are then weighted and summed before being residually concatenated with the initial spectral label sequence. The processing device can then perform layer normalization again on the residually concatenated initial spectral label sequence and perform residual concatenation between the nonlinear transformation result and the sublayer input through a feedforward network sublayer. The processing device can use the sequence output after layer-by-layer computation through multiple Transformer encoding blocks as vibration semantic features.

[0056] This embodiment provides an AI-based data processing and analysis method. By segmenting and linearly projecting enhanced image data and two-dimensional pseudo-color time-frequency spectral data respectively, and adding learnable positional encoding vectors, the method can preserve the internal spatial topological relationships of the enhanced image data and the internal time-frequency spatial positional relationships of the two-dimensional pseudo-color time-frequency spectral data. Nonlinear feature extraction of the initial visual marker sequence using a visual Transformer encoder improves the ability to capture global spatial geometric features and local complex texture features in railway turnout field images. Nonlinear feature extraction of the initial spectral marker sequence using a spectral feature Transformer encoder improves the ability to capture high-frequency anomalous impact features and low-frequency modulation harmonic features in the two-dimensional pseudo-color time-frequency spectral data. The parallel extraction architecture of the visual Transformer encoder and the spectral feature Transformer encoder achieves consistent output of visual semantic features and vibration semantic features in terms of sequence length and feature dimension.

[0057] In one embodiment, S3 may include: S31. Input the visual semantic features and vibration semantic features into the alignment network. Through the linear layer in the alignment network, the visual semantic features and vibration semantic features are mapped to the joint semantic latent space of the same dimension to obtain the mapped visual feature vector and the mapped vibration feature vector. Then, the alignment network is used to perform L2 normalization on the mapped visual feature vector and the mapped vibration feature vector to output the aligned visual semantic features and the aligned vibration semantic features.

[0058] Specifically, the linear layer can be a fully connected computational layer consisting of trainable weight matrices and bias vectors. The joint semantic latent space can be a high-dimensional vector space shared by the visual and vibrational modalities.

[0059] Optionally, the processing device can input visual semantic features into a linear layer in the alignment network, and map the visual semantic features from the original feature dimension to a preset joint semantic latent space dimension through weight matrix multiplication to obtain a mapped visual feature vector. Similarly, the processing device can input vibration semantic features into the same linear layer or a linear layer with the same structure and dimension in the alignment network, and map the vibration semantic features from the original feature dimension to the same joint semantic latent space dimension through weight matrix multiplication to obtain a mapped vibration feature vector. The processing device can calculate the L2 norm of the mapped visual feature vector, and divide each component of the mapped visual feature vector by this L2 norm to obtain the aligned visual semantic features. Likewise, the processing device can calculate the L2 norm of the mapped vibration feature vector, and divide each component of the mapped vibration feature vector by this L2 norm to obtain the aligned vibration semantic features.

[0060] S32. The aligned visual semantic features and aligned vibration semantic features are concatenated along the channel dimension and then input into the purification network. The variational inference network in the purification network predicts the Gaussian posterior distribution of the aligned visual semantic features and the Gaussian posterior distribution of the aligned vibration semantic features, and the visual distribution parameters and vibration distribution parameters are obtained respectively.

[0061] Specifically, the variational inference network can be a neural network structure used to estimate the posterior distribution parameters. The Gaussian posterior distribution can be a multivariate Gaussian distribution followed by the latent variables given the input features. The visual distribution parameters can be the mean and variance vectors of the Gaussian posterior distribution corresponding to the visual semantic features. The vibration distribution parameters can be the mean and variance vectors of the Gaussian posterior distribution corresponding to the vibration semantic features.

[0062] Optionally, the processing device can concatenate and align visual semantic features and vibration semantic features along the channel dimension, and input the concatenated feature sequence into a purification network. The fully connected layer in the variational inference network performs a nonlinear transformation on the concatenated feature sequence, outputting a mean vector and a variance vector, respectively. The processing device can use the mean and variance vectors corresponding to the aligned visual semantic features as visual distribution parameters, and the mean and variance vectors corresponding to the aligned vibration semantic features as vibration distribution parameters.

[0063] Specifically, the mean vector can be used to represent the central location of the Gaussian posterior distribution, and the variance vector can be used to represent the degree of uncertainty in each dimension of the Gaussian posterior distribution.

[0064] For example, the formulas for calculating the mean vector and variance vector of the Gaussian posterior distribution can be: in, The feature sequence obtained by concatenating visual semantic features and vibration semantic features along the channel dimension; For the weight matrix of the mean branch fully connected layer in the variational inference network; This is the bias vector for the fully connected layer of the mean branch in the variational inference network. Let be the mean vector of the Gaussian posterior distribution; The weight matrix of the variance branch fully connected layer in the variational inference network; The bias vector for the fully connected layer of the variance branch in the variational inference network; It is the logarithmic variance vector; and These represent the linear transformation functions for the mean branch and the variance branch, respectively.

[0065] S33. Based on the reparameterization algorithm, the visual distribution parameters and vibration distribution parameters are randomly sampled, and the purified visual semantic features and purified vibration semantic features are output.

[0066] Specifically, the reparameterization algorithm can be a gradient propagation calculation method that transforms the random sampling process into a deterministic calculation and the addition of independent random noise. Random sampling can be a process of drawing samples from a Gaussian posterior distribution.

[0067] Optionally, the processing device can sample a visual random noise vector with the same dimension as the variance vector in the visual distribution parameters from a standard normal distribution. The processing device can perform an exponential operation on the variance vector in the visual distribution parameters and then take the square root to obtain the visual standard deviation vector. The processing device can then perform element-wise multiplication of the visual standard deviation vector and the visual random noise vector, and add the product result element-wise to the mean vector in the visual distribution parameters to obtain the refined visual semantic features. Similarly, the processing device can sample a vibration random noise vector with the same dimension as the variance vector in the vibration distribution parameters from a standard normal distribution. The processing device can perform an exponential operation on the variance vector in the vibration distribution parameters and then take the square root to obtain the vibration standard deviation vector. The processing device can then perform element-wise multiplication of the vibration standard deviation vector and the vibration random noise vector, and add the product result element-wise to the mean vector in the vibration distribution parameters to obtain the refined vibration semantic features.

[0068] This embodiment provides an AI-based data processing and analysis method. By aligning the linear layers in the network to map visual and vibration semantic features to a joint semantic latent space of the same dimension and performing L2 normalization, comparable representations of visual and vibration semantic features in the same metric space can be achieved. By refining the variational inference network in the purification network to predict the Gaussian posterior distributions of aligned visual and vibration semantic features, quantitative estimation of feature uncertainty can be achieved. Random sampling of visual and vibration distribution parameters using a reparameterization algorithm enables propagable gradient calculation during the purification process. By overcoming the variational information bottleneck constraint of the purification network, the ability to filter out modal private noise unrelated to fault diagnosis in visual and vibration semantic features can be improved.

[0069] In one embodiment, S5 may include: S51. Through a linear projection layer, the deep fusion feature sequence is mapped to the same dimensional space as the instruction text embedding vector to obtain a projection fusion feature sequence with soft prompting characteristics.

[0070] Specifically, soft cueing features can be attributes that pass prior information to generative large language models through continuous vectors rather than discrete words.

[0071] For example, the linear projection layer may include a trainable weight matrix, the number of rows of which may be equal to the embedding dimension of the instruction text embedding vector, and the number of columns may be equal to the vector dimension of the deep fusion feature sequence.

[0072] Optionally, the processing device can input the deep fusion feature sequence into the linear projection layer, perform multiplication on the weight matrix in the deep fusion feature sequence and the linear projection layer, and map each vector of the deep fusion feature sequence from the original vector dimension to the same embedding dimension as the instruction text embedding vector to obtain the projection fusion feature sequence.

[0073] Specifically, the projection fusion feature sequence can be consistent with the instruction text embedding vector in terms of vector dimension.

[0074] S52. By concatenating and merging the instruction text embedding vector and the projection fusion feature sequence in the sequence dimension, a combined prompt condition is obtained.

[0075] Specifically, the combined cue conditions can be the autoregressive decoding input of a generative large language model, and the autoregressive decoding input can be the initial sequence used to initiate the word-by-word generation process.

[0076] Optionally, the processing device can read a preset system instruction text, map each word in the system instruction text to a corresponding word vector index using the vocabulary mapping mechanism of a generative large language model, and extract the corresponding word embedding vector from the word embedding matrix according to the word vector index to construct the instruction text embedding vector. The processing device can place the instruction text embedding vector at the beginning of the sequence, place the projection fusion feature sequence at the end of the instruction text embedding vector, and perform concatenation and merging in the sequence dimension to construct the combined prompt conditions.

[0077] This embodiment provides an AI-based data processing and analysis method that maps the deep fusion feature sequence to the same dimensional space as the instruction text embedding vector through a linear projection layer, achieving consistency between the deep fusion feature sequence and the instruction text embedding vector in the vector space dimension. By concatenating and merging the instruction text embedding vector and the projected fusion feature sequence in the sequence dimension, multimodal fusion information and task instruction information can be integrated into a unified input format. By combining cue conditions as autoregressive decoding input for the generative large language model, the generative large language model's ability to understand the physical state reflected by railway turnout field images and collected vibration signals during the diagnostic generation process can be improved.

[0078] In one embodiment, S6 may include: S61. Input the combined cue conditions into the Transformer decoder of the generative large language model, perform masked self-attention calculation on the combined cue conditions through a predefined one-way causal attention mask, hide the lexical information of future time moments, make predictions based on historical words, and obtain the contextual hidden state vector of the current time moment.

[0079] Specifically, the Transformer decoder can be a neural network computational unit composed of multiple stacked masked self-attention sublayers, cross-attention sublayers, and feedforward network sublayers. Internally, the Transformer decoder includes multiple stacked Transformer decoding blocks, each configured with a predefined unidirectional causal attention mask matrix. The unidirectional causal attention mask can be a matrix constraint that restricts each position in the sequence to only observing information from that position and previous positions. The contextual hidden state vector at the current time step can be a vector representation output by the Transformer decoder at a certain time step, incorporating historical information.

[0080] Optionally, the processing device can use the combined cue conditions as the input sequence and input it into the Transformer decoder of the generative large language model. When calculating the self-attention mechanism, the processing device can apply a one-way causal attention mask matrix to the attention score matrix, setting the attention scores corresponding to all positions after the current position in the input sequence to their minimum values. The processing device can obtain a weighted representation that depends only on historical words through masked self-attention calculation, and then perform residual connections and layer normalization on the weighted representation and the input sequence. The processing device can perform a nonlinear transformation on the layer normalized result through a feedforward network sublayer, and then perform residual connections and layer normalization again. The processing device can use the result obtained after layer-by-layer calculation through multiple Transformer decoding blocks as the context hidden state vector at the current time step.

[0081] S62. Based on the contextual hidden state vector at the current moment, calculate the conditional probability distribution of each word, select the word with the highest conditional probability as the output word at the current moment, and use the output word as the historical word at the next moment. Iterate in a loop until the end symbol is generated to obtain a natural language diagnostic report.

[0082] Specifically, the output word can be the word with the highest probability selected from the vocabulary at the current time step. The historical word for the next time step can be the embedding vector representation of the output word at the current time step, which will be included at the end of the input sequence in the next time step.

[0083] Optionally, the processing device can input the current contextual hidden state vector into the output classification layer of the generative large language model, multiply the current contextual hidden state vector and the linear transformation matrix of the output classification layer to obtain a logits vector with a dimension equal to the vocabulary size. The processing device can then input the logits vector into the Softmax activation function to calculate the normalized probability value of each candidate word in the vocabulary, obtaining a conditional probability distribution. The processing device can retrieve the candidate word with the highest probability value from the conditional probability distribution and use it as the output word for the current time step. The processing device can convert the current output word into an embedding vector using a word embedding matrix and append this embedding vector to the end of the input sequence. The processing device can then input the updated input sequence back into the Transformer decoder, repeating the processes of mask self-attention calculation, contextual hidden state vector generation, conditional probability distribution calculation, and output word selection. The processing device can terminate the iterative process when the output word equals a preset end symbol. Finally, the processing device can concatenate all output words generated during the iterative process in the order of generation to construct a natural language diagnostic report.

[0084] Specifically, the output classification layer can be composed of a linear transformation matrix and a Softmax activation function.

[0085] This embodiment provides an AI-based data processing and analysis method that uses a predefined unidirectional causal attention mask to perform masked self-attention calculation on combined prompt conditions and hide lexical information from future moments, thus following the temporal causal relationship in the autoregressive generation process. By calculating the conditional probability distribution of each word based on the contextual hidden state vector at the current moment and selecting the word with the highest conditional probability as the output word, word-by-word generation of diagnostic text vocabulary can be achieved. By iterating the output words as historical words for the next moment until the end symbol is generated, autoregressive complete output of natural language diagnostic reports of arbitrary length can be achieved. Through the autoregressive decoding mechanism of the generative large language model, the coherence and completeness of the language expression of fault diagnosis results in the four dimensions of specific fault location, physical anomaly manifestation, possible causes, and maintenance suggestions can be improved.

[0086] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0087] Based on the same inventive concept, this application also provides a system for implementing the aforementioned artificial intelligence-based data processing and analysis method. The solution provided by this system is similar to the implementation scheme described in the above method; therefore, the specific limitations of one or more embodiments of an artificial intelligence-based data processing and analysis system provided below can be found in the limitations of the artificial intelligence-based data processing and analysis method described above, and will not be repeated here.

[0088] In one exemplary embodiment, such as Figure 2 As shown, an artificial intelligence-based data processing and analysis system 200 is provided to implement the methods in the above-described method embodiments. This system may include: The multimodal data preprocessing module 201 is used to perform low-light enhancement on the acquired railway turnout field images based on the CLAHE and Retinex algorithms to obtain enhanced image data, and to perform continuous wavelet transform on the acquired vibration signals to obtain continuous wavelet transform coefficients. The real part, imaginary part and amplitude square of the continuous wavelet transform coefficients are mapped to the RGB three channels respectively to generate two-dimensional pseudo-color time-frequency spectrum data.

[0089] The dual-modal feature extraction module 202 is used to extract image features from the enhanced image data through a visual Transformer encoder to obtain the visual semantic features of the enhanced image data, and to extract semantic features from the two-dimensional pseudo-color time-frequency spectrum data through a spectral feature Transformer encoder that is structurally symmetrical with the visual Transformer encoder to obtain the vibration semantic features of the two-dimensional pseudo-color time-frequency spectrum data.

[0090] The feature alignment and purification module 203 is used to perform semantic alignment of visual semantic features and vibration semantic features by an alignment network trained under supervision based on the symmetric contrast loss function, and to perform constrained purification of the aligned visual semantic features and the aligned vibration semantic features by a purification network trained under supervision based on the variational information bottleneck loss function, so as to obtain purified visual semantic features and purified vibration semantic features.

[0091] The cross-modal deep fusion module 204 is used to calculate the deep fusion feature sequence by using the purified vibration semantic features as the query matrix and the purified visual semantic features as the key matrix and value matrix based on the multi-head cross-attention mechanism.

[0092] The prompt condition construction module 205 is used to adjust the vector dimension of the deep fusion feature sequence through a linear projection layer, and to concatenate the adjusted deep fusion feature sequence and the instruction text embedding vector to obtain the combined prompt condition; wherein, the instruction text embedding vector is a sequence of word vectors mapped from the system instruction text, and the system instruction text is used to define the roles and analysis tasks of the generative large language model.

[0093] The diagnostic report generation module 206 is used to input combined prompt conditions into a generative large language model based on a Transformer decoder to generate a natural language diagnostic report in an autoregressive manner.

[0094] In one embodiment, the multimodal data preprocessing module 201 can also be used for: The acquired vibration signal is scaled and translated using complex Morlet wavelets. The integral transform values ​​of the vibration signal under different scale factors and translation factors are calculated to obtain the continuous wavelet transform coefficients.

[0095] The real, imaginary, and amplitude squared parts of the continuous wavelet transform coefficients are normalized to obtain normalized real, normalized imaginary, and normalized amplitude squared parts. The normalized real parts are then mapped to the R channel, the normalized imaginary parts to the G channel, and the normalized amplitude squared parts to the B channel, generating two-dimensional pseudo-color time-frequency spectrum data.

[0096] In one embodiment, the dual-modal feature extraction module 202 can also be used for: The enhanced image data is segmented and linearly projected, and a learnable location encoding vector is added to obtain an initial visual label sequence.

[0097] The initial visual label sequence is nonlinearly feature-extracted by a visual Transformer encoder to capture high-dimensional spatial geometric features and complex texture features, thereby obtaining visual semantic features.

[0098] The two-dimensional pseudo-color time-frequency spectral data is divided into blocks and linearly projected, and a learnable position encoding vector is added to obtain the initial spectral label sequence.

[0099] The initial spectral marker sequence is nonlinearly feature-extracted using a spectral feature Transformer encoder to capture high-frequency anomalous impact and low-frequency modulated harmonic features, thereby obtaining vibration semantic features.

[0100] In one embodiment, the feature alignment and purification module 203 can also be used for: Visual semantic features and vibration semantic features are input into the alignment network. The linear layers in the alignment network map the visual semantic features and vibration semantic features to a joint semantic latent space of the same dimension, resulting in mapped visual feature vectors and mapped vibration feature vectors. The alignment network then performs L2 normalization on the mapped visual feature vectors and mapped vibration feature vectors, outputting aligned visual semantic features and aligned vibration semantic features.

[0101] The aligned visual semantic features and aligned vibration semantic features are concatenated along the channel dimension and then input into the purification network. The variational inference network in the purification network predicts the Gaussian posterior distribution of the aligned visual semantic features and the Gaussian posterior distribution of the aligned vibration semantic features, and the visual distribution parameters and vibration distribution parameters are obtained respectively.

[0102] Based on the reparameterization algorithm, visual distribution parameters and vibration distribution parameters are randomly sampled to output purified visual semantic features and purified vibration semantic features.

[0103] In one embodiment, the prompt condition construction module 205 can also be used for: By using a linear projection layer, the deep fusion feature sequence is mapped to the same dimensional space as the instruction text embedding vector, resulting in a projection fusion feature sequence with soft cueing characteristics.

[0104] By concatenating and merging the instruction text embedding vector and the projection fusion feature sequence along the sequence dimension, a combined prompt condition is obtained.

[0105] In one embodiment, the diagnostic report generation module 206 can also be used for: The combined cue conditions are input into the Transformer decoder of the generative large language model. The combined cue conditions are then subjected to masked self-attention computation through a predefined one-way causal attention mask to hide the lexical information of future time steps. Based on historical lexical information, prediction is performed to obtain the contextual hidden state vector of the current time step.

[0106] Based on the contextual hidden state vector at the current moment, the conditional probability distribution of each word is calculated. The word with the highest conditional probability is selected as the output word at the current moment, and the output word is used as the historical word at the next moment. This process is repeated until the end symbol is generated, and a natural language diagnostic report is obtained.

[0107] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the artificial intelligence-based data processing and analysis method described above.

[0108] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0109] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0110] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A data processing and analysis method based on artificial intelligence, characterized in that, The method includes: S1. Perform low-light enhancement on the collected railway turnout field images based on CLAHE and Retinex algorithms to obtain enhanced image data. Perform continuous wavelet transform on the collected vibration signals to obtain continuous wavelet transform coefficients. Map the real part, imaginary part and amplitude square of the continuous wavelet transform coefficients to the RGB three channels respectively to generate two-dimensional pseudo-color time-frequency spectrum data. S2. The enhanced image data is processed by a visual Transformer encoder to extract image features, thereby obtaining the visual semantic features of the enhanced image data. The two-dimensional pseudo-color time-frequency spectrum data is processed by a spectral feature Transformer encoder that is symmetrical to the structure of the visual Transformer encoder to extract the vibration semantic features of the two-dimensional pseudo-color time-frequency spectrum data. S3. The visual semantic features and vibration semantic features are semantically aligned by an alignment network trained under supervision based on a symmetric contrast loss function, and the aligned visual semantic features and aligned vibration semantic features are constrained and purified by a purification network trained under supervision based on a variational information bottleneck loss function, so as to obtain purified visual semantic features and purified vibration semantic features. S4. Based on the multi-head cross-attention mechanism, the purified vibration semantic features are used as the query matrix, and the purified visual semantic features are used as the key matrix and value matrix for calculation to obtain a deep fusion feature sequence. S5. Through a linear projection layer, adjust the vector dimension of the deep fusion feature sequence, and concatenate the adjusted deep fusion feature sequence and the instruction text embedding vector to obtain the combined prompt conditions; wherein, the instruction text embedding vector is a word vector sequence mapped from the system instruction text, and the system instruction text is used to define the roles and analysis tasks of the generative large language model; S6. Input the combined prompt conditions into a generative large language model based on a Transformer decoder to generate a natural language diagnostic report in an autoregressive manner; wherein, the natural language diagnostic report includes the specific fault location, physical abnormality manifestations, possible causes, and maintenance suggestions.

2. The method according to claim 1, characterized in that, The process of performing continuous wavelet transform on the acquired vibration signal, mapping the real part, imaginary part, and squared amplitude of the transform coefficients to the RGB channels respectively, and generating two-dimensional pseudo-color time-frequency spectrum data includes: S11. The acquired vibration signal is scaled and translated using complex Morlet wavelets, and the integral transform value of the vibration signal under different scale factors and translation factors is calculated to obtain the continuous wavelet transform coefficients; wherein, the calculation formula for the continuous wavelet transform coefficients is: in, These are the continuous wavelet transform coefficients; The scaling factor corresponds to the reciprocal of the frequency of the vibration signal; The translation factor is used for time positioning; The vibration signal was collected; For the complex conjugate of the Morlet wavelet; S12. Normalize the real part, imaginary part, and amplitude square of the continuous wavelet transform coefficients to obtain normalized real part, normalized imaginary part, and normalized amplitude square. Map the normalized real part to the R channel, the normalized imaginary part to the G channel, and the normalized amplitude square to the B channel to generate the two-dimensional pseudo-color time-frequency spectrum data.

3. The method according to claim 1, characterized in that, The S2 includes: S21. The enhanced image data is divided into blocks and linearly projected, and a learnable positional encoding vector is added to obtain an initial visual label sequence; wherein, the positional encoding vector is used to preserve spatial topological information; S22. The initial visual marker sequence is nonlinearly extracted using the visual Transformer encoder to capture high-dimensional spatial geometric features and complex texture features, thereby obtaining visual semantic features. S23. The two-dimensional pseudo-color time-frequency spectrum data is divided into blocks and linearly projected and mapped, and the learnable position encoding vector is added to obtain the initial spectrum label sequence. S24. The initial spectral marker sequence is nonlinearly extracted using the spectral feature Transformer encoder to capture high-frequency anomalous impact and low-frequency modulation harmonic features, thereby obtaining vibration semantic features.

4. The method according to claim 1, characterized in that, The S3 includes: S31. Input the visual semantic features and the vibration semantic features into the alignment network, and map the visual semantic features and the vibration semantic features to a joint semantic latent space of the same dimension through the linear layer in the alignment network to obtain the mapped visual feature vector and the mapped vibration feature vector. Then, perform L2 normalization on the mapped visual feature vector and the mapped vibration feature vector through the alignment network to output the aligned visual semantic features and the aligned vibration semantic features. S32. The alignment visual semantic features and the alignment vibration semantic features are concatenated along the channel dimension and then input into the purification network. The Gaussian posterior distribution of the alignment visual semantic features and the Gaussian posterior distribution of the alignment vibration semantic features are predicted by the variational inference network in the purification network to obtain the visual distribution parameters and vibration distribution parameters, respectively. S33. Based on the reparameterization algorithm, the visual distribution parameters and the vibration distribution parameters are randomly sampled, and the purified visual semantic features and purified vibration semantic features are output.

5. The method according to claim 1, characterized in that, The S5 includes: S51. Through the linear projection layer, the deep fusion feature sequence is mapped to the same dimensional space as the instruction text embedding vector to obtain a projection fusion feature sequence with soft prompting characteristics; S52. The instruction text embedding vector and the projection fusion feature sequence are concatenated and merged along the sequence dimension to obtain the combined prompt condition; wherein the combined prompt condition is the autoregressive decoding input of the generative large language model.

6. The method according to claim 1, characterized in that, The S6 includes: S61. Input the combined prompt conditions into the Transformer decoder of the generative large language model, perform mask self-attention calculation on the combined prompt conditions through a predefined one-way causal attention mask, hide the lexical information of future moments, make predictions based on historical words, and obtain the contextual hidden state vector of the current moment. S62. Based on the contextual hidden state vector at the current moment, calculate the conditional probability distribution of each word, select the word with the highest conditional probability as the output word at the current moment, and use the output word as the historical word at the next moment. Iterate in a loop until an end symbol is generated to obtain the natural language diagnostic report.

7. A data processing and analysis system based on artificial intelligence, used to implement the method of any one of claims 1 to 6, characterized in that, The system includes: The multimodal data preprocessing module is used to perform low-light enhancement on the acquired railway turnout field images based on the CLAHE and Retinex algorithms to obtain enhanced image data. It also performs continuous wavelet transform on the acquired vibration signals to obtain continuous wavelet transform coefficients. The real part, imaginary part and amplitude square of the continuous wavelet transform coefficients are mapped to the RGB three channels respectively to generate two-dimensional pseudo-color time-frequency spectrum data. The dual-modal feature extraction module is used to extract image features from the enhanced image data through a visual Transformer encoder to obtain the visual semantic features of the enhanced image data, and to extract semantic features from the two-dimensional pseudo-color time-frequency spectrum data through a spectral feature Transformer encoder that is structurally symmetrical with the visual Transformer encoder to obtain the vibration semantic features of the two-dimensional pseudo-color time-frequency spectrum data. The feature alignment and purification module is used to perform semantic alignment on the visual semantic features and vibration semantic features by an alignment network trained under supervision based on the symmetric contrast loss function, and to perform constrained purification on the aligned visual semantic features and aligned vibration semantic features by a purification network trained under supervision based on the variational information bottleneck loss function, so as to obtain purified visual semantic features and purified vibration semantic features. The cross-modal deep fusion module is used to calculate the deep fusion feature sequence by using the purified vibration semantic features as a query matrix and the purified visual semantic features as a key matrix and value matrix based on a multi-head cross-attention mechanism. The prompt condition construction module is used to adjust the vector dimension of the deep fusion feature sequence through a linear projection layer, and concatenate the adjusted deep fusion feature sequence and the instruction text embedding vector to obtain combined prompt conditions; wherein, the instruction text embedding vector is a word vector sequence mapped from the system instruction text, and the system instruction text is used to define the roles and analysis tasks of the generative large language model; The diagnostic report generation module is used to input the combined prompt conditions into a generative large language model based on a Transformer decoder to generate a natural language diagnostic report in an autoregressive manner.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images