Steel rail head damage classification method and device based on bilinear model, computer equipment, readable storage medium and program product

Abstract

The invention relates to a bilinear model-based steel rail head flaw classification method and device, computer equipment, a computer readable storage medium and a computer program product, relates to the field of target detection, and can improve the accuracy of flaw detection. The method comprises the following steps: acquiring damage image data of a steel rail head, and inputting the damage image data into a trained bilinear model; through a first convolutional neural network and a second convolutional neural network with different network structures in the bilinear model, steel rail head flaw feature extraction is carried out on the flaw image data, and first flaw features and second flaw features of different feature dimensions of the steel rail head are determined; and bilinear feature fusion is conducted on the first damage feature and the second damage feature through a bilinear model, target fusion damage features are obtained, the target fusion damage features are input into a full connection layer in the bilinear model for category determination, and the damage category corresponding to the steel rail head is determined.

Description

Technical Field

[0001] The present application relates to the field of target detection technology, and in particular to a rail head damage classification method, device, computer equipment, computer-readable storage medium and computer program product based on a bilinear model. Background Art

[0002] Currently, rail damage detection mostly relies on ultrasonic nondestructive testing combined with artificial intelligence algorithms. While this method can achieve preliminary damage location and classification, different types of damage in the rail head region, such as abnormal welds, core damage, and joints, have highly similar waveforms, making it difficult for existing methods to effectively distinguish them. This results in a high rate of missed detections and insufficient classification accuracy. Therefore, a recognition method that can improve the classification accuracy of multiple similar types of damage at the rail head is urgently needed. Summary of the Invention

[0003] Based on this, it is necessary to provide a rail head damage classification method, device, computer equipment, computer-readable storage medium and computer program product based on a bilinear model to address the above technical problems.

[0004] In a first aspect, the present application provides a rail head damage classification method based on a bilinear model, comprising:

[0005] Obtaining damage image data of the rail head and inputting it into the trained bilinear model;

[0006] Extracting rail head damage features from the damage image data using a first convolutional neural network and a second convolutional neural network having different network structures in the bilinear model, respectively, to determine first damage features and second damage features of different feature dimensions of the rail head;

[0007] The first damage feature and the second damage feature are bilinearly fused through the bilinear model to obtain a target fused damage feature, and the target fused damage feature is input into the fully connected layer in the bilinear model for category determination to determine the damage category corresponding to the rail head.

[0008] In one embodiment, the representation form of the first damage feature and the representation form of the second damage feature are both in vector form;

[0009] The step of performing bilinear feature fusion on the first damage feature and the second damage feature through the bilinear model to obtain a target fused damage feature includes:

[0010] Performing an outer product operation on the first damage feature in vector form and the second damage feature in vector form through the bilinear model to obtain an intermediate fused damage feature in matrix form;

[0011] According to the vertical dimension or the horizontal dimension of the matrix, the intermediate fusion damage features are summed and pooled, and the target fusion damage features are determined according to the summed and pooled results in vector form.

[0012] In one embodiment, determining the target fusion damage feature based on the sum pooling processing result in vector form includes:

[0013] Determine a normalized scalar corresponding to the sum pooling processing result in the vector form;

[0014] The sum pooling processing result in the vector form is normalized based on the normalized scalar to obtain the target fusion damage feature.

[0015] In one embodiment, performing bilinear feature fusion on the first damage feature and the second damage feature using the bilinear model to obtain a target fused damage feature further includes:

[0016] Performing feature dimensionality reduction processing on the first damage feature and the second damage feature to obtain a first damage feature in vector form and a second damage feature in vector form.

[0017] In one embodiment, obtaining damage image data of the rail head includes:

[0018] Acquiring multiple echo signals corresponding to different ultrasonic incident angles of the rail head;

[0019] An image is constructed based on the waveform morphology change characteristics and echo intensity distribution characteristics of the multiple echo signals to obtain damage image data of the rail head.

[0020] In one embodiment, the bilinear model is trained by the following steps:

[0021] Obtain historical damage image data of a rail head with a known damage category label, input the historical damage image data into the first convolutional neural network and the second convolutional neural network of the bilinear model to be trained to extract damage features, and perform bilinear feature fusion on the extracted first training damage features and second training damage features to obtain a training target fused damage feature;

[0022] Inputting the target fusion damage feature into the fully connected layer of the bilinear model to be trained to obtain a training damage category;

[0023] According to the difference between the training damage category and the damage category label, the structural parameters of the first convolutional neural network, the second convolutional neural network and the fully connected layer of the bilinear model to be trained are updated, and when the training end condition is met, the trained bilinear model is obtained.

[0024] In a second aspect, the present application further provides a rail head damage classification device based on a bilinear model, comprising:

[0025] A rail head damage image data acquisition module is used to acquire the damage image data of the rail head and input it into the trained bilinear model;

[0026] a damage feature dual-branch extraction module, configured to extract rail head damage features from the damage image data using a first convolutional neural network and a second convolutional neural network having different network structures in the bilinear model, respectively, to determine a first damage feature and a second damage feature of different feature dimensions of the rail head;

[0027] A bilinear feature fusion and classification module is used to perform bilinear feature fusion on the first damage feature and the second damage feature through the bilinear model to obtain a target fused damage feature, input the target fused damage feature into the fully connected layer of the bilinear model for category determination, and determine the damage category corresponding to the rail head.

[0028] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0029] Obtaining damage image data of the rail head and inputting it into the trained bilinear model;

[0030] Extracting rail head damage features from the damage image data using a first convolutional neural network and a second convolutional neural network having different network structures in the bilinear model, respectively, to determine first damage features and second damage features of different feature dimensions of the rail head;

[0031] The first damage feature and the second damage feature are bilinearly fused through the bilinear model to obtain a target fused damage feature, and the target fused damage feature is input into the fully connected layer in the bilinear model for category determination to determine the damage category corresponding to the rail head.

[0032] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0033] Obtaining damaged image data of the rail head and inputting it into the trained bilinear model;

[0034] Extracting rail head damage features from the damage image data using a first convolutional neural network and a second convolutional neural network having different network structures in the bilinear model, respectively, to determine first damage features and second damage features of different feature dimensions of the rail head;

[0035] The first damage feature and the second damage feature are bilinearly fused through the bilinear model to obtain a target fused damage feature, and the target fused damage feature is input into the fully connected layer in the bilinear model for category determination to determine the damage category corresponding to the rail head.

[0036] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the following steps:

[0037] Obtaining damage image data of the rail head and inputting it into the trained bilinear model;

[0038] Extracting rail head damage features from the damage image data using a first convolutional neural network and a second convolutional neural network having different network structures in the bilinear model, respectively, to determine first damage features and second damage features of different feature dimensions of the rail head;

[0039] The first damage feature and the second damage feature are bilinearly fused through the bilinear model to obtain a target fused damage feature, and the target fused damage feature is input into the fully connected layer in the bilinear model for category determination to determine the damage category corresponding to the rail head.

[0040] The above-mentioned rail head damage classification method, device, computer equipment, computer-readable storage medium and computer program product based on the bilinear model obtain damage image data of the rail head and input it into the trained bilinear model; use the first convolutional neural network and the second convolutional neural network with different network structures in the bilinear model to extract rail head damage features from the damage image data, respectively, and determine the first damage feature and the second damage feature of different feature dimensions of the rail head; use the bilinear model to perform bilinear feature fusion on the first damage feature and the second damage feature to obtain a target fused damage feature, and input the target fused damage feature into the fully connected layer in the bilinear model for category determination to determine the damage category corresponding to the rail head. In this application, a bilinear model with two convolutional neural networks with different structures is used to extract damage features from the damage image data of the rail head from different damage feature dimensions, and the extraction results are deeply cross-modeled using a bilinear feature fusion method, which effectively enhances the combined expression ability between features and improves the ability to distinguish damage types with similar damage morphology but significant differences in essence. By inputting the fused features into a fully connected neural network for classification, fine-grained recognition of rail head-related damage can be achieved, thereby improving the accuracy of damage detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 1 is a flow chart of a rail head damage classification method based on a bilinear model in one embodiment;

[0043] Figure 2 Schematic diagram of damage image data corresponding to a normal "inverted eight" weld in one embodiment;

[0044] Figure 3 Schematic diagram of damage image data corresponding to an abnormal "inverted eight" weld in one embodiment;

[0045] Figure 4 Schematic diagram of damage image data corresponding to a normal thermite weld in one embodiment;

[0046] Figure 5 Schematic diagram of damage image data corresponding to an abnormal thermite weld in one embodiment;

[0047] Figure 6Schematic diagram of damage image data corresponding to rail head core damage in one embodiment;

[0048] Figure 7 This is a schematic diagram of damage image data corresponding to a normal joint in one embodiment;

[0049] Figure 8 Schematic diagram of damage image data corresponding to an abnormal joint in one embodiment;

[0050] Figure 9 is a schematic flow chart of a rail head damage classification method based on a bilinear model in another embodiment;

[0051] Figure 10 1 is a structural block diagram of a rail head damage classification device based on a bilinear model in one embodiment;

[0052] Figure 11 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0054] In one embodiment, Figure 1 As shown, a rail head damage classification method based on a bilinear model is provided. This embodiment uses the method applied to a terminal as an example for illustration. It is understandable that the method can also be applied to a server, or to a terminal including a terminal and a server, and implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0055] Step S102: obtaining damage image data of the rail head and inputting it into the trained bilinear model.

[0056] The rail head is the part of the track that bears the pressure of the train wheels and is the most critical part in contact with the train wheels. Image data is obtained by scanning the rail head with ultrasonic flaw detection equipment and converting it into visual data of rail head damage.

[0057] For example, the rail head is first scanned using ultrasonic flaw detection equipment. The internal structure of the rail head can be determined by transmitting high-frequency sound waves to the rail head and receiving the reflected signals. To accurately capture the state of the rail head, in some examples, multiple probes can be used to scan the rail head at various angles. For example, ultrasonic equipment can collect ultrasonic signals at multiple angles, such as 70 degrees from the front and outside, 70 degrees from the rear and outside, and 70 degrees from the front and center. The collected signal data is then converted into image data.

[0058] In some embodiments, the damage image data will be pre-processed, such as formatting, denoising, and normalization, to adapt it to the subsequent deep learning model input. The pre-processed damage image data will be input into a pre-trained bilinear model to begin feature extraction and classification tasks.

[0059] In step S104, the first convolutional neural network and the second convolutional neural network with different network structures in the bilinear model are used to extract the damage features of the rail head from the damage image data, thereby determining the first damage features and the second damage features of the rail head in different feature dimensions.

[0060] Among them, damage characteristics refer to the structural information inside the rail head. These characteristics can be reflected in the damage image data of the rail head obtained by ultrasonic flaw detection technology.

[0061] For example, the first convolutional neural network can extract the first morphological features of the rail head from the damaged image data, namely, features related to the initial damage to the rail head. Specifically, the first convolutional neural network is responsible for extracting waveform patterns in the image that reflect local defects. These defects can be on the surface or near the surface of the rail head, such as microcracks, pores, slag inclusions, etc. The reflected waveforms of these areas in the image have specific patterns. Through multi-layer convolution operations, the first convolutional neural network can capture the waveform changes of these damaged areas from the damaged image data and extract key information reflecting these damages through learning convolution filters.

[0062] The second convolutional neural network focuses on extracting secondary damage features of the rail head, specifically those related to deeper damage. These deep damages can be located within the rail head's inner structure and present as difficult-to-detect damage, such as core damage, deep cracks, corrosion, and metal fatigue. Because the waveform characteristics of these deep damages are more complex and difficult to visualize directly in images, the second convolutional neural network, through deeper convolution operations, is able to extract the features of these hidden damages.

[0063] Optionally, the first and second convolutional neural networks of the bilinear model can be trained with different network structures and training objectives to focus on extracting damage features from different dimensions of the damage image data. The first convolutional neural network focuses on preliminary damage identification, while the second convolutional neural network specializes in deeper damage identification. In this way, the first and second convolutional neural networks complement each other and can capture damage information from different dimensions of the rail head.

[0064] In step S106, the first damage feature and the second damage feature are bilinearly fused through a bilinear model to obtain a target fused damage feature. The target fused damage feature is input into the fully connected layer of the bilinear model for category determination to determine the damage category corresponding to the rail head.

[0065] Bilinearity refers to a specific mathematical relationship or function property involving two independent input variables or feature sets. Specifically, when one input variable is fixed, the relationship or function exhibits a linear property with respect to the other input variable. In other words, for a bilinear process that takes two inputs [A] and [B] and produces an output [C], the output [C] varies linearly with input [A] (when [B] is fixed), and the output [C] also varies linearly with input [B] (when [A] is fixed).

[0066] For example, to effectively combine features from these two different dimensions, the terminal uses a bilinear feature fusion method. During this process, the first and second damage features are extracted from their respective convolutional neural networks and fused to generate a new feature in a manner that satisfies a mathematical linear process, including but not limited to matrix transformation and weighted summation. The goal of bilinear fusion is to combine the interactive information between the two feature dimensions to generate a high-dimensional feature representation that contains more potential relationships and richer information. This fusion process can capture the complex relationships between different types of damage on the rail head, allowing the terminal to better understand the inherent patterns of damage.

[0067] The target fused damage feature generated through bilinear feature fusion contains multi-dimensional damage information of the rail head. This feature can fully express the damage pattern of the rail head, including the characteristics of both initial damage and deep damage.

[0068] The target fused damage features are input into the fully connected layer of the bilinear model. The fully connected layer performs a weighted summation of the fused features and generates the output result through nonlinear transformation. The function of the fully connected layer is to map the complex high-dimensional features to the specific damage category of the rail head. Through this processing step, the terminal can classify based on the fused features and finally determine the damage category. Figure 2-Figure 8 As shown, the final determined damage categories of the rail head include but are not limited to normal rail head "inverted eight" shaped welds, normal rail head thermite welds, abnormal rail head "inverted eight" shaped welds, abnormal rail head thermite welds, rail head core damage, normal joints and abnormal joints.

[0069] In this embodiment, a bilinear model with two convolutional neural networks with different structures is used to extract damage features from the damage image data of the rail head from different damage feature dimensions. The extraction results are then subjected to deep cross-modeling using a bilinear feature fusion method. This effectively enhances the combined expression capability between features and improves the ability to distinguish damage types that are similar but have significant essential differences. By inputting the fused features into a fully connected neural network for classification, fine-grained identification of rail head-related damage can be achieved, thereby improving the accuracy of damage detection.

[0070] In an exemplary embodiment, the representation form of the first damage feature and the representation form of the second damage feature are both in vector form;

[0071] In step S104, the first damage feature and the second damage feature are bilinearly fused through a bilinear model to obtain a target fused damage feature, including: performing an outer product operation on the first damage feature in vector form and the second damage feature in vector form through a bilinear model to obtain an intermediate fused damage feature in matrix form; performing sum pooling processing on the intermediate fused damage feature according to the vertical dimension or horizontal dimension of the matrix, and determining the target fused damage feature based on the sum pooling processing result in vector form.

[0072] In a specific implementation, the first damage feature in vector form can be expressed as follows: and secondary damage characteristics Perform outer product operation, recorded as , which is expressed as follows by formula (3).

[0073]

[0074] in Expressed as eigenvalues, , represents the feature dimension,

[0075]

[0076] in, Expressed as eigenvalues, , Represents the feature dimension.

[0077] After the above feature extraction and outer product operation, the intermediate fusion damage feature is obtained ,remember , which is expressed by formula (3).

[0078]

[0079] The outer product operation multiplies each pair of elements of two eigenvectors to generate a new matrix that contains the interaction information between the two feature dimensions.

[0080] Optionally, a first weight coefficient is introduced to weight the interactions between features. Each element in the feature vector is multiplied by the corresponding first weight coefficient, thereby changing the importance of different features in the fusion process. The weight coefficient is learned during the model training phase and can be dynamically adjusted based on the importance of the tasks in the dataset.

[0081] After the outer product operation, the resulting feature matrix serves as the intermediate fusion damage feature. Pooling is then performed on this matrix. The purpose of pooling is to reduce the dimensionality of the high-dimensional feature matrix and extract the most representative feature information. During the pooling process, sum pooling can be used to combine feature information within a region by summing the vertical or horizontal dimensions of the matrix.

[0082] Specifically, the intermediate fusion damage features Perform horizontal dimension sum pooling to obtain the target fusion damage feature in matrix form , which is expressed by formula (4).

[0083]

[0084] in, .

[0085] The target fusion morphological features in matrix form will be obtained Converted into vector form, recorded as ,in , ,

[0086] Optionally, during the summation pooling process, the intermediate fused damage features in matrix form are weighted and summed according to their dimensions, based on the second weight coefficient learned from the model's training data. For example, during pooling, the model can assign higher weights to deep damage features (such as deep cracks) while assigning lower weights to more preliminary damage features. This weighted pooling operation allows the feature vector to retain the most important information during dimensionality reduction while avoiding interference from redundant data.

[0087] In this embodiment, through bilinear feature fusion, the system effectively integrates the different information from the first and second damage features to generate a target fused damage feature with high expressiveness and discriminative power. The introduction of model weight coefficients (first and second weight coefficients) not only enhances the representativeness of the features but also improves the system's ability to recognize complex damage patterns. This significantly improves classification accuracy and robustness when dealing with similar damage patterns but different types.

[0088] In an exemplary embodiment, the target fusion damage feature is determined based on the sum pooling processing result in vector form, including: determining a normalized scalar corresponding to the sum pooling processing result in vector form; normalizing the sum pooling processing result in vector form based on the normalized scalar to obtain the target fusion damage feature.

[0089] Specifically, a normalized scalar is first calculated based on the summed pooled target fusion features. This can be the L2 norm of the pooled vector, or the square root of the sum of the squares of the pooled feature vectors. Each element of the target fusion feature is divided by the calculated normalized scalar to keep the pooled features within a uniform numerical range, preventing the final classification from being affected by excessively large or small feature dimensions.

[0090] The feature vector after summation, pooling, and normalization is the final target fusion damage feature, as shown in Equation (5). This feature vector contains global information about rail head damage. Due to the pooling and normalization process, the relationship between features is more balanced, making it suitable for subsequent classification tasks. The target fusion damage feature fully reflects the damage information extracted from different dimensions and provides optimized input for the subsequent fully connected layer.

[0091]

[0092] In this embodiment, by balancing the contribution of each feature dimension, the classification performance of the final target fused damage feature is improved. The normalization step ensures that the feature vector has high discriminative power and consistency, preventing certain features from dominating the classification process due to scale issues, thereby achieving higher accuracy.

[0093] In an exemplary embodiment, the bilinear model performs bilinear feature fusion on the first damage feature and the second damage feature to obtain a target fused damage feature, further comprising:

[0094] The first damage feature and the second damage feature are subjected to feature dimensionality reduction processing to obtain the first damage feature in vector form and the second damage feature in vector form.

[0095] In one embodiment, two convolutional neural networks extract a first damage feature and a second damage feature from a B-display image of a rail head. These features represent shallow and deep damage information, respectively. Because these two features can be high-dimensional, to reduce computational complexity and improve the model's classification performance, the system processes them through feature dimensionality reduction. Specifically, linear discriminant analysis can be used to reduce the dimensionality of the first and second damage features. By training the first and second damage features based on damage category labels, a low-dimensional feature space is generated. In this space, samples with similar damage categories are clustered together, while samples of different categories are spaced farther apart.

[0096] In this embodiment, a low-dimensional feature representation is generated by performing linear discriminant analysis dimensionality reduction processing on the first damage feature and the second damage feature, thereby providing a concise and efficient input for subsequent feature fusion and classification tasks.

[0097] In an exemplary embodiment, obtaining damage image data of a rail head includes:

[0098] Acquire multiple echo signals corresponding to different ultrasonic incident angles of the rail head; construct an image based on the waveform morphology change characteristics and echo intensity distribution characteristics of the multiple echo signals to obtain damage image data of the rail head.

[0099] Specifically, ultrasonic waves are emitted through different ultrasonic output channels in the rail head, and corresponding echo signals are obtained. The configuration of the ultrasonic output channels in the rail head is shown in Table 1. These channels are arranged at various angles and directions, and are represented by different numbers. These channels can capture reflected wave information from the internal structure of the rail head at various angles and depths, providing comprehensive damage data.

[0100]

[0101] After acquiring ultrasonic echo signals, the terminal analyzes the waveform morphology and echo intensity distribution characteristics of each echo signal. These characteristics can reveal the type and location of rail head damage. Specifically, the waveform morphology characteristics reflect the distortion of the reflected waveform caused by the damage, while the echo intensity distribution characteristics provide information on the depth and type of damage and its impact on ultrasonic signal propagation.

[0102] After feature extraction, each echo signal is converted into a digital data format that reflects the changes in the ultrasonic echo signal. Through further processing, the terminal can extract discriminative features from multiple echo signals and use them for subsequent image construction and damage classification tasks.

[0103] Based on multiple echo signals acquired from each ultrasonic channel, the terminal reconstructs the signals to produce B-mode damage image data of the rail head. The characteristics of each echo signal are mapped to pixel values or pixel blocks in the image, which represent the damage information of the rail head at that angle of incidence. By collecting data from multiple different angles of incidence, the terminal can integrate the echo signals from each angle to construct a complete B-mode image of the rail head.

[0104] Each pixel or pixel block in the damage image data represents the damage at that location on the rail head, and the data from each ultrasonic outgoing channel corresponds to a different area in the image. Using this damage image data, the terminal can determine the damage pattern of the rail head at multiple angles, providing comprehensive input data for subsequent damage classification.

[0105] Take the B-display damaged image data as sample data, denoted as ,in is the number of sample data. The matrix form is shown in formula (6).

[0106]

[0107] Then, through the sample data of the damage defects at the rail head Perform convolutional neural network feature extraction.

[0108] In this embodiment, comprehensive damage information of the rail head is obtained through multi-angle ultrasonic echo signals, and B-display damage image data of the rail head is constructed based on the waveform morphology change characteristics and echo intensity distribution characteristics of these echo signals. This provides more detailed model input, especially when processing similar morphologies but different damage patterns, which can improve classification accuracy and robustness.

[0109] In an exemplary embodiment, the bilinear model is trained by the following steps:

[0110] Historical damage image data of the rail head with a known damage category label is obtained, and the historical damage image data is input into the first convolutional neural network and the second convolutional neural network of the bilinear model to be trained to extract damage features. The extracted first training damage features and the second training damage features are then bilinearly fused to obtain a training target fused damage feature. The target fused damage feature is input into the fully connected layer of the bilinear model to be trained to obtain a training damage category. Based on the difference between the training damage category and the damage category label, the structural parameters of the first convolutional neural network, the second convolutional neural network and the fully connected layer of the bilinear model to be trained are updated. When the training end condition is met, a trained bilinear model is obtained.

[0111] Specifically, during the initial stage of model training, historical damage image data of rail heads with known damage category labels is acquired. This historical damage image data comes from actual rail head ultrasonic flaw detection processes and includes multiple different damage types on rail heads, including "normal welds," "abnormal welds," and "rail head core damage."

[0112] The historical damage image data is fed into a bilinear model to be trained. The model consists of two convolutional neural networks. The first convolutional neural network extracts primary damage features from the image. These features can be associated with surface damage or shallow damage. The second convolutional neural network extracts secondary damage features. These features can be associated with deep damage or internal damage.

[0113] After extracting the primary and secondary damage features, these features are then subjected to bilinear feature fusion. Specifically, the bilinear model combines the two feature vectors using an outer product or similar mathematical operation to generate a new, high-dimensional fused feature. This fused feature incorporates information from both the primary and secondary damage features, reflecting the damage status of the rail head at different levels.

[0114] The fused target damage features are then fed into the fully connected layer. This layer further processes the target damage features through weighting and nonlinear activation, ultimately outputting the damage category for the rail head. During training, the model uses a loss function to measure the difference between the output damage category and the actual label, adjusting the model parameters using a backpropagation algorithm. This algorithm adjusts the structural parameters of the first and second convolutional neural networks, as well as the fully connected layer, based on the error, enabling the model to more accurately identify different types of damage.

[0115] After each training session, the parameters of each layer in the network are updated using gradient descent or other optimization algorithms. These parameters are continuously optimized during the training process until a predetermined stopping criterion is met. This can include a maximum number of training rounds, minimization of error, or performance on a validation set reaching a predetermined threshold. At this point, the model is considered trained and has demonstrated strong damage classification capabilities.

[0116] In this embodiment, the aforementioned training process enables the learning of different rail head damage patterns from historical damage image data, and the use of a bilinear model for feature extraction and fusion. Through training, the model automatically adjusts the parameters of the convolutional neural network and the weights of the fully connected layers, thereby improving the accuracy of rail head damage classification. Especially when faced with complex or morphologically similar damage patterns, the bilinear model enhances damage recognition and improves classification accuracy by fusing feature information from different levels.

[0117] In some embodiments, B-display damage image data is first input, and background enhancement and damage feature enhancement are performed on the image to improve the visibility of damaged rail areas. By adjusting the relevant parameters of the YOLO algorithm and combining the characteristic information of the rail image, it is possible to efficiently locate and identify normal and abnormal rail data. However, due to the incomplete coverage of the abnormal data set, the average detection accuracy of the algorithm is relatively low.

[0118] In some embodiments, K-means clustering is used to analyze data and automatically and adaptively pre-set anchors based on the data, eliminating the difficulty of manually designing anchors in traditional methods. The multi-dimensional constrained anchors generated by the K-means algorithm effectively address issues in damage location and identification. However, due to the small size of the sample dataset and the limited learning capabilities of the model, some missed detections still exist.

[0119] In some embodiments, B-display damage image data is segmented and denoised, rail image feature information is extracted, and a BP neural network is trained to identify and classify damaged areas. This method can automatically detect and classify different types of rail damage, and improves damage detection accuracy through a nonlinear network structure. Although this method can achieve relatively accurate damage classification, it requires training multiple neural networks for different types of damage and relies on manually designed features, so its applicability to specific rail damage is limited.

[0120] In an exemplary embodiment, the rail head damage classification method based on the bilinear model provided in this application is as follows: Figure 9 The steps shown include:

[0121] Step S901: Acquire ultrasonic echo signals from the rail head. Echo signals from the rail head are acquired using multiple ultrasonic probes positioned at different incident angles. Specific incident angles include the front outer 70-degree channel (F_F70), rear outer 70-degree channel (R_F70), and front center 70-degree channel (F_C70), as shown in Table 1. The waveform morphology and echo intensity distribution of each echo signal reflect internal damage information on the rail head. The multi-angle data obtained from these echo signals provides detailed input for subsequent image construction and feature extraction.

[0122] Step S902: Image reconstruction is performed on the echo signals. Based on the acquired echo signals, image reconstruction is performed using waveform damage characteristics and echo intensity distribution features. Each echo signal feature is digitally processed and mapped to image pixel values or pixel blocks, reflecting the damage information of the rail head at that angle. These images can be used to obtain comprehensive damage data for the rail head, providing visual support for subsequent classification.

[0123] Step S903: Extract the first and second damage features. These features are extracted using convolutional neural networks (CNNs). The first CNN (CNN 1) extracts shallow damage features (such as cracks and pores), while the second CNN (CNN 2) extracts deeper damage features (such as rail head core damage and metal fatigue). Each network extracts key features from the damage image data through convolution, pooling, and activation operations, providing input for subsequent feature fusion. Optionally, the first and second damage features are subjected to dimensionality reduction for further processing. The goal of feature dimensionality reduction is to reduce redundant information and computational complexity while retaining the key information that best describes the rail head damage. Dimensionality reduction methods include principal component analysis (PCA), autoencoders, and linear discriminant analysis (LDA). These methods can compress high-dimensional features into low-dimensional features, enhancing model efficiency and accuracy.

[0124] Step S904, bilinear feature fusion is performed on the first damage feature and the second damage feature. After obtaining the first damage feature and the second damage feature, the two features are bilinearly fused. In order to more effectively capture the interaction information between the two features, the outer product operation is used, and the fusion process is optimized by introducing a first weight coefficient. Specifically, a product operation is performed on each pair of feature elements of the first damage feature and the second damage feature to generate a high-dimensional feature matrix, that is, the intermediate fusion damage feature. The outer product operation can not only capture the simple relationship between features, but also reveal the complex interactions between them. In the outer product operation process, the weight coefficient is introduced, and by adjusting the weight of each feature, the more critical features have a greater influence in the final fusion. These weight coefficients are automatically adjusted by the back propagation algorithm during the training process to optimize the performance of the model.

[0125] Subsequently, the generated feature matrix is reduced in dimensionality through pooling, further reducing computational complexity while preserving key information. This pooling operation uses sum pooling to compress the high-dimensional feature matrix into a low-dimensional vector while retaining the most discriminative information. Optionally, a second weighting factor is introduced during sum pooling of the generated feature matrix to further enhance important feature information in the horizontal and vertical dimensions, thereby improving detection accuracy despite diverse injury types.

[0126] Step S905: Normalize the target fusion impairment feature. After completing the outer product operation and pooling, the feature vector is normalized. Normalization normalizes each element in the feature vector so that each feature contributes evenly to the final classification result. First, a normalized scalar of the target fusion impairment feature is calculated, optionally using the L2 norm. Normalization allows different features to be processed at the same scale, preventing the numerical range of a particular feature from being too large or too small, which could dominate the classification process.

[0127] Step S906: Damage classification is performed using the fully connected layer. The vectorized target fusion damage features are input into the fully connected layer for damage classification. In the fully connected layer, the feature vectors are processed through weighted summation and nonlinear activation, ultimately outputting the rail head damage category. This step optimizes weights and biases, enabling the model to effectively identify different types of damage, such as "normal weld," "abnormal weld," and "rail head core damage."

[0128] In this embodiment, the aforementioned steps extract rail head damage features from ultrasonic echo signals. Feature representation and classification are optimized through techniques such as bilinear feature fusion, outer product operations, pooling, and feature dimensionality reduction. This not only effectively identifies damage but also handles morphologically similar damage types, ensuring classification accuracy. Introducing weight coefficients in the feature fusion process allows the model to more flexibly adjust feature importance, thereby improving the ability to identify complex damage.

[0129] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0130] Based on the same inventive concept, embodiments of the present application also provide a bilinear model-based rail head damage classification device for implementing the bilinear model-based rail head damage classification method mentioned above. The solution provided by this device is similar to the solution described in the above-mentioned method. Therefore, the specific limitations of one or more bilinear model-based rail head damage classification device embodiments provided below can be found in the above-mentioned limitations of the bilinear model-based rail head damage classification method, and will not be repeated here.

[0131] In an exemplary embodiment, Figure 10 As shown, a rail head damage classification device based on a bilinear model is provided, comprising: a rail head damage image data acquisition module 101, a damage feature dual-branch extraction module 102, and a bilinear feature fusion and classification module 103, wherein:

[0132] The rail head damage image data acquisition module 101 is used to acquire the damage image data of the rail head and input it into the trained bilinear model;

[0133] a damage feature dual-branch extraction module 102 for extracting rail head damage features from the damage image data using a first convolutional neural network and a second convolutional neural network having different network structures in the bilinear model, respectively, to determine a first damage feature and a second damage feature of the rail head in different feature dimensions;

[0134] The bilinear feature fusion and classification module 103 is used to perform bilinear feature fusion on the first damage feature and the second damage feature through the bilinear model to obtain a target fused damage feature, input the target fused damage feature into the fully connected layer of the bilinear model for category determination, and determine the damage category corresponding to the rail head.

[0135] In one embodiment, the representation form of the first damage feature and the representation form of the second damage feature are both in vector form; the damage feature dual-branch extraction module 102 is also used to perform an outer product operation on the first damage feature in vector form and the second damage feature in vector form through the bilinear model to obtain an intermediate fused damage feature in matrix form; the intermediate fused damage feature is summed and pooled according to the vertical dimension or horizontal dimension of the matrix, and the target fused damage feature is determined based on the summed pooling processing result in vector form.

[0136] In one embodiment, the damage feature dual-branch extraction module 102 is also used to determine the normalized scalar corresponding to the sum pooling processing result in the vector form; based on the normalized scalar, the sum pooling processing result in the vector form is normalized to obtain the target fused damage feature.

[0137] In one embodiment, the damage feature dual-branch extraction module 102 is further configured to perform feature dimensionality reduction processing on the first damage feature and the second damage feature to obtain a first damage feature in vector form and a second damage feature in vector form.

[0138] In one embodiment, the rail head damage image data acquisition module 101 is further configured to acquire a plurality of echo signals corresponding to different ultrasonic incident angles of the rail head;

[0139] An image is constructed based on the waveform morphology change characteristics and echo intensity distribution characteristics of the multiple echo signals to obtain damage image data of the rail head.

[0140] In one embodiment, the rail head damage image data acquisition module 101 includes a bilinear model training module, which is used to obtain historical damage image data of the rail head with a known damage category label, and input the historical damage image data into the first convolutional neural network and the second convolutional neural network of the bilinear model to be trained to extract damage features, and perform bilinear feature fusion on the extracted training first damage features and training second damage features to obtain training target fused damage features; input the target fused damage features into the fully connected layer of the bilinear model to be trained to obtain a training damage category; according to the difference between the training damage category and the damage category label, the structural parameters of the first convolutional neural network, the second convolutional neural network and the fully connected layer of the bilinear model to be trained are updated, and when the training end condition is met, the trained bilinear model is obtained.

[0141] Each module in the aforementioned bilinear model-based rail head damage classification device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a computer device memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0142] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 11 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner. The wireless manner can be achieved through WIFI, a mobile cellular network, near field communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a rail head damage classification method based on a bilinear model.

[0143] Those skilled in the art will understand that Figure 11The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0144] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0145] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0146] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0147] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0148] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0149] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0150] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A rail head damage classification method based on a bilinear model, characterized in that: The method comprises: Obtaining damage image data of the rail head and inputting it into the trained bilinear model; Extracting rail head damage features from the damage image data using a first convolutional neural network and a second convolutional neural network having different network structures in the bilinear model, respectively, to determine first damage features and second damage features of different feature dimensions of the rail head; The first damage feature and the second damage feature are bilinearly fused through the bilinear model to obtain a target fused damage feature, and the target fused damage feature is input into the fully connected layer in the bilinear model for category determination to determine the damage category corresponding to the rail head.

2. The method according to claim 1, characterized in that The representation form of the first damage feature and the representation form of the second damage feature are both in vector form; The step of performing bilinear feature fusion on the first damage feature and the second damage feature through the bilinear model to obtain a target fused damage feature includes: Performing an outer product operation on the first damage feature in vector form and the second damage feature in vector form through the bilinear model to obtain an intermediate fused damage feature in matrix form; According to the vertical dimension or the horizontal dimension of the matrix, the intermediate fusion damage features are summed and pooled, and the target fusion damage features are determined according to the summed and pooled results in vector form.

3. The method according to claim 2, characterized in that Determining target fusion damage features based on the sum pooling processing results in vector form includes: Determine a normalized scalar corresponding to the sum pooling processing result in the vector form; The sum pooling processing result in the vector form is normalized based on the normalized scalar to obtain the target fusion damage feature.

4. The method according to claim 2, characterized in that The bilinear feature fusion of the first damage feature and the second damage feature using the bilinear model to obtain a target fused damage feature further includes: Performing feature dimensionality reduction processing on the first damage feature and the second damage feature to obtain a first damage feature in vector form and a second damage feature in vector form.

5. The method according to claim 1, wherein The obtaining of the damage image data of the rail head includes: Acquiring multiple echo signals corresponding to different ultrasonic incident angles of the rail head; An image is constructed based on the waveform morphology change characteristics and echo intensity distribution characteristics of the multiple echo signals to obtain damage image data of the rail head.

6. The method according to any one of claims 1 to 5, characterized in that The bilinear model is trained by the following steps: Obtain historical damage image data of a rail head with a known damage category label, input the historical damage image data into the first convolutional neural network and the second convolutional neural network of the bilinear model to be trained to extract damage features, and perform bilinear feature fusion on the extracted first training damage features and second training damage features to obtain a training target fused damage feature; Inputting the target fusion damage feature into the fully connected layer of the bilinear model to be trained to obtain a training damage category; According to the difference between the training damage category and the damage category label, the structural parameters of the first convolutional neural network, the second convolutional neural network and the fully connected layer of the bilinear model to be trained are updated, and when the training end condition is met, the trained bilinear model is obtained.

7. A rail head damage classification device based on a bilinear model, characterized in that: The device comprises: A rail head damage image data acquisition module is used to acquire the damage image data of the rail head and input it into the trained bilinear model; a damage feature dual-branch extraction module, configured to extract rail head damage features from the damage image data using a first convolutional neural network and a second convolutional neural network having different network structures in the bilinear model, respectively, to determine a first damage feature and a second damage feature of different feature dimensions of the rail head; A bilinear feature fusion and classification module is used to perform bilinear feature fusion on the first damage feature and the second damage feature through the bilinear model to obtain a target fused damage feature, input the target fused damage feature into the fully connected layer of the bilinear model for category determination, and determine the damage category corresponding to the rail head.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images