Railway wagon fault image intelligent identification data set construction method and device, electronic equipment and computer program product

Abstract

The invention provides a railway wagon fault image intelligent identification data set construction method and device, electronic equipment and a computer program product, and the method comprises the steps: obtaining a local screenshot of a typical fault of a wagon and a complete fault train monitoring image, and taking the local screenshot and the complete fault train monitoring image as an original railway wagon fault image; scoring the original rail wagon fault image to obtain an image quality score value, and cleaning the original rail wagon fault image based on the image quality score value to obtain an effective rail wagon fault image; carrying out image balancing on the effective railway wagon fault image to obtain a balanced fault image; classifying the effective railway wagon fault images to obtain fault classification labels and fault levels; component complete labeling and fault part local labeling are carried out on the equalization fault image to obtain fault labeling data; and based on the balanced fault image, the fault classification label set, the fault level set and the fault labeling data, constructing a railway wagon fault image intelligent identification data set.

Description

Technical Field

[0001] The present invention relates to the field of railway freight transportation technology, and in particular to a method, device, electronic equipment and computer program product for constructing a railway freight car fault image intelligent recognition data set. Background Art

[0002] The Truck Operation Fault Dynamic Image Detection System (TFDS for short) uses a high-speed image acquisition device on the trackside to collect images of the bottom and both sides of the truck body, and provides them to dynamic inspectors for analysis and inspection to prevent truck vehicle operation failures. It is an important part of the vehicle operation safety monitoring system. Up to now, the TFDS system has been widely deployed throughout the railway, with more than 540 sets of networked operation, and an average of more than 950,000 trucks are inspected per day. The huge amount of inspections has led to an extremely heavy workload for manual analysis and inspection. Therefore, in order to improve the efficiency of truck fault image inspection and reduce the workload of dynamic inspectors, it is particularly important to carry out TFDS truck fault image intelligent recognition.

[0003] At present, the image recognition method based on deep learning is an important technical solution to the problem of intelligent recognition of TFDS freight car fault images. The premise for conducting research on this technical route is to establish a scientific and complete image training and verification data set. Although a large amount of freight car image data has been accumulated after the long-term use of the TFDS system, these data have problems such as inconsistent storage format, uneven image quality, uneven distribution of fault samples, missing or inaccurate fault annotation, which greatly restricts the research effect of algorithms based on deep learning. Therefore, in response to the above problems, studying the construction technology of TFDS railway freight car fault image recognition data set has become an urgent task in current image recognition research.

[0004] The establishment of scientific and systematic image datasets provides solid data support for the research and progress of image processing technology. Since 2005, the field of image recognition has continuously produced research results on dataset establishment technology. Mark Everingham, Luc Van Gool and others launched the Computer Vision Challenge and announced the PASCAL VOC dataset; the Microsoft team researched and released the COCO dataset; the Google team researched and released the Open Image dataset, etc. These datasets contain a wide range of image categories and a large amount of data, which greatly accelerated the progress of image recognition technology, enabling computers to reach or even surpass human levels in general scenarios, and promoted major breakthroughs in artificial intelligence technology. However, for some specific industrial scenarios, the performance of the above general scenario datasets is slightly inferior.

[0005] In the industrial world, especially in the field of fault detection, a series of high-quality scene-specific datasets have also emerged. From the bearing remaining life prediction dataset provided by the IEEE PHM 2012 Data Challenge in 2012, to the XJTU-SY bearing dataset established by Professor Lei Yaguo's team from the School of Mechanical Engineering of Xi'an Jiaotong University in 2018, to the rotating machinery multimodal dataset constructed by Wonho Jung and Seong-Hu Kim in 2023, the rich fault and operating status data provide strong data support for the research on automatic detection of industrial faults and promote the process of industrial intelligence. However, due to the unique characteristic attributes of the truck body images collected by TFDS, it is impossible to simply rely on existing general or industrial image datasets when conducting research on its automatic fault identification technology.

[0006] Therefore, conducting research on the construction technology of railway freight car fault image intelligent recognition dataset can not only provide data support for the research on TFDS fault image intelligent recognition algorithm, but also establish an accurate and comprehensive evaluation benchmark for the evaluation and comparison of related algorithms. It is an important basic work for conducting TFDS fault image recognition research. Summary of the invention

[0007] In order to solve the above technical problems, the present application provides a method, device, electronic device and computer program product for constructing a railway freight car fault image intelligent recognition data set, so as to at least solve or alleviate the problems existing in the above-mentioned prior art.

[0008] A method for constructing a data set for intelligent recognition of railway freight car fault images, comprising:

[0009] Obtaining a typical partial screenshot of a freight car fault and a complete monitoring image of a faulty train as the original railway freight car fault image;

[0010] Scoring the original railway freight car fault image to obtain an image quality score value, and cleaning the original railway freight car fault image based on the image quality score value to obtain a valid railway freight car fault image;

[0011] Performing image balancing on the valid railway freight car fault image to obtain a balanced fault image;

[0012] Classifying the valid railway freight car fault images to obtain fault classification labels and fault levels;

[0013] Performing complete component labeling and partial labeling of fault locations on the balanced fault image to obtain fault labeling data;

[0014] Based on the balanced fault image and the fault classification label set, fault level set, and fault annotation data, a railway freight car fault image intelligent recognition data set is constructed.

[0015] A device for constructing a data set for intelligent recognition of railway freight car fault images, comprising:

[0016] A data acquisition unit, used to acquire a local screenshot of a typical fault of a freight car and a monitoring image of a complete faulty train as an original railway freight car fault image;

[0017] a cleaning unit, configured to score the original railway freight car fault image to obtain an image quality score value, and to clean the original railway freight car fault image based on the image quality score value to obtain a valid railway freight car fault image;

[0018] An image balancing unit, used for performing image balancing on the effective railway freight car fault image to obtain a balanced fault image;

[0019] A fault classification unit, used to classify the valid railway freight car fault image to obtain a fault classification label and a fault level;

[0020] A fault labeling unit, used for fully labeling components and partially labeling fault locations on the equalization fault image to obtain fault labeling data;

[0021] A construction unit is used to construct a railway freight car fault image intelligent recognition data set based on the balanced fault image and the fault classification label set, fault level set, and fault annotation data.

[0022] An electronic device comprises a memory and a processor, wherein a computer executable program is stored in the memory, and the processor runs the computer executable program to perform the following steps:

[0023] Obtaining a typical partial screenshot of a freight car fault and a complete monitoring image of a faulty train as the original railway freight car fault image;

[0024] Scoring the original railway freight car fault image to obtain an image quality score value, and cleaning the original railway freight car fault image based on the image quality score value to obtain a valid railway freight car fault image;

[0025] Performing image balancing on the valid railway freight car fault image to obtain a balanced fault image;

[0026] Classifying the valid railway freight car fault images to obtain fault classification labels and fault levels;

[0027] Performing complete component labeling and partial labeling of fault locations on the balanced fault image to obtain fault labeling data;

[0028] Based on the balanced fault image and the fault classification label set, fault level set, and fault annotation data, a railway freight car fault image intelligent recognition data set is constructed.

[0029] A computer program product having a computer executable program stored thereon, wherein the computer executable program is run to perform the following steps:

[0030] Obtaining a typical partial screenshot of a freight car fault and a complete monitoring image of a faulty train as the original railway freight car fault image;

[0031] Scoring the original railway freight car fault image to obtain an image quality score value, and cleaning the original railway freight car fault image based on the image quality score value to obtain a valid railway freight car fault image;

[0032] Performing image balancing on the valid railway freight car fault image to obtain a balanced fault image;

[0033] Classifying the valid railway freight car fault images to obtain fault classification labels and fault levels;

[0034] Performing complete component labeling and partial labeling of fault locations on the balanced fault image to obtain fault labeling data;

[0035] Based on the balanced fault image and the fault classification label set, fault level set, and fault annotation data, a railway freight car fault image intelligent recognition data set is constructed.

[0036] The construction method provided in this application specifically solves a series of technical problems existing in the TFDS truck fault image data mentioned in the background technology, such as the non-uniform storage format, uneven image quality, uneven distribution of fault samples, missing or inaccurate fault annotation, etc., which are discussed in detail as follows:

[0037] (1) First, we obtain local screenshots of typical truck faults and monitoring images of complete faulty trains as original railway truck fault images. This step is equivalent to sorting out data from the source. Images related to truck faults are collected and integrated. No matter how scattered or in different formats they were before, they are first uniformly included in the category of original images, laying the foundation for subsequent standardized processing.

[0038] Then, the original railway freight car fault images are scored to obtain image quality score values, and then the original images are cleaned based on the score values. For example, images with poor image quality and difficulty in clearly presenting fault features are judged as low-scoring images according to the set quality scoring standards and are screened out, thereby removing the poorer images with uneven quality and leaving only valid railway freight car fault images that meet the quality standards. This solves the problem of uneven image quality and, to a certain extent, also performs preliminary normalization of the data, alleviating the confusion caused by inconsistent storage formats, because the subsequent processing is all relatively standardized and valid images that have been screened.

[0039] (2) By performing image balancing operations on valid railway freight car fault images, balanced fault images are obtained. In actual freight car fault image data, there may be a large number of fault images of certain types, while the number of other relatively rare fault images is too small. This imbalance will affect the effect of deep learning algorithm training and lead to poor recognition of rare faults. In the image balancing process, methods such as data expansion (increasing the number of small fault samples through reasonable image transformation) or appropriate deletion of excessive samples will be adopted to make the number of images of different types of faults reach a relatively balanced state, so that the subsequent deep learning-based algorithm can comprehensively and fairly learn the characteristics of various faults during training, effectively solving the problem of uneven distribution of fault samples.

[0040] (3) In response to missing or inaccurate fault labels, this application first classifies valid railway freight car fault images, obtains fault classification labels and fault levels, clarifies key information such as the fault type and severity corresponding to the image, and prepares for subsequent accurate labeling. Then, the balanced fault image is fully labeled with components and partially labeled with fault locations to obtain fault labeling data. In this way, the integrity of each truck component involved in the image and the specific location of the fault are labeled in detail and accurately, making up for the previous lack of fault labeling. At the same time, through standardized labeling processes and requirements, the accuracy of labeling is greatly improved, so that the constructed data set can meet the requirements for accurate labeling data based on deep learning algorithms in terms of labeling, and provide a reliable data foundation for the subsequent research on intelligent recognition algorithms for railway freight car fault images.

[0041] (4) Based on balanced fault images and the organized fault classification label set, fault level set, and fault annotation data, a railway freight car fault image intelligent recognition dataset is constructed. This dataset has been optimized and improved in many aspects, such as image quality, sample balance, and annotation accuracy. It can not only provide strong data support for the research on the TFDS fault image intelligent recognition algorithm, but also because its construction process solves many problems that existed in the previous data, it can establish an accurate and comprehensive evaluation benchmark for the evaluation and comparison of related algorithms. It is in good agreement with the requirements of carrying out the important basic work of TFDS fault image recognition research, and effectively breaks through the previous constraints on the research on TFDS freight car fault image intelligent recognition due to data problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of a method for constructing a railway freight car fault image intelligent recognition data set according to an embodiment of the present application.

[0043] Figure 2 Examples of images with poor image quality and fault features that are difficult to clearly present for comparison with qualified images.

[0044] Figure 3 The following is a diagram of a migration example.

[0045] Figure 4 Schematic diagram of fault image samples of different forms obtained through amplification processing. DETAILED DESCRIPTION

[0046] Figure 1 FIG. 1 is a flow chart of a method for constructing a data set for intelligently identifying railway freight car fault images according to an embodiment of the present application. Figure 1 As shown, the method for constructing a data set for intelligent recognition of railway freight car fault images is characterized by comprising:

[0047] Obtaining a typical partial screenshot of a freight car fault and a complete monitoring image of a faulty train as the original railway freight car fault image;

[0048] Scoring the original railway freight car fault image to obtain an image quality score value, and cleaning the original railway freight car fault image based on the image quality score value to obtain a valid railway freight car fault image;

[0049] Performing image balancing on the valid railway freight car fault image to obtain a balanced fault image;

[0050] Classifying the valid railway freight car fault images to obtain fault classification labels and fault levels;

[0051] Performing complete component labeling and partial labeling of fault locations on the balanced fault image to obtain fault labeling data;

[0052] Based on the balanced fault image and the fault classification label set, fault level set, and fault annotation data, a railway freight car fault image intelligent recognition data set is constructed.

[0053] Figure 1 The solution provided by the embodiment of the present application has at least the following technical advantages:

[0054] (1) First, we obtain local screenshots of typical truck faults and monitoring images of complete faulty trains as original railway truck fault images. This step is equivalent to sorting out data from the source. Images related to truck faults are collected and integrated. No matter how scattered or in different formats they were before, they are first uniformly included in the category of original images, laying the foundation for subsequent standardized processing.

[0055] Then the original railway freight car fault image is scored to obtain an image quality score value, and then the original image is cleaned based on the score value. Figure 2 As shown in the figure, compared with the qualified image (a), there may be too bright images (b), too dark images (c), blurred images (d), etc., which lead to poor image quality and make it difficult to clearly present the fault features. According to the set quality scoring standard, they are judged as low-scoring images and screened out, thereby removing the poorer images with uneven quality, leaving only the valid railway freight car fault images that meet the quality standards, solving the problem of uneven image quality, and at the same time, preliminarily normalizing the data to a certain extent, alleviating the confusion caused by inconsistent storage formats, because the subsequent processing is all relatively standardized and valid images that have been screened.

[0056] (2) By performing image balancing operations on valid railway freight car fault images, balanced fault images are obtained. In actual freight car fault image data, there may be a large number of fault images of certain types, while the number of other relatively rare fault images is too small. This imbalance will affect the effect of deep learning algorithm training and lead to poor recognition of rare faults. In the image balancing process, methods such as data expansion (increasing the number of small fault samples through reasonable image transformation) or appropriate deletion of excessive samples will be adopted to make the number of images of different types of faults reach a relatively balanced state, so that the subsequent deep learning-based algorithm can comprehensively and fairly learn the characteristics of various faults during training, effectively solving the problem of uneven distribution of fault samples.

[0057] (3) In response to missing or inaccurate fault labels, this application first classifies valid railway freight car fault images, obtains fault classification labels and fault levels, clarifies key information such as the fault type and severity corresponding to the image, and prepares for subsequent accurate labeling. Then, the balanced fault image is fully labeled with components and partially labeled with fault locations to obtain fault labeling data. In this way, the integrity of each truck component involved in the image and the specific location of the fault are labeled in detail and accurately, making up for the previous lack of fault labeling. At the same time, through standardized labeling processes and requirements, the accuracy of labeling is greatly improved, so that the constructed data set can meet the requirements for accurate labeling data based on deep learning algorithms in terms of labeling, and provide a reliable data foundation for the subsequent research on intelligent recognition algorithms for railway freight car fault images.

[0058] (4) Based on balanced fault images and the organized fault classification label set, fault level set, and fault annotation data, a railway freight car fault image intelligent recognition dataset is constructed. This dataset has been optimized and improved in many aspects, such as image quality, sample balance, and annotation accuracy. It can not only provide strong data support for the research on the TFDS fault image intelligent recognition algorithm, but also because its construction process solves many problems that existed in the previous data, it can establish an accurate and comprehensive evaluation benchmark for the evaluation and comparison of related algorithms. It is in good agreement with the requirements of carrying out the important basic work of TFDS fault image recognition research, and effectively breaks through the previous constraints on the research on TFDS freight car fault image intelligent recognition due to data problems.

[0059] Optionally, the method further includes: accessing a typical fault local image library to obtain local screenshots of typical freight car faults manually identified during daily TFDS operation; accessing a faulty train whole vehicle image library to obtain complete faulty train monitoring images simulated by TFDS and actually discovered; and using the local screenshots of typical freight car faults and the complete faulty train monitoring images as original railway freight car fault images.

[0060] To this end, by accessing the typical fault local image library to obtain local screenshots of typical faults of freight cars, and accessing the fault train whole vehicle image library to obtain complete fault train monitoring images, image resources from multiple channels can be integrated. These images reflect the fault conditions of freight cars from two perspectives: local details and the overall condition of the whole vehicle, making the original railway freight car fault images more comprehensive and rich in content. For example, local screenshots can clearly show the subtle features of a specific component when it fails, such as wear marks on the axle surface, damage to a part of the brake device, etc.; while the whole vehicle monitoring image helps to understand the location of the fault in the whole train environment and its association with other components. The data set constructed in this way can better represent the various types of faults that may occur in freight cars in actual operation, and provide a more representative data basis for subsequent intelligent recognition algorithms, making them more accurate and reliable when facing fault detection in real scenarios.

[0061] Since it includes both simulated test images and actual discovered images, it takes into account both the theoretically preset fault conditions and the actual fault conditions that occur in actual operations, further enhancing the dataset's coverage of truck faults under different working conditions, avoiding the algorithm's adaptation to fault identification only in specific scenarios and improving its versatility.

[0062] In addition, TFDS has accumulated a large number of manually identified typical partial screenshots of freight car faults and complete fault train monitoring images during daily operation and simulation testing, which are valuable resources. By directly accessing the corresponding image library to obtain these images as raw data, the tedious process of re-collecting a large amount of data is avoided, saving manpower, material resources and time costs. Moreover, the use of existing images that have been manually screened or actually verified has a certain guarantee in data quality, reducing the workload of subsequent data cleaning and other links, which helps to more efficiently build a railway freight car fault image intelligent recognition data set and accelerate the progress of the entire TFDS fault image intelligent recognition research.

[0063] Furthermore, the sources of these images are closely related to the daily workflow of TFDS. Whether it is the local fault conditions identified manually or the whole vehicle fault conditions found in simulation tests and actual situations, they are all accumulated based on the actual business operations of the dynamic image detection system for truck operation faults. The data set constructed based on such data can better match the application scenarios of the TFDS system, and the trained intelligent recognition algorithm can also better meet the actual truck fault detection needs. When put into actual use, it can be more smoothly integrated into the existing truck operation safety monitoring system, and truly play a role in improving the efficiency of truck fault image inspection and reducing the workload of dynamic inspection personnel.

[0064] In a specific application scenario, the technical processing process for obtaining the original railway freight car fault image is as follows:

[0065] (1) Image library access rights and interface settings

[0066] First, you need to ensure that you have the authority to access the typical fault local image library and the faulty train image library, which involves the authority management of the system. Relevant technicians need to apply for appropriate access rights from the corresponding image library management department or system, such as logging in with a specific account and password, or performing authorization verification based on a unified identity authentication platform, to ensure the legitimacy and security of data acquisition.

[0067] At the same time, a stable interface should be established to achieve connection and access to the two image libraries. Standardized network interface protocols, such as RESTful API (Representational State Transfer Application Programming Interface), can be used to define the data format and method of requests and responses (such as GET method for obtaining image data, etc.), to ensure that the required typical fault partial screenshots of trucks and complete fault train monitoring images can be accurately obtained from the image library. For example, a request containing specific query parameters (such as fault type, time range, etc.) is sent to the interface of the typical fault partial image library. The interface queries the database according to the set logic and returns the partial screenshot image data that meets the conditions.

[0068] (2) Image format conversion and compatibility processing

[0069] Different image libraries may store images in different formats, such as JPEG and PNG, and the image resolution, color mode and other parameters may also be different. After acquiring the image, format conversion and compatibility processing are required to make it uniformly meet the requirements of the subsequent dataset construction process. Image processing libraries (such as OpenCV libraries in Python, etc.) can be used to implement format conversion functions, convert images of various formats to a preset standard format (such as JPEG format), and normalize the image resolution (for example, uniformly adjust to a specific width and height pixel value) to ensure that there will be no compatibility errors due to image format and size issues when performing subsequent operations such as image quality scoring and annotation.

[0070] (3) Data integrity check

[0071] In the process of acquiring images from the image library, data integrity verification is required to prevent image data corruption, partial content loss, etc. By calculating the checksum of the image (such as the MD5 checksum, etc.), the corresponding checksum information can be obtained at the same time when the image is acquired, and then the checksum of the acquired image is calculated again locally and compared with the original one. If there is inconsistency, it means that there may be a problem with the image data and it needs to be re-acquired or repaired. In addition, it is also possible to check whether the file header information of the image is complete and correct, and whether the pixel data of the image conforms to the corresponding format specifications, etc., to ensure that each local screenshot of a typical truck fault and the complete fault train monitoring image obtained are complete and usable original railway truck fault images, providing reliable data guarantee for the subsequent construction of high-quality data sets.

[0072] Optionally, scoring the original railway freight car fault image to obtain an image quality score value includes: scoring the original railway freight car fault image based on a constructed image quality evaluation model to obtain an image quality score value.

[0073] Optionally, the image quality assessment model includes a feature extraction network, a rule mapping network, and a quality assessment network; the scoring of the original railway freight car fault image based on the constructed image quality assessment model to obtain an image quality score value includes: based on the feature extraction network, performing semantic feature extraction on the original railway freight car fault image to obtain image semantic features; based on the rule mapping network, mapping the image semantic features to a pre-constructed quality assessment rule description to obtain a rule mapping result; based on the quality assessment network, processing the rule mapping result to obtain the image quality score value.

[0074] To this end, based on the constructed image quality evaluation model, the original railway freight car fault image is scored to obtain an image quality score value, which has the following technical benefits:

[0075] (1) The specially constructed image quality evaluation model is designed for a specific type of image, namely, railway freight car fault images. Different from the general image quality evaluation method, it fully considers the characteristics of freight car fault images, such as the complexity of the structure of freight car components, the diversity of fault characteristics, and the special requirements for image quality in actual detection scenarios. Therefore, it can more accurately evaluate the quality of these images and provide a basis for image screening that meets the requirements for the subsequent construction of a high-quality railway freight car fault image intelligent recognition dataset.

[0076] The model provides a standardized process and method for image quality scoring, avoiding the inconsistency and arbitrariness that may occur when manually judging image quality. No matter which operator uses this model to score the image, as long as the same original railway freight car fault image is input, a relatively objective and stable image quality score value can be obtained according to the established algorithm and parameter settings of the model, ensuring the scientificity and fairness of the image screening process in the entire data set construction process.

[0077] In the actual collection process, railway freight car fault images will present extremely complex and diverse characteristics due to various factors such as the shooting environment (such as light, angle, weather, etc.), the status of the freight car itself (different models, different fault types and locations, etc.). The constructed model can comprehensively consider these complex factors and comprehensively evaluate the image quality by virtue of the collaborative work of multiple networks within it, accurately distinguish which images are high-quality images that can truly provide effective information for subsequent intelligent recognition algorithms, and improve the overall quality and effectiveness of the data set.

[0078] (2) By extracting semantic features from the original railway freight car fault images, we can break through the limitations of the pixel information on the image surface and dig out the deep semantic connotation. For example, we can not only identify the shape and outline of a certain freight car component in the image, but also understand the meaning of the component in the fault scenario, such as the appearance of a certain texture change on the axle means that there may be a wear fault. This will help to evaluate the quality of the image more accurately based on the fault semantic information conveyed by the image, rather than just based on simple standards such as whether the image appearance is clear.

[0079] Semantic feature extraction can focus on key features related to truck failures and filter out some irrelevant background or interference information in the image. For example, in an image containing an entire train of trucks, the fault location (such as semantic features related to the brake failure at the bottom of the truck) can be accurately extracted, and irrelevant information such as the track and surrounding environment can be ignored. This allows subsequent quality evaluation to consider image quality based on these key fault features, avoiding incorrect judgment of whether the image meets the requirements due to interference from irrelevant information, and improving the accuracy of quality scoring.

[0080] Due to the wide variety of railway freight car faults, the presentation of different faults in images varies greatly. Semantic feature extraction can adaptively capture the unique feature representations of various fault conditions, whether it is obvious component damage, slight wear, or some hidden faults, etc., they can be effectively characterized by extracting semantic features, thereby allowing the entire image quality evaluation model to flexibly respond to a variety of fault images, improving its versatility and robustness.

[0081] (3) Pre-constructed quality evaluation rule descriptions are usually formulated based on professional image quality requirements, truck fault detection experience, and relevant technical standards. By mapping the extracted image semantic features to the rule mapping network, the features actually presented by the image can be closely combined with the professional quality evaluation standards. For example, the rule stipulates that images that can clearly present key fault locations and have identifiable texture features are considered high-quality images. Then, through mapping, it can be accurately determined whether an image that extracts the semantic features of axle wear meets this standard, making the quality evaluation more professional and authoritative.

[0082] After mapping the image semantic features to the rule description, we get the rule mapping result, which can intuitively reflect the degree of fit between the image and the quality evaluation rule, which is equivalent to a quantitative representation of the image quality at the rule level. For example, the rule mapping results such as the similarity matrix obtained, the higher the value, the more the image meets the quality evaluation rule requirements, and the more likely it is to be a high-quality image suitable for subsequent data set construction, which facilitates the subsequent accurate quality score value based on the quantitative result, and achieves more refined image quality distinction.

[0083] Different railway freight car fault images have different semantic features due to fault type, shooting angle, etc. The rule mapping network uses a unified quality evaluation rule description as a reference to provide a unified evaluation scale for these various images, ensuring that no matter what fault features the image presents, quality evaluation can be performed under the same rule framework, avoiding confusion in evaluation standards due to feature differences, and ensuring consistency and standardization of the entire image quality evaluation process.

[0084] (4) The quality evaluation network works together through multiple modules (such as vector conversion module, quality mining module, output module, etc.) to comprehensively consider and deeply process the various information contained in the rule mapping results. It does not only rely on simple indicators such as the similarity between a single image and a rule, but also comprehensively analyzes the image quality through operations such as converting the rule mapping results into quality representation tensors and performing multi-layer full-connection processing to mine deeper quality features. It finally gives an accurate score value that can comprehensively reflect the quality status of the image in multiple dimensions, making the scoring result more comprehensive and reliable.

[0085] The design of the quality assessment network is closely coordinated with the previous feature extraction network and rule mapping network to form a complete logical system of the image quality assessment model. It can process the intermediate results obtained by the previous network in a way that meets the overall model objectives and output a suitable quality score value. The form and range of the score value can well adapt to the subsequent cleaning and screening of the original railway freight car fault images, making it easy to directly perform effective image quality control based on this score value in the data set construction process to ensure the high-quality construction of the data set.

[0086] The image quality score obtained through a series of standardized processing steps of the quality evaluation network has a clear and traceable generation process, and the operation of each module gives the score a certain degree of interpretability. At the same time, because the score is generated according to a unified network architecture and processing logic, the scores between different images are highly comparable, which is convenient for operators or subsequent algorithms to intuitively judge the quality of images based on the score, make image selections, and control the overall quality of the data set.

[0087] Optionally, the feature extraction network includes: a convolution module, a pooling module, and an attention module; based on the feature extraction network, the semantic feature extraction of the original railway freight car fault image is performed to obtain image semantic features, including: based on the convolution module, sliding convolution is performed on the original railway freight car fault image to extract local features of the original railway freight car fault image and generate a local feature map of the fault image accordingly; based on the pooling module, downsampling the local feature map of the fault image is performed to obtain a pooled fault feature map; based on the attention module, the following processing is performed to obtain the image semantic features: multi-channel pooling processing is performed on the pooled fault feature map to obtain a corresponding plurality of fault feature vectors; dimensionality reduction processing is performed on the plurality of fault feature vectors to obtain fault feature intensity vectors; nonlinear transformation is performed on each fault feature intensity vector to obtain a fault importance feature vector; spatial dimension transformation is performed on each fault importance feature vector to obtain a corresponding spatial feature map; multi-head attention analysis is performed on the spatial feature map to obtain a self-attention weight, and the spatial feature map is fused based on the self-attention weight to obtain the image semantic features.

[0088] Optionally, the rule mapping network includes a feature embedding module, a rule representation module, a rule matching module, and an attention module; based on the rule mapping network, the image semantic features are mapped to a pre-constructed quality evaluation rule description to obtain a rule mapping result, including: based on the feature embedding module, the image semantic features are transformed into a created fault semantic space to obtain a fault semantic vector; based on the rule representation module, the pre-constructed image quality evaluation rules are encoded to obtain an evaluation rule encoding vector; based on the rule matching module, the evaluation rule encoding vector is matched with the fault semantic vector to obtain a similarity matrix; based on the attention module, the similarity matrix and the fault semantic vector are subjected to attention analysis to obtain an attention weight matrix as the rule mapping result.

[0089] Optionally, the quality assessment network includes a vector conversion module, a quality mining module, and an output module; based on the quality assessment network, the rule mapping result is processed to obtain the image quality score value, including: based on the vector conversion module, the rule mapping result is converted into a quality representation tensor; based on the quality mining module, the quality representation tensor is cascaded with multi-layer full connection processing to generate an image quality feature representation; based on the output module, the image quality feature representation is linearly transformed and multiplied with a set scoring weight matrix to obtain the image quality score value.

[0090] In a specific application scenario, based on the constructed image quality evaluation model, the original railway freight car fault image is scored to obtain the image quality score value. When the network design is as follows:

[0091] (1) Feature extraction network

[0092] Convolutional module

[0093] Convolution kernel initialization first requires determining parameters such as the size, number, and step length of the convolution kernel. For example, the convolution kernel size can be set to 3×3, the number to 64 (which means 64 different feature maps will be generated), and the step length to 1. This setting is convenient for effectively capturing local features on the image without losing too much detail information. The initial weight values ​​of these convolution kernels are usually assigned according to random initialization or based on a specific initialization algorithm (such as Xavier initialization, He initialization, etc.) to ensure that the network can start training and converge smoothly.

[0094] The sliding convolution operation slides the initialized convolution kernel on the original railway freight car fault image according to the set step size. At each sliding position, the convolution kernel performs a convolution operation with the pixel value of the corresponding area of ​​the image (i.e., the corresponding elements are multiplied and summed), thereby extracting the features of the local area. For example, for a freight car fault image of size 256×256, the convolution kernel starts from the upper left corner and slides to the right and downward in sequence. Convolution calculation is performed on each small 3×3 area to generate the corresponding eigenvalues. After the sliding convolution of the entire image, 64 different local feature maps of the fault image are obtained. Each feature map represents the local feature information of the image under different convolution kernels.

[0095] After the convolution operation, the activation function (such as ReLU activation function) is usually applied to perform nonlinear transformation on the convolution result to enhance the network's expressiveness so that the extracted features can better reflect the nonlinear characteristics of the image. For example, the part of the eigenvalue after the convolution operation that is less than 0 is set to 0, and the part greater than 0 is retained, so as to highlight the more significant local features in the image.

[0096] Pooling module

[0097] Pooling method selection Common pooling methods include maximum pooling and average pooling. Here you can choose the maximum pooling method, which operates on the local feature map of the fault image by defining a pooling window size (such as 2×2) and a pooling step size (such as 2).

[0098] Taking the maximum pooling as an example of downsampling operation, the pooling window slides on the local feature map of the fault image, and each time the pixel with the largest pixel value in the window is selected as the output, thereby reducing the size of the original feature map by half (if the pooling window is 2×2 and the step size is 2), and the purpose of downsampling is achieved. For example, a 16×16 local feature map of a fault image will obtain an 8×8 pooled fault feature map after a 2×2 maximum pooling operation. This process reduces the amount of data while retaining the key local features of the image, reduces the complexity of subsequent calculations, and also makes the features have a certain degree of translation invariance, which helps to improve the robustness of the features.

[0099] Attention Module

[0100] Multi-channel pooling is performed on the pooled fault feature map. The multi-channels here correspond to the multiple feature map channels generated by the previous convolution module (such as the 64 channels mentioned above). For the pooled fault feature map of each channel, global average pooling or global maximum pooling is used to convert the two-dimensional feature map into a one-dimensional fault feature vector. In this way, multiple fault feature vectors corresponding to the number of channels are obtained, and each vector summarizes the feature information of the image from different channel perspectives.

[0101] Dimensionality reduction processing In order to reduce the dimension of feature vectors and avoid problems such as excessive subsequent calculations and overfitting, principal component analysis (PCA), linear discriminant analysis (LDA) or fully connected layers can be used to perform dimensionality reduction processing on multiple fault feature vectors to obtain fault feature intensity vectors, making the features more compact and retaining key information. For example, a fault feature vector with a higher dimension (such as 100 dimensions) can be reduced to a 20-dimensional fault feature intensity vector.

[0102] Nonlinear transformation applies nonlinear activation functions (such as Sigmoid function, Tanh function, etc.) to the fault feature intensity vector after dimensionality reduction to obtain the fault importance feature vector. Through this nonlinear transformation, the complex relationship between features is further explored, and the feature components that are more important for judging the image semantics are highlighted. For example, the Sigmoid function can map the value of the feature intensity vector to between 0 and 1, reflecting the importance of different feature components.

[0103] Spatial dimension transformation and multi-head attention analysis perform spatial dimension transformation on each fault importance feature vector and restore it to a two-dimensional spatial feature map, which is convenient for subsequent analysis of spatial relationships. Then, a multi-head attention mechanism is used to set the number of heads (e.g., 8 heads). Each head performs self-attention calculation on the spatial feature map based on different weight matrices to obtain self-attention weights under different heads. Finally, these self-attention weights are fused (e.g., concatenated and then linearly transformed) to perform weighted fusion on the spatial feature map based on the fused self-attention weights, so that the network can focus on more critical semantic areas in the image, thereby obtaining the final image semantic features. This method can adaptively capture the correlation between different positions and different features of the image and strengthen the extraction of fault semantic information.

[0104] (2) Rule Mapping Network

[0105] The rule mapping network is responsible for mapping and associating the image semantic features obtained from the feature extraction network with the pre-built quality evaluation rule description. It includes a feature embedding module, a rule representation module, a rule matching module, and an attention module. Through the collaborative work of these modules, a rule mapping result is generated that can reflect the degree of fit between the image and the rules.

[0106] Feature Embedding Module

[0107] Spatial transformation and embedding receives the image semantic features from the feature extraction network, and first transforms the image semantic features from their original feature space to the created fault semantic space through a fully connected layer or a specific embedding algorithm (such as Word2Vec-like ideas applied to the image feature space) to obtain a fault semantic vector. This process is equivalent to giving the image semantic features a new representation in the semantic space corresponding to the quality evaluation rule, which is convenient for subsequent matching and comparison with the rule description. For example, the image semantic features with an original dimension of 100 are converted into a fault semantic vector with a dimension of 50 in the fault semantic space through a fully connected layer, so that it can better adapt to the vector form output by the rule representation module.

[0108] Rule representation module

[0109] Rule encoding initialization For the pre-built image quality evaluation rule description (usually in the form of text or structured rule statements), it is necessary to encode it so that it can be processed in the network. The word vector model (if the rule is in text form) or the custom encoding method (for structured rules) can be used to give each rule an initial vector representation. For example, the rule "the texture of the fault part in the image is clearly identifiable" can be encoded as an evaluation rule encoding vector of a fixed dimension (such as 50 dimensions). Each element value in the vector represents the feature representation of the rule in different dimensions.

[0110] The rule vector is updated by training the network (image samples with labeled quality levels can be combined with corresponding rules for supervised learning), and the weight value of the evaluation rule encoding vector is continuously adjusted to make it more accurately reflect the essential meaning of the rule and its association with image features. After multiple rounds of training, the final evaluation rule encoding vector that can effectively represent the rule is obtained. These vectors will be used to match the fault semantic vector later.

[0111] Rule matching module

[0112] Similarity calculation matches the fault semantic vector from the feature embedding module with the evaluation rule encoding vector from the rule representation module. The commonly used matching method is to calculate the similarity between the two, such as using the cosine similarity calculation method. For each fault semantic vector and the corresponding evaluation rule encoding vector, the cosine value of the angle between them is calculated as the similarity value, so that a similarity matrix can be obtained. Each element in the matrix represents the similarity between a certain image semantic feature and a certain quality evaluation rule. The value range is usually between -1 and 1. The higher the value, the more similar it is and the more it meets the requirements of the corresponding rule.

[0113] Attention Module

[0114] Attention weight calculation is based on the obtained similarity matrix and fault semantic vector for attention analysis. Here, a content-based attention mechanism can be used. By defining a suitable weight calculation function (such as Softmax function, etc.), according to the value of each element in the similarity matrix and the relevant information of the fault semantic vector, different attention weights are assigned to the association between each fault semantic vector and the rule, and an attention weight matrix is ​​generated. This matrix is ​​the rule mapping result, which can highlight the importance of the image semantic features under different rules, that is, it reflects the focus and compliance of the image in various quality evaluation rules.

[0115] (3) Quality Evaluation Network

[0116] The quality evaluation network takes the rule mapping result (attention weight matrix) obtained by the rule mapping network as input, and undergoes a series of processing such as the vector conversion module, quality mining module, and output module to finally generate a score value that can accurately measure the image quality and comprehensively evaluate whether the image meets the requirements of the dataset construction.

[0117] Vector conversion module

[0118] After the tensor transformation operation receives the rule mapping result (attention weight matrix), it is first converted into a quality representation tensor through tensor operations such as reshaping and splicing, so that it has a structural form suitable for subsequent network layer processing. For example, if the attention weight matrix is ​​two-dimensional, it is reshaped into a three-dimensional tensor form according to the input requirements of the fully connected layer in the subsequent quality mining module, and it can be normalized (such as using batch normalization and other methods) to make the data distribution more in line with the requirements of network training, which is convenient for subsequent more stable mining of quality features.

[0119] Quality Mining Module

[0120] Multi-layer fully connected processing performs cascaded multi-layer fully connected processing on the quality characterization tensor. By setting multiple fully connected layers (such as setting 3 fully connected layers, the number of neurons in each layer decreases successively, such as 128, 64, 32, etc.), each fully connected layer performs a linear transformation (i.e., matrix multiplication operation) on the input data and then applies an activation function (such as ReLU activation function) for nonlinear transformation, gradually excavating a deeper level of image quality feature representation, and extracting features that can better reflect the overall quality status of the image from the original rule mapping related information. After multi-layer information fusion and feature extraction, a more representative image quality feature representation is generated, so that it can comprehensively consider the comprehensive performance of the image under multiple quality evaluation rules.

[0121] Output Module

[0122] Linear transformation and score generation Finally, the image quality feature representation obtained by the quality mining module is linearly transformed (implemented through a fully connected layer) and converted into a one-dimensional vector form, which is then multiplied by the set score weight matrix. This score weight matrix is ​​learned based on image samples with labeled quality levels during the network training process. The final image quality score is obtained by multiplying the two, for example, a score value between 0 and 100 is obtained, representing the degree of image quality. The higher the value, the better the image quality, and the more it meets the image quality requirements of the subsequent dataset construction.

[0123] Optionally, classifying the valid railway freight car fault images to obtain fault classification labels and fault levels includes: classifying the valid railway freight car fault images according to railway freight car maintenance procedure description data and historical fault data of railway freight cars to obtain fault classification labels and fault levels.

[0124] Optionally, the classifying the valid railway freight car fault image according to the railway freight car maintenance procedure description data and the railway freight car historical fault data to obtain a fault classification label and a fault level includes: extracting fault description terms from the railway freight car maintenance procedure description data to construct a fault rule dictionary according to the extracted fault description terms, wherein each entry in the fault rule dictionary is constructed with a fault classification label and an initial clue of the fault level; cleaning the railway freight car historical fault data to extract fault descriptions therefrom and establishing an association relationship between the fault description and the historical fault image, wherein the fault description includes at least one of a location where the fault occurs, a type of fault, and a fault frequency; and matching the valid railway freight car fault image with the fault rule dictionary based on the association relationship to obtain a fault classification label and a fault level.

[0125] To this end, the above-mentioned technical processing of classifying the valid railway freight car fault images according to the railway freight car maintenance procedure description data and the historical fault data of the railway freight car to obtain the fault classification label and fault level has the following technical advantages:

[0126] (1) The railway freight car maintenance procedure description data itself is a professional guidance document summarized based on the long-term operation and maintenance practice of railway freight cars, covering the standard definitions and judgment rules related to various types of faults. Combined with the historical fault data of railway freight cars, it can fully take into account the actual faults that have occurred in actual operation. Based on this, the effective railway freight car fault images are classified to ensure that the classification results are closely aligned with the actual business scenarios and professional technical standards of railway freight cars, so that the subsequently constructed data sets and the intelligent recognition algorithms trained based on the data sets can meet the needs of actual fault detection and maintenance work.

[0127] The comprehensive use of these two data sources avoids the one-sidedness and errors that may occur when relying solely on single data or subjective judgment for classification. The maintenance procedure description data provides a theoretically standardized and accurate fault classification framework, while the historical fault data records and verifies the fault conditions from the perspective of actual occurrence. The two verify and complement each other, and can more accurately classify the fault conditions reflected in the image, thereby improving the accuracy and reliability of fault classification labels and fault level determination, and providing a high-quality annotation data foundation for subsequent intelligent recognition algorithms.

[0128] Since the classification is based on actual business and historical experience, the fault classification labels and corresponding fault level information contained in the constructed data set can better reflect the diversity and complexity of freight car faults in the real world. Such a data set can play a better role when applied to the intelligent recognition of freight car fault images on different railway lines and in different operating environments. It is more versatile and helps to promote the effective application of railway freight car fault detection technology in a wider range, effectively reduce the workload of dynamic inspection personnel and improve overall work efficiency.

[0129] (2) By extracting fault description terms and constructing a fault rule dictionary, it is equivalent to sorting out and normalizing the scattered textual fault descriptions in the maintenance procedures, providing a set of standardized rule systems for subsequent classification work. The fault classification label and initial clues of the fault level constructed by each entry clearly define the position and belonging relationship of different faults in the classification system, so that there is a basis for classifying fault images and judgments are made according to unified standards, avoiding the classification inconsistency caused by different personnel's understanding of the procedures, and ensuring the scientificity and standardization of the classification process.

[0130] As a reference index, the fault rule dictionary can quickly locate the possible corresponding fault classification labels and hierarchical information when matching valid railway freight car fault images. For example, when the fault characteristics of a specific component are shown in the image, by looking up the corresponding entries of the related fault description terms in the dictionary, the preliminary classification clues can be quickly obtained, and then further combined with other information for accurate classification, which greatly improves the efficiency of classification. Especially when faced with a large number of fault images that need to be classified and labeled, this guiding role is more obvious, which helps to speed up the progress of the entire data set construction.

[0131] Although the procedure description data is relatively fixed, the method of extracting terms to build a dictionary can flexibly respond to new fault conditions or more detailed fault classification requirements to a certain extent. When a new fault feature description is found, its corresponding term can be added to the dictionary, and the corresponding classification label and hierarchical clues can be assigned according to a certain logic, so that the fault classification system can keep pace with the times and continuously improve, better cover various possible truck fault types, and maintain the adaptability and extensibility of the classification rules.

[0132] (3) Historical fault data is often accumulated in large quantities during actual operation, which may contain some inaccurate, incomplete or redundant information. Through cleaning operations, these noise data can be removed and truly useful fault description information can be extracted, such as the location, type, frequency and other key contents of the fault. In this way, when establishing the association relationship with the historical fault image in the future, it is possible to focus on the relationship between the core fault features and the corresponding image, making the association more accurate and clear, providing reliable data support for accurately matching effective railway freight car fault images and fault rule dictionaries, and avoiding classification errors caused by historical data quality problems.

[0133] Analyzing the correlation between fault descriptions and historical fault images can help us discover some implicit rules in the occurrence, development, and presentation characteristics of faults in actual operations. For example, certain types of faults always occur frequently in specific truck component locations, or the frequency of occurrence of specific faults has a certain trend over time. These rules can serve as additional auxiliary information to help determine which fault classification label and corresponding fault level a new valid railway truck fault image is more likely to belong to when classifying it, further improving the accuracy of classification, and also providing valuable reference for in-depth understanding of the actual situation of truck faults.

[0134] Based on the association established after cleaning the data, the classification process not only relies on the theoretical rules in the maintenance procedures, but also fully considers the actual historical fault conditions, realizing a data-driven classification method. This method can better capture the fault characteristics and classification connections that actually exist in actual business but may not be described in detail in the procedures, making the classification results more realistic, making up for the possible shortcomings of simple rule-driven classification, and improving the entire classification process's ability to process fault images in different actual scenarios.

[0135] (4) Matching valid railway freight car fault images with fault rule dictionaries through association relationships is actually combining the standard fault classification rules obtained from maintenance procedures with the actual fault characteristic rules mined from historical fault data, and comprehensively considering various information to determine the fault classification label and fault level corresponding to each image. In this way, the advantages of both can be fully utilized to more accurately classify the complex and diverse fault conditions presented in the images. Whether it is a common fault or some relatively special and rare faults, it can be classified into the appropriate classification category as accurately as possible, thereby improving the comprehensiveness and accuracy of the classification.

[0136] Matching and classification are performed under the unified association relationship and fault rule dictionary framework, ensuring that image classification work by different operators or batches can follow the same process and standards, and the classification results are consistent and coherent. This is crucial for building a high-quality, standardized intelligent recognition dataset for railway freight car fault images, making the fault classification labels and level information in the dataset reliable and stable, facilitating subsequent algorithm development, training, and evaluation based on the dataset, and ensuring the stability and effectiveness of the entire intelligent recognition system.

[0137] Accurate fault classification labels and fault levels are very important annotation information for fault images. The high-quality annotation data obtained based on the above matching method can enable the intelligent recognition algorithm to better learn the characteristic manifestations of different fault types and their severity in the image, and improve the algorithm's classification and recognition capabilities for fault images. In turn, in actual applications, it can more accurately judge the truck fault situation, achieve efficient fault detection and early warning, and give full play to the practical value of building data sets and developing intelligent recognition technology.

[0138] In a specific application scenario, the technical implementation process of classifying the valid railway freight car fault images according to the railway freight car maintenance procedure description data and the historical fault data of the railway freight car to obtain the fault classification label and fault level is as follows:

[0139] (1) Extract fault description terms from railway freight car maintenance procedure description data and construct a fault rule dictionary

[0140] Data acquisition and preprocessing:

[0141] First, we need to obtain the railway freight car operation and maintenance procedures description data, which usually exists in the form of documents (such as Word, PDF format, etc.). It is necessary to extract its content through the corresponding text reading tool and convert it into plain text format to facilitate subsequent text processing operations.

[0142] Preprocess the extracted plain text, such as removing punctuation marks, extra spaces, line breaks and other irrelevant characters in the text, unifying the case of the text (generally converting it to lowercase), and organizing the text into a standardized format to facilitate subsequent term extraction operations.

[0143] Fault description term extraction method:

[0144] Rule-based method: Terms can be extracted according to some pre-set grammatical rules and customary expressions in professional fields. For example, noun phrases (usually composed of adjective + noun or noun + noun, etc.) are defined as possible fault description terms, and the maintenance procedure text is tagged with part-of-speech tags using part-of-speech tagging tools (such as part-of-speech tagging functions in natural language processing libraries), and then phrases that meet the set rules are screened out as candidate terms. For example, "brake wear" and "axle cracks" are extracted according to the rule of "component + fault phenomenon".

[0145] Statistical method: Calculate the frequency of occurrence of words in the text, the co-occurrence frequency between words, and other statistical information, and extract the words or phrases that appear frequently and are related to the fault as fault description terms. You can use word frequency statistics tools (such as the Counter class in Python, etc.) to count the number of times each word appears in the text, pay special attention to and extract those frequent word combinations that are meaningful in the context of railway freight car faults, such as "wheel tread scratches" that are mentioned many times in the regulations, which can be easily screened out.

[0146] Machine learning method: Use labeled fault description term sample data (if there are a small number of manually labeled examples) to train a classification model, such as support vector machine (SVM), convolutional neural network (CNN) text classification model in deep learning, etc., input the maintenance procedure text into the model, and let the model determine whether each word or phrase belongs to the fault description term, in this way, automatically extract relevant terms.

[0147] Build the fault rule dictionary:

[0148] For each fault description term extracted, a corresponding entry is created. In each entry, a fault classification label is assigned based on the overall framework of fault classification in railway freight car expertise and maintenance procedures, for example, "brake wear" is classified under the classification label "brake system failure"; at the same time, the initial clues of the fault level are determined, for example, "brake system failure" can be set as the first-level fault category, and "brake wear" belongs to the second-level specific fault type under it, so as to build a hierarchical fault rule dictionary.

[0149] A suitable data structure can be used to store the fault rule dictionary, such as the dictionary data structure in Python, with the fault description term as the key and the corresponding structure composed of fault classification labels, fault levels and other information (such as a dictionary or class instance containing classification labels, levels and other fields) as the value, to facilitate subsequent query and matching operations.

[0150] (2) Clean the historical fault data of railway freight cars and establish correlation relationships

[0151] Data cleaning operation details:

[0152] Unified data format: The historical fault data of railway freight cars may come from various sources, such as database records, Excel tables, and electronic texts of paper records. First, the data from these different sources must be in a unified format, for example, they must all be converted into structured database tables (such as MySQL, SQLite, and other databases), and the table structure must be defined, including fields such as the time of the fault, freight car number, location of the fault, fault type, and fault description, to facilitate subsequent centralized processing.

[0153] Missing value processing: Check whether there are missing values ​​in each field. For missing values ​​of key fields (such as fault type, fault location, etc.), a reasonable filling method can be used. If it is a numerical field (such as fault frequency-related data), it can be filled according to statistical values ​​such as mean and median; for text fields (such as fault description), reasonable speculation and supplement can be made based on other records of the same type of fault or combined with railway freight car professional knowledge, or marked as "unknown" for further manual verification.

[0154] Outlier processing: Identify and process outliers by setting a reasonable range or based on statistical analysis methods (such as the 3σ principle, etc.). For example, if the fault frequency is extremely high or extremely low and obviously exceeds the normal fluctuation range, it is necessary to verify that it may be caused by recording errors. These abnormal data need to be corrected or eliminated to ensure the rationality and reliability of the data.

[0155] Duplicate data processing: Use the database's duplicate checking function or write code logic (such as comparing whether the values ​​of each field are exactly the same) to find and delete duplicate historical fault records, retain a complete and accurate record, and avoid duplicate data from interfering with subsequent analysis and the establishment of association relationships.

[0156] Extract fault descriptions and establish associations:

[0157] Fault description extraction: Accurately extract fault description related information from the cleaned historical fault data, focusing on key content such as the location of the fault, the type of fault, and the frequency of the fault. This information can be obtained through database query statements (such as SELECT statements in SQL) or written code logic (such as traversing the data structure in Python to extract the corresponding field content), and organized into a format that is easy to analyze, such as summarizing the location and frequency of faults of different trucks under the same fault type.

[0158] Establish association relationship: associate the extracted fault description information with the corresponding historical fault image. One feasible implementation method is to add a foreign key field to the historical fault image table in the database, pointing to the primary key of the historical fault data table (such as the fault record number), and reflect the association between them through this database relationship; or build a mapping relationship data structure in the data processing code (such as a dictionary structure, with the historical fault image file name or number as the key and the corresponding fault description structure as the value) to ensure that each historical fault image can correspond to its accurate fault description information, providing a basis for subsequent matching with valid railway freight car fault images.

[0159] (3) Match the valid railway freight car fault image with the fault rule dictionary based on the association relationship to obtain the fault classification label and fault level

[0160] Image feature extraction and representation:

[0161] First, it is necessary to extract features from valid railway freight car fault images so that they can be matched with the term descriptions in the fault rule dictionary. Common image feature extraction methods in the field of computer vision can be used, such as using a convolutional neural network (CNN) model (such as the classic VGG, ResNet and other model structures), inputting valid railway freight car fault images into a pre-trained CNN model (which can be pre-trained on a large-scale general image dataset and then fine-tuned according to the characteristics of railway freight car fault images), and extracting feature vector representations of the images. These feature vectors can reflect key information such as freight car components and fault morphology in the image to a certain extent, which is convenient for subsequent matching operations.

[0162] Alternatively, traditional manual feature extraction methods can be used, such as extracting the image's color histogram, texture features (such as gray-level co-occurrence matrix features), shape features (such as contour-based shape descriptors), etc., and combining these manual features of different dimensions into a feature vector to characterize each valid railway freight car fault image. Although manual features may be slightly weaker in expressiveness than deep features, they can also play an effective matching role in some specific simple scenarios.

[0163] Matching algorithm and logic:

[0164] Similarity calculation and matching: For each valid railway freight car fault image with extracted features, calculate the similarity between its feature vector and the feature representation corresponding to each entry in the fault rule dictionary (feature representation can be extracted for each entry in advance, or represented by word vectors or other methods according to its semantics). Commonly used similarity calculation methods include cosine similarity and Euclidean distance. For example, when using cosine similarity, the cosine value of the angle between the image feature vector and the feature representation vector of the entry is calculated to measure their similarity. The higher the similarity, the more likely the image is to meet the fault type represented by the entry.

[0165] Multi-feature fusion matching (if multiple feature extraction methods are used): If multiple types of features such as deep features and manual features are extracted at the same time, the similarity results of different features can be combined through weighted fusion and other methods to determine the final matching degree. For example, a higher weight (such as 0.7) is given to the similarity of deep features, and a lower weight (such as 0.3) is given to the similarity of manual features, and then the weighted sum is obtained to obtain the comprehensive similarity, and the matching between the image and the fault rule dictionary entries is judged based on the comprehensive similarity.

[0166] Hierarchical matching and screening: Matching and screening are performed based on the fault hierarchy structure set in the fault rule dictionary. Matching starts with the high-level fault category to determine which large fault category range the image roughly belongs to (such as first determining whether it belongs to the first-level category such as "running gear fault" or "brake system fault"), and then further accurately matching is performed in the corresponding low-level specific fault types, gradually narrowing the scope, and finally determining the fault classification label that best matches the image features and the corresponding fault level (the level can be determined based on the classification standards for different fault severities in the maintenance regulations).

[0167] Result output and verification:

[0168] The matched fault classification labels and fault levels are output as the final results. They can be stored in the metadata of the corresponding valid railway freight car fault images in the form of annotation information (such as adding fields to the image database to record classification labels and level information), or generating special annotation files (such as XML, JSON and other format files, recording the file name of each image, the corresponding classification label and level, etc.), which is convenient for subsequent data set construction and use.

[0169] To ensure the accuracy of the classification results, a certain verification mechanism can be set up, such as randomly selecting some classified images and having professional railway freight car technicians conduct manual review to check whether the classification labels and levels are consistent with the actual fault conditions presented in the images. If many errors are found, it is necessary to adjust and optimize the previous matching algorithms, feature extraction methods, etc. to continuously improve the accuracy of classification.

[0170] Optionally, performing image balancing on the valid railway freight car fault images to obtain balanced fault images includes: performing oversampling processing on valid railway freight car fault images with a small number of samples to obtain increased fault image samples; generating fault scene images in a real environment based on the increased fault image samples according to a simulated fault by dismantling a real vehicle; and migrating a few fault morphologies to a fault-free image based on the fault scene images in the real environment to obtain a balanced fault image.

[0171] Optionally, the method described in the method also includes: determining valid railway freight car fault images with a small sample size in accordance with at least one of the following methods: covering more than 95% of the vehicle models in use on the entire road, and performing random sampling according to the frequency ratio of vehicle models; clearly distinguishing the age of vehicles, and selecting images of vehicles that are newly-factory used or have been used for more than or equal to 3 years after factory repair and vehicles that are newly-factory used or have been used for less than 3 years after factory repair in a ratio of 1:1; selecting images in different weather conditions in a ratio of 1:1 between normal weather and abnormal weather; selecting images under different lighting conditions in the morning, noon and evening in a ratio of 1:1:1; covering the 367 TFDS fault types that have been sorted out.

[0172] To this end, the above-mentioned technical processing process of obtaining a balanced fault image and determining a valid railway freight car fault image with a small sample size has the following technical advantages:

[0173] (1) When constructing a dataset for intelligent recognition, the imbalanced distribution of fault samples is a common and influential problem, that is, the number of images of some faults is large, while the number of images corresponding to other faults is too small. By performing image balancing on the effective railway freight car fault images, the number of image samples of various faults is relatively balanced, which allows the intelligent recognition algorithms such as deep learning trained on the dataset to learn the characteristics of different faults more comprehensively and evenly, avoiding the algorithm from overfitting the fault types with a large number of samples and insufficient recognition ability for the fault types with a small number of samples, thereby improving the algorithm's comprehensive recognition accuracy and generalization ability for various faults, so that the algorithm can detect different types of freight car faults more stably and reliably in practical applications.

[0174] The balanced fault images obtained through image balancing cover more comprehensive and rich fault forms and scenarios, and can more truly reflect the various fault conditions that may occur in the actual operation of railway freight cars. This makes the constructed data set more representative, and both common faults and relatively rare faults can be reasonably reflected in the data set, providing data support that is more in line with actual business needs for the subsequent development of the fault image intelligent recognition system, which helps to improve the applicability and effectiveness of the system in the complex and diverse actual railway transportation environment.

[0175] (2) Oversampling is performed on fault images with a small number of samples, which directly increases the number of image samples of these relatively rare faults. For example, for minor faults in certain specific parts or rare faults that only occur under specific working conditions, the original samples are scarce and it is difficult for the intelligent recognition algorithm to fully learn their features. After increasing the number of samples through oversampling, the algorithm has more data to capture the unique manifestations of these faults, such as the texture and shape changes of the fault part, which helps to improve the recognition ability of these rare faults and avoid missing the detection of important but rare faults due to insufficient samples.

[0176] The oversampling operation is performed on the basis of existing samples, which can retain the key feature information in the original fault image to the greatest extent. Unlike some methods that generate new images but may introduce uncertain factors, it is a copy of existing real samples or a moderate transformation based on existing samples (such as some oversampling algorithms will make small random perturbations on the basis of the original samples, etc.), ensuring that the newly added samples can still accurately reflect the actual situation of the corresponding fault, making subsequent analysis, training and other operations based on these expanded samples more reliable, and can better fit the actual fault characteristics to optimize the intelligent recognition algorithm.

[0177] (3) The fault scenarios of railway freight cars in actual operation are complex and diverse, and are affected by many factors. Simply relying on the few existing fault images may not be able to fully cover the fault presentation methods in these real scenarios. By dismantling the real car to simulate the fault, and generating fault scene images in the real environment based on the increased number of samples, it is possible to artificially reproduce various possible actual fault conditions, such as simulating the appearance of a freight car failure under different road conditions, different operating speeds, different vehicle loads, etc., so that the fault image is closer to the actual performance in the real operating environment, making the fault characteristics learned by the intelligent recognition algorithm more in line with the actual scenario, and improving the algorithm's ability to accurately identify faults in the face of various complex situations in real railway transportation.

[0178] This method can create more diverse fault scene images, further expanding the types and forms of fault images based on the original samples. Different simulated fault settings, different real environment simulation parameters, etc. will produce different fault images, injecting more new elements into the data set, enriching the content of the data set, and helping the algorithm learn fault features from more angles and more comprehensively, enhancing its adaptability and robustness to changes in different fault scenarios, and better coping with the ever-changing fault conditions in practical applications.

[0179] (4) In the actual collected railway freight car images, there are often relatively more fault-free images. By migrating a few fault morphologies to these fault-free images (for example, Figure 3As shown in the figure, we can make full use of the existing fault-free image resources to increase the number of fault images, quickly expand the fault image samples without increasing the data acquisition cost too much, and achieve the balance of fault samples. For example, by cleverly integrating some specific fault part features into the corresponding part images of fault-free vehicles, we can generate a large number of new fault images that conform to the actual situation in a short time, thereby improving the efficiency and economy of data set construction.

[0180] Since the fault morphology migration is based on the fault scene images in the real environment, the migrated images can still maintain the realism consistent with the actual railway freight car operating environment, and the background and overall style of the images are consistent with the original fault-free images and fault scene images. The balanced fault images generated in this way can achieve a high level in overall visual effect and actual scene fit, avoiding the image that looks unreal or out of touch with the actual situation due to unreasonable generation methods, ensuring the stability and reliability of the image quality in the data set, and facilitating the intelligent recognition algorithm to better learn and identify fault characteristics in real scenes.

[0181] (5) By covering more than 95% of the models in use on the road and randomly sampling according to the frequency ratio of the models, we can fully consider the differences in the structure and component characteristics of trucks of different models and the probability of failure in actual operation. Trucks of different models have different types of failures and frequencies due to differences in design, manufacturing process, and service life. Determining the failure images with a small sample size in this way can ensure that no rare failure conditions under any common model are missed, so that the data set can fully reflect the diversity of failures of various types of trucks in actual operation, and improve the compatibility and recognition ability of the intelligent recognition system for failures of trucks of different models.

[0182] Clearly distinguish between the age of vehicles, and select images of vehicles that are new or have been used for more than 3 years after factory repair and vehicles that are new or have been used for less than 3 years after factory repair in a 1:1 ratio, taking into account the impact of vehicle aging, wear and tear on faults over time. The types and characteristics of faults that occur in new and old vehicles are often quite different. This selection method can evenly cover the fault images of trucks at different stages of use, allowing the intelligent recognition algorithm to learn the fault manifestations under different degrees of wear, more accurately judge the fault conditions of vehicles in different states, and enhance the accuracy and adaptability of the algorithm when detecting faults in trucks of different ages.

[0183] Images under different weather conditions were selected in a 1:1 ratio between normal weather and abnormal weather, and images under different light conditions were selected in a 1:1:1 ratio in the morning, afternoon, and evening, fully taking into account the impact of external environmental factors on the acquisition of truck fault images and the presentation of fault characteristics. Weather conditions (such as sunny days, rainy days, snowy days, etc.) and light conditions (light intensity and angle at different time periods) will change the clarity, color, contrast and other features of the fault part in the image. By selecting images under these conditions in a balanced manner, the data set includes fault images under the influence of various environmental factors, which helps the algorithm learn the ability to accurately identify faults under different environmental interferences, and improves its robustness and practicality in actual complex and changeable natural environments and operation scenarios at different time periods.

[0184] It covers 367 TFDS fault types that have been sorted out, ensuring that all known fault conditions of railway freight cars can be taken into account and will not be ignored due to the small number of samples of certain fault types. This comprehensive coverage ensures the integrity of the data set in terms of fault types, allowing the intelligent recognition algorithm to fully learn the image features of different types of faults, improve the comprehensive recognition ability of various faults, avoid blind spots in the recognition of some fault types, and enable the constructed data set and the intelligent recognition system developed based on it to truly meet the actual needs of railway freight car fault detection.

[0185] In a specific application scenario, the technical processing involved in obtaining the balanced fault image and determining the effective railway freight car fault image with a small sample size is as follows:

[0186] (1) Determine the effective railway freight car fault images with a small sample size

[0187] Covering more than 95% of all road-use models, and random sampling based on the frequency ratio of vehicle models:

[0188] Vehicle model data collection and collation: First, it is necessary to obtain relevant information on all railway freight vehicle models. This information can be extracted from the railway department's vehicle management database, including vehicle model number, vehicle model name, ownership of each vehicle model, and operation distribution on different lines. At the same time, the frequency data of each vehicle model can be counted. The number of times different vehicle models pass through specific inspection points can be recorded through monitoring equipment installed along the railway (such as the TFDS system itself or other vehicle counting devices), and summarized and collated according to a certain time period (such as monthly or annually) to form a statistical report on the frequency of each vehicle model.

[0189] Determine the sampling ratio: According to the number of vehicle models and the frequency of vehicle passing, calculate the proportion of each vehicle model in the total number of vehicles passing, and use this as a basis to determine the random sampling ratio for each vehicle model. For example, if a certain vehicle model has a large number of vehicles and a high frequency of vehicles passing, and its proportion in the total number of vehicles passing is 20%, then during random sampling, the fault image samples of this vehicle model can be extracted at a relatively high ratio; and for some vehicles with a small number of vehicles and a low frequency of vehicles passing, even if their proportion is small (but the overall coverage must ensure that more than 95% of the operating models are covered), sampling should be carried out at an appropriate ratio to ensure that the fault images with a small number of samples that may appear under these models are not missed.

[0190] Random sampling implementation: In the fault image data set corresponding to each vehicle model (which can be stored in an image database or file system, with the vehicle model number as a classification identifier), random sampling is performed according to a determined sampling ratio using a random number generation algorithm (such as the random function provided by the programming language, the `random` module in Python, etc.). For example, a certain vehicle model is set to be sampled at a ratio of 10%, and an index value within the total number of fault images of the vehicle model is randomly generated, and the image corresponding to the index is selected as a sample. These randomly sampled images are collected for subsequent judgment of whether the sample size is too small and for further image balancing processing.

[0191] Clearly distinguish the age of vehicles, and select images of vehicles that are new from the factory or have been repaired for more than 3 years and vehicles that are new from the factory or have been repaired for less than 3 years in a 1:1 ratio:

[0192] Vehicle information acquisition and newness judgment: Obtain relevant information such as the factory date and factory repair records of each freight car from the railway vehicle management system to judge the newness of the vehicle. For vehicles with factory repair records, determine whether it is greater than or equal to 3 years or less than 3 years based on the time it has been put into use after the factory repair. For example, if a vehicle is put into use again after factory repair in 2020, and it has been more than 3 years as of the current time (such as 2024), it will be classified as a used car; if it has been put into use for less than 3 years after the factory repair, it will be classified as a new car.

[0193] Image classification and balanced selection: The railway freight car fault images are classified and sorted according to the age of the vehicles they belong to, and stored in different data sets or folders (labeled as new vehicle fault image set and old vehicle fault image set). Then, by traversing the two data sets, counters or index control are used to select images from the new vehicle fault image set and the old vehicle fault image set in a 1:1 ratio, ensuring that the final selected images can evenly cover the fault conditions of vehicles of different ages, which is convenient for subsequent analysis of the number of fault samples of vehicles at different use stages and image balancing operations.

[0194] Select images under different weather conditions according to the 1:1 ratio of normal weather to abnormal weather:

[0195] Weather information annotation and image classification: For existing railway freight car fault images, check the weather record information when the corresponding image was collected (this information may be recorded in the metadata of the image, or reflected in the environmental monitoring data associated with the image acquisition system, such as the meteorological sensor records of the TFDS system, etc.), and divide the images into normal weather image sets (such as images collected in sunny, cloudy and other weather conditions) and abnormal weather image sets (such as images collected in heavy rain, heavy snow, dense fog and other weather conditions) according to weather conditions.

[0196] Balanced sampling operation: Count the number of images in the normal weather image set and the abnormal weather image set respectively, and then select the same number of images from the larger image set by random sampling (also using a random number generation algorithm, such as randomly selecting within the index range of the corresponding image set) based on the smaller one, so that the images under different weather conditions finally selected can meet the 1:1 ratio requirement, ensuring that the data set covers fault image samples under the influence of different weather environments, providing a comprehensive data foundation for subsequent image balancing and overall data set construction.

[0197] Select images under different lighting conditions in the morning, afternoon and evening in a 1:1:1 ratio:

[0198] Light condition judgment and image grouping: Based on the timestamp information of image acquisition (usually stored in the metadata of the image), the time period corresponding to the light condition is roughly judged, and the images are divided into three image groups with different light conditions: morning (such as images collected within 1-2 hours after sunrise), afternoon (such as images collected about 2 hours before and after noon), and evening (such as images collected within 1-2 hours before sunset). Of course, the brightness, contrast and other characteristics of the image itself (calculated and analyzed by image processing algorithms) can also be combined to further assist in confirming whether the light condition classification is accurate.

[0199] Proportional balanced selection: Count the number of images in each light condition image group, take the group with the least number as the standard, and select the corresponding number of images from the other two groups by random sampling, to ensure that the final selected images under different light conditions in the morning, afternoon, and evening reach a 1:1:1 ratio. For example, if the number of images collected in the morning is the least, then randomly select the same number of images as the morning images from the image sets collected in the afternoon and evening, so that the number of images under the three light conditions is balanced, so that the data set fully includes the sample situations in which different light conditions affect the fault images, which is convenient for subsequent image balance processing and the construction of a data set that is more in line with actual application scenarios.

[0200] Covering the 367 TFDS fault types that have been sorted out: Fault type labeling and verification: First, based on the existing TFDS fault type classification standards (usually compiled by the railway department after long-term practice and professional research, with detailed definitions and descriptions of various types of freight car faults), the existing railway freight car fault images are labeled with fault types. This can be done through manual labeling (professional railway vehicle inspection personnel view the image and label it after determining the fault type based on the fault characteristics) combined with computer-assisted labeling (using the existing intelligent recognition algorithm for preliminary judgment, and then manually reviewing and confirming), to ensure that each image is accurately labeled with the corresponding TFDS fault type.

[0201] Statistics and screening: Statistics are collected for images with marked fault types to check the number of images for each fault type. Images corresponding to fault types with a small number of images are screened out to ensure that all 367 TFDS fault types are covered and that no fault images with a small number of samples are missed. This provides a complete data basis for subsequent image balancing operations such as oversampling on these images with a small number of samples, ensuring the comprehensiveness of the dataset in terms of fault type coverage.

[0202] (2) Oversampling of valid railway freight car fault images with small sample size

[0203] Select oversampling algorithm: You can use a variety of oversampling algorithms, such as random oversampling, SMOTE (Synthetic Minority Oversampling Technique) and its variants. Random oversampling is simply to randomly copy minority class samples to increase their number; the SMOTE algorithm interpolates and synthesizes new samples based on adjacent samples in the feature space of minority class samples. For example, for a minority class sample point in a two-dimensional feature space, find its k nearest neighbor sample points, and then synthesize new sample points between them in a certain ratio, thereby generating more fault image samples.

[0204] Feature space construction: If you use an oversampling algorithm based on feature space operations such as SMOTE, you need to first construct the feature space of the sample. For railway freight car fault images, you can extract the feature vector of the image to represent the image. Common methods include using convolutional neural networks (CNN) to extract deep features (input the image into a pre-trained CNN model, such as ResNet, VGG, etc., and take the feature vector output by the last fully connected layer), or extract traditional manual features (such as color histograms, texture features, shape features, etc., and combine them into a feature vector) as a representation of the sample in the feature space, which is convenient for subsequent operations such as calculating the distance between samples and finding adjacent samples, and then synthesizing new samples.

[0205] Oversampling operation execution: Random oversampling: If the random oversampling algorithm is used, directly in the valid railway freight car fault image dataset with a small sample number, according to the set multiple (for example, if you want to expand the sample number to 2 times, 3 times, etc. of the original), use the random number generator to randomly select the samples to be copied within the index range of the dataset, and then copy the selected samples and add them to the dataset to increase the number of samples.

[0206] SMOTE and its variants: Taking the SMOTE algorithm as an example, for each minority class sample (i.e., the sample point corresponding to the fault image with a small sample size), first determine its k value (usually select a suitable k value within a certain range through cross-validation, such as k=5, k=10, etc.), find its k nearest neighbor sample points (the nearest neighbors are determined by calculating the distance between sample points in the feature space, such as Euclidean distance, cosine distance, etc., and commonly used libraries such as `scikit-learn` in Python have corresponding functions to implement distance calculation and nearest neighbor search), then synthesize new sample points between the sample and the adjacent samples according to a preset synthesis ratio (such as 0.5, indicating that a new sample is synthesized at the midpoint between two adjacent samples), add the images corresponding to these newly synthesized samples to the data set, and repeat this process until the desired sample size expansion target is reached and an increase in fault image samples is obtained.

[0207] (3) Based on the actual vehicle dismantling simulation failure, the fault scene image in the real environment is generated based on the increased number of fault image samples

[0208] Preparation for real vehicle dismantling and simulating failures: Choose appropriate real vehicles: Select representative vehicles from railway freight vehicles for dismantling and simulating failures, covering different models and different degrees of oldness, so as to ensure that the simulated failure scenarios are more universal and practical. For example, select several common models of freight vehicles, including new vehicles that have just been manufactured and old vehicles that have been in operation for many years, to ensure that the actual scenarios of vehicles in different states when failures occur can be simulated.

[0209] Set up a simulated fault environment: In a special simulation site (which can be a vehicle maintenance base of the railway department or other places with corresponding conditions), build an environment similar to the actual railway operation, including laying tracks, setting different road conditions (such as straight tracks, curves, slopes, etc.), and equip with corresponding detection equipment (such as cameras, used to collect fault scene images, and the installation position and angle of the camera should be similar to the actual image collection of the TFDS system), simulate different operating speeds (control the vehicle running speed through traction equipment), vehicle loads (by adding simulated goods of different weights, etc.), and other conditions, so as to reproduce the various working conditions of railway freight cars in actual operation as realistically as possible.

[0210] Determine the type and location of simulated faults: Based on the sorted TFDS fault types and common fault conditions of actual railway freight cars, determine the type of fault to be simulated and the specific location of occurrence, such as simulated axle wear failure, brake component damage failure, etc., and arrange professional technicians to artificially create these fault conditions on the actual vehicle in a scientific and reasonable manner to ensure that the simulated faults are consistent with the actual fault characteristics and occurrence principles, and prepare for the subsequent generation of real fault scenario images.

[0211] Generate fault scene images based on the increased number of fault image samples: Fault reproduction and image acquisition: Place a real vehicle with simulated faults in a simulated environment for operation, and perform multiple operation tests according to different set working conditions (such as different speeds, loads, road conditions, etc.). During the operation, use pre-set cameras and other detection equipment to collect images of the vehicle when it fails. The acquisition frequency can be set according to actual needs and equipment performance (such as collecting 1 or more frames of images per second), and a large number of fault scene images are obtained. These images not only contain simulated fault features, but also reflect the relationship between the fault and the vehicle as a whole, the track, and the surrounding environment in a real environment, making the image closer to the fault situation in actual operation.

[0212] Image screening and organization: Screen a large number of fault scene images collected, remove those that do not meet the quality requirements due to poor acquisition angles, blurred images, etc., retain high-quality images that are clear and can accurately present fault characteristics and real environmental information, and then classify and organize these images according to simulated fault types, working conditions, etc., to facilitate subsequent operations such as fault morphology migration based on these images, and further expand the balanced fault image data set.

[0213] (4) Based on the fault scene images in the real environment, a few fault morphologies are transferred to the fault-free images to obtain a balanced fault image.

[0214] Fault morphology extraction and representation: Image feature extraction: For fault scene images in real environments, extract key features that can characterize the fault morphology. Deep learning methods (such as using CNN models to extract deep features) or traditional manual feature extraction methods (such as extracting texture features and shape features of the fault part) can also be used to convert each fault scene image into a corresponding feature vector representation, so that the fault morphology can be quantified and analyzed in the form of feature vectors, which is convenient for subsequent migration operations.

[0215] Feature selection and simplification: If the extracted feature vector has a high dimension, the feature vector can be simplified through feature selection algorithms (such as screening out features with strong correlation with fault morphology based on correlation analysis, information gain, etc.) or feature dimensionality reduction algorithms (such as principal component analysis PCA, projecting high-dimensional feature vectors into low-dimensional space while retaining the main fault morphology information) to reduce computational complexity and improve the efficiency and accuracy of fault morphology migration.

[0216] Selection and preparation of fault-free images: Screening of suitable fault-free images: Fault-free images are selected from the existing railway freight car image dataset. The selection of these images should take into account certain similarities with the fault scene images in terms of vehicle model, shooting angle, background environment, etc., so that the images generated after the fault morphology is migrated look more natural and realistic, and conform to the actual railway freight car operation scene. For example, fault-free images of the same vehicle model and collected on the same or similar sections are preferentially selected as the basic images for migration.

[0217] Image preprocessing: Preprocess the selected fault-free images, such as adjusting the image size and resolution to make them consistent with the size specifications of the fault scene image, to facilitate subsequent operations such as feature fusion; the image can also be normalized (such as normalizing the pixel value to a specific range) to ensure that different images have better compatibility and stability during numerical calculation and fusion.

[0218] Fault morphology migration operation: Feature fusion method selection: A variety of feature fusion methods can be used to achieve fault morphology migration, such as methods based on image editing (such as pasting the image block of the fault part extracted from the fault scene image to the corresponding position of the fault-free image through image synthesis technology, and performing edge fusion, color adjustment and other processing to make it look naturally transitioned); or image generation methods based on deep learning (such as using the idea of ​​generative adversarial network GAN to train a generative model, using the fault-free image and the fault morphology feature vector as input to generate a new image that integrates the fault morphology, and using the adversarial training mechanism to make the generated image as realistic as possible and consistent with the actual fault characteristics).

[0219] Optimization of migration effect: After the fault morphology migration, the quality of the generated image is evaluated and optimized to check whether the image has obvious splicing marks, inconsistent colors, unclear fault morphology, etc. The migration effect can be judged by manual visual inspection combined with computer image quality evaluation indicators (such as the clarity, contrast, structural similarity and other indicators of the calculated image). For images with poor results, they are processed again by adjusting the migration parameters (such as the weight of feature fusion, the position and size of image synthesis, etc.) or improving the migration method (such as changing different GAN architectures, image synthesis algorithms, etc.) until a high-quality balanced fault image that looks natural and real and accurately presents the fault morphology is obtained, thereby achieving the equalization of fault samples and improving the construction of the data set.

[0220] Optionally, the method described in the method further includes: executing the following steps to generate a real vehicle dismantling simulation failure: according to different failure types, dismantling the railway freight car and simulating the failure setting to restore the conditions when the failure occurs in real time and generate the real vehicle dismantling simulation failure accordingly.

[0221] In a specific application scenario, according to different fault types, railway freight cars are disassembled and fault settings are simulated to restore the conditions when the fault occurs in real time and generate a real car disassembly simulation fault library based on this, which may include:

[0222] (1) Fault type sorting and analysis

[0223] First, it is necessary to comprehensively collect the fault classification data related to railway freight cars. The sources of these data include the "Railway Freight Car Operation and Maintenance Regulations" formulated by the railway department, past fault statistics reports, and professional technical research literature. These data are sorted out and analyzed to sort out the common fault types of various key component systems such as the running gear, braking system, coupler and buffer device, and body structure, such as wheel set tread scratches and peeling of the running gear, brake shoe eccentric wear and brake cylinder leakage of the braking system, and poor three-state function of the coupler of the coupler and buffer device, etc., to form a detailed and comprehensive list of railway freight car fault types to ensure that the subsequent simulated fault settings can cover various typical fault conditions.

[0224] According to the actual application scenario requirements, fault frequency, degree of harm and other factors, the fault types that need to be simulated are determined from the many fault types sorted out. For example, priority is given to those fault types that directly affect driving safety and have a relatively high probability of occurrence in actual operation, while also taking into account some faults that occur less frequently but have serious consequences once they occur, ensuring that the simulated faults are representative and have practical application value, laying the foundation for generating high-quality real vehicle dismantling simulation faults and corresponding fault scene images.

[0225] (2) Real vehicle selection and preparation

[0226] According to the type of fault to be simulated, select the appropriate railway freight car model. Freight cars of different models differ in structural design, component specifications, etc., and some faults are more likely to occur on specific models. For example, for fault simulations involving differences in bogie structure, it is necessary to select models with corresponding bogie types for disassembly and simulation operations. Usually, common models with large numbers of typical structural characteristics in railway operations, such as C70, P70 and other models of freight cars, are selected. This not only facilitates the acquisition of real vehicle resources, but also ensures that the simulation results are of reference value to most actual operating vehicles.

[0227] After selecting the actual vehicle, conduct a comprehensive and detailed inspection of the initial state of the vehicle, and record the original condition of each component of the vehicle, including the degree of wear of the components, surface integrity, connection tightness, etc. This can be done through professional inspection tools (such as calipers to measure component dimensions, flaw detectors to detect whether there are internal cracks, etc.) combined with manual visual inspection to ensure that the vehicle is in a known and recordable state before disassembly, so as to accurately evaluate the changes after the simulated fault setting, and also provide a reference basis for the accuracy of the simulation, so as to avoid the judgment of the simulated fault effect affected by the abnormal initial state of the vehicle itself.

[0228] (3) Disassembly process and key points

[0229] According to the selected vehicle model and the type of fault to be simulated, a detailed disassembly plan and process should be developed to clearly define the order of disassembly, the tools required, and the operating specifications for each step. For example, for a simulated brake system fault, the disassembly plan may start with the disassembly of the brake cylinder, followed by the disassembly of the connecting pipes, brake shoes and other components, from outside to inside and from the whole to the part, to ensure that the disassembly process is orderly and does not cause unnecessary damage to vehicle components. The disassembly process must comply with the relevant standards and safety requirements for railway freight car maintenance, while also taking into account the convenience of subsequent simulated fault settings and reassembly and restoration work.

[0230] During the disassembly process, each disassembled component is marked with its name, location, number (if any), etc., so that the original state of the vehicle can be accurately restored during subsequent reassembly. At the same time, key information such as the disassembly process, component appearance characteristics, and connection methods are recorded in detail by taking photos, recording videos, and writing down texts, and a disassembly file is established. This not only helps to ensure the accuracy of disassembly and assembly work, but also provides detailed reference information for subsequent analysis of fault simulation effects and comparison with actual fault conditions.

[0231] (4) Simulation fault setting

[0232] For different types of faults, we conduct in-depth analysis of the actual causes of the faults to determine the corresponding means of simulating the faults. For example, for the tread scratch fault of the wheelset, the cause may be uneven braking force, wheel locking, etc. during braking. In the simulation, we can artificially create the fault form of tread scratch by setting specific braking parameters (such as adjusting the uneven pressure of the brake cylinder, simulating wheel locking conditions, etc.); for the wear fault of components, we can use the simulation method of accelerated wear, such as adding a certain wear material to the contact surface of a specific component, so that it can quickly produce the effect similar to that after long-term wear in actual operation under the simulated operating environment, to ensure that the simulated fault is as close to the real fault condition as possible in terms of the cause and manifestation.

[0233] In the process of setting up simulated faults, various parameters involved (such as force, running speed, running time, etc.) must be accurately controlled and recorded. Professional test instruments (such as pressure sensors, speed sensors, etc.) can be used to monitor and adjust parameters in real time to ensure that the conditions of simulated faults meet the expected settings. For example, when simulating a brake cylinder leakage fault, the pressure sensor is used to accurately control the air pressure changes in the brake cylinder so that it leaks at a preset leakage rate. At the same time, the changes in relevant indicators such as vehicle braking performance during the entire process are observed and recorded. By comparing and analyzing the data with the actual fault case, the accuracy and rationality of the simulated fault setting can be verified.

[0234] (5) On-site restoration of the conditions when the failure occurred

[0235] In a special simulation site (such as the test track area in the railway vehicle maintenance base), it is built and arranged according to the common environmental conditions of actual railway freight car operation. In addition to laying standard tracks, different road conditions must be simulated, such as setting track sections with different slopes (achieved by adjusting the height difference of the track foundation), curves (built according to the actual railway curve radius standards), etc., to restore the impact of vehicle failures caused by changes in road conditions during actual driving; at the same time, consider the simulation of weather factors (which can be achieved by building a simple artificial climate chamber or conducting tests under specific weather conditions), such as simulating the failure performance of the vehicle braking system in a wet state in a rainy environment, so that the environmental conditions when the simulated failure occurs are as close to the real railway operation scenario as possible.

[0236] The running speed, running time, start-stop times and other operating conditions of the real vehicle in the simulated environment are controlled by traction equipment to match the actual conditions when the fault occurs. For example, for some fatigue fault simulations caused by long-term continuous operation, the real vehicle is allowed to run continuously on the simulated track for a corresponding period of time according to the average running speed of the vehicle in actual operation and the corresponding continuous operation time requirements. At the same time, the need to simulate frequent start-up and stop processes is considered in combination with the fault type to fully reproduce the actual conditions when the fault occurs, ensuring that the simulated fault can accurately reflect the characteristics and impacts under real operating conditions.

[0237] (6) Generate real car dismantling simulation failure and subsequent inspection

[0238] After completing the simulated fault setting and restoring the fault conditions on site, the simulated fault status of the vehicle is carefully checked and confirmed. Through visual inspection, re-measurement of relevant parameters of components with professional testing tools (such as flaw detectors, measuring tools, etc.), and observation of the morphological characteristics of the faulty parts, the final state of the simulated fault is compared and verified with the expected fault type performance to ensure that the simulated fault is accurately generated and conforms to the basic characteristics of the actual fault. At the same time, the final state of the simulated fault is recorded in detail in various forms such as text, images, and videos to form a complete simulated fault data archive, which provides an accurate basis for subsequent work such as generating fault scene images and data analysis based on this.

[0239] After confirming the generation of the simulated fault, the overall safety of the vehicle and the subsequent recoverability must also be checked. Ensure that the simulated fault setting will not cause irreversible damage to the vehicle, and that the vehicle can be restored to a normal safe operating state after the simulated fault is removed and reassembled. Follow the relevant regulations and standards for railway freight car safety inspections and key safety-related parts (such as coupler connection strength, running gear stability, etc.) to conduct key inspections and assessments to ensure that the entire simulated fault process is carried out under the premise of safety, controllability and recoverability, so that the actual vehicle can be cyclically used for simulations of different fault types and related research work.

[0240] Optionally, the oversampling of valid railway freight car fault images with a small number of samples to obtain increased fault image samples includes: determining a transformation range for image amplification based on a set image amplification model; transforming the valid railway freight car fault images based on the transformation range to generate fault image samples of different forms; and adjusting the overall brightness of the fault image samples of different forms based on a set brightness factor and offset to obtain increased fault image samples.

[0241] In a specific application scenario, the technical implementation process of oversampling the valid railway freight car fault images with a small number of samples to obtain increased fault image samples is as follows:

[0242] (1) Determine the transformation range of image amplification based on the set image amplification model

[0243] Image augmentation model selection and initialization: Select an image generation model based on deep learning, such as a generative adversarial network (GAN) or a variational autoencoder (VAE), or use a traditional image transformation model (such as a model based on a combination of geometric transformations such as affine transformation and elastic transformation). Taking GAN as an example, first select a suitable GAN architecture (such as DCGAN, WGAN, etc.), and then initialize the model according to the characteristics of the railway freight car fault image (such as image size, number of channels, etc.), including determining the number of network layers, the number of neurons in each layer, the size of the convolution kernel, and other parameters. For traditional image transformation models, it is necessary to determine which specific geometric transformation operations to use, their order, parameter range, etc.

[0244] Basis and method for determining transformation range: Determine the transformation range based on the actual characteristics of railway freight car fault images and the consideration of the rationality of generating new samples. For example, for the size transformation range of the image, the size distribution of the existing fault images can be analyzed, and the width and height scaling range can be set in combination with the reasonable size variation range of railway freight car components in the image. For example, the width can vary between 0.8 and 1.2 times of the original size, and the height can vary between 0.9 and 1.1 times; for the rotation angle range, considering the normal angle deviation of railway freight cars in actual shooting scenes, it can be set to rotate between -15 degrees and 15 degrees; for the translation range, according to the position of the freight car in the image and the principle of keeping the key components intact and visible, set the horizontal and vertical translation pixel range, etc. These ranges can be determined by statistically analyzing a large number of annotated railway freight car fault images and combining the experience of railway vehicle inspection professionals to ensure that the generated new samples are both diverse and in line with the actual situation.

[0245] (2) Based on the transformation range, transform the valid railway freight car fault image to generate fault image samples of different forms.

[0246] Geometric transformations:

[0247] Scaling operation: If the image scaling is implemented based on affine transformation, the scaling matrix is ​​constructed to transform each pixel coordinate of the original image according to the set scaling ratio range (such as the width and height scaling ratios mentioned above), and then the pixel value of the transformed coordinate position is calculated using an interpolation algorithm (such as bilinear interpolation) to obtain the scaled image. For example, for a 500×300 image of a railway freight car fault, it is scaled according to the width scaling ratio of 1.1 and the height scaling ratio of 1.05. After calculation and interpolation processing, a new image with a larger size is obtained, and the freight car parts and faulty parts in the image are also scaled accordingly, presenting a different visual form.

[0248] Rotation operation: Also based on the affine transformation, the rotation matrix is ​​constructed, and the image is rotated according to the set rotation angle range (such as 5 degrees). The rotation center can be selected from the center of the image or determine the appropriate center point based on factors such as the location of key parts of the truck. After the rotation, blank areas may appear at the edge of the image. Through the filling algorithm (such as filling with background color or using mirror filling), it becomes a complete image, thereby changing the relative angle relationship between the fault part in the image and the entire truck, and generating a new shape.

[0249] Translation operation: According to the set horizontal and vertical translation pixel range (such as horizontal translation of 10 to 20 pixels), the pixel coordinates of the image are directly translated, and the entire image is moved a corresponding distance in the plane, so that the positions of the truck and the fault part in the image are changed, and a new sample shape is generated. At the same time, attention should be paid to processing the part that exceeds the boundary of the original image after translation (such as cropping or repeated filling to ensure image integrity).

[0250] Elastic transformation: If elastic transformation is introduced to increase sample diversity, by generating a random displacement field on the image (a displacement vector field that conforms to a certain distribution law can be generated based on a two-dimensional Gaussian function or other methods), the pixels of the image can be slightly elastically deformed according to the displacement field. This simulates the image morphology changes that may be caused by factors such as vibration and slight changes in shooting angle during the actual shooting process of a truck, allowing the fault site to present a more natural morphological difference, enriching the diversity of the newly generated samples.

[0251] Color transformation: In addition to geometric transformation, color-related transformations can also be performed, such as changing the image's hue, saturation, brightness and other color attributes. By adjusting the values ​​of the corresponding channels in a reasonable color space (such as the HSV color space), the image can be made to present different color styles while keeping the fault features recognizable, thereby increasing the diversity of samples. For example, in the HSV space, the hue value is randomly increased or decreased within a certain range to generate samples of the same fault image with different color tones, just like a truck fault presents different visual effects under different lighting conditions.

[0252] Combining multiple transformations to generate diverse samples:

[0253] In order to generate more fault image samples with different morphologies, the above-mentioned multiple transformation operations are usually combined in a certain order and probability. For example, a scaling operation is first performed with a certain probability (such as 0.6), and then a rotation operation is determined with a probability of 0.4, and then a color transformation is performed with a probability of 0.3. Through the random combination of different transformations, multiple new samples with obvious morphological differences can be generated from an original valid railway freight car fault image, which can better expand the number and diversity of samples.

[0254] (3) Based on the set brightness factor and offset, the overall brightness of the fault image samples of different forms is adjusted to obtain more fault image samples.

[0255] Brightness adjustment principle and implementation method:

[0256] In image processing, brightness adjustment can be achieved by linearly transforming the pixel values ​​of the image. For a color image (usually in RGB color mode), each pixel consists of the values ​​of three channels: red (R), green (G), and blue (B). After setting the brightness factor (such as a value range between 0.8 and 1.2) and the offset (such as an integer range from -30 to 30), the brightness is adjusted by transforming each channel value of each pixel using the following formula:

[0257] R neew =clip(R old ×brightnessFacttor+offset, 0, 255)

[0258] G new =clip(G old ×brightnessFactor+offset, 0, 255)

[0259] B new =clip(B old ×brightnessFactor+offset, 0, 255)

[0260] Among them, R old , G old , B old are the red, green, and blue channel values ​​of the original pixel, respectively. new , G new , B newis the new channel value after adjusting the brightness. The clip function is used to limit the calculation result to the legal pixel value range of 0 (indicating the darkest, i.e. black) to 255 (indicating the brightest, i.e. white). By traversing each pixel of the image and calculating according to the above formula, the overall brightness can be adjusted. For example, when the brightness factor is 1.1 and the offset is 10, the image will become brighter overall. Different combinations of brightness factors and offsets can simulate the effects of fault images taken under different light intensity environments, further increasing the diversity of samples.

[0261] Selection and optimization of brightness adjustment parameters: The specific values ​​of the brightness factor and offset can be determined based on the analysis of the brightness variation range of actual railway freight car fault images under different lighting conditions. You can first select a small number of sample images, manually try different combinations of brightness factors and offsets, observe the effect after adjustment, and combine the judgment of railway vehicle inspection professionals on the recognizability of fault image features under different lighting conditions to determine the appropriate value range and several commonly used sets of parameter combinations. Then, use these optimized parameters when adjusting the brightness of a large number of fault image samples of different forms to ensure that the adjusted images have both diversity in brightness and that the fault features are still clearly identifiable, meeting the requirements of subsequent data set construction and intelligent recognition algorithm training.

[0262] Therefore, the effective railway freight car fault images with a small number of samples are oversampled to obtain more fault image samples, which has the following technical benefits:

[0263] (1) By reasonably setting the transformation range of image amplification, the diversity of samples can be increased as much as possible while ensuring that the generated new fault image samples conform to the actual operation and shooting scenes of railway freight cars. Different transformation ranges can make the original image produce a variety of reasonable changes in size, angle, position, etc., simulating the various forms of freight car faults under different viewing angles and different states, avoiding the generation of invalid samples that deviate too much from the actual situation or do not conform to physical logic, so that the expanded sample set covers various possible fault conditions more comprehensively, providing richer learning materials for the intelligent recognition algorithm.

[0264] Adapting to the characteristics of railway freight car fault images: Railway freight car fault images have their own characteristics, such as the relatively fixed structure of freight car parts, and the specific positional relationship between the fault part and the whole vehicle. According to the transformation range determined by these characteristics, the image can be transformed in a targeted manner, so that the generated new samples can be diversified while maintaining these inherent characteristics, better meeting the needs of subsequent fault identification and classification tasks based on such samples, improving the fit between the data set and the actual application scenarios, and helping to improve the accuracy and generalization ability of the intelligent recognition algorithm in real railway freight car fault detection.

[0265] (2) Performing multiple transformation operations on valid railway freight car fault images to generate samples of different forms directly increases the number of samples and solves the problem of data set imbalance caused by the small number of samples of some fault images. At the same time, samples of different forms can show different characteristic manifestations of the same fault from multiple angles and dimensions. For example, the rotated image can show the appearance of the fault part at different angles, and the translated image can reflect the visual effects of the fault at different positions on the freight car, etc., which enriches the characteristic representation of the fault and allows the intelligent recognition algorithm to learn more comprehensive and detailed fault characteristic information, thereby more accurately identifying fault conditions in various forms and enhancing the robustness of the algorithm and its adaptability to complex scenarios.

[0266] In the actual operation of railway freight cars and the process of image acquisition, the fault image itself will present many different forms due to the influence of vehicle movement, the position and angle changes of the shooting equipment, environmental factors, etc. By artificially performing these transformation operations to generate samples, it is equivalent to simulating various changes in real scenes, making the constructed data set closer to reality. After the intelligent recognition algorithm is trained based on such a data set, it can better identify and judge the ever-changing fault images in actual operations, which improves the practicality and reliability of the algorithm in practical applications.

[0267] (3) Railway freight cars run at different times (such as day and night), in different weather conditions (sunny, cloudy, rainy, etc.), and in different geographical locations (different light intensities). By setting the brightness factor and offset to adjust the overall brightness of the sample, the fault image effects under these different lighting environments can be simulated, further expanding the diversity of the samples in the lighting dimension, so that the data set covers a more comprehensive range of actual situations, which helps the intelligent recognition algorithm learn the ability to accurately identify faults under various lighting conditions and avoid the problem of algorithm recognition performance degradation due to changes in lighting.

[0268] In practical applications, intelligent recognition algorithms need to detect railway freight car fault images collected under different lighting environments. The samples with brightness adjustment are used in data set construction and algorithm training, which enables the algorithm to better adapt to changes in lighting and improve its feature extraction and recognition capabilities for fault images with different brightness. This enables the algorithm to stably and accurately detect freight car faults in actual complex and changeable lighting scenes, improves the robustness and versatility of the algorithm, and better meets the actual needs of railway freight car fault detection in different environments.

[0269] The fault image samples of different forms obtained based on the above-mentioned amplification process are exemplified as follows: Figure 4As shown, (a) is the original image before amplification, and (b)-(j) are the amplified first to ninth fault image samples.

[0270] Optionally, the method of migrating a minority of fault forms to a fault-free image based on the fault scene image in the real environment to obtain a balanced fault image includes: deforming the fault scene image in the real environment according to a selected fault area to obtain a deformed fault scene image; determining a minority of fault forms based on the deformed fault scene image to migrate the minority of fault forms to a fault-free image and thereby obtain a balanced fault image.

[0271] Optionally, in a specific application scenario, migrating a small number of fault forms to a fault-free image based on the fault scene image in the real environment to obtain a balanced fault image includes the following technical processing processes:

[0272] (1) According to the selected fault area, the fault scene image in the real environment is deformed to obtain a deformed fault scene image

[0273] First of all, professional railway vehicle inspection personnel are required to accurately identify and mark the fault area in the fault scene image in the real environment based on the structural knowledge of railway freight cars and the manifestation of fault characteristics. The fault area can be determined by manually drawing a bounding box on the image (using image annotation tools such as LabelImg, etc., the annotated information can be saved in XML or JSON format, and the coordinate position, category and other information of the fault area can be recorded) or by using image segmentation algorithms to automatically identify the specific area where the fault is located (for example, using a semantic segmentation model based on a convolutional neural network, such as U-Net, DeepLab, etc., after training on annotated fault image data sets, it can output the segmentation result of whether each pixel belongs to the fault area or the normal area). The selected fault area should cover the key fault part and its surrounding related parts as accurately as possible to ensure that the subsequent deformation operation can fully reflect the fault morphology and facilitate migration to a fault-free image.

[0274] According to the selected fault area, an affine transformation matrix is ​​constructed to achieve image deformation. For example, if the fault area needs to be stretched, rotated, or beveled, the corresponding matrix can be constructed by determining the parameters required for the affine transformation (such as translation, rotation angle, scaling, etc.). For a two-dimensional image, the affine transformation matrix is ​​generally a 2×3 matrix (expressed in homogeneous coordinates). The coordinate transformation is achieved by multiplying the coordinates of each pixel in the fault area with the matrix, and then the pixel value of the transformed coordinate position is calculated using an interpolation algorithm (such as bilinear interpolation), so that the fault area is deformed according to the set affine transformation rules. For example, in order to make the fault part more suitable for the corresponding position in the fault-free image to be migrated in terms of shape and angle, the fault area can be rotated around a certain point by a certain angle and appropriately scaled to obtain the deformed effect.

[0275] When a more complex and realistic nonlinear deformation of the fault area is required, a Thin Plate Spline (TPS) transformation can be used. The TPS transformation controls the deformation of the image based on control points. First, several control points are selected around and inside the fault area (the selection of control points can be determined based on the fault morphology and the target deformation effect, usually at key boundaries, feature points, etc.), and then the target position of each control point is set. The entire deformation field is determined by solving a set of equations based on a thin plate spline function, so that the pixels in the fault area are smoothly and naturally deformed nonlinearly according to this deformation field. For example, when simulating irregular deformation of the fault area caused by actual conditions such as vehicle vibration and slight distortion of components, the TPS transformation can better achieve this complex deformation effect, so that the deformed fault scene image can more realistically reflect the actual fault morphological changes.

[0276] During the image deformation operation, it is necessary to continuously adjust the deformation parameters (such as angle and scaling ratio in affine transformation, control point position and target position in TPS transformation, etc.) to achieve the ideal deformation effect. It is possible to observe visually whether the deformed fault area meets the actual fault migration requirements in terms of shape and connection with surrounding components. At the same time, a comprehensive judgment is made based on the fusion effect after the fault morphology is subsequently migrated to the fault-free image, and the parameters are continuously fine-tuned to ensure that the deformed fault scene image can not only highlight the characteristics of the fault morphology, but also facilitate the subsequent migration operation and look natural and reasonable.

[0277] (2) Based on the deformed fault scene image, determine a minority of fault forms, so as to migrate the minority of fault forms to the fault-free image and thereby obtain a balanced fault image

[0278] For the fault area in the deformed fault scene image, feature descriptors that can characterize the fault morphology are extracted. Commonly used feature descriptors include scale-invariant feature transform (SIFT) features and speeded-up robust features (SURF) features. These features can reflect the key morphological information of the fault part, such as texture, shape, and edge, to a certain extent, and have the advantages of scale and rotation invariance. Taking SIFT features as an example, by constructing the scale space of the image, detecting the extreme points in the scale space, determining the main direction for each extreme point and generating the corresponding feature vector, a set of SIFT feature descriptors of the fault area is obtained. These feature descriptors can be used for subsequent matching and migration operations to accurately capture the key features of the fault morphology.

[0279] The fault area in the deformed fault scene image is cropped out (or the entire image containing the fault area is directly used as input, and the subsequent regional attention mechanism is used to focus on the fault area), and input into a pre-trained CNN model (such as classic models such as ResNet and VGG, or a model fine-tuned on a railway freight car fault image dataset). The output of a certain layer of the model (such as the feature map output by the last fully connected layer or convolutional layer after global average pooling) is taken as the deep feature vector. This vector can reflect the comprehensive characteristics of the fault morphology from a more abstract and high-level perspective. Combined with traditional feature descriptors, it can more comprehensively describe the fault morphology and provide richer information for accurate migration.

[0280] For the deformed fault scene image from which the fault morphological feature descriptor is extracted and the selected fault-free image, firstly, the region matching the fault morphological feature descriptor is searched in the fault-free image. By calculating the similarity between the feature descriptors (such as using the measurement methods such as Euclidean distance and cosine similarity, generally using the nearest neighbor matching combined with the ratio test and other strategies to screen reliable matching pairs), the position corresponding to the fault morphological feature in the fault-free image is found. Then, according to the matching result, the fault area in the deformed fault scene image is migrated to the corresponding position of the fault-free image through the image synthesis technology (such as using the Poisson fusion algorithm, which is based on the gradient information of the image and can make the boundary transition after fusion natural and avoid obvious splicing marks; or a simple weighted average fusion algorithm, which fuses the pixel values ​​of the fault area and the corresponding area of ​​the fault-free image according to the set weight) to realize the migration of the fault morphology and generate a preliminary balanced fault image.

[0281] The generative model is constructed using the idea of ​​GAN. The fault-free image and the fault morphological features extracted from the deformed fault scene image (which can be a combination of the traditional feature descriptors and deep feature vectors mentioned above) are used as inputs. The generative model is trained to learn how to naturally fuse the fault morphology into the fault-free image to generate a new image. During the training process, the adversarial training mechanism (the generator generates the fused image, the discriminator determines whether the generated image is real, and the two are constantly optimized) is used to make the generated image as realistic as possible and consistent with the actual fault characteristics. Finally, the trained generative model is used to transfer the fault morphology of a large number of fault-free images to obtain balanced fault images. This method can better handle complex image fusion situations and generate more natural and high-quality balanced fault images, but it requires more training data and computing resources.

[0282] The quality of the migrated balanced fault image is checked to see if there are any unclear fault morphology, unnatural fusion with the fault-free image (such as obvious edge traces, color incoordination, etc.), and overall image quality degradation (such as blur, distortion, etc.). The migration effect is judged by manual visual inspection combined with computer image quality evaluation indicators (such as calculating the structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR) of the image to quantitatively evaluate the image quality). For the problems found, targeted measures are taken to optimize them, such as adjusting the parameters of the fusion algorithm (such as the gradient weight in Poisson fusion), reselecting a more appropriate fault-free image (if the fusion is poor due to large differences in the image itself), or improving the feature extraction and matching methods (if the fault morphology is not accurately migrated), etc., to ensure that the quality of the generated balanced fault image meets the requirements.

[0283] In order to ensure the rationality and accuracy of the balanced fault images, verification work is required. Some balanced fault images can be extracted and reviewed by railway vehicle inspection professionals to check whether the fault morphology after migration conforms to the actual fault characteristics and whether the presentation on the fault-free image is reasonable and consistent with the actual fault situation. At the same time, the balanced fault images generated by different batches and different methods are compared to ensure that the balanced fault images in the overall data set are consistent and coherent in terms of quality, fault morphology, etc., so as to facilitate the subsequent training and application of intelligent recognition algorithms based on this data set.

[0284] The above-mentioned method of migrating a few fault forms to a fault-free image based on the fault scene image in the real environment to obtain a balanced fault image has the following technical advantages:

[0285] (1) The fault-free image of a railway freight car may differ from the fault scene image in terms of structural layout, shooting angle, and location of freight car components. By deforming the fault area, it can be better adapted to the corresponding position and overall structure in the fault-free image to be migrated. For example, the angle or shape of a component in the fault-free image is inconsistent with the fault area in the fault scene image. Adjusting the fault area by deformation means such as rotation and stretching can make the fault shape more naturally integrated into the fault-free image after migration, improve the rationality and realism of the fusion, avoid the situation where the fault shape is not coordinated with the surrounding environment, and make the generated balanced fault image more consistent with the appearance characteristics and visual effects of the actual railway freight car.

[0286] During the deformation process, the fault area can be appropriately adjusted as needed to highlight the key features of the fault morphology, making it easier to identify and learn when it is subsequently migrated to a fault-free image. For example, some subtle but important fault features (such as tiny cracks on the axle, local wear marks on brake components, etc.) can be made more obvious through deformation operations such as magnification and angle adjustment, so that the intelligent recognition algorithm can more accurately capture these fault morphology information, improve the algorithm's sensitivity and recognition ability to different fault morphologies, and thus improve the accuracy of the algorithm trained based on the balanced fault image data set in actual fault detection.

[0287] (2) The number of fault-free images collected during the operation of railway freight cars is often relatively large. By migrating a few fault morphologies extracted from deformed fault scene images to these fault-free images, the number of fault image samples can be quickly expanded without increasing the cost of additional image acquisition too much, effectively solving the problem of uneven distribution of fault samples, making the data set more comprehensive. Covering various possible fault conditions, improving the integrity and representativeness of the data set, providing richer and more balanced learning materials for the intelligent recognition algorithm, and enhancing the algorithm's comprehensive recognition ability for different fault types.

[0288] In actual railway freight car operation, faults may occur on different individual vehicles, and they are similar to the appearance of vehicles in a fault-free state in general, except for differences in fault morphology in specific parts. The balanced fault image generated by migrating the fault morphology to the fault-free image can better simulate this actual situation, making the data set closer to the actual railway freight car operation scene, so that the intelligent recognition algorithm can be trained based on such a data set. When facing the actual freight car fault detection task, it can more accurately identify the fault conditions hidden under the normal appearance, improve the practicality and generalization ability of the algorithm in practical applications, and better serve the safe operation monitoring of railway freight cars.

[0289] By accurately extracting and migrating fault morphology to fault-free images, the intelligent recognition algorithm can be presented with the manifestation of fault features in different backgrounds and vehicle states in a more targeted manner, allowing the algorithm to learn fault feature information from more angles and more comprehensively. Compared with relying solely on the original fault image, this method enables the algorithm to better understand the difference and correlation between fault features and normal vehicle appearance, strengthen the memory and recognition ability of fault features, and thus more keenly detect faults in actual detection, reduce the occurrence of misjudgments and missed judgments, and improve the overall performance of intelligent recognition of railway freight car fault images.

[0290] Optionally, performing image balancing on the valid railway freight car fault images to obtain balanced fault images includes: performing undersampling processing on valid railway freight car fault images with a large number of samples to obtain reduced fault image samples; determining fault image samples that require data undersampling among the reduced fault image samples based on the distribution of railway freight car fault samples; clustering the fault image samples that require data undersampling to generate image clustering results; and performing random undersampling in each cluster of the image clustering results to generate a balanced fault image.

[0291] Optionally, the specific technical implementation process of performing image balancing on the valid railway freight car fault image to obtain a balanced fault image includes:

[0292] (1) Under-sampling is performed on valid railway freight car fault images with a large number of samples to obtain reduced fault image samples.

[0293] First, all valid railway freight car fault images need to be classified and counted according to classification standards such as fault type, and the number of images in each category (for example, according to classification methods such as different component faults and faults of different causes) is counted. By setting a reasonable threshold (this threshold can be determined based on the overall size of the data set, the expected balance, and past experience, for example, when the number of a certain type of fault image exceeds 10% of the total number of images, it is determined to be a category with a large number of samples), find out those fault image categories whose sample numbers exceed the threshold and determine them as objects that need to be undersampled.

[0294] The most basic random undersampling method can be used, that is, using a random number generator (in programming languages, Python's `random` module has a corresponding random function) in a large number of samples of fault image category data sets, according to the preset sampling ratio (for example, if you want to reduce the number of images of this type to half of the original, set the sampling ratio to 0.5) to randomly select the image samples to be retained, and remove the unselected images from the data set, thereby obtaining a reduced number of fault image samples. This method is simple and direct, but it may lose some valuable information. However, in some cases where the uniformity of data distribution is not extremely high, it can quickly achieve a preliminary reduction in the number of samples.

[0295] Alternatively, if you want to retain representative samples as much as possible during the undersampling process, you can use a distance-based undersampling method. First, convert the fault image with a large number of samples into a feature vector representation (for example, by extracting the color histogram, texture features, deep features extracted based on convolutional neural networks, etc. of the image, and combining these features into a vector), and then calculate the distance between the samples (common distance measurement methods include Euclidean distance, cosine distance, etc., and you can use relevant machine learning libraries, such as the distance calculation function provided in Python's `scikit-learn`). According to the distance information, select those images that are relatively far away from other samples for retention, that is, retain those samples that are relatively dispersed and more representative in the feature space, and remove some samples with close distances and high feature similarity, so as to achieve undersampling while minimizing information loss, so that the reduced fault image samples are more diverse and representative.

[0296] (2) Determining the fault image samples that need to be undersampled among the reduced fault image samples according to the distribution of the fault samples of the railway freight car

[0297] Collect and organize the distribution information of railway freight car fault samples in different dimensions, such as statistical analysis according to vehicle model, fault location, fault severity and other dimensions. For example, count the frequency of various faults in freight cars of different models, observe which models have some fault image samples that are still too concentrated in the corresponding dimension distribution of the entire data set despite the initial undersampling process, or analyze whether the number of samples with faults occurring in different parts of the freight car (such as the running gear, braking system, car body, etc.) is evenly distributed, and whether the proportion of fault samples with different severity is reasonable. Through these multi-dimensional analyses, we can fully understand the current distribution status of fault samples and find out the fault image sample categories or subsets that are still unbalanced in the overall distribution and need further undersampling adjustment.

[0298] According to the above analysis results of the fault sample distribution, combined with the actual operation characteristics of railway freight cars and the requirements of fault detection, determine which specific reduced fault image samples still need to continue data undersampling operations. For example, if it is found that the minor fault image samples of a certain part of a certain model still account for too high a proportion of the entire model-related fault samples after the first round of undersampling, affecting the balance of the data set in different fault severity dimensions, then such fault image samples will be determined as objects that need further undersampling, so that the overall sample distribution of the data set can be optimized through more refined undersampling operations in the future.

[0299] (3) Cluster the fault image samples that need data undersampling to generate image clustering results

[0300] For the fault image samples that need to be further undersampled, information that can effectively characterize the image features is extracted to facilitate clustering operations. A variety of feature extraction methods can also be used, such as extracting traditional manual features of images (including color features, such as color histograms, color moments, etc.; texture features, such as gray-level co-occurrence matrix features, local binary pattern features, etc.; shape features, such as contour-based Fourier descriptors, Hu moments, etc.), or using deep learning models (such as pre-trained convolutional neural networks, taking the feature vector of a certain layer of the feature map output after appropriate processing as the feature representation of the image) to obtain more abstract and high-level features, converting each fault image sample into a corresponding feature vector form, and providing a data basis for clustering analysis.

[0301] Select at least one of the following clustering algorithms to perform clustering operations on these feature vectors:

[0302] If the number of clusters to be clustered is roughly known in advance (it can be estimated based on preliminary observations of fault image samples and related domain knowledge, for example, it is expected that a certain fault can be roughly divided into 3-5 categories under different forms), the K-Means clustering algorithm can be used. First, randomly initialize K cluster centers (K is the preset number of clusters), then calculate the distance from each sample to these cluster centers (such as Euclidean distance), divide the samples into the clusters with the closest cluster centers, and then recalculate the new cluster centers of each cluster (usually the mean of the feature vectors of the samples in the cluster), and repeat this process until the cluster centers no longer change significantly or the preset number of iterations is reached, and finally obtain the clustering results of K clusters, and the image samples in each cluster are relatively close in feature space.

[0303] For situations where the specific number of clusters is unclear or different levels of clustering structures are desired, a hierarchical clustering algorithm can be used. It has two modes: agglomerative and divisive. Taking the agglomerative mode as an example, each sample is initially treated as a separate cluster, and then similar clusters are gradually merged based on the similarity between samples (measured by distance metric) to form larger and larger clusters, and finally a tree-shaped clustering structure is constructed. Different numbers of clusters can be divided at different levels as needed, and the hierarchical relationship and similarity between samples can be intuitively displayed, which is convenient for subsequent undersampling operations based on clusters at different levels.

[0304] If the distribution of fault image samples in the feature space has areas with different densities, that is, some areas have dense samples and some areas have sparse samples, the DBSCAN density clustering algorithm is more suitable. It divides clusters based on the density information of the samples, and divides density-connected samples into the same cluster. It can automatically discover clusters of different shapes and densities. It has a better clustering effect for fault image samples with irregular distribution characteristics. For example, it can discover some sample clusters that are relatively unique in the performance of specific fault forms and form relatively independent density areas, which helps to perform subsequent undersampling operations more accurately.

[0305] (4) Performing random undersampling in each cluster of the image clustering result to generate a balanced fault image

[0306] According to the overall balancing goal of the data set and the number of samples in each cluster, determine the proportion of random undersampling in each cluster. A uniform undersampling ratio can be used (for example, each cluster is set to be undersampled at a ratio of 50%, and the number of samples in the cluster is halved), or different undersampling ratios can be used according to factors such as the size and importance of the cluster. For example, for clusters with a large number of samples and relatively important (such as clusters where common fault forms are located), a higher undersampling ratio (such as 60%) can be set, while for clusters with a small number of samples or relatively minor ones, a lower undersampling ratio (such as 30%) can be set. Through this differentiated undersampling ratio setting, the sample distribution can be adjusted more finely, so that the final generated balanced fault image can achieve a more ideal balanced state in all aspects.

[0307] In each cluster, a random number generator is used to randomly select the image samples to be retained according to the determined undersampling ratio, and the unselected samples are removed from the cluster. For example, in a cluster containing 100 images, if the set undersampling ratio is 50%, 50 images are randomly selected to be retained, and the remaining 50 images are removed. After completing such random undersampling operations for all clusters, the remaining image samples are combined together to generate a balanced fault image data set. At this time, the number of fault image samples of different fault types and forms in the data set is more reasonably and balanced, which is more in line with the requirements of subsequent intelligent recognition algorithm training and overall railway freight car fault image analysis.

[0308] To this end, performing image balancing on the effective railway freight car fault image to obtain a balanced fault image has the following technical benefits:

[0309] (1) In the railway freight car fault image dataset, fault image categories with a large number of samples may occupy a large amount of storage space and computing resources, which will increase the time cost and hardware resource requirements in subsequent data analysis, algorithm training and other operations. By reducing the number of such image samples through undersampling, the scale of the dataset can be effectively controlled, making it easier to manage and process, speeding up operations such as data reading, feature extraction and algorithm training, and improving overall work efficiency. This efficiency improvement effect is more obvious when processing large-scale railway freight car fault image datasets.

[0310] The existence of a large number of repeated or highly similar fault image samples may lead to data redundancy, causing the intelligent recognition algorithm to over-focus on the fault features represented by these samples during the learning process, while ignoring other relatively rare but equally important fault conditions, affecting the algorithm's generalization ability and comprehensive recognition ability for different fault types. Undersampling processing can eliminate a portion of samples with high similarity, optimize the initial distribution of samples to a certain extent, lay the foundation for further fine-tuning the sample distribution and achieving a more balanced data set construction, and guide the algorithm to learn various fault features more comprehensively and balancedly.

[0311] (2) The distribution of railway freight car fault samples is affected by many factors, such as differences in vehicle models, fault locations, and fault severity. Simply undersampling the whole sample may not completely solve the problem of uneven sample distribution, and may also lead to new imbalances in some key dimensions. By analyzing the distribution of fault samples in different dimensions in detail, we can accurately locate those fault image samples that still affect the overall balance, and make further undersampling adjustments to these samples in a targeted manner to ensure that the data set can achieve a relatively reasonable and balanced distribution state in all dimensions, so that the intelligent recognition algorithm can better learn the fault characteristics in different situations based on such a comprehensive and balanced data set, and improve the algorithm's accurate recognition rate of various railway freight car faults in practical applications.

[0312] Considering the business characteristics of the actual operation and fault detection of railway freight cars, different fault sample distributions reflect different practical problems and focus points. For example, in actual maintenance work, the fault handling methods and attention levels for different models, different parts, and different degrees of severity are different. By determining the under-sampling objects based on these actual business-related sample distributions, the constructed data set can be more closely aligned with actual business needs, making the subsequent intelligent recognition system developed based on this data set more practical and instructive in actual applications, and better assisting railway vehicle inspection personnel in fault diagnosis and maintenance decisions.

[0313] (3) Although the railway freight car fault images are all centered around the theme of freight car faults, there are complex internal similarities and differences between different samples in terms of fault morphology and manifestations. Through clustering operations, fault image samples with similar features can be clustered together, and the internal structural relationship of these samples in the feature space can be mined to discover the grouping of different fault morphologies, which helps to gain a deeper understanding of the diversity of faults and the relationship between different morphologies, providing a basis for more reasonable undersampling operations in the future. At the same time, it also provides a valuable data basis for further analysis of the evolution law and characteristic patterns of railway freight car faults.

[0314] Directly performing random undersampling on all samples may mistakenly delete some samples with unique but relatively rare fault forms, or fail to fully consider the similarity differences between samples. However, performing undersampling within each cluster after clustering can more specifically select samples to be retained or removed based on the common characteristics and distribution of samples within the cluster, ensuring that while reducing the number of samples, representative samples of different fault forms are retained to the greatest extent, making the undersampling operation more reasonable and scientific, avoiding problems such as information loss or unreasonable sample distribution caused by undersampling, and further optimizing the quality and balance of the data set.

[0315] (4) Random undersampling is performed within each cluster formed by clustering. The undersampling ratio can be flexibly set according to the specific conditions of each cluster (such as cluster size, sample importance, etc.), so as to achieve more precise control and adjustment of the number of samples, so that the final generated balanced fault image can achieve an ideal balanced state in terms of different fault forms and different feature groupings, solving the problem of uneven sample distribution caused by various complex factors. The data set can comprehensively and evenly cover various fault conditions that may occur in the actual operation of railway freight cars, providing high-quality and balanced learning data for the intelligent recognition algorithm, and improving the algorithm's comprehensive recognition ability and generalization ability for different fault types and forms.

[0316] The random undersampling method is used to select the samples to be retained in each cluster, which can retain the diversity and randomness of the samples to a certain extent. Even within the same cluster, there may be subtle differences between samples. Random undersampling allows these differences to have a chance to be retained, avoiding the problem of sample homogeneity that may be caused by over-regular sample selection, enriching the diversity of the data set, allowing the intelligent recognition algorithm to be exposed to more diverse fault feature manifestations, which helps to improve the robustness and adaptability of the algorithm when facing the actual complex and changeable railway freight car fault images, and better play its fault detection and recognition functions.

[0317] Optionally, the complete component labeling and local labeling of the fault location of the balanced fault image to obtain fault labeling data includes: obtaining labeling strategy data for component integrity and fault location locality to label the balanced fault images of the same vehicle model, the same component and the same fault type, and generating image labels including the component as a whole and fault details; based on a detailed fault type description, performing detailed labeling on the balanced fault image to obtain both a complete labeling of the component as a whole and a local labeling of the fault location; generating fault labeling data based on the image labels of the component as a whole and the fault details, the complete labeling of the component as a whole, and the local labeling of the fault location.

[0318] Optionally, in a specific application scenario, performing complete component labeling and partial labeling of fault locations on the equalization fault image to obtain fault labeling data includes the following technical processing:

[0319] The following are the detailed technical implementation details of the above technical processing links and the corresponding technical benefits:

[0320] (1) Obtain the labeling strategy data of component integrity and fault location locality to label the balanced fault images of the same vehicle model, the same component and the same fault type, and generate image labels containing the overall component and fault details.

[0321] Collect standard documents developed by the railway department on railway freight car component structure, fault diagnosis, and image annotation, such as "Railway Freight Car Operation and Maintenance Regulations" and "Railway Freight Car Components Atlas". These documents specify in detail the standard structure of each component of different models of railway freight cars, normal state characteristics, and the definition, manifestation, and judgment criteria of various types of faults. Extract key information related to component integrity and locality of faults from them, and organize them to form the basic part of the annotation strategy data to ensure that the annotation work meets the authoritative requirements of the industry and actual business specifications.

[0322] Collect successful cases and accumulated experience data from past railway freight car fault image annotation projects, and check the annotation methods, annotation key points, problems encountered and solutions used by professional annotators when processing images of similar vehicle models, components and fault types. Summarize and summarize these historical experiences, extract universal and effective annotation ideas and techniques, and add them to the annotation strategy data to make it more in line with actual operating conditions and improve the practicality and operability of the annotation strategy.

[0323] Experts in the field of railway vehicle inspection are invited to participate in the formulation of the labeling strategy, and their professional insights, rules of thumb, and some detailed knowledge that is difficult to reflect through written standards in their long-term practical work on the integrity judgment of different components and the identification of fault locations are listened to. At the same time, field research is conducted at the railway freight car maintenance site to observe the actual status of the components on the actual vehicle and the specific manifestations after the failure occurs. These actual observations on the spot are fed back into the labeling strategy data, so that the labeling strategy can more accurately reflect the actual status of the railway freight car and enhance the accuracy and reliability of the labeling.

[0324] First, the images of balancing faults are classified and sorted, and grouped according to the truck model, the specific components involved, and the fault type, to ensure that the images in the same group have the same attributes of model, component, and fault type. For example, all balancing fault images involving the C70 model, running gear wheelset components, and the fault type of tread abrasion are grouped together to facilitate subsequent unified labeling operations according to the corresponding labeling strategy.

[0325] According to the requirements of the component integrity in the annotation strategy data, the components in the image are annotated as a whole. Image annotation tools (such as LabelImg, RectLabel and other open source tools, which support drawing a variety of annotation shapes such as rectangular boxes and polygons, and can save annotation information in common data formats such as XML and JSON) can be used to draw a suitable annotation box around the outer contour of the component (for regular-shaped components, such as wheels, they are usually annotated with rectangular boxes; for irregular-shaped components, such as some complex structural components of the hook and buffer device, polygon annotation can be used to more accurately cover their range), and at the same time, the name of the component, the model to which it belongs, and the degree of integrity (such as qualitative descriptions such as intact, partially damaged, etc., can also add specific quantitative indicators according to actual conditions, such as the percentage of component wear, etc.) are recorded in the annotation information to generate image label content that reflects the overall situation of the component.

[0326] For the fault part, the specific location and range of the fault are more accurately marked on the image in accordance with the localization of the fault part in the annotation strategy. Image annotation tools are also used to mark the fault part with smaller and more focused annotation shapes (such as more accurate demarcation of smaller fault areas with rectangular or circular frames; accurate delineation of the boundaries of irregular and scattered fault areas with polygons). The fault type (such as the specific morphological description of tread scratches, detailed feature information such as depth range), the position relationship of the fault part relative to the overall component (such as the upper left corner of the component, the lower middle, etc.) and other relevant details (such as the direction of the fault texture, color changes, etc., which help further describe the fault characteristics) are recorded in detail in the annotation information, so that the image label can fully present the fault details. Through such annotation operations, image labels containing the overall component and fault details are generated for each group of balanced fault images of the same vehicle model, the same component and the same fault type.

[0327] (2) Based on the detailed fault type description, the balanced fault image is carefully labeled to obtain a complete labeling of the entire component and a local labeling of the fault location.

[0328] For each type of railway freight car fault, the description is further refined and decomposed into multiple key feature dimensions for description. For example, for the brake shoe eccentric wear fault in the braking system, the detailed description can include the specific location of the brake shoe eccentric wear (left side, right side or a local area), the degree of eccentric wear (wear range in millimeters), the morphological characteristics of eccentric wear (such as whether it is uniform wear, special shapes formed by local excessive wear, etc.), and the relationship with surrounding components (whether it affects the normal operation of the brake beam, etc.). Through this detailed decomposition, a detailed, comprehensive and operational fault type description system is formed, providing a clear and detailed basis for subsequent detailed labeling.

[0329] Carefully observe and analyze each balanced fault image, and extract various feature information related to the fault from the image, not only limited to the obvious appearance of the fault part, but also including its relationship with the surrounding parts in terms of color, texture, spatial position, etc., as well as the overall state change of the entire part in the image, etc. For example, when annotating an axle crack fault image, in addition to annotating the local features such as the position, length, and width of the crack itself, attention should also be paid to the overall surface condition of the axle, whether there are other potential abnormal signs, and the relative position relationship between the crack part and other structures on the axle (such as the journal, axle body, etc.), etc., to ensure that the annotated information can fully reflect the fault situation.

[0330] According to the detailed fault type description after refinement, multi-dimensional annotation information is added to each image in the image annotation tool. In addition to the basic contents of the overall component annotation and the local annotation of the fault location mentioned above, each refined fault feature dimension is annotated and recorded one by one. For example, for the brake shoe eccentric wear fault image, in addition to marking the overall integrity of the brake shoe, the specific location of the eccentric wear and other conventional information, the specific numerical value of the eccentric wear degree, the detailed text description of the eccentric wear morphology and the observation results of the association with the brake beam should also be accurately recorded, so that the annotated image can present very detailed and comprehensive fault information, including both the complete annotation of the overall component and the local annotation of the fault location, providing rich and accurate data support for subsequent fault analysis, intelligent recognition algorithm training and other work.

[0331] (3) Generate fault annotation data based on image labels of the entire component and fault details, complete annotation of the entire component, and local annotation of the fault location

[0332] Integrate the various annotation information obtained through the previous steps (image labels of the entire component and fault details, complete annotation of the entire component, and local annotation of the fault location) and convert them into a standard data format for subsequent storage, management, and use. The common data format can be JSON, which has good readability and versatility and can easily represent complex hierarchical data. For example, organize the annotation information of each image into a JSON object, which contains the image file name, vehicle model, component name, fault type, and details of each annotation dimension (such as the overall annotation information of the component as a sub-object, including fields such as completeness; the local annotation information of the fault location as another sub-object, including multiple fields such as fault location and feature description), to ensure that all annotation data are consistent in structure and format.

[0333] Clean the integrated annotated data to check whether there are problems such as incomplete annotation information, incorrect annotation format, logical contradictions (such as the annotated fault location exceeds the overall annotation range of the component, etc.). By writing a data verification script (which can be implemented using programming languages ​​such as Python with relevant data processing and verification libraries), the annotated data is traversed and checked, and any problems found are corrected or marked in a timely manner. If necessary, return to the previous annotation step to re-verify the annotation situation to ensure the quality and accuracy of the annotated data so that it meets the requirements of subsequent data analysis and algorithm application.

[0334] Choose an appropriate storage method based on the scale and usage scenarios of the annotated data. For smaller-scale annotated data, you can use the local file system for storage, and classify and store them according to a certain directory structure (such as establishing a multi-layer directory according to vehicle model, component category, etc., and storing the corresponding annotated data files in the corresponding directory) to facilitate search and management. If the amount of annotated data is large, you can consider using a database for storage, such as relational databases such as MySQL and SQLite. By designing a reasonable table structure (for example, creating an image information table, a component annotation table, a fault annotation table, etc., and establishing the relationship between each table through foreign key associations), the annotated data can be stored to facilitate complex query, retrieval, and data association analysis.

[0335] If the annotation data will continue to change and improve with the development of railway freight car technology and the update of annotation standards, it is necessary to establish a version control mechanism. Version control systems (such as Git, etc.) can be used to manage the annotation data files or databases, record the updated content, update time, and update reasons of each annotation data, and facilitate the backtracking and comparison of different versions of annotation data, so as to ensure the maintainability and long-term effectiveness of the annotation data, so that it can continue to provide accurate data support for railway freight car fault image related work.

[0336] To this end, the balanced fault image is fully labeled with components and partially labeled with fault locations, and the fault labeling data is obtained by the following techniques:

[0337] (1) By referring to authoritative railway industry standards, analyzing historical experience, and combining expert knowledge and field research to obtain labeling strategy data, it is possible to formulate unified and scientific specifications and standards for labeling work. In this way, different labelers can follow the same labeling principles and methods when labeling balanced fault images of the same vehicle model, the same component, and the same fault type, avoiding labeling inconsistencies caused by differences in personal subjective understanding or non-standard operations, ensuring the standardization and consistency of the labeling results, making the labeled data more reliable and comparable, and facilitating subsequent data analysis, algorithm training, and collaborative communication between different teams based on the labeled data.

[0338] The labeling strategy data provides a clear and definite operation guide for labelers. Labelers do not need to explore the labeling method from scratch. They can directly locate the labeling focus quickly according to the established strategy, select the appropriate labeling tools and methods for labeling operations, thereby improving the efficiency of labeling work. At the same time, because the strategy data has been verified and optimized in many aspects, labeling based on it can more accurately reflect the overall situation of the component and the detailed characteristics of the fault, reduce the occurrence of labeling errors and omissions, ensure the accuracy of the labeling information, and provide a high-quality labeling data foundation for subsequent work such as intelligent recognition of fault images.

[0339] (2) Describing the fault type in detail and annotating it accordingly can refine the originally general fault information into a multi-dimensional and specific feature description, greatly enriching the content of the annotated data. By recording the detailed features of the fault in different aspects, such as location, degree, shape, and relationship with surrounding components, the real performance of the fault in the image is presented more comprehensively and in-depth, improving the ability to express fault features, so that the annotated data can more accurately depict the differences between different faults, providing richer and more detailed learning materials for the intelligent recognition algorithm, helping the algorithm to better understand and identify various complex railway freight car faults and improve the accuracy and precision of fault recognition.

[0340] In practical applications, railway freight car fault detection requires accurate knowledge of the details of the fault in order to make effective maintenance and processing decisions. This carefully annotated data is closely integrated with actual business needs. The rich fault information it contains can directly provide detailed references for railway vehicle inspection and maintenance personnel in actual work, helping them to more intuitively and comprehensively understand the problems reflected by the fault images, improve the efficiency and accuracy of fault diagnosis, and enhance the practicality and guiding value of annotated data in the actual operation and maintenance of railway freight cars, so that the annotated data not only serves the training of intelligent recognition algorithms, but can also be better integrated into the entire railway freight car fault detection and maintenance business process.

[0341] (3) Integrate the annotation information obtained at different stages and dimensions to generate unified fault annotation data, changing the original situation of scattered annotation information and inconsistent formats, and realizing standardized data management. Subsequent operations such as data analysis, algorithm training, and data sharing can be performed based on this integrated annotation data, avoiding the inconvenience caused by inconsistent data formats and scattered storage, improving the efficiency and convenience of data management, and facilitating centralized maintenance, updating, and long-term preservation of annotation data, ensuring the effective use of annotation data resources.

[0342] The integrated fault labeling data covers rich information on the overall components and fault details, and can meet the needs of different application scenarios. For the training of intelligent recognition algorithms, it provides comprehensive and accurate labeling samples, which helps the algorithm learn more complete fault feature patterns and improve algorithm performance; for the daily work of railway vehicle inspection personnel, these labeling data can be used as a reference for fault diagnosis, helping them to quickly and accurately determine the fault situation; at the same time, when conducting statistical analysis of railway freight car fault data, research on fault evolution laws, etc., rich labeling data can also provide detailed data support, so that fault labeling data can play an important role in multiple fields and work links, and improve the comprehensive utilization value of data.

[0343] The present application also provides a device for constructing a railway freight car fault image intelligent recognition data set, which includes:

[0344] A data acquisition unit is used to acquire a local screenshot of a typical fault of a freight car and a complete fault train monitoring image as an original railway freight car fault image; a cleaning unit is used to score the original railway freight car fault image to obtain an image quality score value, and based on the image quality score value, clean the original railway freight car fault image to obtain a valid railway freight car fault image; an image balancing unit is used to perform image balancing on the valid railway freight car fault image to obtain a balanced fault image; a fault classification unit is used to classify the valid railway freight car fault image to obtain a fault classification label and a fault level; a fault labeling unit is used to perform complete component labeling and partial labeling of the fault part on the balanced fault image to obtain fault labeling data; a construction unit is used to construct a railway freight car fault image intelligent recognition data set based on the balanced fault image and the fault classification label set, the fault level set, and the fault labeling data.

[0345] An embodiment of the present application also provides an electronic device, which includes a memory and a processor, wherein a computer executable program is stored on the memory, and the processor runs the computer executable program to perform the following steps: obtaining a local screenshot of a typical fault of a freight car and a complete fault train monitoring image as an original railway freight car fault image; scoring the original railway freight car fault image to obtain an image quality score value, and based on the image quality score value, cleaning the original railway freight car fault image to obtain a valid railway freight car fault image; performing image balancing on the valid railway freight car fault image to obtain a balanced fault image; classifying the valid railway freight car fault image to obtain a fault classification label and a fault level; performing complete component labeling and partial labeling of the fault location on the balanced fault image to obtain fault labeling data; and constructing a railway freight car fault image intelligent recognition data set based on the balanced fault image and the fault classification label set, fault level set, and fault labeling data.

[0346] The embodiment of the present application also provides a computer program product having a computer executable program stored thereon, and the computer executable program is run to execute the following steps: obtaining a local screenshot of a typical freight car fault and a complete fault train monitoring image as an original railway freight car fault image; scoring the original railway freight car fault image to obtain an image quality score value, and based on the image quality score value, cleaning the original railway freight car fault image to obtain a valid railway freight car fault image; performing image balancing on the valid railway freight car fault image to obtain a balanced fault image; classifying the valid railway freight car fault image to obtain a fault classification label and a fault level; fully labeling the components of the balanced fault image and partially labeling the fault location to obtain fault labeling data; constructing a railway freight car fault image intelligent recognition data set based on the balanced fault image and the fault classification label set, fault level set, and fault labeling data. For exemplary explanations of each of the above steps, please refer to the above Figure 1 Records of.

[0347] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for constructing a data set for intelligent recognition of railway freight car fault images, characterized in that: include: Obtaining a typical partial screenshot of a freight car fault and a complete monitoring image of a faulty train as the original railway freight car fault image; Scoring the original railway freight car fault image to obtain an image quality score value, and cleaning the original railway freight car fault image based on the image quality score value to obtain a valid railway freight car fault image; Performing image balancing on the valid railway freight car fault image to obtain a balanced fault image; Classifying the valid railway freight car fault images to obtain fault classification labels and fault levels; Performing complete component labeling and partial labeling of fault locations on the balanced fault image to obtain fault labeling data; Based on the balanced fault image and the fault classification label set, fault level set, and fault annotation data, a railway freight car fault image intelligent recognition data set is constructed.

2. The method for constructing a railway freight car fault image intelligent recognition dataset according to claim 1, characterized in that: The method further comprises: Access the typical fault local image library to obtain local screenshots of typical truck faults manually identified during the daily operation of TFDS; Access the faulty train image library to obtain complete faulty train monitoring images found in TFDS simulation tests and actual cases; The typical local screenshot of the freight car fault and the complete fault train monitoring image are used as the original railway freight car fault image.

3. The method for constructing a railway freight car fault image intelligent recognition dataset according to claim 1, characterized in that: Scoring the original railway freight car fault image to obtain an image quality score value includes: scoring the original railway freight car fault image based on a constructed image quality evaluation model to obtain an image quality score value.

4. The method for constructing a railway freight car fault image intelligent recognition dataset according to claim 3 is characterized in that: The image quality evaluation model includes a feature extraction network, a rule mapping network, and a quality evaluation network; the image quality evaluation model constructed based on which the original railway freight car fault image is scored to obtain an image quality score value includes: Based on the feature extraction network, semantic features are extracted from the original railway freight car fault image to obtain image semantic features; Based on the rule mapping network, the image semantic features are mapped to a pre-constructed quality evaluation rule description to obtain a rule mapping result; Based on the quality evaluation network, the rule mapping result is processed to obtain the image quality score value.

5. The method for constructing a railway freight car fault image intelligent recognition dataset according to claim 1, characterized in that: The valid railway freight car fault images are classified according to the railway freight car maintenance procedure description data and the railway freight car historical fault data to obtain the fault classification label and fault level, including: Extracting fault description terms from the maintenance procedure description data of the railway freight car, so as to construct a fault rule dictionary based on the extracted fault description terms, wherein each entry in the fault rule dictionary is constructed with a fault classification label and an initial clue of the fault level; Cleaning the historical fault data of the railway freight car to extract fault descriptions therefrom and establish an association relationship between the fault descriptions and historical fault images, wherein the fault descriptions include at least one of a location where the fault occurred, a type of the fault, and a fault frequency; Based on the association relationship, the valid railway freight car fault image is matched with the fault rule dictionary to obtain a fault classification label and a fault level.

6. The method for constructing a railway freight car fault image intelligent recognition dataset according to claim 1, characterized in that: The performing image balancing on the valid railway freight car fault image to obtain a balanced fault image includes: Oversampling is performed on valid railway freight car fault images with a small number of samples to obtain increased fault image samples; According to the real vehicle dismantling simulation failure, generating a fault scene image in a real environment based on the increased fault image samples; Based on the fault scene image in the real environment, a few fault forms are transferred to the fault-free image to obtain a balanced fault image.

7. The method for constructing a railway freight car fault image intelligent recognition dataset according to claim 1, characterized in that: The performing image balancing on the valid railway freight car fault image to obtain a balanced fault image includes: Under-sampling is performed on valid railway freight car fault images with a large number of samples to obtain reduced fault image samples; Determining, according to the distribution of railway freight car fault samples, fault image samples that need to be undersampled among the reduced fault image samples; Clustering fault image samples that require data undersampling to generate image clustering results; Random undersampling is performed in each cluster of the image clustering result to generate a balanced fault image.

8. A device for constructing a data set for intelligent identification of railway freight car fault images, characterized in that: include: A data acquisition unit, used to acquire a local screenshot of a typical fault of a freight car and a monitoring image of a complete faulty train as an original railway freight car fault image; a cleaning unit, configured to score the original railway freight car fault image to obtain an image quality score value, and to clean the original railway freight car fault image based on the image quality score value to obtain a valid railway freight car fault image; An image balancing unit, used for performing image balancing on the effective railway freight car fault image to obtain a balanced fault image; A fault classification unit, used to classify the valid railway freight car fault image to obtain a fault classification label and a fault level; A fault labeling unit, used for fully labeling components and partially labeling fault locations on the equalization fault image to obtain fault labeling data; A construction unit is used to construct a railway freight car fault image intelligent recognition data set based on the balanced fault image and the fault classification label set, fault level set, and fault annotation data.

9. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein a computer executable program is stored in the memory, and the processor runs the computer executable program to perform the following steps: Obtaining a typical partial screenshot of a freight car fault and a complete monitoring image of a faulty train as the original railway freight car fault image; Scoring the original railway freight car fault image to obtain an image quality score value, and cleaning the original railway freight car fault image based on the image quality score value to obtain a valid railway freight car fault image; Performing image balancing on the valid railway freight car fault image to obtain a balanced fault image; Classifying the valid railway freight car fault images to obtain fault classification labels and fault levels; Performing complete component labeling and partial labeling of fault locations on the balanced fault image to obtain fault labeling data; Based on the balanced fault image and the fault classification label set, fault level set, and fault annotation data, a railway freight car fault image intelligent recognition data set is constructed.

10. A computer program product, characterized in that A computer executable program is stored thereon, and the computer executable program is run to perform the following steps: Obtaining a typical partial screenshot of a freight car fault and a complete monitoring image of a faulty train as the original railway freight car fault image; Scoring the original railway freight car fault image to obtain an image quality score value, and cleaning the original railway freight car fault image based on the image quality score value to obtain a valid railway freight car fault image; Performing image balancing on the valid railway freight car fault image to obtain a balanced fault image; Classifying the valid railway freight car fault images to obtain fault classification labels and fault levels; Performing complete component labeling and partial labeling of fault locations on the balanced fault image to obtain fault labeling data; Based on the balanced fault image and the fault classification label set, fault level set, and fault annotation data, a railway freight car fault image intelligent recognition data set is constructed.

Images