Public work equipment disease identification method and device

Abstract

The invention provides a public service equipment disease identification method and device. The public service equipment disease identification method comprises the steps of obtaining a to-be-identified public service equipment image; inputting the to-be-recognized public work equipment image into a trained equipment recognition model to obtain a public work equipment recognition result corresponding to the to-be-recognized public work equipment image; based on the public service equipment identification result, determining a first disease identification model corresponding to the equipment type in the public service equipment identification result from a disease identification model group, and inputting the to-be-identified public work equipment image into the trained first disease identification model to obtain an equipment disease identification result corresponding to the to-be-identified public work equipment image. The disease identification method can automatically identify the diseases of different types of public work equipment, and improves the model generalization ability and the disease identification efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of engineering equipment defect recognition, and in particular to an engineering equipment defect recognition method and device. Background Art

[0002] Timely inspection and repair of equipment defects are crucial to ensure the safety and stability of high-speed railway operations. Affected by environmental factors such as seasonal weather changes, spring and autumn are the periods when equipment accidents are most likely to occur in high-speed railway maintenance sections. Therefore, there are a large number of equipment to be inspected, which brings great challenges to equipment maintenance and inspection. Traditional defect identification methods mainly rely on manual observation and judgment, and have high requirements on the professional level and experience of inspectors. However, due to the large number and variety of defects in maintenance equipment in spring and autumn, manual identification is not only time-consuming and labor-intensive, but also prone to misdetection, resulting in low efficiency and accuracy of maintenance in spring and autumn. Therefore, the development of an automated defect identification system is of great significance to improving the efficiency and accuracy of railway maintenance equipment maintenance in spring and autumn.

[0003] With the development of image detection technology and computer vision technology in recent years, many studies on automatic defect recognition methods based on deep learning convolutional neural networks have been carried out in China, and good recognition and classification effects have been achieved on a certain basis. For example: In the invention patent application with publication number CN114066878A, a method for automatic bridge defect recognition based on the YOLOv5 deep learning algorithm is disclosed; the invention mainly includes the following steps: (1) Screenshot the bridge defect video and construct a relevant data set. (2) Introduce the attention mechanism module YOLOv5 algorithm, optimize its network structure and loss function, and train the defect recognition model in batches. (3) Observe the convergence and fitting of the model during the training process, and continuously adjust the hyperparameters. (4) Use the trained model to identify defects, output the defect category and positioning results, and generate an XML file. Although the existing automatic bridge defect recognition method can automatically recognize bridge defects and improve recognition efficiency compared with manual recognition methods, the existing automatic bridge defect recognition method is only for specific bridge types, resulting in poor model generalization ability and low recognition efficiency, and cannot complete the recognition of defects of other types of engineering equipment other than bridges. Therefore, how to improve the model generalization ability and recognition efficiency in the automatic recognition of engineering equipment defects is a technical problem that needs to be solved urgently. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a method and device for identifying defects in engineering equipment to eliminate or improve one or more defects in the prior art.

[0005] One aspect of the present invention provides a method for identifying defects in civil engineering equipment, the method comprising:

[0006] Acquire images of the engineering equipment to be identified;

[0007] Inputting the to-be-identified engineering equipment image into a trained equipment recognition model to obtain an engineering equipment recognition result corresponding to the to-be-identified engineering equipment image;

[0008] Based on the engineering equipment identification result, a first disease identification model corresponding to the equipment type in the engineering equipment identification result is determined from the disease identification model group, and the engineering equipment image to be identified is input into the trained first disease identification model to obtain the equipment disease identification result corresponding to the engineering equipment image to be identified.

[0009] In some embodiments of the present invention, the device identification model includes a first convolutional layer, a first flatten layer, a first fully connected layer, a first dropout layer and a first output layer; and / or,

[0010] The disease recognition model includes a second convolutional layer, a maximum pooling layer, a residual block, a global average pooling layer, a second flatten layer, a second fully connected layer, a second dropout layer and a second output layer.

[0011] In some embodiments of the present invention, the disease recognition model group includes multiple disease recognition models, and the multiple disease recognition models correspond to different equipment types; and / or,

[0012] Inputting the to-be-identified maintenance equipment image into the first trained defect recognition model to obtain an equipment defect recognition result corresponding to the to-be-identified maintenance equipment image, including:

[0013] Dividing the to-be-recognized engineering equipment image into a plurality of candidate frames;

[0014] Inputting the image in each candidate frame into the first trained defect recognition model to obtain the equipment defect recognition result of the to-be-recognized maintenance equipment image, wherein the equipment defect recognition result includes the defect area and the defect type;

[0015] Drawing a boundary box of the defective area on the image of the engineering equipment to be identified based on the identified defective area;

[0016] The disease type is marked in the disease area boundary box based on the identified disease type.

[0017] In some embodiments of the present invention, determining a first disease identification model corresponding to the equipment type in the work equipment identification result from a disease identification model group based on the work equipment identification result includes:

[0018] The first disease identification model corresponding to the equipment type in the engineering equipment identification result is determined from the disease identification model group through conditional logic or dictionary query.

[0019] In some embodiments of the invention, the method comprises:

[0020] Construct a sample data set, pre-train an initial equipment recognition model based on the sample data set to obtain a trained equipment recognition model, and pre-train each initial disease recognition model based on the sample data set to obtain a trained disease recognition model; wherein the sample data in the sample data set includes engineering equipment image sample data, engineering equipment recognition result sample data, and equipment disease recognition result sample data.

[0021] In some embodiments of the present invention, pre-training an initial device recognition model based on the sample data set to obtain a trained device recognition model includes:

[0022] Obtaining the first convolutional layer of the device recognition model based on the convolutional layer transfer learning of the pre-trained VGG16 model; and / or,

[0023] Based on the sample data set, each initial disease recognition model is pre-trained to obtain each trained disease recognition model, including: based on the convolutional layer transfer learning of the pre-trained ResNet50 model, the second convolutional layer of the disease recognition model is obtained.

[0024] In some embodiments of the present invention, constructing a sample data set includes:

[0025] Collecting image sample data of various types of engineering equipment corresponding to the engineering equipment;

[0026] Label the equipment type and disease type of the engineering equipment image sample data corresponding to each type of engineering equipment;

[0027] Dividing the labeled multiple engineering equipment image sample data into multiple sub-datasets based on equipment types;

[0028] Performing data cleaning and data enhancement on the sample data of the engineering equipment images in each of the sub-data sets;

[0029] Each of the sub-datasets after data cleaning and data enhancement is divided into a training set, a validation set and a test set.

[0030] In some embodiments of the present invention, the equipment types include turnout, track, rail, regulator, weld, ballastless track, ballasted track and ballastless track bed; and / or,

[0031] The loss function of the device identification model is: Where N represents the number of sample data, K represents the number of device categories, t n，k represents the true label of the nth sample on the kth device category, y n，k represents the model's predicted probability that the nth sample belongs to the kth device category;

[0032] The regularization algorithm of the disease recognition model is: Among them, L 0 represents the cross entropy loss, λ represents the regularization coefficient, and w i represents the weight, and m represents the number of samples.

[0033] According to another aspect of the present invention, a system for identifying defects in engineering equipment is also disclosed, including a processor, a memory, and a computer program stored in the memory, wherein the processor is used to execute the computer program, and when the computer program is executed, the system implements the steps of the method described in any of the above embodiments.

[0034] According to another aspect of the present invention, a computer-readable storage medium is disclosed, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0035] The method and device for identifying defects in engineering equipment disclosed in the above-mentioned embodiment of the present invention firstly identifies the equipment type of the engineering equipment image to be identified based on the equipment identification model, then determines the first defect identification model corresponding to the equipment type, and identifies the equipment defect type in the engineering equipment image to be identified based on the first defect identification model. The method accurately identifies and locates equipment defects based on the defect identification model matching the equipment type, which can not only identify equipment defects of various types of engineering equipment, but also improve the accuracy and efficiency of identifying engineering equipment defects.

[0036] In addition, the first convolutional layer of the equipment identification model of the present application is obtained based on VGG16 model transfer learning, and the second convolutional layer of the disease identification model is obtained based on ResNet50 model transfer learning, thereby reducing the labeling cost of sample data during the pre-training process of the disease identification model and the equipment identification model, and further improving the efficiency of engineering equipment disease identification.

[0037] In addition to the above, the engineering equipment defect identification method and device of the present application draws a defect area boundary box on the engineering equipment image to be identified based on the identified defect area, and marks the defect type in the defect area boundary box based on the identified defect type. This method realizes the visual display of the defect area and improves the intuitiveness and interpretability of the identification results.

[0038] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0039] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention. The components in the drawings are not drawn to scale, but are only for illustrating the principles of the present invention. In order to facilitate the illustration and description of some parts of the present invention, the corresponding parts in the drawings may be enlarged, that is, they may become larger relative to other components in the exemplary device actually manufactured according to the present invention. In the drawings:

[0041] Figure 1 The figure is a flow chart of a method for identifying faults of civil engineering equipment according to an embodiment of the present invention.

[0042] Figure 2 It is a schematic flow chart of a method for identifying faults of civil engineering equipment according to another embodiment of the present invention.

[0043] Figure 3 Schematic diagram of the architecture of a system for identifying faults in civil engineering equipment according to an embodiment of the present invention.

[0044] Figure 4 FIG. 4 is a schematic diagram of the architecture of a device identification model according to an embodiment of the present invention.

[0045] Figure 5 FIG. 4 is a schematic diagram of the architecture of a disease recognition model according to an embodiment of the present invention.

[0046] Figure 6 FIG. 4 is a schematic diagram of the architecture of a system for identifying faults in construction equipment according to another embodiment of the present invention.

[0047] Figure 7 This is a flow chart of a method for identifying defects in civil engineering equipment according to an embodiment of the present invention applied to a special maintenance project in spring and autumn. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0049] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0050] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0051] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0052] The existing automatic identification method for engineering equipment defects, especially the automatic identification method for high-speed railway engineering equipment defects, has a high reliance on manual annotation and weak generalization ability of the model when performing defect identification, because the training process of the traditional convolutional neural network depends on a large number of manually labeled data sets and is trained for specific types of defects. Therefore, the existing automatic identification method for engineering equipment defects has a high reliance on manual annotation and a weak generalization ability of the model, which not only has low recognition efficiency, but also is difficult to adapt to the identification tasks of other types of defects other than the trained defect types. Therefore, in view of the problems of strong data dependence and limited generalization ability of the existing automatic identification method for engineering equipment defects, the present application discloses a method and device for identifying engineering equipment defects, which can be applied to the identification of different engineering equipment defects in different projects. In the following, the engineering equipment defect identification method is described in detail by taking the spring and autumn special maintenance project of the high-speed railway as an example, but it should be understood that the spring and autumn special maintenance project of the high-speed railway does not constitute a limitation on the application scenario of the engineering equipment defect identification method and device of the present application. In addition, the defect identification method and device of the present application can also be applied to other maintenance projects.

[0053] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0054] Figure 1 FIG. 1 is a flow chart of a method for identifying a fault in a construction equipment according to an embodiment of the present invention. Figure 1 As shown, the method at least includes steps S10 to S30.

[0055] Step S10: Acquire the image of the engineering equipment to be identified.

[0056] In this step, the maintenance equipment images to be identified may be spring and autumn special maintenance images of high-speed railway equipment uploaded by users, and may specifically be images taken by photographic equipment.

[0057] Step S20: inputting the to-be-recognized engineering equipment image into a trained equipment recognition model to obtain an engineering equipment recognition result corresponding to the to-be-recognized engineering equipment image.

[0058] In this step, the input and output of the equipment recognition model are the image of the engineering equipment to be recognized and the recognition result of the engineering equipment, respectively. The recognition result of the engineering equipment includes the equipment type of the engineering equipment.

[0059] Step S30: Based on the engineering equipment identification result, determine the first defect identification model corresponding to the equipment type in the engineering equipment identification result from the defect identification model group, and input the engineering equipment image to be identified into the trained first defect identification model to obtain the equipment defect identification result corresponding to the engineering equipment image to be identified.

[0060] In this step, based on the equipment type determined in step S20, a first disease recognition model corresponding to the equipment type is determined from the disease recognition model group, and then the defects in the engineering equipment image to be identified are identified based on the first disease recognition model. The disease recognition model group includes multiple disease recognition models, and the multiple disease recognition models are respectively used to identify the defects of a type of engineering equipment; specifically, the disease recognition model group includes multiple disease recognition models, and the multiple disease recognition models correspond to different equipment types. Therefore, when performing disease recognition on the engineering equipment image to be identified, this step uses one of the disease recognition models in the disease recognition model group to perform recognition, and the first disease recognition model used is an exclusive model that is highly matched with the engineering equipment to be identified. Therefore, this method is based on setting a plurality of different types of disease recognition models, so that the method is applicable to different engineering equipment, thereby improving the generalization ability of the model, and improving the accuracy and efficiency of the recognition results.

[0061] Furthermore, the equipment recognition model may include a first convolutional layer, a first flatten layer, a first fully connected layer, a first dropout layer and a first output layer, and the disease recognition model may include a second convolutional layer, a maximum pooling layer, a residual block, a global average pooling layer, a second flatten layer, a second fully connected layer, a second dropout layer and a second output layer.

[0062] In one embodiment, the equipment identification model can be constructed based on the VGG16 model, and the disease identification model can be constructed based on the ResNet50 model.

[0063] The VGG16 model uses a deep convolutional neural network pre-trained with the open source dataset ImageNet. It has a simple architecture and high computational efficiency, and is suitable for the classification of device types in the first phase of this application, such as Figure 4As shown, the VGG16 model contains five convolutional layer blocks and three fully connected layers. When building a device recognition model based on the VGG16 model, this application retains the five convolutional layer blocks, removes the three fully connected layers in the VGG16 model, and adds a flatten layer, a fully connected layer, a dropout layer and an output layer in sequence on the basis of the five convolutional layer blocks to meet the needs of the device type recognition task. Specifically, the first convolutional layer block includes two 64*3*3 convolutional layers, the second convolutional layer block includes two 128*3*3 convolutional layers, the third convolutional layer block includes three 256*3*3 convolutional layers, the fourth convolutional layer block includes three 512*3*3 convolutional layers, and the fourth convolutional layer block includes three 512*3*3 convolutional layers. In the equipment recognition model constructed based on the VGG16 model of the present application, the five convolutional layer blocks in the first convolutional layer retain the pre-trained weights of the VGG16 model on ImageNet, and can extract the mid- and bottom-level features of the image, such as texture, shape, and color; the first flatten layer converts the multi-dimensional feature matrix processed by the convolutional layer and the pooling layer into a one-dimensional vector, and transitions to the fully connected layer; the first fully connected layer contains 512 neurons, which enhances the model's ability to capture high-level features of the equipment type, enables the model to obtain higher generalization capabilities, and can adapt to equipment type classification tasks; the first dropout layer randomly discards neurons at a ratio of 0.5, which helps prevent overfitting; the first output layer can contain 8 neurons, and the number of neurons in the first output layer is the same as the number of types of engineering equipment.

[0064] In the first output layer of the device identification model, the Softmax activation function outputs probability vectors of eight device types, such as Where x is a multidimensional vector, x i is the i-th element in the vector, and K in the formula is the total number of equipment types; when the number of neurons in the output layer is 8, the output result of the output layer is a 1*8 matrix, recorded as V_result=[v_1, v_2, v_3, v_4, v_5, v_6, v_7, v_8], where v_i represents the probability of the i-th equipment type. It can be understood that the number of equipment types of the engineering equipment listed in this embodiment is only an example, and in other embodiments, the equipment types can be more.

[0065] The ResNet50 model uses a deep residual neural network that has been pre-trained on the open source dataset ImageNet. The introduction of residual blocks effectively solves the problems of gradient vanishing and gradient exploding. Compared with the VGG16 model, features such as residual connections enable it to have higher performance in deep network structures and can capture more and more complex features. Therefore, the disease recognition model built based on the ResNet50 model can process more detailed and multi-level image features and enable the model to achieve a higher generalization ability.

[0066] like Figure 5 As shown, the ResNet50 model includes a 64*7*7 convolution layer, a maximum pooling layer with a stride of 2, four residual blocks (the number of convolution kernels are 64, 128, 256, and 512 respectively), a global average pooling layer, and a fully connected layer. When constructing a disease recognition model based on the ResNet50 model, this application retains all the convolution layers and residual blocks in the ResNet50 basic network for extracting underlying features in the image; removes the fully connected layer in the ResNet50 model, and adds a flatten layer, a fully connected layer with 1024 neurons, a dropout layer with a ratio of 0.5, and an output layer with 6 neurons, so that the disease recognition model can extract more complex high-level image features suitable for disease recognition tasks and complete disease type classification tasks. In this defect recognition model, the number of neurons in the output layer is 6 because there are five types of defects corresponding to each type of equipment. In addition, maintenance equipment may also be free of defects. Therefore, the six neurons correspond to five types of defects and no defects. Taking the spring and autumn maintenance defect images of high-speed railway turnout equipment as an example, the corresponding defect recognition results can be wear, block drop, gap separation, abrasion, rust and no defect. Specifically, the final output result of the defect recognition model is a 1*6 matrix, denoted as R a _result = [R n _1, R n _2, R n _3, R n _4, R n _5, R n _6], where R a _b represents the probability of the bth disease type under equipment type a, where R a _6 indicates the probability that this type of equipment is free of defects.

[0067] In addition, since different engineering equipment corresponds to different disease types, the present application sets up a disease identification model group so that each type of equipment corresponds to a disease identification model. In the first stage, after obtaining the equipment type label corresponding to the highest probability based on the equipment identification model, in the second stage, according to the correspondence between the equipment type and the disease identification model, the first disease identification model corresponding to the equipment type in the engineering equipment identification result is determined from the disease identification model group through conditional logic or dictionary query, that is, effective serial processing is performed based on the equipment identification model and the disease identification model to realize dynamic selection of the corresponding disease identification model based on the equipment type.

[0068] In the above embodiment, device type recognition and disease type recognition are performed in two stages respectively. In the first stage, a device recognition model based on a pre-trained VGG16 model is used to classify the input image and identify the type of device. The recognition result is used as the basis for selecting the disease recognition model in the second stage. In the second stage, based on the device type prediction result in the first stage, a corresponding disease recognition model based on the ResNet50 model is dynamically selected to perform disease type recognition on the same input image, and one of the six fixed disease types is identified as the disease recognition result of the input image. Exemplarily, Figure 6 The schematic diagram of the architecture of the maintenance equipment defect recognition system applied to the spring and autumn maintenance defect images of high-speed railway turnout equipment is shown in FIG. Figure 6 As shown, firstly, the equipment recognition model constructed based on the VGG16 model recognizes the type of maintenance equipment in the image, and further, according to the equipment recognition result, the first defect recognition model corresponding to the equipment type in the image is selected from the defect recognition model group constructed based on the ResNet50 model (the turnout equipment defect recognition model selected in this embodiment) and the defects in the image are recognized based on the first defect recognition model. In the spring and autumn maintenance project of high-speed railways, the types of maintenance equipment may include turnouts, lines, rails, regulators, welds, ballastless tracks, ballasted tracks and ballastless track beds, and the corresponding defect recognition model group includes defect recognition models such as turnout defect recognition model, line defect recognition model, rail defect recognition model, regulator defect recognition model, weld defect recognition model, ballastless track defect recognition model, ballasted track defect recognition model and ballastless track bed defect recognition model.

[0069] In some embodiments, the image of the engineering equipment to be identified is input into the first trained disease recognition model to obtain the equipment disease recognition result corresponding to the image of the engineering equipment to be identified, including the following steps: dividing the image of the engineering equipment to be identified into multiple candidate frames; inputting the image in each of the candidate frames into the first trained disease recognition model to obtain the equipment disease recognition result of the image of the engineering equipment to be identified, the equipment disease recognition result including the disease area and the disease type; drawing a disease area boundary box on the image of the engineering equipment to be identified based on the identified disease area; marking the disease type in the disease area boundary box based on the identified disease type.

[0070] In the above embodiment, the disease identification results are further visualized to improve the intuitiveness of the disease identification results. For example, tool libraries such as Keras and Matplotlib can be used to realize the visualization of the disease area; first, Keras is used to load the trained device identification model to identify the device type of the input image; according to the identified device type, the corresponding disease identification model is selected to identify the disease area; each image is divided into multiple candidate frames using sliding window technology, each candidate frame has a size of 64*64 pixels and a step size of 32 pixels, and then these candidate frames are classified by the disease identification model to predict the boundary box and disease type of the disease area; finally, Matplotlib and OpenCV cv2.rectangle and other functions are used to draw these boundary boxes on the original image, and the cv2.putText function is used to mark the disease type label above the box, and finally an intuitive visualization image of the disease area is generated and displayed. In addition, it should be noted that the disease recognition model built based on the ResNet50 model requires the input image size to be 256*256 pixels, while the candidate frame size after the sliding window technology is 64*64 pixels. Therefore, before the pre-training of each disease recognition model begins, it is necessary to use the resize function in the OpenCV library to resize the image part in each candidate frame to a size of 256*256 to adapt to the input size requirements of the disease recognition model.

[0071] In some other embodiments, the method for identifying defects in civil engineering equipment further includes the following steps: constructing a sample data set, pre-training an initial equipment recognition model based on the sample data set to obtain a trained equipment recognition model, and pre-training each initial defect recognition model based on the sample data set to obtain a trained defect recognition model; wherein the sample data in the sample data set includes civil engineering equipment image sample data, civil engineering equipment recognition result sample data, and equipment defect recognition result sample data.

[0072] In addition, since the equipment recognition model is constructed based on the VGG16 model, and the disease recognition model is constructed based on the ResNet50 model, the first convolutional layer of the equipment recognition model can be obtained based on the convolutional layer transfer learning of the pre-trained VGG16 model, and / or the second convolutional layer of the disease recognition model can be obtained based on the convolutional layer transfer learning of the pre-trained ResNet50 model.

[0073] Furthermore, when constructing a sample data set, the steps include: collecting engineering equipment image sample data corresponding to various types of engineering equipment; annotating the engineering equipment image sample data corresponding to various types of engineering equipment with equipment type and disease type; dividing the labeled multiple engineering equipment image sample data into multiple sub-data sets based on equipment type; performing data cleaning and data enhancement on the engineering equipment image sample data in each of the sub-data sets; and dividing each of the sub-data sets after data cleaning and data enhancement into a training set, a validation set, and a test set.

[0074] When collecting images, collect images of various types of maintenance equipment, including images of equipment without diseases and equipment affected by diseases. For example, the images can be taken by mobile devices in workshops along the high-speed railway in spring and autumn, or obtained from the open source dataset of high-speed railway maintenance equipment diseases. After obtaining the image samples, the obtained images are classified and labeled according to the equipment category and disease type, and the label content is the equipment category and disease type; for example, the equipment category can be marked as E, and the disease type can be marked as D. The special maintenance equipment in spring and autumn is divided into eight categories: turnouts, lines, rails, regulators, welds, ballastless tracks, ballasted tracks and ballastless trackbeds, and the equipment categories are marked as E. 1 、E 2 、E 3 、E 4 、E 5 、E 6 、E 7 and E 8 Each type of equipment corresponds to five different types of diseases, and the disease is marked as d 1 d 2 d 3 d 4 d 5 and 6 , d 1 to d 5 For five different diseases, d 6 Represents no disease. Since different types of equipment correspond to different diseases, the d in each type of equipment 1 d 2 d 3 d 4 d 5The specific diseases represented are different; that is, a certain type of disease image under each type of equipment is an independent sub-dataset, such as E n {d m}, n∈[1,8], m∈[1,6]. In the process of data cleaning, the quality of the labeled image data is checked, blurred and duplicate images are filtered, the images are uniformly set to jpeg format, and the size is uniformly adjusted to 224*224. In the process of data enhancement, it is necessary to ensure that the number of images in each sub-dataset can be balanced; if there are insufficient images in a sub-dataset, data enhancement is used to randomly rotate, translate, add noise, and transform the color of each existing image in the sub-dataset, so that the sample data corresponding to various diseases under various types of engineering equipment is roughly balanced. In the data set division step, the labeled image set can be divided into training set, validation set, and test set in a ratio of 8:1:1. The training set and validation set will be used for model training, and the test set will be used for the final model evaluation; it should be understood that no matter which data set is used, it is necessary to ensure that each E n {d m The number of images in the sub-datasets is balanced to a certain extent.

[0075] In addition, in the above embodiment, since the five convolutional layer blocks of the device recognition model retain the initial image classification features obtained by the VGG16 model through pre-training, during the training process of the device recognition model, the convolution part of the VGG16 model can be frozen, and only the newly added fully connected layer and output layer in the device recognition model can be trained. This method allows the model to extract high-level features of the device image more efficiently. Exemplarily, when training the device recognition model, forward iteration and reverse iteration methods are used to adjust the learning parameter weights. The forward iteration calculates the results and makes predictions based on the input and set parameters and functions, while the reverse iteration learns and optimizes the parameters based on the gradient value given by the loss function. In one embodiment, the cross entropy, which is the most widely used in multi-category classification tasks, is used as the loss function of the device recognition model, such as In this formula, N represents the number of sample data, K represents the number of device categories, for example, K = 8, t n，k represents the true label of the nth sample on the kth device category, y n，kIndicates the model's predicted probability that the nth sample belongs to the kth device category. During training, the Adam optimizer can be used to adjust the model's adaptive learning rate through faster convergence speed; and the tuning of hyperparameters largely determines the performance of the device recognition model and is the key to this training; the hyperparameters that need to be debugged in this model include learning rate, batch size, and dropout ratio. The learning rate is initialized to 0.001, the batch size is initialized to 64, and the dropout ratio is initialized to 0.5. In the specific training process, the accuracy and loss of the model on the training set and the validation set will be used as the basis for adjusting the hyperparameters. For example, the following adjustment mechanism can be adopted: if the validation accuracy is low, the generalization of the model can be improved by increasing the batch size; if the convergence speed of the training curve is slow, increase the learning rate; if the training curve lacks stability and has obvious noise, reduce the learning rate; if both the training and validation accuracy are relatively low, it means that the model complexity is not enough, and the model can learn more features by reducing the dropout ratio; if the training accuracy is high and the validation accuracy is low, it means that the model is overfitted on the training set, and the model's dependence on the training set can be reduced by increasing the dropout ratio to enhance the generalization of the model; based on the above adjustment mechanism, after continuous adjustment and optimization of the hyperparameters, the best model parameter combination is saved as the final device recognition model.

[0076] When the equipment recognition model reaches the ideal accuracy, the various disease recognition models built based on the ResNet50 model are further pre-trained. Specifically, multiple disease recognition models are trained separately, and the input of the disease recognition model is the image grouped according to the equipment type and disease type. Similar to the equipment recognition model, since the convolution layer and each residual layer block of the disease recognition model retain the initial image classification features obtained by the ResNet50 model through pre-training, in this training, the convolution part of the ResNet50 model will be frozen, and only the newly added fully connected layer and output layer will be trained. This method allows the model to extract high-level features of disease images more efficiently. The ResNet50 model can also be trained based on the cross-entropy loss function during pre-training.

[0077] Unlike the above-mentioned equipment identification model, the ResNet50 model has strong gradient stability due to its residual block structure, so it is less dependent on the Adam optimizer in dealing with gradient vanishing. In addition, the fast convergence characteristics of the Adam optimizer can easily cause deep neural networks to fall into local optimal solutions, limiting their performance; in order to better train the ResNet50 model, you can choose to use the SGD optimizer with momentum (SGD with Momentum) and set the momentum parameter to 0.9; this optimizer is not only suitable for deep neural network training, but also helps control the update speed of the model in the later stages of training, making it easier to find the global optimal solution. Since the disease recognition model network is deeper, L regularization can be added to prevent overfitting. L regularization is such as Among them, L in this formula 0 represents the cross entropy loss, λ represents the regularization coefficient, and w i represents the weight, and m represents the number of samples; that is, the disease recognition model adds a regularization term in the cross entropy loss function, which is proportional to the sum of the squares of the model weights. Its introduction helps to limit the growth rate of the weight and improve the generalization ability of the model. The regularization coefficient λ can be initialized to 0.01.

[0078] When training the disease recognition model, the hyperparameters that need to be debugged also include learning rate, batch size, dropout ratio and regularization coefficient λ. The initialization and adjustment mechanism of the first three are the same as the equipment recognition model and will not be elaborated here. The regularization coefficient λ is initialized to 0.01, and its adjustment mechanism is as follows: when the model is underfitting, that is, the training and verification accuracy are relatively low, reduce λ to reduce the penalty on the weight so that the model can obtain more features; when the model is overfitting, and the training accuracy is high but the verification accuracy is low, increase λ to limit the growth of the weight and improve the generalization ability of the model. Based on the above adjustment mechanism, after continuous adjustment and optimization of the hyperparameters, the best parameter combination of each model is saved as the final eight disease recognition models.

[0079] In addition, the confusion matrix can be used for the final model evaluation. The confusion matrix is ​​divided into four quadrants: TP (true positive), FN (false negative), FP (false positive) and TN (true negative). The equipment identification model and each disease identification model will be evaluated separately. The precision rate (P), recall rate (TPR) and F1 score are used to evaluate the performance of each model on the test set. The calculation of P, TPR and F1 score is as follows: And for the multi-class classification models such as the equipment identification model and the disease identification model mentioned above, the macro-average model is used for evaluation, and the quantitative indicators are as follows: Among them, M P represents the macro average precision, M TPR represents the macro average recall, M F1 represents the macro-average F1 score. For the device recognition model, N represents the number of device categories (in one embodiment, the number of device categories N is 8); for the disease recognition model, N represents the number of disease types corresponding to a certain device category (in one embodiment, the number of disease types N is 6).

[0080] For the above-mentioned method for identifying defects in civil engineering equipment, since equipment type identification and defect type identification are performed serially, the models of the two stages can be trained and optimized in sequence. For example, the equipment identification model built based on VGG16 is first trained to achieve the ideal accuracy and confirm its performance, and then each defect identification model built based on the ResNet model is trained. The inventors found in experiments that the equipment identification model and each defect identification model are trained and optimized in sequence, and finally verified based on the verification set, which can achieve the ideal accuracy.

[0081] For example, the method for identifying defects in civil engineering equipment of the present application can be applied to the special maintenance of civil engineering equipment in spring and autumn, such as Figure 7 As shown, in this embodiment, first, the spring and autumn special maintenance images of engineering equipment are collected, and the images are preprocessed such as classification, annotation, data cleaning and enhancement; then the training set, data set and test set are divided according to the ratio of 8:1:1; further, the equipment recognition model is constructed based on the pre-trained VGG16 model, and the disease recognition models corresponding to various types of equipment are constructed based on the pre-trained ResNet50 model; and the effective serial connection of the equipment recognition model and each disease recognition model is realized; further, the new fully connected layer of the equipment recognition model is trained and optimized, and the new fully connected layer of each disease recognition model is trained and optimized; a visualization image of the disease area is generated; and finally, the test set is used to evaluate the serial model composed of the equipment recognition model and the disease recognition model. This method realizes automatic disease recognition based on the serial architecture of VGG16 and ResNet50, and divides the recognition process into two stages: equipment recognition and disease recognition. The first stage adopts the equipment recognition model, and dynamically selects the disease recognition model of the second stage according to the equipment recognition result to perform disease recognition. Finally, the diseased area is displayed visually, which improves the intuitiveness and interpretability of the recognition results and ensures the accuracy and efficiency of spring and autumn maintenance.

[0082] Accordingly, the present invention provides a system for identifying defects in construction equipment, including a processor, a memory, and a computer program stored in the memory, wherein the processor is used to execute the computer program, and when the computer program is executed, the system implements the steps of the method described in any of the above embodiments. The system combines the equipment identification model with the defect identification model to achieve efficient and accurate identification of equipment categories and corresponding defect types and visual display of defect areas.

[0083] For example, Figure 3 As shown in the figure, the system architecture can be built with a multi-layer system, which is divided into data layer, business layer, computing layer and display layer from bottom to top. The data layer is responsible for the data storage and management of the system. Combined with the characteristics and access requirements of the image data of railway engineering equipment defects, a relational database suitable for user information, equipment information, defect type information and defect recognition results is designed, and a classified storage structure suitable for the original image and feature-annotated image path is designed to achieve efficient data reading and analysis. The business layer is responsible for the business logic processing of the system. Aiming at the business needs of automatic identification of defects in spring and autumn maintenance of engineering equipment, it mainly realizes functions such as image upload processing, interaction with the computing layer, and system business process management (such as user login, permission verification, and data query). The computing layer is the core part of the system. It adopts a computing architecture based on transfer learning to realize equipment type feature extraction, defect area detection and classification, model optimization and training, etc., providing key technical support for the system and generating defect recognition results. The display layer is responsible for the front-end display of the system and user interaction. It designs a friendly interactive interface based on user needs, realizes functions such as uploading equipment disease images and visualizing disease identification results, and supports interactive functions such as disease image screening to improve user experience and data management efficiency.

[0084] In the data layer, this application uses the relational database PostgreSQL to store structured spring and autumn special maintenance equipment disease data, which mainly includes user information, equipment information, disease type information and identification results. For example, the design overview of the data layer is shown in the following table:

[0085]

[0086]

[0087] Specifically, the stored user information includes user ID, user name, password, etc., which is used to manage system users and access rights; device information includes device ID, device type, device location, etc., which is used to identify and classify different devices; disease type information includes disease type, disease name and description, and is dynamically associated with device type information; identification results include data such as device ID, disease type ID, disease location coordinates and identification time, which are used to store the results of disease identification.

[0088] In addition, in the data layer, through the extension of PostGIS, the system can support the storage and management of data of various geometric types. For example, the two-dimensional coordinates (X, Y) of the disease represent the position of the disease on the plane image, which can be efficiently stored and processed in PostGIS. At the same time, by adding field information such as a unique identifier (ID) and a timestamp (timestamp) to each piece of data, a one-to-one correspondence between data and version is achieved. Finally, for semi-structured and unstructured data, this system uses the MongoDB database. The specific storage content includes uploaded disease images and annotated feature images, and the file types are all JPEG or PNG; at the same time, this application uses MongoDB combined with GridFS to achieve shard storage of large files. The storage record of each image file contains metadata such as a unique identifier, file type, and upload time; shard storage can ensure that scattered non-relational data is uniformly managed in the database, which is convenient for rapid reading and retrieval of disease image data.

[0089] In one embodiment, the display layer is a user-friendly interactive interface designed for user needs, including functions such as image uploading, recognition result display, and image filtering. The front-end display layer is developed using the Vue.js framework and combined with Element UI to realize interface componentization and aesthetics. Vue.js is a lightweight framework with the advantages of flexibility and efficiency, which is convenient for realizing complex front-end interactions and dynamic data binding. In the image upload module, the image upload function is realized using the File API of HTML5. Users can click the upload button to upload images of the spring and autumn special maintenance of engineering equipment. Batch upload is supported and the upload progress is displayed; the uploaded images will be verified for format and size restrictions on the front end, and uploaded to the back-end server through Ajax requests after verification. In the recognition result display module, the recognition result data is obtained by interacting with the back-end API. The recognition result of each image is displayed in the card component, including information such as image preview, equipment type, and disease type; at the same time, ECharts is used to realize data visualization, including pie charts to display equipment type distribution and bar charts to display disease type statistics. In the image screening module, the front-end manages the image data and screening conditions through the Vuex state management library. Users can filter the uploaded images according to the equipment type, disease type, date and other conditions, and view the identification results after screening. In summary, the display layer design combines Vue.js with the Element UI framework selection and the above-mentioned functional design to realize the efficient, convenient and visual front-end application of the automatic identification system for the spring and autumn special maintenance of high-speed railway maintenance equipment, meeting the various needs of users such as uploading pictures, viewing identification results and filtering pictures.

[0090] exist Figure 2In the specific embodiment shown, the system first receives and uploads the images of high-speed railway maintenance equipment in the spring and autumn special maintenance at the business layer, and then the computing layer subsequently starts the automatic disease recognition model based on the serial architecture of the equipment recognition model and the disease recognition model. In the computing layer, the equipment type is first identified through the equipment recognition model; according to the equipment recognition result, the corresponding disease recognition model is dynamically selected to identify the equipment disease; further, the identified disease information is transmitted to the display layer, and visualized by image annotation and disease area highlighting. Based on multi-model fusion technology, the system provides key technical support for the efficient and accurate identification of spring and autumn maintenance diseases, while improving the intuitiveness and interpretability of the recognition results.

[0091] In summary, the engineering equipment disease recognition method based on the serial architecture of VGG16 and ResNet50 of this application outputs the recognition result of equipment type + disease type, adopts the transfer learning method, and through the learning ability of the pre-trained model on the large general data set ImageNet, and fine-tunes it in combination with specific task data, the model can handle different equipment types and disease types, thereby significantly improving the generalization ability and scope of application of the system. In addition, the implementation of this application divides the spring and autumn special maintenance data of high-speed railway engineering equipment according to the data structure, stores it in different databases, and adopts classified storage and sharding storage to improve data access efficiency.

[0092] In addition to the above, this application adopts a phased model design. The first phase uses VGG16-based identification of device types, and the second phase dynamically associates device types to select corresponding ResNet50-based device disease identification types, so that the model can efficiently identify different device types and disease types, improving the generalization ability of the system model. In addition, when performing equipment disease identification in the prior art, it relies on a large number of training samples with manual annotations, which makes it difficult to obtain data sets and has high manual annotation costs. This application uses transfer learning technology to further train the equipment identification model and the disease identification model based on the pre-trained VGG16 and ResNet50 models, and only requires less labeled data to achieve good recognition results, thereby greatly reducing the cost of data acquisition and annotation.

[0093] This application also uses databases such as Keras and OpenCV to generate visualization images of diseased areas, and through the visualization design of the front-end application, it can intuitively display the recognition results, including equipment type, disease type and its distribution; based on this, users can quickly know the maintenance status through a combination of pictures and texts, which improves the interpretability of the results and user experience. In addition, this application uses the JavaScript framework for front-end development, and improves the reusability and maintainability of the code through componentized design; each functional module is independently encapsulated to facilitate subsequent functional expansion and system maintenance, ensuring the long-term efficient and stable operation of the system.

[0094] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned edge computing server deployment method are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0095] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0096] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0097] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0098] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for identifying defects in civil engineering equipment, characterized in that: The method comprises: Acquire images of engineering equipment to be identified; Inputting the to-be-identified engineering equipment image into a trained equipment recognition model to obtain an engineering equipment recognition result corresponding to the to-be-identified engineering equipment image; Based on the engineering equipment identification result, a first disease identification model corresponding to the equipment type in the engineering equipment identification result is determined from the disease identification model group, and the engineering equipment image to be identified is input into the trained first disease identification model to obtain the equipment disease identification result corresponding to the engineering equipment image to be identified.

2. The method for identifying defects in construction equipment according to claim 1, characterized in that: The device identification model includes a first convolutional layer, a first flatten layer, a first fully connected layer, a first dropout layer and a first output layer; and / or, The disease recognition model includes a second convolutional layer, a maximum pooling layer, a residual block, a global average pooling layer, a second flatten layer, a second fully connected layer, a second dropout layer and a second output layer.

3. The method for identifying damage to construction equipment according to claim 2, characterized in that: The disease identification model group includes multiple disease identification models, and the multiple disease identification models correspond to different equipment types; and / or, Inputting the to-be-identified maintenance equipment image into the first trained defect recognition model to obtain an equipment defect recognition result corresponding to the to-be-identified maintenance equipment image, including: Dividing the to-be-recognized engineering equipment image into a plurality of candidate frames; Inputting the image in each candidate frame into the first trained defect recognition model to obtain the equipment defect recognition result of the to-be-recognized maintenance equipment image, wherein the equipment defect recognition result includes the defect area and the defect type; Drawing a boundary box of the defective area on the image of the engineering equipment to be identified based on the identified defective area; The disease type is marked in the disease area boundary box based on the identified disease type.

4. The method for identifying damage to construction equipment according to claim 3, characterized in that: Determining a first defect recognition model corresponding to the equipment type in the work equipment recognition result from the defect recognition model group based on the work equipment recognition result includes: The first disease identification model corresponding to the equipment type in the engineering equipment identification result is determined from the disease identification model group through conditional logic or dictionary query.

5. The method for identifying damage to construction equipment according to claim 3, characterized in that: The method comprises: Construct a sample data set, pre-train an initial equipment recognition model based on the sample data set to obtain a trained equipment recognition model, and pre-train each initial disease recognition model based on the sample data set to obtain a trained disease recognition model; wherein the sample data in the sample data set includes engineering equipment image sample data, engineering equipment recognition result sample data, and equipment disease recognition result sample data.

6. The method for identifying damage to construction equipment according to claim 5, characterized in that: Pre-training the initial device recognition model based on the sample data set to obtain a trained device recognition model, including: The first convolutional layer of the device recognition model is obtained based on the convolutional layer transfer learning of the pre-trained VGG16 model; and / or, Based on the sample data set, each initial disease recognition model is pre-trained to obtain each trained disease recognition model, including: based on the convolutional layer transfer learning of the pre-trained ResNet50 model, the second convolutional layer of the disease recognition model is obtained.

7. The method for identifying damage to construction equipment according to claim 5, characterized in that: Building the sample dataset involves: Collecting image sample data of various types of engineering equipment corresponding to the engineering equipment; Label the equipment type and disease type of the engineering equipment image sample data corresponding to each type of engineering equipment; Dividing the labeled multiple engineering equipment image sample data into multiple sub-datasets based on equipment types; Performing data cleaning and data enhancement on the sample data of the engineering equipment images in each of the sub-data sets; Each of the sub-datasets after data cleaning and data enhancement is divided into a training set, a validation set and a test set.

8. The method for identifying damage to construction equipment according to any one of claims 5 to 7, characterized in that: Equipment types include switches, tracks, rails, regulators, welds, ballastless track, ballasted track and ballastless trackbed; and / or, The loss function of the device identification model is: Where N is the number of sample data, K is the number of device categories, and t n，k represents the true label of the nth sample on the kth device category, y n，k represents the model's predicted probability that the nth sample belongs to the kth device category; The regularization algorithm of the disease recognition model is: Among them, L0 represents the cross entropy loss, λ represents the regularization coefficient, and w i represents the weight, and m represents the number of samples.

9. A system for identifying defects in construction equipment, comprising a processor, a memory, and a computer program stored in the memory, characterized in that: The processor is used to execute the computer program. When the computer program is executed, the system implements the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images