Method and device for detecting fracture fault of complex framework, electronic equipment and medium

Abstract

The invention discloses a composite framework fracture fault detection method and device, electronic equipment and a medium, and relates to the technical field of railway transportation, and the method comprises the steps: obtaining a target image of a to-be-detected composite framework; segmenting a complex framework region in the target image to obtain a complex framework binary image; performing fault recognition on the target image through a pre-trained fault recognition model to obtain crack region information; according to the crack area information, carrying out binarization on a crack area in the target image to obtain a crack binary image; responding to the situation that a crack area in the crack binary image coincides with a complex framework area in the complex framework binary image, and obtaining a fault detection result representing fracture of the to-be-detected complex framework; wherein the fault identification model comprises a residual network, a cross-stage partial feature fusion network and a feature selection anchor-frame-free residual network which are connected in sequence. The positioning accuracy of the complex framework and the crack area can be improved, the diversity and expression ability of the features are enhanced, and fracture identification can be more accurately carried out on the complex framework.

Description

Technical Field

[0001] This application relates to the technical field of railway transportation, and more specifically, to a method, device, electronic device, and medium for detecting the fracture fault of a composite frame. Background Art

[0002] In a railway transportation system, as a structural component, the integrity and stability of a composite frame affect the safe operation of a train. However, during long-term use, the composite frame may experience fracture faults, which not only affect the normal operation of the vehicle but may also lead to safety accidents. Therefore, it is necessary to detect the fracture faults of the composite frame.

[0003] For example, it is possible to rely on pixel features and fixed threshold settings to detect the fracture faults of the composite frame. However, in the face of complex fault patterns, it is impossible to fully capture the key features. Especially when the feature patterns change significantly, the risks of missed detection and false detection increase significantly, which not only reduces the reliability of fault detection but also increases potential safety hazards.

[0004] In summary, how to accurately detect the fracture faults of the composite frame is an urgent problem to be solved by those skilled in the art at present. Summary of the Invention

[0005] The purpose of this application is to provide a method for detecting the fracture fault of a composite frame, which can, to a certain extent, solve the technical problem of how to accurately detect the fracture fault of a composite frame. This application also provides a device, an electronic device, and a computer-readable storage medium for detecting the fracture fault of a composite frame.

[0006] To achieve the above purpose, this application provides the following technical solutions:

[0007] In the first aspect, a method for detecting the fracture fault of a composite frame is provided, including:

[0008] Obtain a target image of the composite frame to be detected;

[0009] Segment the composite frame area in the target image to obtain a binary image of the composite frame;

[0010] Perform fault recognition on the target image through a pre-trained fault recognition model to obtain crack area information;

[0011] According to the crack area information, binarize the crack area in the target image to obtain a binary image of the crack;

[0012] In response to the crack area in the binary image of the crack coinciding with the composite frame area in the binary image of the composite frame, obtain a fault detection result indicating that the composite frame to be detected is fractured;

[0013] Among them, the fault recognition model includes a residual network, a cross-stage partial feature fusion network, and a residual network with feature selection and without anchor boxes connected in sequence.

[0014] On the other hand, the fault recognition is performed on the target image through a pre-trained fault recognition model to obtain crack area information, including:

[0015] The residual network is used to extract features from the target image to obtain an original feature map, and the dimension of the original feature map is less than a set value;

[0016] The cross-stage partial feature fusion network is used to fuse the features of the original feature map to obtain an intermediate feature map;

[0017] The residual network with feature selection and without anchor boxes is used to classify and perform position regression on the intermediate feature map to obtain class information and corresponding position information;

[0018] According to the class information and the position information, the crack area information is determined in the target image.

[0019] On the other hand, the residual network with feature selection and without anchor boxes is used to classify and perform position regression on the intermediate feature map to obtain class information and corresponding position information, including:

[0020] The residual network with feature selection and without anchor boxes is used to classify and perform position regression on the intermediate feature map to obtain class information and corresponding position information, and the class information includes cracks, water flow, chalk, shadows, and foreign objects.

[0021] On the other hand, before the fault recognition is performed on the target image through a pre-trained fault recognition model to obtain crack area information, it further includes:

[0022] Obtain a training sample set;

[0023] The initial fault recognition model is used to perform fault recognition on the images in the training sample set to obtain recognition results;

[0024] According to the training sample set and the recognition results, the loss function value of the fault recognition model is generated;

[0025] Based on the loss function value, the fault recognition model is adjusted to obtain the trained fault recognition model;

[0026] Among them, the generation formula of the loss function value includes:

[0027] ;

[0028] ;

[0029] Among them, represents the value of the loss function; represents the total number of pixels within all valid bounding boxes; represents the hierarchical index in the feature pyramid; represents the position coordinates; represents the th layer of valid bounding boxes in the feature pyramid, represents the bounding box, represents valid; represents the intersection over union of the predicted bounding box and the ground truth bounding box; represents the distance between the center points of the predicted bounding box and the ground truth bounding box; represents the diagonal length of the smallest bounding rectangle containing the predicted bounding box and the ground truth bounding box; represents the width of the ground truth bounding box, represents the height of the ground truth bounding box; represents the width of the predicted bounding box, represents the height of the predicted bounding box; represents a set value.

[0030] On the other hand, obtaining a training sample set, including:

[0031] Obtaining a train image obtained by photographing a training vehicle;

[0032] Marking and region drawing the complex frame components in the train image, and cropping out the first image of the known complex frame in the train image;

[0033] Based on the first image, generating a marking file of the known complex frame, the marking file including an image name, a detection category, and complex frame region coordinates;

[0034] Determining the fault information of the known complex frame;

[0035] Performing data enhancement and contrast enhancement processing on the first image to obtain a second image;

[0036] Using the second image, the marking file, and the fault information as a training sample set.

[0037] On the other hand, before performing fault recognition on the images in the training sample set through an initial fault recognition model, it further includes:

[0038] Obtaining the model parameters of the visual object recognition data set;

[0039] Initializing the network parameters of the initial fault recognition model based on the model parameters.

[0040] On the other hand, segmenting the complex frame region in the target image to obtain a binary image of the complex frame, including:

[0041] Segmenting the complex frame region in the target image through a semantic segmentation network to obtain a binary image of the complex frame, where the pixel value of the complex frame region is 1 and the pixel value of other regions is 0;

[0042] Binarizing the crack region in the target image according to the crack region information to obtain a binary crack image, including:

[0043] Binarizing the crack region in the target image through the semantic segmentation network according to the crack region information to obtain a binary crack image, where the pixel value of the crack region is 1 and the pixel value of the non-crack region is 0.

[0044] On the other hand, in response to the crack region in the binary crack image coinciding with the complex frame region in the binary complex frame image, a fault detection result indicating that the to-be-detected complex frame is broken is obtained, including:

[0045] Performing an AND operation on the binary complex frame image and the binary crack image to generate an image to be processed;

[0046] Detecting whether there is a pixel region with a value of 1 in the image to be processed;

[0047] In response to the existence of a pixel region with a value of 1 in the image to be processed, a fault detection result indicating that the to-be-detected complex frame is broken is obtained.

[0048] In a second aspect, a complex frame breakage fault detection device is provided, including:

[0049] A first acquisition module for acquiring a target image of a to-be-detected complex frame;

[0050] A first segmentation module for segmenting the complex frame region in the target image to obtain a binary image of the complex frame;

[0051] A first recognition module for performing fault recognition on the target image through a pre-trained fault recognition model to obtain crack region information;

[0052] A second segmentation module for binarizing the crack region in the target image according to the crack region information to obtain a binary crack image;

[0053] A first detection module for obtaining a fault detection result indicating that the to-be-detected complex frame is broken in response to the crack region in the binary crack image coinciding with the complex frame region in the binary complex frame image;

[0054] Among them, the fault identification model includes a residual network, a cross-stage partial feature fusion network, and a feature selection anchor-free residual network connected in sequence.

[0055] In a third aspect, an electronic device is provided, including:

[0056] A memory for storing a computer program;

[0057] A processor for implementing the steps of any of the above complex frame breakage fault detection methods when executing the computer program.

[0058] In a fourth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above complex frame breakage fault detection methods are implemented.

[0059] For a complex frame breakage fault detection method provided in this application, a target image of a complex frame to be detected is obtained; the complex frame area in the target image is segmented to obtain a complex frame binary image; the target image is subjected to fault identification through a pre-trained fault identification model to obtain crack area information; according to the crack area information, the crack area in the target image is binarized to obtain a crack binary image; in response to the crack area in the crack binary image coinciding with the complex frame area in the complex frame binary image, a fault detection result indicating that the complex frame to be detected is broken is obtained; among them, the fault identification model includes a residual network, a cross-stage partial feature fusion network, and a feature selection anchor-free residual network connected in sequence. In this application, segmenting the target image according to the complex frame area and the crack area can improve the positioning accuracy of the complex frame and the crack area. In this way, when performing breakage detection based on the obtained binary image, false alarms caused by similar breakage faults at non-complex frame positions can be avoided, and the detection accuracy of complex frame breakage faults can be improved; moreover, in this application, the cross-stage partial feature fusion network is introduced into the feature selection anchor-free residual network, which can combine the cross-stage feature information fusion ability of the cross-stage partial feature fusion network to enhance the diversity and expression ability of features, thereby solving the problem of missed detection of complex frames caused by loss of feature information, improving the breakage fault recognition rate of the network model for complex frames, and thus more accurately identifying the breakage faults of complex frames. A complex frame breakage fault detection device, an electronic device, and a computer-readable storage medium provided in this application also solve the corresponding technical problems. Description of the Drawings

[0060] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to the provided drawings.

[0061] Figure 1 It is a flowchart of a method for detecting complex frame breakage faults provided by an embodiment of the present application;

[0062] Figure 2 It is a schematic diagram of an image of a complex frame with a breakage fault;

[0063] Figure 3 It is a schematic diagram of a cross-stage partial feature fusion network;

[0064] Figure 4 It is a schematic diagram of a fault recognition model;

[0065] Figure 5 It is a schematic diagram of the training of a fault recognition model;

[0066] Figure 6 It is a schematic diagram of the structure of a device for detecting complex frame breakage faults provided by an embodiment of the present application;

[0067] Figure 7 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application;

[0068] Figure 8 It is another schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Specific embodiments

[0069] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0070] Please refer to Figure 1 , Figure 1 It is a flowchart of a method for detecting complex frame breakage faults provided by an embodiment of the present application.

[0071] A method for detecting complex frame breakage faults provided by an embodiment of the present application may include the following steps:

[0072] Step S101: Obtain the target image of the complex frame to be detected.

[0073] In practical applications, since the broken fault detection is performed on the complex structure, the corresponding information of the complex structure to be detected needs to be obtained. Considering that images can record the information of the complex structure to be detected and are easy to process, the target image of the complex structure to be detected can be obtained first.

[0074] It should be noted that the target image can be acquired by image acquisition devices such as cameras and sensors. To facilitate the understanding of this process, taking a railway freight car as an example, the process of obtaining the target image can be as follows: Install trackside imaging devices on both sides of the railway. After the freight car passes through the high-definition industrial line array camera, a high-definition train image is obtained. Then, according to prior knowledge such as hardware devices, wheelbase information, and relevant positions, the area of the side complex structure components is intercepted from the high-definition train image to obtain the initial complex structure image. If the initial complex structure image meets the image detection requirements, the initial complex structure image can be used as the target image; if the initial complex structure image does not meet the image detection requirements, the initial complex structure image is adjusted to obtain the target image. For example, if the initial complex structure image has different image brightness levels due to the influence of the camera's angle and distance, in order to avoid the broken area of the complex structure not being clearly observable due to the image being too dark, local adaptive contrast enhancement processing can be performed on the initial complex structure image, or other enhancement processing can be continued on the initial complex structure image according to actual needs to obtain the target image.

[0075] Step S102: Segment the complex structure area in the target image to obtain a binary image of the complex structure.

[0076] In practical applications, due to the shape of the complex structure, there may be other objects in the target image besides the complex structure. For example, there is a broken complex structure image as Figure 2 shown. To exclude the influence of other objects on the broken fault identification of the complex structure, the complex structure area in the target image can be segmented to obtain a binary image of the complex structure that separates the complex structure area from other areas.

[0077] Step S103: Perform fault identification on the target image through a pre-trained fault identification model to obtain crack area information.

[0078] In practical applications, a neural network model can be used to identify faults in the target image. For example, the fault in the target image can be identified through a pre-trained fault identification model to obtain crack area information; and the fault identification model includes a residual network (ResNet), a cross-stage partial feature fusion network (CSP-PAN), and a feature selection anchor-free residual network (FSAF-RetinaNet) connected in sequence.

[0079] In a specific application scenario, during the process of identifying faults in a target image through a pre-trained fault identification model to obtain crack area information, a residual network can be used to extract features from the target image to obtain an original feature map, and the dimension of the original feature map is less than a set value; a cross-stage partial feature fusion network is used to fuse the features of the original feature map to obtain an intermediate feature map; a feature selection anchor-free residual network is used to classify and perform position regression on the intermediate feature map to obtain class information and corresponding position information; based on the class information and position information, crack area information is determined in the target image. That is, the residual network is used to extract features from the target image to obtain low-dimensional feature maps of different layers, the cross-stage partial feature fusion network is used to fuse the low-dimensional feature maps of different layers to obtain an intermediate feature map, and the feature selection anchor-free residual network is used to classify and perform position regression on the intermediate feature map to obtain the fault class information in the target image and the position information corresponding to the fault class information.

[0080] It should be noted that the structures of the components in the fault identification model can be flexibly determined according to actual needs. For example, the structure of the CSP-PAN network can be as Figure 3 shown. Specifically, the feature map of the third layer can be represented by C3, the feature map of the fourth layer can be represented by C4, and the feature map of the fifth layer can be represented by C5. The feature maps of the third layer, the fourth layer, and the fifth layer are simultaneously input into the CSP-PAN network. A 1x1 convolution is performed on the feature map of the third layer to output a feature map with a set number of channels. For example, a 1x1 convolution is performed on the 96-channel feature map of the third layer to output a 96-channel feature map; a 1x1 convolution is performed on the feature map of the fourth layer to output a feature map with a set number of channels. For example, a 1x1 convolution is performed on the 192-channel feature map of the fourth layer to output a 96-channel feature map; a 1x1 convolution is performed on the feature map of the fifth layer to output a feature map with a set number of channels. For example, a 1x1 convolution is performed on the 384-channel feature map of the fifth layer to output a 96-channel feature map; the output feature maps are input into the CSP module for cross-stage partial aggregation of features. The CSP module divides the feature maps into two parts, processes and merges them separately to enhance the diversity of features. Among them, the feature maps of the fourth layer and the fifth layer are upsampled and downsampled to align the feature maps of different layers, and channel splicing is performed. The processed feature maps are combined together to form a richer feature representation; finally, the CSP-PAN structure outputs intermediate feature maps of multiple scales, that is, the third-layer output P3, the fourth-layer output P4, the fifth-layer output P5, and the sixth-layer output P6 are obtained.

[0081] For another example, the FSAF-RetinaNet network can include a class subnet Class Subnet and a bounding box subnet BoxSubnet. At this time, the structure of the fault identification model is asFigure 4 As shown; correspondingly, in the FSAF-RetinaNet network, each intermediate feature map first passes through the class subnet, which is composed of multiple convolutional layers. For example, for a feature map of W×H×256, after 4 convolutions, a feature map of W×H×K is output, where K is the number of classes; at the same time, the feature map also passes through the box subnet, which is also composed of multiple convolutional layers. For example, for a feature map of W×H×256, after 4 convolutions, a feature map of W×H×4A is output, where A is the number of anchor boxes.

[0082] In specific application scenarios, the types of fault information detected by the FSAF-RetinaNet network can be flexibly determined according to actual needs. For example, considering that due to the complex structure, there are sometimes water flow traces, chalk traces, and the image features of the shadow areas with foreign objects hanging and the crack areas at the breakage are similar, the faults are classified into five categories: cracks, water flow, chalk, shadows, and foreign objects. That is, in the process of classifying the intermediate feature map and performing position regression through the FSAF-RetinaNet network to obtain the class information and the corresponding position information, the FSAF-RetinaNet network can be used to classify the intermediate feature map and perform position regression to obtain the class information and the corresponding position information. The class information can include cracks, water flow, chalk, shadows, and foreign objects, etc.

[0083] In actual applications, the fault recognition model needs to be trained. That is, before using the pre-trained fault recognition model to identify faults in the target image and obtain the crack area information, the fault recognition model also needs to be trained. The training process can be as shown in Figure 5 and includes the following steps:

[0084] Step S2011: Obtain the training sample set.

[0085] This training sample set is used to identify the fault types of the complex structure for the fault recognition model and can include complex structure images, complex structure marking information, complex structure fault information, etc.

[0086] Step S2012: Use the initial fault recognition model to identify faults in the images in the training sample set to obtain the recognition results.

[0087] Step S2013: Generate the loss function value of the fault recognition model according to the training sample set and the recognition results.

[0088] Step S2014: Adjust the fault recognition model based on the loss function value to obtain the trained fault recognition model.

[0089] Among them, the generation formula of the loss function value can include:

[0090] ;

[0091] ;

[0092] Among them, represents the loss function value; represents the total number of pixels within all valid bounding boxes; represents the hierarchical index in the feature pyramid; represents the position coordinates; represents the th layer of valid bounding boxes in the feature pyramid, represents the bounding box, represents valid; represents the intersection over union of the predicted bounding box and the ground truth bounding box; represents the distance between the center point of the predicted bounding box and the center point of the ground truth bounding box, which can be Euclidean clustering, etc.; represents the diagonal length of the smallest enclosing rectangle containing the predicted bounding box and the ground truth bounding box; represents the width of the ground truth bounding box, represents the height of the ground truth bounding box; represents the width of the predicted bounding box, represents the height of the predicted bounding box; represents a set value, which can take etc. Among them, the functions of each part of the loss are as follows: This part of the loss predicts that the center point of the predicted bounding box is as close as possible to the center point of the ground truth bounding box, and the distance is smaller, the smaller this part of the loss; This part of the loss predicts that the aspect ratio of the predicted bounding box is consistent with the aspect ratio of the ground truth bounding box. By minimizing the aspect ratio difference, the shape accuracy of the predicted bounding box can be improved; FR comprehensively considers the overlap degree, center point distance, and aspect ratio consistency to comprehensively measure the difference between the predicted bounding box and the ground truth bounding box. Using the FR function can improve the bounding box regression performance, thereby improving the overall detection accuracy. In other words, the loss function of the present application can comprehensively measure the difference between the predicted bounding box and the ground truth bounding box, effectively improving the algorithm detection accuracy.

[0093] It should be noted that the generation formula of the loss function value, that is, the derivation process of the loss function can be as follows: Considering that in the anchor box branch, the anchor box mechanism is used for object detection, that is, by classifying and regressing each anchor box, the category and location of the object are predicted. In the anchor-free branch, object detection is performed on the feature map. Through the feature selection mechanism, the most representative features are selected for object classification and location regression. From the feature map output by the classification sub-network, the category probability of each location is extracted. The probability of each category is calculated through the Softmax (normalized exponential function) function. From the feature map output by the regression sub-network, the bounding box coordinates of each location are extracted, and the offset between the predicted box and the ground truth box is calculated through the loss function. For the predicted bounding box, the non-maximum suppression algorithm is applied to retain the optimal detection result. Finally, the category and location coordinates of each object are output. During training, the total classification loss of the anchor-free branch in the image is the sum of the losses of all non-ignored regions and is normalized to the total number of pixels within all valid box regions, that is:

[0094] ;

[0095] represents the focal position loss at the feature level at position ; represents the FocalLoss function;

[0096] If the regression loss is defined, the loss function will be:

[0097] ;

[0098] .

[0099] In a specific application scenario, during the process of obtaining a training sample set, train images obtained by photographing a training vehicle can be acquired. During this process, considering that the quality of the multiple-frame images is vulnerable to weather and human factors, such as rain, snow, mud stains, light brightness, etc., and the multiple-frame images at different stations vary greatly due to differences in equipment installation angles, distances, etc., therefore, during the process of collecting image data, various images under different conditions should be covered as much as possible to ensure the diversity of train images, and then ensure the diversity of the multiple-frame images used for training; mark and draw regions for the multiple-frame components in the train images, and crop the first image of the known multiple-frame in the train images; based on the first image, generate a label file for the known multiple-frame, and the label file includes image names, detection categories, multiple-frame region coordinates, etc. The label file can be an xml file, and the multiple-frame region coordinates can include the upper left and lower right coordinates of the multiple-frame region, etc.; determine the fault information of the known multiple-frame; perform data augmentation and contrast enhancement processing on the first image to obtain a second image to improve sample diversity and then improve the generalization of the algorithm. Data augmentation can include randomly rotating, translating, scaling, and mirroring the image, etc.; use the second image, the label file, and the fault information as the training sample set.

[0100] In a specific application scenario, if the parameters of the fault recognition model are initialized in advance, the training efficiency of the recognition model can be improved. That is, before the initial fault recognition model performs fault recognition on the images in the training sample set, the model parameters of the visual object recognition data set can also be obtained, such as obtaining the model parameters of the ImageNet data set; initialize the network parameters of the initial fault recognition model based on the model parameters.

[0101] Step S104: Binarize the crack region in the target image according to the crack region information to obtain a crack binary image.

[0102] Step S105: In response to the crack region in the crack binary image coinciding with the multiple-frame region in the multiple-frame binary image, obtain a fault detection result indicating that the to-be-detected multiple-frame is broken.

[0103] In practical applications, when a crack falls outside the complex framework, there is also a breakage fault in the target image. However, this breakage fault is not the breakage fault of the complex framework. To exclude this situation and more accurately detect the breakage fault of the complex framework, the crack region in the target image can be binarized according to the crack region information to obtain a crack binary image that distinguishes the crack region from the non-crack region. Then, it is detected whether the crack region in the crack binary image coincides with the complex framework region in the complex framework binary image. In response to the coincidence of the crack region in the crack binary image and the complex framework region in the complex framework binary image, a fault detection result indicating that the complex framework to be detected is broken is obtained. In response to the non-coincidence of the crack region in the crack binary image and the complex framework region in the complex framework binary image, a fault detection result indicating that the complex framework to be detected is not broken is obtained.

[0104] In practical applications, in the process of segmenting the complex framework region in the target image to obtain the complex framework binary image, the complex framework region in the target image can be segmented through a semantic segmentation network to obtain the complex framework binary image, where the pixel value of the complex framework region is 1 and the pixel value of other regions is 0. In the process of binarizing the crack region in the target image according to the crack region information to obtain the crack binary image, the crack region in the target image can be binarized through a semantic segmentation network according to the crack region information to obtain the crack binary image, where the pixel value of the crack region is 1 and the pixel value of the non-crack region is 0. Correspondingly, in the process of obtaining a fault detection result indicating that the complex framework to be detected is broken in response to the coincidence of the crack region in the crack binary image and the complex framework region in the complex framework binary image, the complex framework binary image and the crack binary image can be subjected to an AND operation to generate an image to be processed. It is detected whether there is a pixel region with a value of 1 in the image to be processed. In response to the existence of a pixel region with a value of 1 in the image to be processed, a fault detection result indicating that the complex framework to be detected is broken is obtained. In this way, by simply setting the pixel values of the complex framework region and the crack region in the target image to 1 and then performing an AND operation once, it is possible to simply and quickly detect whether the crack region in the crack binary image coincides with the complex framework region in the complex framework binary image, further improving the efficiency of detecting the breakage fault of the complex framework.

[0105] A method for detecting the fracture fault of a complex frame provided by the present application includes: obtaining a target image of the complex frame to be detected; segmenting the complex frame area in the target image to obtain a binary image of the complex frame; performing fault recognition on the target image through a pre-trained fault recognition model to obtain crack area information; binarizing the crack area in the target image according to the crack area information to obtain a binary crack image; in response to the coincidence between the crack area in the binary crack image and the complex frame area in the binary complex frame image, obtaining a fault detection result indicating that the complex frame to be detected is fractured; wherein, the fault recognition model includes a residual network, a cross-stage partial feature fusion network, and a feature selection anchor-free residual network connected in sequence. In the present application, segmenting the target image according to the complex frame area and the crack area can improve the positioning accuracy of the complex frame and the crack area. In this way, when performing fracture detection based on the obtained binary image, false alarms caused by similar fracture faults at non-complex frame positions can be avoided, and the detection accuracy of the complex frame fracture fault can be improved; and the present application introduces the cross-stage partial feature fusion network into the feature selection anchor-free residual network, which can combine the cross-stage feature information fusion ability of the cross-stage partial feature fusion network, enhance the diversity and expression ability of features, thereby solving the problem of missed detection of complex frames caused by loss of feature information, improving the fracture fault recognition rate of the network model for complex frames, and thus more accurately recognizing the fracture fault of complex frames.

[0106] Please refer to Figure 6 , Figure 6 which is a schematic structural diagram of a device for detecting the fracture fault of a complex frame provided by an embodiment of the present application.

[0107] A device for detecting the fracture fault of a complex frame provided by an embodiment of the present application may include:

[0108] A first acquisition module 101, configured to acquire a target image of the complex frame to be detected;

[0109] A first segmentation module 102, configured to segment the complex frame area in the target image to obtain a binary image of the complex frame;

[0110] A first recognition module 103, configured to perform fault recognition on the target image through a pre-trained fault recognition model to obtain crack area information;

[0111] A second segmentation module 104, configured to binarize the crack area in the target image according to the crack area information to obtain a binary crack image;

[0112] A first detection module 105, configured to obtain a fault detection result indicating that the complex frame to be detected is fractured in response to the coincidence between the crack area in the binary crack image and the complex frame area in the binary complex frame image;

[0113] Among them, the fault recognition model includes a residual network, a cross-stage partial feature fusion network, and a residual network with feature selection and no anchor boxes connected in sequence.

[0114] A complex frame breakage fault detection device provided by an embodiment of the present application, the first recognition module may include:

[0115] A first extraction unit, configured to extract features from a target image through a residual network to obtain an original feature map, and the dimension of the original feature map is less than a set value;

[0116] A first fusion unit, configured to perform feature fusion on the original feature map through a cross-stage partial feature fusion network to obtain an intermediate feature map;

[0117] A first recognition unit, configured to perform classification and position regression on the intermediate feature map through a residual network with feature selection and no anchor boxes to obtain category information and corresponding position information;

[0118] A first determination unit, configured to determine crack region information in the target image according to the category information and the position information.

[0119] A complex frame breakage fault detection device provided by an embodiment of the present application, the first recognition unit may be used for: performing classification and position regression on the intermediate feature map through a residual network with feature selection and no anchor boxes to obtain category information and corresponding position information, and the category information includes cracks, water flow, chalk, shadows, and foreign objects.

[0120] A complex frame breakage fault detection device provided by an embodiment of the present application may further include:

[0121] A second acquisition module, configured to acquire a training sample set before the first recognition module performs fault recognition on the target image through a pre-trained fault recognition model to obtain crack region information;

[0122] A second recognition module, configured to perform fault recognition on the images in the training sample set through an initial fault recognition model to obtain a recognition result;

[0123] A first generation module, configured to generate a loss function value of the fault recognition model according to the training sample set and the recognition result;

[0124] A first adjustment module, configured to adjust the fault recognition model based on the loss function value to obtain a trained fault recognition model;

[0125] Among them, the generation formula of the loss function value includes:

[0126] ;

[0127] ;

[0128] Among them, represents the loss function value; represents the total number of pixels within all valid bounding boxes; represents the hierarchical index in the feature pyramid; represents the position coordinates; represents the th layer of valid bounding boxes in the feature pyramid, represents the bounding box, represents valid; represents the intersection over union (IoU) between the predicted bounding box and the ground truth bounding box; represents the distance between the center points of the predicted bounding box and the ground truth bounding box; represents the diagonal length of the smallest bounding rectangle enclosing the predicted bounding box and the ground truth bounding box; represents the width of the ground truth bounding box, represents the height of the ground truth bounding box; represents the width of the predicted bounding box, represents the height of the predicted bounding box; represents the set value.

[0129] A complex frame breakage fault detection device provided by an embodiment of the present application, the second acquisition module may include:

[0130] A first acquisition unit, configured to acquire a train image obtained by photographing a training vehicle;

[0131] A first marking unit, configured to mark and draw regions of complex frame components in the train image, and crop out a first image of the known complex frame in the train image;

[0132] A first generation unit, configured to generate a marking file of the known complex frame based on the first image, where the marking file includes an image name, a detection category, and complex frame region coordinates;

[0133] A second determination unit, configured to determine the fault information of the known complex frame;

[0134] A first processing unit, configured to perform data enhancement and contrast enhancement processing on the first image to obtain a second image;

[0135] A first setting unit, configured to use the second image, the marking file, and the fault information as a training sample set.

[0136] A complex frame breakage fault detection device provided by an embodiment of the present application may further include:

[0137] A third acquisition module, configured to acquire the model parameters of the visual object recognition data set before the second recognition module performs fault recognition on the images in the training sample set through an initial fault recognition model;

[0138] A first initialization unit for initializing the network parameters of an initial fault identification model based on model parameters.

[0139] A complex frame breakage fault detection device provided by an embodiment of the present application. The first segmentation module may include:

[0140] A first segmentation unit for segmenting the complex frame area in the target image through a semantic segmentation network to obtain a complex frame binary image, where the pixel value of the complex frame area is 1 and the pixel value of other areas is 0;

[0141] The second segmentation module may include:

[0142] A second segmentation unit for binarizing the crack area in the target image through a semantic segmentation network according to the crack area information to obtain a crack binary image, where the pixel value of the crack area is 1 and the pixel value of the non-crack area is 0.

[0143] A complex frame breakage fault detection device provided by an embodiment of the present application. The first detection module may include:

[0144] A second processing unit for performing an AND operation on the complex frame binary image and the crack binary image to generate an image to be processed;

[0145] A first detection unit for detecting whether there is a pixel area with a value of 1 in the image to be processed; in response to the presence of a pixel area with a value of 1 in the image to be processed, a fault detection result indicating the breakage of the complex frame to be detected is obtained.

[0146] The present application also provides an electronic device and a computer-readable storage medium, both of which have the corresponding effects of a complex frame breakage fault detection method provided by an embodiment of the present application. Please refer to Figure 7 , Figure 7 which is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

[0147] An electronic device provided by an embodiment of the present application includes a memory 201 and a processor 202. A computer program is stored in the memory 201, and when the processor 202 executes the computer program, the steps of the complex frame breakage fault detection method described in any of the above embodiments are implemented.

[0148] Please refer to Figure 8, another electronic device provided by an embodiment of the present application may further include: an input port 203 connected to the processor 202, configured to transmit an externally input command to the processor 202; a display unit 204 connected to the processor 202, configured to display the processing result of the processor 202 to the outside; a communication module 205 connected to the processor 202, configured to implement communication between the electronic device and the outside. The display unit 204 may be a display panel, a laser scanning display, etc.; the communication methods adopted by the communication module 205 include but are not limited to Mobile High-Definition Link (MHL), Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI), wireless connections: Wireless Fidelity (WiFi), Bluetooth communication technology, low-power Bluetooth communication technology, communication technology based on IEEE802.11s.

[0149] A computer-readable storage medium provided by an embodiment of the present application stores a computer program, and when the computer program is executed by a processor, it implements the steps of the complex frame breakage fault detection method described in any of the above embodiments.

[0150] The computer-readable storage medium involved in the present application includes Random Access Memory (RAM), memory, Read-Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROM (Compact Disc Read-Only Memory), or any other form of storage medium well-known in the technical field.

[0151] For the description of the relevant parts in a complex frame breakage fault detection device, an electronic device, and a computer-readable storage medium provided by an embodiment of the present application, please refer to the detailed description of the corresponding parts in a complex frame breakage fault detection method provided by an embodiment of the present application, which will not be elaborated here. In addition, the parts of the above technical solutions provided by the embodiments of the present application that are consistent with the corresponding technical solutions in the prior art in terms of implementation principles are not described in detail to avoid excessive elaboration.

[0152] It should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0153] The foregoing description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting a composite frame breakage fault, characterized in that: include: Acquire a target image of the complex frame to be detected; Segmenting the complex frame region in the target image to obtain a complex frame binary image; Performing fault recognition on the target image through a pre-trained fault recognition model to obtain crack area information; Binarizing the crack region in the target image according to the crack region information to obtain a crack binary image; In response to the crack region in the crack binary image being coincident with the complex frame region in the complex frame binary image, a fault detection result characterizing the fracture of the complex frame to be detected is obtained; The fault recognition model includes a sequentially connected residual network, a cross-stage partial feature fusion network, and a feature selection anchor-free residual network.

2. The method for detecting fracture failure of a composite frame according to claim 1, characterized in that: The target image is subjected to fault recognition through a pre-trained fault recognition model to obtain crack area information, including: Extracting features of the target image through the residual network to obtain an original feature map, wherein the dimension of the original feature map is smaller than a set value; Performing feature fusion on the original feature map through the cross-stage partial feature fusion network to obtain an intermediate feature map; Classify and regress the intermediate feature map through the feature selection anchor-free residual network to obtain category information and corresponding position information; The crack region information is determined in the target image according to the category information and the position information.

3. The method for detecting fracture failure of a composite frame according to claim 2, characterized in that: The intermediate feature map is classified and positionally regressed by the feature selection anchor-free residual network to obtain category information and corresponding position information, including: The intermediate feature map is classified and regressed by the feature selection anchor-free residual network to obtain category information and corresponding position information, wherein the category information includes cracks, water flow, chalk, shadows and foreign matter.

4. The method for detecting fracture failure of a composite frame according to claim 1, characterized in that: Before performing fault recognition on the target image by using a pre-trained fault recognition model to obtain crack area information, the method further includes: Obtain a training sample set; Performing fault recognition on the images in the training sample set by using an initial fault recognition model to obtain a recognition result; Generating a loss function value of the fault identification model according to the training sample set and the identification result; Adjusting the fault identification model based on the loss function value to obtain the trained fault identification model; The loss function value generation formula includes: ； ； in, represents the loss function value; Indicates the total number of pixels in all valid frames; Represents the level index in the feature pyramid; Indicates the location coordinates; Represents the feature pyramid The effective bounding box of the layer, represents the bounding box, Indicates validity; Represents the intersection-over-union ratio of the predicted bounding box and the true bounding box; Represents the distance between the center point of the predicted bounding box and the center point of the true bounding box; Represents the diagonal length of the minimum bounding rectangle that contains the predicted bounding box and the true bounding box; represents the width of the true bounding box, Indicates the height of the true bounding box; represents the width of the predicted bounding box, Represents the height of the predicted bounding box; Indicates the set value.

5. The method for detecting fracture failure of a composite frame according to claim 4, characterized in that: Get the training sample set, including: Acquire a train image obtained by photographing a training vehicle; Marking and region drawing of the complex frame components in the train image, and intercepting a first image of the known complex frame in the train image; Based on the first image, generating a label file of a known complex frame, the label file including an image name, a detection category, and coordinates of a complex frame region; Determine the fault information of known complex structures; Performing data enhancement and contrast enhancement processing on the first image to obtain a second image; The second image, the marking file and the fault information are used as a training sample set.

6. The method for detecting fracture failure of a composite frame according to claim 4, characterized in that: Before performing fault recognition on the images in the training sample set by using the initial fault recognition model, the method further includes: Get the model parameters of the visual object recognition dataset; The network parameters of the initial fault identification model are initialized based on the model parameters.

7. The method for detecting fracture failure of a composite frame according to claim 1, characterized in that: Segmenting the complex frame region in the target image to obtain a complex frame binary image includes: Segment the complex frame region in the target image through a semantic segmentation network to obtain a complex frame binary image, wherein the pixel value of the complex frame region is 1, and the pixel values ​​of other regions are 0; The step of binarizing the crack region in the target image according to the crack region information to obtain a crack binary image comprises: Through the semantic segmentation network, the crack area in the target image is binarized according to the crack area information to obtain a crack binary image, wherein the pixel value of the crack area is 1, and the pixel value of the non-crack area is 0.

8. The method for detecting fracture failure of a composite frame according to claim 7, characterized in that: In response to the crack region in the crack binary image being coincident with the complex frame region in the complex frame binary image, a fault detection result characterizing the fracture of the complex frame to be detected is obtained, including: Performing AND processing on the complex frame binary image and the crack binary image to generate an image to be processed; Detecting whether there is a pixel region with a value of 1 in the image to be processed; In response to the presence of a pixel region with a value of 1 in the image to be processed, a fault detection result indicating that the complex frame to be detected is broken is obtained.

9. A composite frame breakage fault detection device, characterized in that: include: A first acquisition module is used to acquire a target image of the complex frame to be detected; A first segmentation module is used to segment the complex frame area in the target image to obtain a complex frame binary image; A first recognition module is used to perform fault recognition on the target image through a pre-trained fault recognition model to obtain crack area information; A second segmentation module is used to binarize the crack region in the target image according to the crack region information to obtain a crack binary image; A first detection module, configured to obtain a fault detection result indicating that the complex frame to be detected is broken in response to the crack region in the crack binary image being coincident with the complex frame region in the complex frame binary image; The fault recognition model includes a sequentially connected residual network, a cross-stage partial feature fusion network, and a feature selection anchor-free residual network.

10. An electronic device, characterized in that: include: Memory for storing computer programs; A processor is used to implement the steps of the composite frame breakage fault detection method as claimed in any one of claims 1 to 8 when executing the computer program.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the complex frame breakage fault detection method according to any one of claims 1 to 8 are performed.

Images