Composite frame break fault detection method, device, electronic equipment and medium

Abstract

The present application discloses a method, device, electronic device, and medium for detecting a fractured composite frame, which relates to the field of railway transportation technology. The method comprises the following steps: obtaining a target image of a composite frame to be detected; segmenting the composite frame region in the target image to obtain a composite frame binary image; performing fault identification on the target image using a pre-trained fault identification model to obtain crack region information; binarizing the crack region in the target image based on the crack region information to obtain a crack binary image; and obtaining a fault detection result indicating fractured composite frame to be detected in response to the overlap between the crack region in the crack binary image and the composite frame region in the composite frame binary image. The fault identification model comprises a sequentially connected residual network, a cross-stage partial feature fusion network, and a feature selection anchor-free residual network. The method can improve the positioning accuracy of the composite frame and the crack region, enhance the diversity and expression capability of features, and more accurately identify fractured composite frames.

Description

Technical Field

[0001] The present application relates to the field of railway transportation technology, and more specifically, to a method, device, electronic equipment and medium for detecting a fracture fault of a composite frame. Background Art

[0002] In railway transportation systems, the integrity and stability of the multiframe, a structural component, impact the safe operation of trains. However, multiframes can break over time, potentially impacting the normal operation of trains and potentially leading to accidents. Therefore, multiframe breakage detection is essential.

[0003] For example, we can rely on pixel features and fixed threshold settings to detect fracture faults in complex structures. However, when faced with complex fault morphologies, we cannot fully capture key features, especially when the feature morphology changes significantly. The risk of missed detection and false detection increases significantly, which not only reduces the reliability of fault detection but also increases potential safety hazards.

[0004] In summary, how to accurately detect the breakage fault of a composite frame is a problem that needs to be solved urgently by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a method for detecting a breakage fault of a composite frame, which can solve the technical problem of how to accurately detect a breakage fault of a composite frame to a certain extent. This application also provides a device for detecting a breakage fault of a composite frame, an electronic device, and a computer-readable storage medium.

[0006] In order to achieve the above objectives, this application provides the following technical solutions:

[0007] In a first aspect, a method for detecting a fracture fault of a composite frame is provided, comprising:

[0008] Acquire a target image of the complex frame to be detected;

[0009] Segmenting the complex frame region in the target image to obtain a complex frame binary image;

[0010] Performing fault recognition on the target image using a pre-trained fault recognition model to obtain crack area information;

[0011] Binarizing the crack region in the target image according to the crack region information to obtain a crack binary image;

[0012] In response to the crack region in the crack binary image being coincident with the re-frame region in the re-frame binary image, a fault detection result indicating that the re-frame to be detected is broken is obtained;

[0013] The fault recognition model includes a sequentially connected residual network, a cross-stage partial feature fusion network, and a feature selection anchor-free residual network.

[0014] On the other hand, the target image is subjected to fault recognition using a pre-trained fault recognition model to obtain crack area information, including:

[0015] Performing feature extraction on the target image through the residual network to obtain an original feature map, wherein the dimension of the original feature map is smaller than a set value;

[0016] Performing feature fusion on the original feature map through the cross-stage partial feature fusion network to obtain an intermediate feature map;

[0017] Classify and regress the intermediate feature map using the feature selection anchor-free residual network to obtain category information and corresponding position information;

[0018] The crack region information is determined in the target image according to the category information and the position information.

[0019] On the other hand, the intermediate feature map is classified and regressed by the residual network without anchor boxes through the feature selection to obtain category information and corresponding position information, including:

[0020] The intermediate feature map is classified and regressed by the residual network without anchor boxes through the feature selection to obtain category information and corresponding position information, wherein the category information includes cracks, water flow, chalk, shadows and foreign objects.

[0021] On the other hand, before performing fault recognition on the target image using a pre-trained fault recognition model to obtain crack area information, the method further includes:

[0022] Obtain a training sample set;

[0023] Performing fault recognition on the images in the training sample set using an initial fault recognition model to obtain a recognition result;

[0024] Generating a loss function value of the fault recognition model according to the training sample set and the recognition result;

[0025] Adjusting the fault identification model based on the loss function value to obtain the trained fault identification model;

[0026] The loss function value generation formula includes:

[0027] ;

[0028] ;

[0029] in, represents the loss function value; Indicates the total number of pixels in all valid frames; Represents the level index in the feature pyramid; Indicates location coordinates; Represents the feature pyramid The effective bounding box of the layer, represents the bounding box, Indicates validity; represents the intersection-over-union ratio of the predicted bounding box and the true bounding box; Represents the distance between the center point of the predicted bounding box and the center point of the true bounding box; Represents the diagonal length of the minimum bounding rectangle containing the predicted bounding box and the true bounding box; represents the width of the ground-truth bounding box, Indicates the height of the ground-truth bounding box; represents the width of the predicted bounding box, Represents the height of the predicted bounding box; Indicates the set value.

[0030] On the other hand, obtain a training sample set, including:

[0031] Acquire a train image obtained by photographing a training vehicle;

[0032] Marking and region-drawing the complex frame components in the train image, and extracting a first image of the known complex frame in the train image;

[0033] Based on the first image, generating a label file of a known complex frame, the label file including an image name, a detection category, and coordinates of a complex frame area;

[0034] Determine the fault information of known complex structures;

[0035] performing data enhancement and contrast enhancement processing on the first image to obtain a second image;

[0036] The second image, the marked file and the fault information are used as a training sample set.

[0037] On the other hand, before performing fault recognition on the images in the training sample set using the initial fault recognition model, the method further includes:

[0038] Get the model parameters of the visual object recognition dataset;

[0039] The network parameters of the initial fault identification model are initialized based on the model parameters.

[0040] On the other hand, segmenting the complex frame region in the target image to obtain a complex frame binary image includes:

[0041] Segmenting the complex frame region in the target image through a semantic segmentation network to obtain a complex frame binary image, wherein the pixel value of the complex frame region is 1, and the pixel values of other regions are 0;

[0042] Binarizing the crack region in the target image according to the crack region information to obtain a crack binary image includes:

[0043] Through the semantic segmentation network, the crack area in the target image is binarized according to the crack area information to obtain a crack binary image, wherein the pixel value of the crack area is 1 and the pixel value of the non-crack area is 0.

[0044] On the other hand, in response to the crack region in the crack binary image being coincident with the re-frame region in the re-frame binary image, a fault detection result indicating that the re-frame to be detected is broken is obtained, including:

[0045] Performing AND processing on the complex frame binary image and the crack binary image to generate an image to be processed;

[0046] Detecting whether there is a pixel region with a value of 1 in the image to be processed;

[0047] In response to the presence of a pixel region with a value of 1 in the image to be processed, a fault detection result indicating that the complex structure to be detected is broken is obtained.

[0048] In a second aspect, a device for detecting a fracture fault of a composite frame is provided, comprising:

[0049] A first acquisition module is used to acquire a target image of the complex frame to be detected;

[0050] A first segmentation module is used to segment the complex frame area in the target image to obtain a complex frame binary image;

[0051] A first recognition module is used to perform fault recognition on the target image using a pre-trained fault recognition model to obtain crack area information;

[0052] A second segmentation module is used to binarize the crack area in the target image according to the crack area information to obtain a crack binary image;

[0053] a first detection module configured to obtain a fault detection result indicating that the complex frame to be detected is broken in response to a crack region in the crack binary image overlapping with a complex frame region in the complex frame binary image;

[0054] The fault recognition model includes a sequentially connected residual network, a cross-stage partial feature fusion network, and a feature selection anchor-free residual network.

[0055] According to a third aspect, an electronic device is provided, including:

[0056] memory for storing computer programs;

[0057] A processor is used to implement the steps of any of the above-mentioned complex frame breakage fault detection methods when executing the computer program.

[0058] In a fourth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the steps of any of the above-mentioned complex frame breakage fault detection methods are performed.

[0059] The present application provides a method for detecting a complex frame break fault, which comprises the following steps: obtaining a target image of a complex frame to be detected; segmenting the complex frame region in the target image to obtain a binary image of the complex frame; performing fault identification on the target image using a pre-trained fault recognition model to obtain crack region information; binarizing the crack region in the target image based on the crack region information to obtain a crack binary image; and obtaining a fault detection result representing the break of the complex frame to be detected in response to the overlap between the crack region in the crack binary image and the complex frame region in the complex frame binary image; wherein the fault recognition model comprises a sequentially connected residual network, a cross-stage partial feature fusion network, and a feature selection residual network without an anchor frame. In the present application, the target image is segmented according to the complex frame area and the crack area, which can improve the positioning accuracy of the complex frame and the crack area. In this way, if the fracture detection is performed based on the obtained binary image, it can avoid the false alarm of similar fracture faults in non-complex frame positions, and improve the detection accuracy of complex frame fracture faults; and the present application introduces the cross-stage partial feature fusion network into the residual network of feature selection without anchor frame, which can combine the cross-stage feature information fusion capability of the cross-stage partial feature fusion network to enhance the diversity and expression capability of features, thereby solving the problem of missed detection of complex frames due to the loss of feature information, improving the recognition rate of complex frame fracture faults by the network model, and thus more accurately identifying complex frame fracture faults. The complex frame fracture fault detection device, electronic device and computer-readable storage medium provided by the present application also solve the corresponding technical problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0061] Figure 1 A flowchart of a method for detecting a fracture fault of a composite frame provided in an embodiment of the present application;

[0062] Figure 2 This is a schematic diagram of a complex structure with a fracture fault;

[0063] Figure 3 Schematic diagram of the cross-stage partial feature fusion network;

[0064] Figure 4 is a schematic diagram of the fault identification model;

[0065] Figure 5 This is a schematic diagram of the training of the fault recognition model;

[0066] Figure 6 A schematic structural diagram of a composite frame breakage fault detection device provided in an embodiment of the present application;

[0067] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0068] Figure 8 Another structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0070] See also Figure 1 , Figure 1 A flowchart of a method for detecting a composite frame breakage fault provided in an embodiment of the present application.

[0071] The present invention provides a method for detecting a fracture fault of a composite frame, which may include the following steps:

[0072] Step S101: Acquire a target image of a complex frame to be detected.

[0073] In practical applications, since fracture fault detection is performed on a composite frame, the corresponding information of the composite frame to be detected needs to be obtained. Considering that the image can record the information of the composite frame to be detected and is easy to process, the target image of the composite frame to be detected can be obtained first.

[0074] It should be noted that the target image can be acquired using image acquisition equipment such as cameras and sensors. To facilitate understanding of this process, taking a railway freight car as an example, the target image acquisition process can be as follows: trackside imaging equipment is set up on both sides of the railway. After the freight car passes through a high-definition industrial linear array camera, a high-definition train image is acquired. Then, based on prior knowledge such as hardware equipment, wheelbase information, and related positions, the area of the side re-frame components is cut out from the high-definition train image to obtain an initial re-frame image. If the initial re-frame image meets the image detection requirements, the initial re-frame image can be used as the target image. If the initial re-frame image does not meet the image detection requirements, the initial re-frame image is adjusted to obtain the target image. For example, if the initial re-frame image has different image brightness levels due to the camera angle and distance, to avoid the image being too dark and the broken area of the re-frame being unable to be clearly observed, the initial re-frame image can be subjected to local adaptive contrast enhancement processing. Alternatively, other enhancement processing can be continued on the initial re-frame image as needed to obtain the target image.

[0075] Step S102: Segment the complex frame region in the target image to obtain a complex frame binary image.

[0076] In practical applications, due to the limitation of the shape of the complex frame, other objects besides the complex frame may exist in the target image, such as a broken complex frame image. Figure 2 As shown, in order to exclude other objects from affecting the identification of fracture faults of the complex frame, the complex frame region in the target image can be segmented to obtain a complex frame binary image that distinguishes the complex frame region from other regions.

[0077] Step S103: performing fault recognition on the target image using a pre-trained fault recognition model to obtain crack area information.

[0078] In practical applications, neural network models can be used to identify faults in target images. For example, a pre-trained fault recognition model can be used to identify faults in target images and obtain crack area information. Fault recognition models include sequentially connected residual networks (ResNet), cross-stage partial feature fusion networks (CSP-PAN), and feature selection anchor-free residual networks (FSAF-RetinaNet).

[0079] In a specific application scenario, when a pre-trained fault recognition model is used to identify a target image and obtain crack region information, the target image can be feature extracted through a residual network to obtain an original feature map whose dimension is less than a set value; the original feature map is fused through a cross-stage partial feature fusion network to obtain an intermediate feature map; the intermediate feature map is classified and regressed through a feature selection anchor-free residual network to obtain category information and corresponding location information; based on the category information and location information, the crack region information is determined in the target image. That is, the residual network is used to extract features from the target image to obtain low-dimensional feature maps of different layers, the cross-stage partial feature fusion network is used to fuse features of low-dimensional feature maps of different layers to obtain an intermediate feature map, and the feature selection anchor-free residual network is used to classify and regress the intermediate feature map to obtain fault category information in the target image and the location information corresponding to the fault category information.

[0080] It should be noted that the structure of each component in the fault identification model can be flexibly determined according to actual needs. For example, the structure of the CSP-PAN network can be as follows: Figure 3 As shown, specifically, the third layer feature map can be represented by C3, the fourth layer feature map can be represented by C4, and the fifth layer feature map can be represented by C5. The feature maps of the third layer feature map, the fourth layer feature map and the fifth layer feature map are input into the CSP-PAN network at the same time, and the feature map of the third layer feature map is subjected to 1x1 convolution, and the feature map of the set channel is output. For example, the feature map of the 96-channel feature map of the third layer feature map is subjected to 1x1 convolution, and the feature map of the 96-channel feature map is output; the feature map of the fourth layer feature map is subjected to 1x1 convolution, and the feature map of the set channel is output. For example, the feature map of the 192-channel feature map of the fourth layer feature map is subjected to 1x1 convolution, and the feature map of the 96-channel feature map is output; the feature map of the fifth layer feature map is subjected to 1x1 convolution, and the feature map of the set channel is output. Set the feature map of the channel, for example, perform 1x1 convolution on the 384-channel feature map of the fifth-layer feature map, and output a 96-channel feature map; input the output feature map into the CSP module for cross-stage partial aggregation of features, and the CSP module divides the feature map into two parts, processes and merges them separately to enhance the diversity of features, wherein the feature maps of the fourth-layer feature map and the fifth-layer feature map are upsampled and downsampled, the feature maps of different layers are aligned, and the channels are spliced, and the processed feature maps are combined together to form a richer feature representation; finally, the CSP-PAN structure outputs intermediate feature maps of multiple scales, that is, the third-layer output P3, the fourth-layer output P4, the fifth-layer output P5, and the sixth-layer output P6 are obtained.

[0081] For example, the FSAF-RetinaNet network can include the Class Subnet and the Box Subnet. In this case, the structure of the fault identification model is as follows: Figure 4 As shown in the figure; correspondingly, in the FSAF-RetinaNet network, each intermediate feature map first passes through the category subnet Class Subnet, which is composed of multiple convolutional layers. For example, there is a W×H×256 feature map. After 4 convolutions, it outputs a W×H×K feature map, where K is the number of categories; at the same time, the feature map also passes through the box subnet Box Subnet, which is also composed of multiple convolutional layers. For example, there is a W×H×256 feature map. After 4 convolutions, it outputs a W×H×4A feature map, where A is the number of anchor boxes.

[0082] In specific application scenarios, the type of fault information detected by the FSAF-RetinaNet network can be flexibly determined according to actual needs. For example, considering that complex structures sometimes have traces of water flow, chalk marks, and shadows of hanging foreign objects, and the image features of cracks at broken parts are similar, the faults are classified into five categories: cracks, water flow, chalk, shadows, and foreign objects. That is, in the process of classifying and regressing the intermediate feature maps through the FSAF-RetinaNet network to obtain category information and corresponding position information, the intermediate feature maps can be classified and regressed through the FSAF-RetinaNet network to obtain category information and corresponding position information. The category information can include cracks, water flow, chalk, shadows, and foreign objects.

[0083] In practical applications, the fault recognition model needs to be trained. That is, before the target image is fault-identified using the pre-trained fault recognition model and crack area information is obtained, the fault recognition model needs to be trained. The training process can be shown in Figure 5 and includes the following steps:

[0084] Step S2011: Obtain a training sample set.

[0085] This training sample set is used to identify the fault type of the complex frame for the fault recognition model, and may include complex frame images, complex frame marking information, and complex frame fault information;

[0086] Step S2012: performing fault recognition on the images in the training sample set using the initial fault recognition model to obtain recognition results.

[0087] Step S2013: Generate a loss function value of the fault recognition model based on the training sample set and the recognition result.

[0088] Step S2014: adjusting the fault recognition model based on the loss function value to obtain a trained fault recognition model.

[0089] Among them, the loss function value generation formula may include:

[0090] ;

[0091] ;

[0092] in, Represents the loss function value; Indicates the total number of pixels in all valid frames; Represents the level index in the feature pyramid; Indicates location coordinates; Represents the feature pyramid The effective bounding box of the layer, represents the bounding box, Indicates validity; represents the intersection-over-union ratio of the predicted bounding box and the true bounding box; Represents the distance between the center point of the predicted bounding box and the center point of the true bounding box, which can be Euclidean clustering, etc. Represents the diagonal length of the minimum bounding rectangle containing the predicted bounding box and the true bounding box; represents the width of the ground-truth bounding box, Indicates the height of the ground-truth bounding box; represents the width of the predicted bounding box, Represents the height of the predicted bounding box; Indicates the set value, which can be Etc. Among them, the functions lost in each part are as follows: This part of the loss predicts that the center point of the bounding box is as close as possible to the center point of the real bounding box. The smaller it is, the smaller this part of the loss will be; This loss ensures that the aspect ratio of the predicted bounding box is consistent with that of the true bounding box. By minimizing the aspect ratio difference, the shape accuracy of the predicted bounding box can be improved. FR comprehensively measures the difference between the predicted and true bounding boxes by comprehensively considering the degree of overlap, center point distance, and aspect ratio consistency. Using the FR function can improve bounding box regression performance, thereby improving overall detection accuracy. In other words, the loss function of this application can comprehensively measure the difference between the predicted and true bounding boxes, effectively improving the algorithm's detection accuracy.

[0093] It should be noted that the formula for generating the loss function value, that is, the derivation process of the loss function can be: Considering that in the anchor box branch, the anchor box mechanism is used for target detection, that is, by classifying and regressing each anchor box, the category and position of the target are predicted. In the anchor-free branch, target detection is performed on the feature map. Through the feature selection mechanism, the most representative features are selected for target classification and position regression. From the feature map output by the classification subnetwork, the category probability of each position is extracted. The probability of each category is calculated by the Softmax (normalized exponential function) function. From the feature map output by the regression subnetwork, the bounding box coordinates of each position are extracted, and the offset between the predicted box and the true box is calculated by the loss function. For the predicted bounding box, the non-maximum suppression algorithm is applied to retain the optimal detection result. Finally, the category and position coordinates of each target are output. During the training process, the total classification loss of the anchor-free branch in the image is the sum of the losses of all non-ignored areas and normalized to the total number of pixels in all valid box areas, that is:

[0094] ;

[0095] Indicates the position at the feature level The focus position loss, Represents the FocalLoss loss function;

[0096] If regression loss is defined, the loss function is:

[0097] ;

[0098] .

[0099] In specific application scenarios, in the process of obtaining the training sample set, the train images obtained by shooting the training vehicles can be obtained. In this process, considering that the quality of the complex frame image is easily affected by weather and human factors, such as rain, snow, mud stains, light and dark, and the images of different stations are affected by the differences in equipment installation angles, distances, etc., resulting in large differences in the complex frame images, therefore, in the process of collecting image data, efforts should be made to cover images under various conditions to ensure the diversity of train images, and then ensure the diversity of complex frame images used for training; mark the complex frame components in the train image and draw the area, and cut out the train image A first image of a known complex frame in an image; based on the first image, generating a labeling file of the known complex frame, the labeling file including the image name, detection category, complex frame area coordinates, etc., the labeling file may be an XML file, and the complex frame area coordinates may include the coordinates of the upper left corner and the lower right corner of the complex frame area, etc.; determining the fault information of the known complex frame; performing data enhancement and contrast enhancement processing on the first image to obtain a second image, so as to improve sample diversity and then improve the generalization of the algorithm, the data enhancement may include randomly rotating, translating, scaling and mirroring the image, etc.; using the second image, the labeling file and the fault information as a training sample set.

[0100] In specific application scenarios, if the parameters of the fault recognition model are initialized in advance, the training efficiency of the recognition model can be improved. That is, before the initial fault recognition model is used to identify faults in images in the training sample set, the model parameters of the visual object recognition dataset can be obtained, such as the model parameters of the ImageNet dataset; and the network parameters of the initial fault recognition model are initialized based on the model parameters.

[0101] Step S104: Binarize the crack region in the target image according to the crack region information to obtain a crack binary image.

[0102] Step S105: In response to the crack region in the crack binary image being coincident with the re-frame region in the re-frame binary image, a fault detection result indicating that the re-frame to be detected is broken is obtained.

[0103] In practical applications, when the crack falls outside the complex frame, there is also a break fault in the target image, but the break fault at this time is not a complex frame break fault. In order to eliminate this situation and more accurately detect the break fault of the complex frame, the crack area in the target image can be binarized according to the crack area information to obtain a crack binary image that distinguishes the crack area from the non-crack area; then, it is detected whether the crack area in the crack binary image coincides with the complex frame area in the complex frame binary image. In response to the fact that the crack area in the crack binary image coincides with the complex frame area in the complex frame binary image, a fault detection result is obtained that indicates that the complex frame to be detected is broken. In response to the fact that the crack area in the crack binary image does not coincide with the complex frame area in the complex frame binary image, a fault detection result is obtained that indicates that the complex frame to be detected is not broken.

[0104] In practical applications, in the process of segmenting the complex frame region in the target image to obtain a complex frame binary image, the complex frame region in the target image can be segmented using a semantic segmentation network to obtain a complex frame binary image, wherein the pixel value of the complex frame region is 1 and the pixel value of other regions is 0. In the process of binarizing the crack region in the target image based on the crack region information to obtain a crack binary image, the crack region in the target image can be binarized using a semantic segmentation network based on the crack region information to obtain a crack binary image, wherein the pixel value of the crack region is 1 and the pixel value of the non-crack region is 0. Accordingly, in the process of obtaining a fault detection result indicating that the complex frame to be detected is broken in response to the crack region in the crack binary image coinciding with the complex frame region in the complex frame binary image, the complex frame binary image and the crack binary image can be processed to generate a processed image; and the existence of a pixel region with a value of 1 in the processed image is detected; and in response to the existence of a pixel region with a value of 1 in the processed image, a fault detection result indicating that the complex frame to be detected is broken is obtained. In this way, it is only necessary to determine the pixel values of the complex frame area and the crack area in the target image as 1, and then perform an AND operation, so as to easily and quickly detect whether the crack area in the crack binary image and the complex frame area in the complex frame binary image overlap, thereby further improving the efficiency of complex frame breakage fault detection.

[0105] The present application provides a method for detecting a complex frame break fault, which comprises the following steps: obtaining a target image of a complex frame to be detected; segmenting the complex frame region in the target image to obtain a binary image of the complex frame; performing fault identification on the target image using a pre-trained fault recognition model to obtain crack region information; binarizing the crack region in the target image based on the crack region information to obtain a crack binary image; and obtaining a fault detection result representing the break of the complex frame to be detected in response to the overlap between the crack region in the crack binary image and the complex frame region in the complex frame binary image; wherein the fault recognition model comprises a sequentially connected residual network, a cross-stage partial feature fusion network, and a feature selection residual network without an anchor frame. In the present application, the target image is segmented according to the complex frame area and the crack area, which can improve the positioning accuracy of the complex frame and crack area. In this way, if the fracture detection is performed based on the obtained binary image, the false alarm of similar fracture faults in non-complex frame positions can be avoided, and the detection accuracy of complex frame fracture faults can be improved; and the present application introduces the cross-stage partial feature fusion network into the feature selection anchor-free residual network, which can be combined with the cross-stage feature information fusion capability of the cross-stage partial feature fusion network to enhance the diversity and expression ability of features, thereby solving the problem of missed detection of complex frames due to the loss of feature information, improving the network model's recognition rate of complex frame fracture faults, and thus more accurately identifying complex frame fracture faults.

[0106] See also Figure 6 , Figure 6 This is a structural diagram of a composite frame breakage fault detection device provided in an embodiment of the present application.

[0107] An embodiment of the present application provides a device for detecting a fracture fault of a composite frame, which may include:

[0108] The first acquisition module 101 is used to acquire a target image of a complex frame to be detected;

[0109] The first segmentation module 102 is used to segment the complex frame area in the target image to obtain a complex frame binary image;

[0110] The first recognition module 103 is used to perform fault recognition on the target image using a pre-trained fault recognition model to obtain crack area information;

[0111] The second segmentation module 104 is used to binarize the crack area in the target image according to the crack area information to obtain a crack binary image;

[0112] A first detection module 105 is configured to obtain a fault detection result indicating that the complex frame to be detected is broken in response to a crack region in the crack binary image being coincident with a complex frame region in the complex frame binary image;

[0113] Among them, the fault recognition model includes a sequentially connected residual network, a cross-stage partial feature fusion network, and a feature selection anchor-free residual network.

[0114] In an embodiment of the present application, a device for detecting a fracture of a composite frame is provided, wherein a first identification module may include:

[0115] A first extraction unit is used to extract features of the target image through a residual network to obtain an original feature map, where the dimension of the original feature map is smaller than a set value;

[0116] The first fusion unit is used to fuse the original feature map through a cross-stage partial feature fusion network to obtain an intermediate feature map;

[0117] The first recognition unit is used to classify and regress the intermediate feature map through a residual network with feature selection and no anchor box to obtain category information and corresponding position information;

[0118] The first determining unit is used to determine crack area information in the target image according to the category information and the position information.

[0119] An embodiment of the present application provides a complex frame break fault detection device, in which the first recognition unit can be used to: classify and regress the intermediate feature map through a feature selection anchor-free residual network to obtain category information and corresponding position information, wherein the category information includes cracks, water flow, chalk, shadows and foreign objects.

[0120] The embodiment of the present application provides a device for detecting a fracture fault of a composite frame, which may further include:

[0121] The second acquisition module is used for the first recognition module to perform fault recognition on the target image using a pre-trained fault recognition model to obtain a training sample set before obtaining crack area information;

[0122] The second recognition module is used to perform fault recognition on the images in the training sample set using the initial fault recognition model to obtain recognition results;

[0123] A first generation module is used to generate a loss function value of a fault recognition model based on a training sample set and recognition results;

[0124] A first adjustment module is used to adjust the fault recognition model based on the loss function value to obtain a trained fault recognition model;

[0125] Among them, the formula for generating the loss function value includes:

[0126] ;

[0127] ;

[0128] in, Represents the loss function value; Indicates the total number of pixels in all valid frames; Represents the level index in the feature pyramid; Indicates location coordinates; Represents the feature pyramid The effective bounding box of the layer, represents the bounding box, Indicates validity; represents the intersection-over-union ratio of the predicted bounding box and the true bounding box; Represents the distance between the center point of the predicted bounding box and the center point of the true bounding box; Represents the diagonal length of the minimum bounding rectangle containing the predicted bounding box and the true bounding box; represents the width of the ground-truth bounding box, Indicates the height of the ground-truth bounding box; represents the width of the predicted bounding box, Represents the height of the predicted bounding box; Indicates the set value.

[0129] In an embodiment of the present application, a device for detecting a fracture of a composite frame is provided, wherein the second acquisition module may include:

[0130] A first acquisition unit is used to acquire a train image obtained by photographing a training vehicle;

[0131] a first marking unit, configured to mark and region-draw the complex frame components in the train image, and to extract a first image of the known complex frame in the train image;

[0132] A first generating unit is configured to generate a label file of a known complex frame based on the first image, wherein the label file includes an image name, a detection category, and coordinates of a complex frame region;

[0133] A second determining unit is used to determine fault information of a known complex architecture;

[0134] a first processing unit, configured to perform data enhancement and contrast enhancement processing on the first image to obtain a second image;

[0135] The first setting unit is configured to use the second image, the marking file, and the fault information as a training sample set.

[0136] The embodiment of the present application provides a device for detecting a fracture fault of a composite frame, which may further include:

[0137] A third acquisition module is configured to acquire model parameters of the visual object recognition dataset before the second recognition module performs fault recognition on the images in the training sample set using the initial fault recognition model;

[0138] The first initialization unit is configured to initialize network parameters of the initial fault identification model based on the model parameters.

[0139] In an embodiment of the present application, a device for detecting a fracture of a composite frame is provided, wherein the first segmentation module may include:

[0140] The first segmentation unit is used to segment the complex frame area in the target image through a semantic segmentation network to obtain a complex frame binary image, wherein the pixel value of the complex frame area is 1 and the pixel value of other areas is 0;

[0141] The second segmentation module may include:

[0142] The second segmentation unit is used to binarize the crack area in the target image according to the crack area information through the semantic segmentation network to obtain a crack binary image, where the pixel value of the crack area is 1 and the pixel value of the non-crack area is 0.

[0143] An embodiment of the present application provides a device for detecting a fracture fault of a composite frame, wherein a first detection module may include:

[0144] The second processing unit is used to perform AND processing on the complex frame binary image and the crack binary image to generate an image to be processed;

[0145] The first detection unit is used to detect whether there is a pixel area with a value of 1 in the image to be processed; in response to the existence of a pixel area with a value of 1 in the image to be processed, a fault detection result indicating that the complex structure to be detected is broken is obtained.

[0146] The present application also provides an electronic device and a computer-readable storage medium, both of which have the corresponding effects of the composite frame breakage fault detection method provided in the embodiment of the present application. Figure 7 , Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0147] An electronic device provided in an embodiment of the present application includes a memory 201 and a processor 202. The memory 201 stores a computer program. When the processor 202 executes the computer program, the steps of the complex frame breakage fault detection method described in any of the above embodiments are implemented.

[0148] See also Figure 8Another electronic device provided in an embodiment of the present application may further include: an input port 203 connected to the processor 202 for transmitting commands inputted from the outside to the processor 202; a display unit 204 connected to the processor 202 for displaying the processing results of the processor 202 to the outside world; and a communication module 205 connected to the processor 202 for enabling communication between the electronic device and the outside world. The display unit 204 may be a display panel, a laser scanning display, etc. The communication method adopted by the communication module 205 includes but is not limited to Mobile High-Definition Link (MHL), Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI), wireless connection: Wireless Fidelity (WiFi), Bluetooth communication technology, Bluetooth low energy communication technology, and communication technology based on IEEE802.11s.

[0149] An embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the complex frame breakage fault detection method described in any of the above embodiments are implemented.

[0150] The computer-readable storage medium involved in this application includes random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs (Compact Disc Read-Only Memory), or any other form of storage medium known in the technical field.

[0151] For the description of the relevant parts of the device for detecting a fractured composite frame, the electronic device, and the computer-readable storage medium provided in the embodiments of the present application, please refer to the detailed description of the corresponding parts in the method for detecting a fractured composite frame provided in the embodiments of the present application, and no further description is given here. In addition, the parts of the above-mentioned technical solutions provided in the embodiments of the present application that are consistent with the corresponding technical solutions in the prior art in the implementation principles are not described in detail to avoid excessive elaboration.

[0152] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0153] The above description of the disclosed embodiments will enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting a fracture fault of a composite frame, characterized in that: include: Acquire a target image of the complex frame to be detected; Segmenting the complex frame region in the target image to obtain a complex frame binary image; Performing fault recognition on the target image using a pre-trained fault recognition model to obtain crack area information; Binarizing the crack region in the target image according to the crack region information to obtain a crack binary image; In response to the crack region in the crack binary image being coincident with the re-frame region in the re-frame binary image, a fault detection result indicating that the re-frame to be detected is broken is obtained; The fault recognition model includes a sequentially connected residual network, a cross-stage partial feature fusion network, and a feature selection anchor-free residual network.

2. The method for detecting a fracture fault of a composite frame according to claim 1, wherein: Fault recognition is performed on the target image using a pre-trained fault recognition model to obtain crack area information, including: Performing feature extraction on the target image through the residual network to obtain an original feature map, wherein the dimension of the original feature map is smaller than a set value; Performing feature fusion on the original feature map through the cross-stage partial feature fusion network to obtain an intermediate feature map; Classify and regress the intermediate feature map using the feature selection anchor-free residual network to obtain category information and corresponding position information; The crack region information is determined in the target image according to the category information and the position information.

3. The method for detecting a fracture fault of a composite frame according to claim 2, wherein: The intermediate feature map is classified and regressed by the residual network without anchor boxes through the feature selection to obtain category information and corresponding position information, including: The intermediate feature map is classified and regressed by the residual network without anchor boxes through the feature selection to obtain category information and corresponding position information, wherein the category information includes cracks, water flow, chalk, shadows and foreign objects.

4. The method for detecting fracture failure of a composite frame according to claim 1, wherein: Before performing fault recognition on the target image using a pre-trained fault recognition model to obtain crack area information, the method further includes: Obtain a training sample set; Performing fault recognition on the images in the training sample set using an initial fault recognition model to obtain a recognition result; Generating a loss function value of the fault recognition model according to the training sample set and the recognition result; Adjusting the fault identification model based on the loss function value to obtain the trained fault identification model; The loss function value generation formula includes: ； ； in, represents the loss function value; Indicates the total number of pixels in all valid frames; Represents the level index in the feature pyramid; Indicates location coordinates; Represents the feature pyramid The effective bounding box of the layer, represents the bounding box, Indicates validity; represents the intersection-over-union ratio of the predicted bounding box and the true bounding box; Represents the distance between the center point of the predicted bounding box and the center point of the true bounding box; Represents the diagonal length of the minimum bounding rectangle containing the predicted bounding box and the true bounding box; represents the width of the ground-truth bounding box, Indicates the height of the ground-truth bounding box; represents the width of the predicted bounding box, Represents the height of the predicted bounding box; Indicates the set value.

5. The method for detecting a fracture fault of a composite frame according to claim 4, characterized in that: Get the training sample set, including: Acquire a train image obtained by photographing a training vehicle; Marking and region-drawing the complex frame components in the train image, and extracting a first image of the known complex frame in the train image; Based on the first image, generating a label file of a known complex frame, the label file including an image name, a detection category, and coordinates of a complex frame area; Determine the fault information of known complex structures; performing data enhancement and contrast enhancement processing on the first image to obtain a second image; The second image, the marked file and the fault information are used as a training sample set.

6. The method for detecting fracture failure of a composite frame according to claim 4, characterized in that: Before performing fault recognition on the images in the training sample set using the initial fault recognition model, the method further includes: Get the model parameters of the visual object recognition dataset; The network parameters of the initial fault identification model are initialized based on the model parameters.

7. The method for detecting a fracture fault of a composite frame according to claim 1, wherein: Segmenting the complex frame region in the target image to obtain a complex frame binary image includes: Segmenting the complex frame region in the target image through a semantic segmentation network to obtain a complex frame binary image, wherein the pixel value of the complex frame region is 1, and the pixel values of other regions are 0; Binarizing the crack region in the target image according to the crack region information to obtain a crack binary image includes: Through the semantic segmentation network, the crack area in the target image is binarized according to the crack area information to obtain a crack binary image, wherein the pixel value of the crack area is 1 and the pixel value of the non-crack area is 0.

8. The method for detecting fracture failure of a composite frame according to claim 7, characterized in that: In response to the crack region in the crack binary image being coincident with the re-frame region in the re-frame binary image, a fault detection result indicating that the re-frame to be detected is broken is obtained, including: Performing AND processing on the complex frame binary image and the crack binary image to generate an image to be processed; Detecting whether there is a pixel region with a value of 1 in the image to be processed; In response to the presence of a pixel region with a value of 1 in the image to be processed, a fault detection result indicating that the complex structure to be detected is broken is obtained.

9. A composite frame breakage fault detection device, characterized in that: include: A first acquisition module is used to acquire a target image of the complex frame to be detected; A first segmentation module is used to segment the complex frame area in the target image to obtain a complex frame binary image; A first recognition module is used to perform fault recognition on the target image using a pre-trained fault recognition model to obtain crack area information; A second segmentation module is used to binarize the crack area in the target image according to the crack area information to obtain a crack binary image; a first detection module configured to obtain a fault detection result indicating that the complex frame to be detected is broken in response to a crack region in the crack binary image overlapping with a complex frame region in the complex frame binary image; The fault recognition model includes a sequentially connected residual network, a cross-stage partial feature fusion network, and a feature selection anchor-free residual network.

10. An electronic device, characterized in that: include: memory for storing computer programs; A processor is configured to implement the steps of the composite frame breakage fault detection method according to any one of claims 1 to 8 when executing the computer program.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the composite frame breakage fault detection method according to any one of claims 1 to 8 are performed.

Images