Surface defect detection method, terminal, and computer-readable storage medium

Abstract

The application provides a surface defect detection method, a terminal and a computer readable storage medium, the surface defect detection method comprises: acquiring a current video frame, the current video frame containing a target object; the target object comprising a rope-shaped object; performing surface detection on the target object to obtain surface information of the target object; performing semantic segmentation on the current video frame to obtain width detection information of the target object; and determining whether the target object in the current video frame has a defect based on the surface information and the width detection information corresponding to the target object. The application performs surface detection on the target object in the current video frame to obtain the surface information of the target object, performs semantic segmentation on the current video frame to determine the width detection information of each target object, determines whether the target object has a defect according to the surface information and the width detection information of the target object, and thus improves the detection accuracy of the target object.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a surface defect detection method, terminal, and computer-readable storage medium. Background Technology

[0002] In the elevator industry, traction steel cables are an important traction tool, and the condition of the elevator's steel cables has a significant impact on the elevator's operation.

[0003] During operation, steel wire ropes are subjected to various stresses, leading to various dangerous situations. For example, under continuous tension, the diameter of the steel wire rope will become smaller and smaller; if the steel wire rope is exposed to air for a long time, moisture will cause an oxidation reaction on the surface of the steel wire rope, or sludge will form on the surface; and internal wires of the traction steel wire rope may break. Therefore, the task of regularly inspecting elevator traction steel wire ropes is particularly important. Summary of the Invention

[0004] The main technical problem solved by this invention is to provide a surface defect detection method, terminal and computer-readable storage medium, thereby solving the problem of low detection accuracy of steel wire ropes in traction elevators in the prior art.

[0005] To solve the above-mentioned technical problems, the first technical solution adopted by the present invention is: to provide a surface defect detection method, the surface defect detection method comprising:

[0006] Get the current video frame, which contains the target object; the target object includes a rope-like object.

[0007] Perform surface detection on the target object to obtain its surface information;

[0008] Perform semantic segmentation on the current video frame to obtain the width detection information of the target object;

[0009] Based on the surface information and width detection information corresponding to the target object, determine whether there are defects in the target object in the current video frame.

[0010] Among them, the surface information includes at least one of the following: sludge detection information, broken wire detection information, rust detection information, and wear detection information;

[0011] Surface inspection is performed on the target object to obtain its surface information, including:

[0012] The target detection network performs surface detection on the target object in the current video frame, and obtains the target object's detection box, detection confidence, and surface information; the target detection network is a YOLOX network structure.

[0013] This includes performing semantic segmentation on the current video frame to obtain the width detection information of the target object, including:

[0014] The current video frame is semantically segmented using a semantic segmentation network to obtain the mask image of the current video frame;

[0015] Based on the mask image, determine the width detection information of the target object in the current video frame.

[0016] The semantic segmentation network includes a block segmentation module, at least two feature extraction modules, and a pooling layer. The block segmentation module and the at least two feature extraction modules are cascaded in sequence.

[0017] The current video frame is semantically segmented using a semantic segmentation network to obtain a mask image of the current video frame, including:

[0018] The current video frame is divided into blocks by the block segmentation module to obtain the region blocks corresponding to the current video frame.

[0019] The feature map is obtained by sequentially extracting features from the feature extraction modules in at least two feature extraction modules and then fusing the features.

[0020] The feature map is pooled using a pooling layer to obtain the corrected feature map;

[0021] The corrected feature map is upsampled and interpolated to obtain the mask image of the current video frame.

[0022] Among them, at least two feature extraction modules include a first feature extraction module and a second feature extraction module;

[0023] Feature maps are obtained by sequentially extracting features from and fusing features from at least two feature extraction modules on the region block, including:

[0024] The first feature map is obtained by extracting features from the region block using the first feature extraction module.

[0025] The second feature map is obtained by extracting features from the first feature map using the second feature extraction module.

[0026] The second feature map is upsampled and fused with the first feature map to obtain a new feature map.

[0027] Among them, based on the mask image, the width detection information of the target object in the current video frame is determined, including:

[0028] Adjust the target object in the mask image to the reference orientation;

[0029] Based on the width information of the target object at each height position in the reference direction, determine the width histogram corresponding to the target object;

[0030] Based on the width histogram, calculate the width information corresponding to each preset step size of the target object;

[0031] Based on the width information corresponding to each preset step size, the width detection information of the target object is determined.

[0032] Specifically, based on the surface information and width detection information corresponding to the target object, it is determined whether the target object in the current video frame has defects, including:

[0033] Based on the surface information and width detection information of the target object, the detection category of the target object is determined; the detection category includes at least one of normal, sludge, broken wire, rust, wear and thinning.

[0034] If the detection category of the target object is any one of sludge, broken wire, rust, wear, and thinning, then it is determined that there is a defect on the surface of the target object in the current video frame.

[0035] The current video frame includes at least two target objects;

[0036] Surface defect detection methods also include:

[0037] The Sinkhorn algorithm is used to associate the same target object in the current video frame with the same target object in previous historical video frames.

[0038] When the number of video frames showing defects on the surface of the target object reaches a preset number, an alarm is issued and the detection information of the target object is displayed.

[0039] To solve the above-mentioned technical problems, the second technical solution adopted by the present invention is to provide a terminal, which includes a memory, a processor, and a computer program stored in the memory and running on the processor. The processor is used to execute program data to implement the steps in the above-mentioned surface defect detection method.

[0040] To solve the above-mentioned technical problems, the third technical solution adopted by the present invention is to provide a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps in the above-mentioned surface defect detection method.

[0041] The beneficial effects of this invention are as follows: Unlike existing technologies, this invention provides a surface defect detection method, terminal, and computer-readable storage medium. The surface defect detection method includes acquiring a current video frame, where the current video frame contains a target object; the target object includes a rope-like object; performing surface detection on the target object to obtain surface information of the target object; performing semantic segmentation on the current video frame to obtain width detection information of the target object; and determining whether a defect exists in the target object in the current video frame based on the surface information and width detection information corresponding to the target object. This application improves the detection accuracy of the target object by performing surface detection on the target object in the current video frame to obtain surface information of the target object, performing semantic segmentation on the current video frame to determine the width detection information of each target object, and determining whether a defect exists in the target object based on the surface information and width detection information. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a schematic flowchart of the surface defect detection method provided by the present invention;

[0044] Figure 2 This is a schematic flowchart of an embodiment of the surface defect detection method provided by the present invention;

[0045] Figure 3 yes Figure 2 A flowchart illustrating a specific embodiment of step S21 in the provided surface defect detection method;

[0046] Figure 4 This is a schematic diagram of a specific embodiment of the target detection network provided by the present invention;

[0047] Figure 5 yes Figure 2 A flowchart illustrating a specific embodiment of step S22 in the provided surface defect detection method;

[0048] Figure 6 This is a schematic diagram of the structure of a specific embodiment of the semantic segmentation network provided by the present invention;

[0049] Figure 7 This is a structural schematic diagram of a specific embodiment of the current video frame provided by the present invention;

[0050] Figure 8This is a flowchart illustrating a specific embodiment of the surface defect detection method provided by the present invention;

[0051] Figure 9 (a) is an image of a specific embodiment of the present invention that includes a target object;

[0052] Figure 9 (b) is Figure 9 The mask image corresponding to the image in (a);

[0053] Figure 9 (c) is an image of another specific embodiment of the present invention that includes a target object;

[0054] Figure 9 (d) is Figure 9 The mask image corresponding to the image in (c);

[0055] Figure 10 This is a schematic diagram of the detection results for each target object in a specific embodiment of the current video frame;

[0056] Figure 11 This is a schematic diagram of the framework of an embodiment of the surface defect detection device provided by the present invention;

[0057] Figure 12 This is a schematic diagram of the framework of an embodiment of the terminal provided by the present invention;

[0058] Figure 13 A schematic diagram of a framework of an embodiment of a computer-readable storage medium provided by the present invention. Detailed Implementation

[0059] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0060] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0061] In this article, the term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "more" in this article means two or more objects.

[0062] To enable those skilled in the art to better understand the technical solution of the present invention, the following describes in further detail a surface defect detection method provided by the present invention in conjunction with the accompanying drawings and specific embodiments.

[0063] Please see Figure 1 , Figure 1 This is a schematic flowchart of the surface defect detection method provided by the present invention. This embodiment provides a surface defect detection method, which includes the following steps.

[0064] S11: Get the current video frame.

[0065] Specifically, an image acquisition device is used to capture an image of the target object, resulting in the current video frame. The current video frame contains the target object, which may include rope-like objects. For example, the target object could be a steel wire rope, nylon rope, polyester rope, etc.

[0066] S12: Perform surface detection on the target object to obtain the surface information of the target object.

[0067] Specifically, the surface information includes at least one of the following: sludge detection information, wire breakage detection information, rust detection information, and wear detection information.

[0068] In one embodiment, a target detection network is used to perform surface detection on target objects in the current video frame, obtaining the target object's detection bounding box, detection confidence score, and surface information; wherein the target detection network is a YOLOX network structure; wherein the detection categories include normal, sludge, broken wire, rust, wear, and thinning.

[0069] S13: Perform semantic segmentation on the current video frame to obtain the width detection information of the target object.

[0070] Specifically, the current video frame is semantically segmented using a semantic segmentation network to obtain a mask image of the current video frame; based on the mask image, the width detection information of the target object in the current video frame is determined.

[0071] In one specific embodiment, the semantic segmentation network includes a block segmentation module, at least two feature extraction modules, and a pooling layer, with the block segmentation module and the at least two feature extraction modules cascaded sequentially. The block segmentation module performs a block segmentation operation on the current video frame to obtain the corresponding region blocks. The feature extraction modules from the at least two feature extraction modules sequentially extract features from the region blocks and perform feature fusion to obtain a feature map. The pooling layer performs a pooling operation on the feature map to obtain a corrected feature map. Based on the corrected feature map, a mask image of the current video frame is obtained.

[0072] In one embodiment, at least two feature extraction modules include a first feature extraction module and a second feature extraction module. The first feature extraction module extracts features from a region block to obtain a first feature map. The second feature extraction module extracts features from the first feature map to obtain a second feature map. The second feature map is upsampled and fused with the first feature map to obtain a feature map. The corrected feature map is then upsampled and interpolated to obtain a mask image for the current video frame.

[0073] In one embodiment, the target object in the mask image is adjusted to a reference direction; based on the width information of the target object at each height position in the reference direction, the width histogram corresponding to the target object is determined; based on the width histogram of the target object, the width detection information of the target object is determined.

[0074] In one specific embodiment, the width information corresponding to each preset step size of the target object is calculated based on the width histogram; and the width detection information of the target object is determined based on the width information corresponding to each preset step size.

[0075] In one embodiment, the width information corresponding to each preset step size is sorted, and the width information in the middle position is selected as the image width of the target object; based on the image width of the target object and the preset ratio parameter, the width detection information of the target object is determined.

[0076] S14: Based on the surface information and width detection information corresponding to the target object, determine whether there is a defect in the target object in the current video frame.

[0077] Specifically, based on the surface information and width detection information of the target object, the detection category of the target object is determined; the detection category includes at least one of normal, sludge, broken wire, rust, wear, and thinning; in response to the target object's detection category being any one of sludge, broken wire, rust, wear, and thinning, it is determined that there is a defect on the surface of the target object in the current video frame.

[0078] In one embodiment, in response to the target object's width detection information not conforming to a preset width, the target object's detection category is determined to be thinner; in response to the target object's sludge detection information exceeding a sludge threshold, the target object's detection category is determined to be sludge; in response to the target object's wire breakage detection information exceeding a wire breakage threshold, the target object's detection category is determined to be wire breakage; in response to the target object's corrosion detection information exceeding a corrosion threshold, the target object's detection category is determined to be corrosion; and in response to the target object's wear detection information exceeding a wear threshold, the target object's detection category is determined to be wear.

[0079] In one embodiment, the current video frame includes at least two target objects. The same target object in the current video frame is associated with historical video frames preceding the current video frame using the Sinkhorn algorithm. When the number of video frames containing surface defects of the target object reaches a preset number, an alarm is issued and the detection information of the target object is displayed.

[0080] The surface defect detection method provided in this embodiment includes acquiring a current video frame, which contains a target object; the target object includes a rope-like object; performing surface detection on the target object to obtain surface information of the target object; performing semantic segmentation on the current video frame to obtain width detection information of the target object; and determining whether there is a defect in the target object in the current video frame based on the surface information and width detection information corresponding to the target object. This application improves the detection accuracy of the target object by performing surface detection on the target object in the current video frame to obtain surface information, performing semantic segmentation on the current video frame to determine the width detection information of each target object, and determining whether there is a defect in the target object based on the surface information and width detection information.

[0081] Please see Figure 2 , Figure 2 This is a schematic flowchart of an embodiment of the surface defect detection method provided by the present invention. This embodiment provides a surface defect detection method, which includes the following steps.

[0082] S21: Train the object detection network.

[0083] Please see Figure 3 and Figure 4 , Figure 3 yes Figure 2 A flowchart illustrating a specific embodiment of step S21 in the provided surface defect detection method; Figure 4 This is a schematic diagram of a specific embodiment of the target detection network provided by the present invention.

[0084] Specifically, to improve the convergence speed and accuracy of the target detection network, the target detection network can be a YOLOX network structure. In this embodiment, the target detection network includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, a seventh convolutional layer, an eighth convolutional layer, and a ninth convolutional layer. The first convolutional layer is connected to the second convolutional layer, the second convolutional layer is connected to the third and fourth convolutional layers respectively, the third convolutional layer is cascaded with the fifth and sixth convolutional layers in sequence, the fourth convolutional layer is connected to the seventh convolutional layer, and the seventh convolutional layer is connected to the eighth and ninth convolutional layers respectively.

[0085] S211: Obtain multiple first sample images containing the target.

[0086] Specifically, the first sample image contains targets, and the first sample image is associated with the bounding boxes and surface information of each target. Since the bounding boxes are manually labeled, the confidence level of the label corresponding to the target is 1. In this embodiment, the target can be a steel wire rope. Each first sample image can contain multiple targets, and each target has an independent bounding box and label category. The surface information can be categorized into five types: normal, sludge, broken wire, rust, and wear.

[0087] In this embodiment, the first sample image can be augmented using the Mosaic method and the MixUp method.

[0088] In one embodiment, different positive and negative sample mechanisms are used, specifically, the number and type of the first sample images can be determined based on SimOTA. This can further reduce training time, avoid the introduction of additional parameters, and improve the detection accuracy of the object detection network.

[0089] S212: Detect the target in the first sample image using an object detection network to obtain the target's bounding box, prediction confidence, and prediction surface information.

[0090] Specifically, the first sample image is input into the object detection network. The first convolutional layer in the object detection network extracts features from the first sample image to obtain a first sample feature map. The second convolutional layer extracts features from the first sample feature map to obtain a second sample feature map. The third and fourth convolutional layers extract features from the second sample feature map, respectively. The third convolutional layer produces a third sample feature map, and the fourth convolutional layer produces a fourth sample feature map. The convolutional algorithms in the third and fourth convolutional layers are different. Specifically, the third convolutional layer extracts the category features of the target, while the fourth convolutional layer extracts the location features of the target. The fifth and sixth convolutional layers sequentially extract features from the third sample feature map to obtain the predicted surface information of each target in the first sample image. The seventh convolutional layer extracts features from the fourth sample feature map to obtain the fifth sample feature map. The eighth and ninth convolutional layers extract features from the fifth sample feature map, so that the eighth convolutional layer obtains the prediction boxes corresponding to each target in the first sample image based on the fifth sample feature map, and the ninth convolutional layer obtains the prediction confidence scores corresponding to each target in the first sample image based on the fifth sample feature map. The first, third, fourth, fifth, and seventh convolutional layers all use 3x3 convolutional kernels; the second, sixth, eighth, and ninth convolutional layers all use 1x1 convolutional kernels.

[0091] S213: Iteratively train the target detection network based on the sum of the error values ​​between the predicted bounding box and the labeled bounding box corresponding to the same target in the first sample image, the error values ​​between the prediction confidence and the labeled confidence, and the error values ​​between the predicted surface information and the labeled surface information.

[0092] Specifically, the first error value between the predicted bounding box and the labeled bounding box corresponding to the same target in the first sample image, the second error value between the prediction confidence and 1 corresponding to the same target, and the third error value between the predicted surface information and the labeled surface information corresponding to the same target are calculated. The first error value, the second error value, and the third error value are summed, and the target detection network is iteratively trained based on the summed error value.

[0093] In an optional embodiment, the results of the target detection network are backpropagated, and the weights of the target detection network are corrected based on the sum of the error values ​​between the predicted bounding box and the labeled bounding box corresponding to the same target in the first sample image, the error value between the prediction confidence and 1, and the error value between the predicted surface information and the labeled surface information, thereby achieving the training of the target detection network.

[0094] The first sample image is input into the object detection network, which detects each object in the first sample image. Training of the object detection network stops when the sum of the error values ​​between the predicted bounding box and the labeled bounding box corresponding to the same object, the error value between the prediction confidence and 1, and the error value between the predicted surface information and the labeled surface information is less than a preset threshold. The preset threshold can be set manually, for example, 1% or 5%.

[0095] S22: Train the semantic segmentation network.

[0096] Please see Figure 5 and Figure 6 , Figure 5 yes Figure 2 A flowchart illustrating a specific embodiment of step S22 in the provided surface defect detection method; Figure 6 This is a schematic diagram of a specific embodiment of the semantic segmentation network provided by the present invention.

[0097] Specifically, since the trained semantic segmentation network needs to process images of the steel cables of a traction elevator, and the scene where the steel cables are located is complex with poor lighting, and the images are constantly affected by dim lighting, this embodiment uses a transformer-like encoding / decoding structure. The backbone network uses the attention structure from the Swing Transformer to progressively downsample to 1 / 32 of the original image resolution; then, it integrates the feature maps from all modules and upsamples to 1 / 4 of the original resolution of the second sample image in one go. The semantic segmentation network includes a block module, at least two feature extraction modules, and a pooling layer, with the block module and the at least two feature extraction modules cascaded sequentially.

[0098] In this embodiment, the semantic segmentation network includes four cascaded feature extraction modules. The four feature extraction modules can have identical structures, each including a cascaded feature extraction layer and an attention submodule.

[0099] S221: Obtain multiple second sample images containing the target.

[0100] Specifically, the second sample image contains a target and is associated with a real mask image. In this embodiment, the target can be a steel wire rope. Furthermore, each second sample image can contain multiple targets.

[0101] S222: Semantically segment the second sample image using a semantic segmentation network to obtain the predicted mask image of the second sample image.

[0102] Specifically, the second sample image is input into the semantic segmentation network. The segmentation module in the semantic segmentation network segments the second sample image into blocks to obtain the corresponding region blocks. The feature extraction modules in the four feature extraction modules extract features from the region blocks in sequence and perform feature fusion to obtain a feature map.

[0103] Specifically, the four feature extraction modules include a first feature extraction module, a second feature extraction module, a third feature extraction module, and a fourth feature extraction module. The first feature extraction module extracts features from the region block to obtain a first feature map; the second feature extraction module extracts features from the first feature map and performs downsampling to obtain a second feature map; the third feature extraction module extracts features from the second feature map and performs downsampling to obtain a third feature map; and the fourth feature extraction module extracts features from the third feature map and performs downsampling to obtain a fourth feature map. By extracting features from the first, second, third, and fourth feature maps respectively, corresponding first, second, third, and fourth updated feature maps are obtained. The second updated feature map is upsampled and fused with the first updated feature map to obtain a first fused feature map; the third updated feature map is upsampled and fused with the first fused feature map to obtain a second fused feature map; and the fourth updated feature map is upsampled and fused with the second fused feature map to obtain a third fused feature map. Feature extraction is then performed on the third fused feature map to obtain a feature map.

[0104] A corrected feature map is obtained by pooling the feature map using a pooling layer. Specifically, pooling is performed on the feature map to fill holes in the mask image corresponding to the second sample image, thereby effectively eliminating holes and obtaining the corrected feature map. Holes of different sizes are filled by adjusting the pooling parameters of the pooling layer. In other words, the holes in the corrected feature map have been filled. Upsampling the corrected feature map can further eliminate holes more effectively.

[0105] Based on the corrected feature map, a predicted mask image for the second sample image is obtained. Specifically, the corrected feature map is convolved to obtain a corrected feature map, and then interpolated to obtain the predicted mask image. The predicted mask image can have the same size and resolution as the second sample image.

[0106] S223: Iteratively train the semantic segmentation network based on the error value between the real mask image and the predicted mask image corresponding to the second sample image.

[0107] Specifically, the error value between the ground truth mask image and the predicted mask image corresponding to the second sample image is calculated, and the semantic segmentation network is iteratively trained based on this error value. In one embodiment, the error value between the ground truth mask image and the predicted mask image corresponding to the second sample image is calculated using the CE loss function.

[0108] In one optional embodiment, the results of the semantic segmentation network are backpropagated, and the weights of the semantic segmentation network are corrected based on the error value between the real mask image and the predicted mask image corresponding to the second sample image, thereby training the semantic segmentation network.

[0109] The second sample image is input into the semantic segmentation network, which performs semantic segmentation on the second sample image. Training of the semantic segmentation network stops when the error between the ground truth mask image and the predicted mask image corresponding to the second sample image is less than a preset threshold. The preset threshold can be set by the user, such as 1% or 5%.

[0110] Please see Figure 7 and 8 , Figure 7 This is a structural schematic diagram of a specific embodiment of the current video frame provided by the present invention; Figure 8 This is a flowchart illustrating a specific embodiment of the surface defect detection method provided by the present invention.

[0111] In one embodiment, to determine the true size of the target object, it is necessary to first establish a scale between the target object in the image and the actual target object. Specifically, a checkerboard calibration plate for calibrating the camera is used as the standard scale, with each black square having a true size of 50 millimeters, and each black square having approximately 50 pixels in an 8MP camera. The entire checkerboard has 10 black and white squares horizontally and 8 black and white squares vertically. The checkerboard calibration plate and the steel wire rope are placed at the same depth, parallel to the vertical direction of the target object, and close to the target object.

[0112] S23: Get the current video frame.

[0113] Specifically, an image acquisition device is used to capture an image of the target object, resulting in the current video frame. The current video frame contains the target object, which may include rope-like objects. For example, the target object could be a steel wire rope, nylon rope, polyester rope, or other metal rope.

[0114] In this embodiment, the steel wire rope is a steel wire rope. When the steel wire rope is stationary, the image acquisition device captures images of the steel wire rope of the traction elevator. When the steel wire rope is in motion, the image acquisition device pauses image acquisition of the steel wire rope of the traction elevator.

[0115] An image acquisition device captures an image of the target object, resulting in a current video frame. The current video frame contains both the target object and the checkerboard calibration board. In one embodiment, the current video frame includes at least two target objects. In this embodiment, the image acquisition device is a monocular camera.

[0116] S24: Perform surface detection on the target objects in the current video frame using the target detection network to obtain the detection bounding box, detection confidence, and surface information of the target objects.

[0117] Specifically, the target detection network trained in step S21 above performs target object detection on the current video frame, obtaining the detection bounding box, detection confidence score, and surface information of each target object contained in the current video frame. Specifically, the surface information includes at least one of the following: sludge detection information, broken wire detection information, rust detection information, and wear detection information.

[0118] The method described in this embodiment can simultaneously and directly detect five defects on the surface of the wire rope: sludge, broken wires, rust, wear, and thinning.

[0119] S25: Perform semantic segmentation on the current video frame using a semantic segmentation network to obtain the mask image of the current video frame.

[0120] Please see Figure 9 (a), 9(b), 9(c) and 9(d), Figure 9 (a) is an image of a specific embodiment of the present invention that includes a target object; Figure 9 (b) is Figure 9 The mask image corresponding to the image in (a); Figure 9 (c) is an image of another specific embodiment of the present invention that includes a target object; Figure 9 (d) is Figure 9 (c) is the mask image corresponding to the image.

[0121] Specifically, the semantic segmentation network trained in step S22 above performs semantic segmentation on the target object in the current video frame to obtain the mask image corresponding to the current video frame.

[0122] In one embodiment, the current video frame is segmented into blocks by a segmentation module in a semantic segmentation network to obtain region blocks corresponding to the current video frame. Feature extraction modules from at least two feature extraction modules sequentially extract features from and fuse these region blocks to obtain feature maps. Pooling layers are then applied to the feature maps to obtain corrected feature maps. Based on the corrected feature maps, a mask image of the current video frame is obtained.

[0123] In one embodiment, at least two feature extraction modules include a first feature extraction module and a second feature extraction module. The first feature extraction module extracts features from the region block to obtain a first feature map; the second feature extraction module extracts features from the first feature map to obtain a second feature map; the second feature map is upsampled and fused with the first feature map to obtain a final feature map.

[0124] Specifically, the four feature extraction modules include a first feature extraction module, a second feature extraction module, a third feature extraction module, and a fourth feature extraction module. The first feature extraction module extracts features from the region block corresponding to the current video frame to obtain a first feature map; the second feature extraction module extracts features from the first feature map and performs downsampling to obtain a second feature map; the third feature extraction module extracts features from the second feature map and performs downsampling to obtain a third feature map; and the fourth feature extraction module extracts features from the third feature map and performs downsampling to obtain a fourth feature map. By extracting features from the first, second, third, and fourth feature maps respectively, corresponding first, second, third, and fourth updated feature maps are obtained. The second updated feature map is upsampled and fused with the first updated feature map to obtain a first fused feature map; the third updated feature map is upsampled and fused with the first fused feature map to obtain a second fused feature map; and the fourth updated feature map is upsampled and fused with the second fused feature map to obtain a third fused feature map. Feature extraction is then performed on the third fused feature map to obtain a feature map.

[0125] A corrected feature map is obtained by pooling the feature map using a pooling layer. Specifically, pooling is performed on the feature map to fill holes in the mask image corresponding to the current video frame, thereby effectively eliminating holes and obtaining the corrected feature map. Different sized holes are filled by adjusting the pooling parameters of the pooling layer. In other words, the holes in the corrected feature map have been filled. Upsampling the corrected feature map can further eliminate holes more effectively.

[0126] Based on the corrected feature map, a mask image for the current video frame is obtained. Specifically, the corrected feature map is convolved to obtain a corrected feature map, and then interpolated to obtain a mask image. The mask image can have the same size and resolution as the current video frame.

[0127] The corrected feature map is upsampled and interpolated to obtain the mask image of the current video frame.

[0128] S26: Based on the mask image, determine the width detection information of the target object in the current video frame.

[0129] Specifically, to facilitate obtaining the width detection information of the target object, the target object in the mask image is adjusted to a reference direction. In one embodiment, the orientation angle between the target object and the vertical direction is extracted using a minimum bounding rectangle. The target object is then rotated to the vertical direction based on the orientation angle.

[0130] Specifically, based on the width information of the target object at various height positions in the vertical direction, a width histogram corresponding to the target object is determined; based on the width histogram of the target object, width detection information of the target object is determined. In one embodiment, the diameter information of the target object at various height positions in the vertical direction is measured as the width information of the target object at each height position, and a width histogram is created.

[0131] Specifically, based on the width histogram, the width information corresponding to each preset step length of the target object is calculated. In one embodiment, to make the data more representative, a preset step length is set, and the average width of the target object in the width histogram corresponding to each preset step length in the vertical direction is calculated. The average width information is used as the width information corresponding to each preset step length.

[0132] Specifically, the width detection information of the target object is determined based on the width information corresponding to each preset step size. In one specific embodiment, the width information corresponding to each preset step size is sorted, and the width information in the middle position is selected as the image width of the target object; the width detection information of the target object is determined based on the image width of the target object and a preset ratio parameter. That is, the width detection information of the target object is determined based on the actual size of the black squares in the checkerboard calibration board in the image, the size of the black squares in the mask image, and the image width of the target object.

[0133] The method in this embodiment can quickly and efficiently determine the width detection information of the target object, and does not require a specific camera to capture images of the target object, thus saving costs.

[0134] S27: Based on the surface information and width detection information corresponding to the target object, determine whether there is a defect in the target object in the current video frame.

[0135] Figure 10 This is a schematic diagram of the detection results for each target object in a specific embodiment of the current video frame.

[0136] Specifically, based on the surface information and width detection information of the target object, the detection category of the target object is determined; the detection category includes at least one of normal, sludge, broken wire, rust, wear, and thinning; in response to the target object's detection category being any one of sludge, broken wire, rust, wear, and thinning, it is determined that the surface of the target object in the current video frame has a defect. In response to the target object's detection category being normal, it is determined that the surface of the target object in the current video frame does not have a defect.

[0137] In one embodiment, in response to the target object's width detection information not conforming to a preset width, the target object's detection category is determined to be thinner; in response to the target object's sludge detection information exceeding a sludge threshold, the target object's detection category is determined to be sludge; in response to the target object's wire breakage detection information exceeding a wire breakage threshold, the target object's detection category is determined to be wire breakage; in response to the target object's corrosion detection information exceeding a corrosion threshold, the target object's detection category is determined to be corrosion; and in response to the target object's wear detection information exceeding a wear threshold, the target object's detection category is determined to be wear.

[0138] In this embodiment, if there are multiple target objects in the current video frame, it is necessary to associate each detected target object with the target objects detected in historical video frames.

[0139] S28: Based on the Sinkhorn algorithm, associate the same target object in the current video frame with the same target object in the previous historical video frames.

[0140] Specifically, assume that the information on N steel wire ropes detected in historical video frames is represented as a set r of size n; and the information on M steel wire ropes detected in the current video frame is represented as a set c of size m; the information on M steel wire ropes needs to be matched with the information on N steel wire ropes. Therefore, all allocation schemes are defined as shown in Formula 1 below.

[0141]

[0142] The cost function of each steel wire rope corresponding to the detected steel wire rope is stored in matrix H. The matrix H is constructed according to the position information and the target box size information, as shown in Formula 2 below.

[0143] H=α*Δdist+(1-α)*Δarea (Formula 2)

[0144] In the formula: Δdist represents the distance between the centers of the detection boxes of each target object, and Δarea represents the difference in the area of ​​the detection boxes of each target object.

[0145] The final problem can be expressed as shown in Formula 3 below.

[0146]

[0147] The optimal solution is shown in Formula 4 below.

[0148]

[0149] Formula 4 above can be used to calculate the probability of each target object in the historical video frame matching each target object in the current video frame. Associating the two target objects with the highest probability values ​​indicates that the target object in the current video frame corresponding to the highest probability value is the same target object as the target object in the historical video frame.

[0150] The above method can efficiently match target objects in the current video frame with target objects in historical video frames. This ensures that each target object in the current video frame has continuous status information on the timeline. Whether to issue an alarm can be determined based on the status of the target object within a preset time period.

[0151] S29: In response to the number of video frames showing defects on the surface of the target object reaching a preset number, an alarm is issued and the detection information of the target object is displayed.

[0152] Specifically, if the number of video frames showing surface defects on the target object reaches a preset number within a preset time period, a defect is determined to exist in the target object, an alarm is issued, and the detection information of the target object is displayed. The displayed detection information includes the target object's detection frame and the detection type. The displayed detection type can be any one of the following: sludge, broken wire, rust, wear, and thinning.

[0153] The surface defect detection method provided in this embodiment acquires a current video frame containing a target object, including a rope-like object; performs surface detection on the target object to obtain its surface information; performs semantic segmentation on the current video frame to obtain the target object's width detection information; and determines whether the target object in the current video frame has a defect based on the surface information and width detection information. This application improves the detection accuracy of the target object by performing surface detection on the target object in the current video frame, determining the width detection information of each target object through semantic segmentation, and determining whether the target object has a defect based on the surface information and width detection information.

[0154] Please see Figure 11 , Figure 11 This is a schematic diagram of the framework of an embodiment of the surface defect detection device provided by the present invention. This embodiment provides a surface defect detection device 60, which includes an acquisition module 61, a first detection module 62, a second detection module 63, and a determination module 64.

[0155] The acquisition module 61 is used to acquire the current video frame, which contains the target object; the target object includes a rope-like object.

[0156] The first detection module 62 is used to perform surface detection on the target object and obtain the surface information of the target object.

[0157] The second detection module 63 is used to perform semantic segmentation on the current video frame to obtain the width detection information of the target object.

[0158] The determination module 64 is used to determine whether there are defects in the target object in the current video frame based on the surface information and width detection information corresponding to the target object.

[0159] The surface defect detection device provided in this embodiment obtains the surface information of the target object by performing surface detection on the target object in the current video frame, determines the width detection information of each target object by performing semantic segmentation on the current video frame, and determines whether the target object has defects based on the surface information and width detection information, thereby improving the detection accuracy of the target object.

[0160] Please see Figure 12 , Figure 12 This is a schematic diagram of a terminal embodiment provided by the present invention. The terminal 80 includes a memory 81 and a processor 82 coupled to each other. The processor 82 is used to execute program instructions stored in the memory 81 to implement the steps of any of the above-described surface defect detection method embodiments. In a specific implementation scenario, the terminal 80 may include, but is not limited to, a microcomputer or a server. In addition, the terminal 80 may also include mobile devices such as laptops and tablets, which are not limited here.

[0161] Specifically, processor 82 controls itself and memory 81 to implement the steps of any of the above-described surface defect detection method embodiments. Processor 82 can also be referred to as a CPU (Central Processing Unit). Processor 82 may be an integrated circuit chip with signal processing capabilities. Processor 82 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 82 can be implemented using integrated circuit chips.

[0162] Please see Figure 13 , Figure 13This is a schematic diagram of a framework of an embodiment of a computer-readable storage medium provided by the present invention. The computer-readable storage medium 90 stores program instructions 901 that can be executed by a processor. The program instructions 901 are used to implement the steps of any of the above-described embodiments of the surface defect detection method.

[0163] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0164] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0165] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0166] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0167] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0168] The above are merely embodiments of the present invention and do not limit the scope of patent protection of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for detecting surface defects, characterized in that, The surface defect detection method includes: Obtain the current video frame, wherein the current video frame contains the target object; the target object includes a rope-like object. Surface detection is performed on the target object to obtain its surface information; Perform semantic segmentation on the current video frame to obtain the width detection information of the target object; Based on the surface information and width detection information corresponding to the target object, it is determined whether the target object in the current video frame has defects; The step of performing semantic segmentation on the current video frame to obtain the width detection information of the target object includes: The current video frame is semantically segmented using a semantic segmentation network to obtain a mask image of the current video frame; the semantic segmentation network includes a block module, at least two feature extraction modules, and a pooling layer, wherein the block module and the at least two feature extraction modules are cascaded in sequence. Based on the mask image, determine the width detection information of the target object in the current video frame; The step of determining the width detection information of the target object in the current video frame based on the mask image includes: Adjust the target object in the mask image to the reference orientation; Based on the width information of the target object at each height position in the reference direction, determine the width histogram corresponding to the target object; Based on the width histogram, calculate the width information corresponding to each preset step size of the target object; Based on the width information corresponding to each preset step size, the width detection information of the target object is determined; The step of performing semantic segmentation on the current video frame using a semantic segmentation network to obtain the mask image of the current video frame includes: The current video frame is divided into blocks by the block-segmentation module to obtain the region blocks corresponding to the current video frame. The feature extraction modules in the at least two feature extraction modules sequentially extract features from the region block and perform feature fusion to obtain a feature map; The feature map is pooled using the pooling layer to obtain the corrected feature map; The corrected feature map is upsampled and interpolated to obtain the mask image of the current video frame.

2. The surface defect detection method according to claim 1, characterized in that, The surface information includes at least one of the following: sludge detection information, broken wire detection information, rust detection information, and wear detection information; The step of performing surface detection on the target object to obtain the surface information of the target object includes: The target object in the current video frame is surface detected by an object detection network to obtain the detection bounding box, detection confidence, and surface information of the target object; wherein, the object detection network is a YOLOX network structure.

3. The surface defect detection method according to claim 1, characterized in that, The at least two feature extraction modules include a first feature extraction module and a second feature extraction module; The step of sequentially extracting features from the region block and fusing features through the feature extraction modules of the at least two feature extraction modules to obtain a feature map includes: The first feature extraction module extracts features from the region block to obtain a first feature map. The second feature map is obtained by extracting features from the first feature map using the second feature extraction module. The second feature map is upsampled and fused with the first feature map to obtain the feature map.

4. The surface defect detection method according to claim 2, characterized in that, The step of determining whether the target object in the current video frame has defects based on the surface information and width detection information corresponding to the target object includes: Based on the surface information of the target object and the width detection information, the detection category of the target object is determined; the detection category includes at least one of normal, sludge, broken wire, rust, wear, and thinning. If the detection category of the target object is any one of sludge, broken wire, rust, wear, and thinning, then it is determined that there is a defect on the surface of the target object in the current video frame.

5. The surface defect detection method according to claim 1, characterized in that, The current video frame includes at least two of the target objects; The surface defect detection method further includes: The Sinkhorn algorithm is used to associate the same target object in the current video frame with the same target object in the previous historical video frames. When the number of video frames showing defects on the surface of the target object reaches a preset number, an alarm is issued and the detection information of the target object is displayed.

6. A terminal, characterized in that, The terminal includes a memory, a processor, and a computer program stored in the memory and running on the processor, the processor being used to execute program data to implement the steps in the surface defect detection method as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the surface defect detection method as described in any one of claims 1 to 5.

Images