Fault detection method, device, electronic device, and storage medium

Abstract

The present disclosure provides a fault detection method, device, electronic device, and storage medium, belonging to the field of image processing technology. The method includes: obtaining an optical image of a target structure at the current moment and extracting a multi-scale feature map of the optical image; determining a preliminary binary segmentation mask map corresponding to the boundary of the target structure contained in the optical image based on the multi-scale feature map and prompt information, wherein the prompt information indicates the boundary information of the target structure contained in the image; and comparing the preliminary binary segmentation mask map with a preset mask map to determine the fault type of the target structure, wherein different preset mask maps correspond to different fault types. The technical solution provided by the embodiments of the present disclosure can quickly identify the fault type of the target structure by comparing the preliminary binary segmentation mask map of the acquired optical image, thereby improving the accuracy of fault identification.

Description

Technical Field

[0001] The present disclosure belongs to the field of image processing technology, and in particular relates to a fault detection method, device, electronic device, and storage medium. Background Art

[0002] With the rapid development of wind power generation, the structural health of key parts of wind turbines, such as wind turbine blades, towers, and nacelles, is of great importance. Therefore, fault detection of wind turbines is a very important technical means.

[0003] In related technologies, fault detection of wind turbine blades, towers, nacelles and other parts of wind power equipment mainly relies on manual visual inspection. This detection method is not only inefficient, but also has high operational risks and low accuracy in determining the fault type. Summary of the Invention

[0004] The embodiments of the present disclosure provide a solution to solve the problems in the related art of low working efficiency, high operational risk, and low accuracy in determining the fault type.

[0005] In a first aspect, the present disclosure provides a fault detection method, the method comprising:

[0006] Acquire an optical image of the target structure at a current moment, and extract a multi-scale feature map of the optical image;

[0007] Determining a preliminary binary segmentation mask corresponding to a boundary of the target structure contained in the optical image based on the multi-scale feature map and the prompt information, wherein the prompt information indicates boundary information of the target structure contained in the image;

[0008] The preliminary binary segmentation mask image and the preset mask image are compared to determine the fault type of the target structure, and different preset mask images correspond to different fault types.

[0009] In a second aspect, the present disclosure provides a fault detection device, the device comprising:

[0010] an acquisition unit, configured to acquire an optical image of the target structure at a current moment and extract a multi-scale feature map of the optical image;

[0011] a determining unit, configured to determine a preliminary binary segmentation mask corresponding to a boundary of the target structure contained in the optical image based on the multi-scale feature map and prompt information, wherein the prompt information indicates boundary information of the target structure contained in the image;

[0012] The comparison unit is used to compare the preliminary binary segmentation mask image with a preset mask image to determine the fault type of the target structure, where different preset mask images correspond to different fault types.

[0013] In a third aspect, the present disclosure provides an electronic device, comprising:

[0014] processor; and

[0015] a memory for storing executable instructions of the processor;

[0016] The processor is configured to execute any method in the first aspect or any possible implementation of the first aspect by executing the executable instructions.

[0017] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any method in the first aspect or any possible implementation of the first aspect.

[0018] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, comprising computer instructions, which, when executed by a processor, implement any method in the first aspect or any possible implementation of the first aspect.

[0019] The technical solution provided by the present disclosure obtains an optical image of the target structure at the current moment and extracts a multi-scale feature map of the optical image; based on the multi-scale feature map and prompt information, determines a preliminary binary segmentation mask map corresponding to the boundary of the target structure contained in the optical image, wherein the prompt information indicates the boundary information of the target structure contained in the image; compares the preliminary binary segmentation mask map with a preset mask map to determine the fault type of the target structure, wherein different preset mask maps correspond to different fault types. The technical solution provided by each embodiment of the present disclosure determines the fault type of the target structure in the optical image by obtaining a preliminary binary segmentation mask map corresponding to the optical image and comparing it with a preset mask map. This detection method can avoid manual visual detection, improve the recognition accuracy of target structure faults, and improve work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0021] Figure 1 A flowchart of a fault detection method provided in one embodiment of the present disclosure;

[0022] Figure 2 A schematic structural diagram of a fault detection device provided in one embodiment of the present disclosure;

[0023] Figure 3 A schematic structural diagram of an electronic device provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] The embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to be used to explain the present disclosure, but should not be understood as limiting the present disclosure.

[0025] The terms "first" and "second" and the like in the specification, claims, and drawings of the embodiments of the present disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the embodiments of the present disclosure described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or apparatus.

[0026] The fault detection method provided in the embodiments of the present disclosure can be run on a terminal device or a server. The terminal device can be a local terminal device, including wearable devices such as VR (Virtual Reality), AR (Augmented Reality), and MR (Mixed Reality). The server can be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0027] With the rapid development of wind power generation, the structural health of key parts of wind turbines, such as wind turbine blades, towers, and nacelles, is of great importance. Therefore, fault detection of wind turbines is a very important technical means.

[0028] In related technologies, fault detection of wind turbine blades, towers, nacelles and other parts of wind power equipment mainly relies on manual visual inspection. This detection method is not only inefficient, but also has high operational risks and low accuracy in determining the fault type.

[0029] Figure 1This is a flowchart of a fault detection method provided by an exemplary embodiment of the present disclosure. The method can be applied to a device with a data processing function and includes at least the following steps S101-S103:

[0030] S101, acquiring an optical image of a target structure at a current moment, and extracting a multi-scale feature map of the optical image.

[0031] In some embodiments, the present application is mainly applied to wind turbine inspection scenarios. Specifically, wind turbines in various sites are inspected using drones.

[0032] In some embodiments, the optical image is output by an optical camera. Specifically, during the drone inspection process, the optical camera on the drone captures the wind turbine and outputs an optical image. The optical image can be an RGB image frame, represented as: ;

[0033] described is the RGB image frame output by the optical camera, is the space to which the RGB image frame belongs, is the height of the optical image, The width of the image.

[0034] In some embodiments, in order to reduce the dependence on the uniqueness of the scene where the target structure is located, the present application introduces the Segment Anything Model (SAM) as the core perception model. The SAM internally builds an encoder based on the Vision Transformer (ViT) architecture, performs global semantic feature extraction based on the acquired optical image of the target structure, and outputs a multi-scale feature map of the optical image.

[0035] S102: Determine a preliminary binary segmentation mask image corresponding to a boundary of the target structure contained in the optical image based on the multi-scale feature map and the prompt information.

[0036] In some embodiments, the hint information indicates boundary information of a target structure contained in the image.

[0037] In some embodiments, according to the core perception model SAM introduced in the above steps, after extracting the multi-scale feature map of the optical image, it can output a preliminary binary segmentation mask map corresponding to the boundary of the target structure based on the prompt information.

[0038] In other embodiments, the prompt information also includes points, lines, and text of target structures in the image, etc. Its main function is to guide the model to segment the image.

[0039] In some embodiments, determining a preliminary binary segmentation mask corresponding to the boundary of the target structure contained in the optical image based on the multi-scale feature map and the prompt information includes steps S11-S12:

[0040] S11, obtaining prompt information of the target structure, and converting the prompt information into a prompt vector.

[0041] S12, based on the prompt vector, segmenting the multi-scale feature map to obtain a boundary feature map of the target structure, and converting the boundary feature map into a preliminary binary segmentation mask map.

[0042] In some embodiments, with respect to the above-mentioned step S11, the prompt information may be obtained manually or automatically generated by a device.

[0043] Furthermore, converting the prompt information into a prompt vector includes: converting the prompt information into a prompt vector by using a preset encoder.

[0044] In some embodiments, in step S12 above, during the actual conversion process, an attention mechanism is used to evaluate the similarity between the region containing the target structure in the optical image and the prompt information, thereby completing the segmentation of the multi-scale feature map, obtaining a boundary feature map of the target structure, and finally converting the boundary feature map into a preliminary binary segmentation mask. The preliminary binary segmentation mask is a two-dimensional matrix of the same size as the corresponding image, with each pixel value being 0 or 1, indicating which pixels in the image belong to the target structure (such as the blades, hub, tower, etc. of the wind turbine) and which belong to the background.

[0045] The binary segmentation mask image includes the area where the target structure in the optical image is located.

[0046] S103 : Compare the preliminary binary segmentation mask image with a preset mask image to determine the fault type of the target structure.

[0047] In some embodiments, different preset mask images correspond to different fault types. For example, the target structure may be a key component of a wind turbine, such as blades, a hub, or a tower. Specifically, when the target structure is a wind turbine, the mask image corresponding to a normal wind turbine is different from the mask image corresponding to a cracked wind turbine blade. Therefore, by comparing the preset mask images, the type of fault present in the target structure can be determined.

[0048] In some embodiments, comparing the preliminary binary segmentation mask image with a preset mask image to determine the fault type of the target structure includes steps S21-S22:

[0049] S21, determining the problem area in the preliminary binary segmentation mask image.

[0050] S22: Determine the fault type of the target structure based on the problem area and the preset mask image.

[0051] In some embodiments, with respect to the above step S21 , the problem area indicates an area where mask errors exist.

[0052] Furthermore, the determining of the problem area in the preliminary binary segmentation mask image includes steps S211-S213:

[0053] S211, extracting low-level visual structure information from the optical image.

[0054] S212 , calculating the geometric overlap between the edge region of the preliminary binary segmentation mask image and the significant edge region, or calculating a consistency index between the gradient direction change trend of the preliminary binary segmentation mask image and the gradient direction change trend extracted from the optical image.

[0055] S213: If the geometric overlap is less than a first preset value or the consistency index does not meet the standard, the edge area where the geometric overlap is less than the first preset value or the area corresponding to the consistency index not meeting the standard is used as the problem area.

[0056] In some embodiments, with respect to the above-mentioned step S211, the low-level visual structure information package includes at least: a gradient direction change trend and a significant edge region.

[0057] Furthermore, during the actual inspection process, if there is a large deviation between the preliminary binary mask image and the actual edge area of ​​the target structure, such as a significantly low edge overlap rate, broken boundaries, or disordered gradient directions, the device will determine that there is a problem with the mask in that area and mark it as a problem area.

[0058] In some embodiments, before comparing the preliminary binary segmentation mask image with a preset mask image to determine the fault type of the target structure, the method includes steps S31-S33:

[0059] S31, extracting a boundary contour point set of the target structure in the auxiliary image.

[0060] S32 , based on the boundary contour point set and the preliminary binary segmentation mask image, calculating the ratio of the area of ​​the boundary contour point set that is not covered by the boundary mask of the target structure in the preliminary binary segmentation mask image.

[0061] S33: If the area ratio is greater than a second preset value, it is determined that there is a problem with the preliminary binary segmentation mask image, and the preliminary binary segmentation mask image is reacquired; otherwise, there is no problem.

[0062] In some embodiments, with respect to the above step S31, the auxiliary image is an image of the target structure at the current moment, wherein the auxiliary image is an infrared image or a radar point cloud image.

[0063] In some embodiments, for the above step S32, the calculation process is as follows:

[0064]

[0065] Among them, the is the boundary contour point set, is the boundary mask of the target structure in the preliminary binary segmentation mask image, is the regional proportion.

[0066] Furthermore, the second preset value is preferably 0.25. When it is greater than 0.25, it is determined that there is a problem with the preliminary binary segmentation mask image.

[0067] In other embodiments, to determine whether there is a problem with the preliminary binary segmentation mask map, the method further includes: based on the boundary contour point set and the preliminary binary segmentation mask map, calculating the geometric offset between the boundary mask of the target structure in the preliminary binary segmentation mask map and the boundary contour point set; if the geometric offset is greater than a third preset value, it is determined that there is a problem with the preliminary binary segmentation mask map, and the preliminary binary segmentation mask map is reacquired; otherwise, there is no problem.

[0068] Furthermore, the specific calculation process is as follows:

[0069]

[0070] Among them, the is a pixel point of the boundary mask, is a pixel point in the boundary contour point set, is the geometric offset.

[0071] Furthermore, the third preset value is preferably 5. Specifically, when When the value is greater than 5, it is determined that there is a problem with the preliminary binary segmentation mask image.

[0072] In some embodiments, in order to determine whether the auxiliary image is an infrared image or a radar point cloud image, a judgment is made based on the scores of the two. The specific method is as follows: Steps S41-S45:

[0073] S41, calculating a first related parameter of the infrared image and a second related parameter of the radar point cloud image.

[0074] S42: Calculate a first comprehensive score of the infrared image based on the first related parameters.

[0075] S43: Calculate a second comprehensive score of the radar point cloud image based on the second related parameters.

[0076] S44: If the first comprehensive score is greater than the second comprehensive score, the auxiliary image is the infrared image.

[0077] S45: If the first comprehensive score is not greater than the second comprehensive score, the auxiliary image is the radar point cloud image.

[0078] In some embodiments, with respect to the above-mentioned step S41, the first related parameters include: Laplace variance of the edge of the target structure, average length of the linear structure detected based on Hough transform, and infrared hot spot density.

[0079] In some embodiments, with respect to the above-mentioned step S41, the second related parameters include: Laplace variance of the edge of the target structure, average length of the linear structure detected based on Hough transform, and laser occlusion ratio.

[0080] Furthermore, the formula for calculating the comprehensive score is as follows:

[0081]

[0082] Among them, the is the i-th comprehensive score, is the score of the k-th evaluation indicator, is the weighting coefficient.

[0083] In this embodiment, in order to facilitate understanding of the above formula for calculating the comprehensive score, a specific embodiment is given as follows:

[0084] The auxiliary images are radar point cloud images and infrared images, so the comprehensive score refers to the score of the radar point cloud image or the score of the infrared image. That is, if i is 1 or 2, the first comprehensive score refers to the infrared image score, and the second comprehensive score refers to the radar point cloud image score.

[0085] The k refers to the number of data points included in the relevant parameters. For example, when the first comprehensive score is a radar point image score, the first evaluation indicator score refers to the evaluation score of the Laplace variance of the target structure edge, the second evaluation indicator score refers to the evaluation score of the average length of the linear structure detected at the target structure edge based on the Hough transform, and the third evaluation indicator score refers to the evaluation score of the laser obstruction ratio at the target structure edge. When the second comprehensive score is an infrared image score, the first evaluation indicator score refers to the evaluation score of the Laplace variance of the target structure edge, the second evaluation indicator score refers to the evaluation score of the average length of the linear structure detected at the target structure edge based on the Hough transform, and the third evaluation indicator score refers to the evaluation score of the infrared hotspot density at the target structure edge.

[0086] In this embodiment, the comprehensive scores of the radar point cloud image and the infrared image are calculated, and finally, based on the comprehensive scores, the image with the highest comprehensive score is determined as the auxiliary image.

[0087] In other embodiments, the difference in comprehensive scores between the radar point cloud image and the infrared image is calculated to determine whether the auxiliary image is a radar point cloud image or an infrared image. Specifically, the difference in comprehensive scores between the radar point cloud image and the infrared image is calculated. If the difference in comprehensive scores is greater than a preset threshold, the image with the higher comprehensive score is selected as the auxiliary image. If the difference in comprehensive scores is less than the preset threshold, the current image is used as the auxiliary image.

[0088] In some embodiments, when the auxiliary image is the infrared image, extracting a boundary contour point set of the target structure in the auxiliary image includes steps S51-S52:

[0089] S51, performing enhancement processing on the infrared image to extract a temperature distribution characteristic map of the target structure.

[0090] S52, extracting a boundary contour point set of the target structure based on the temperature distribution feature map of the infrared image.

[0091] In some embodiments, in order to improve segmentation accuracy, the present application needs to perform a quality assessment on the obtained preliminary binary segmentation mask, and then improve the preliminary binary segmentation mask. The specific steps are as follows:

[0092] Step 1: Based on the boundary alignment, boundary offset distance and boundary closure of the preliminary binary segmentation mask image, the structural integrity and boundary accuracy of the preliminary binary segmentation mask image are evaluated to determine whether the preliminary binary segmentation mask image is a normal cutting image or an abnormal cutting image.

[0093] Step 2: If the preliminary binary segmentation mask image is an abnormal cut image, extract the abnormal area in the preliminary binary segmentation mask image, and generate supplementary prompt information based on the abnormal area.

[0094] Step 3: Determine enhanced prompt input based on the millimeter wave image and the supplementary prompt information.

[0095] Specifically, based on the abnormal area extracted in step 2, construct the supplementary prompt information, that is, the supplementary prompt information set , and set the supplementary prompt information The vector is passed to the prompt guidance layer of the SAM model in the form of an embedded vector to strengthen the model's attention and recognition of abnormal areas. It is then converted into the enhanced prompt input of the prompt guidance layer of the SAM module through a linear mapping module. The calculation formula is as follows:

[0096]

[0097] Among them, each supplementary prompt information in the supplementary prompt information set consists of three parts: prompt type code, spatial position information, and local image feature embedding. Each supplementary prompt information is uniformly encoded as a prompt vector t, and all prompt vectors are stacked to form a supplementary prompt information set. .

[0098] Furthermore, in the actual process, the attention mechanism inside the prompt guidance layer will interactively fuse the prompt vector with the feature representation of the image, so that the model will focus more on the The indicated abnormal area can be used to repair the problems such as boundary discontinuity and structural fracture in the preliminary binary segmentation mask image.

[0099] Step 4: Generate an updated binary segmentation mask map based on the enhanced prompt input and the preliminary binary segmentation mask map.

[0100] Specifically, according to the updated binary segmentation mask image obtained in step 4, the quality assessment of the binary segmentation mask image can be continued, and the operations of steps 1 to 4 are continued until the updated binary segmentation mask image is a normal cutting image.

[0101] Furthermore, in order to avoid unnecessary calculations, the number of times steps 1 to 4 are continued is preferably 3. If more than three iterations are performed, the calculation is terminated.

[0102] The technical solution provided by the present disclosure obtains an optical image of the target structure at the current moment and extracts a multi-scale feature map of the optical image; based on the multi-scale feature map and prompt information, determines a preliminary binary segmentation mask map corresponding to the boundary of the target structure contained in the optical image, wherein the prompt information indicates the boundary information of the target structure contained in the image; compares the preliminary binary segmentation mask map with a preset mask map to determine the fault type of the target structure, wherein different preset mask maps correspond to different fault types. The technical solution provided by each embodiment of the present disclosure determines the fault type of the target structure in the optical image by obtaining a preliminary binary segmentation mask map corresponding to the optical image and comparing it with a preset mask map. This detection method can avoid manual visual detection, improve the recognition accuracy of target structure faults, and improve work efficiency.

[0103] Figure 2 A schematic structural diagram of a training device for an image recognition model provided by an exemplary embodiment of the present disclosure;

[0104] The device includes: an acquisition unit 201, a determination unit 202, and a comparison unit 203;

[0105] An acquisition unit 201 is configured to acquire an optical image of a target structure at a current moment and extract a multi-scale feature map of the optical image;

[0106] a determining unit 202 configured to determine a preliminary binary segmentation mask corresponding to a boundary of the target structure contained in the optical image based on the multi-scale feature map and prompt information, wherein the prompt information indicates boundary information of the target structure contained in the image;

[0107] The comparing unit 203 is configured to compare the preliminary binary segmentation mask image with a preset mask image to determine the fault type of the target structure. Different preset mask images correspond to different fault types.

[0108] In some embodiments, the apparatus is configured to determine, based on the multi-scale feature map and the prompt information, a preliminary binary segmentation mask corresponding to the boundary of the target structure contained in the optical image, and the apparatus is specifically configured to:

[0109] Obtaining prompt information of the target structure and converting the prompt information into a prompt vector;

[0110] Based on the prompt vector, the multi-scale feature map is segmented to obtain a boundary feature map of the target structure, and the boundary feature map is converted into a preliminary binary segmentation mask map.

[0111] In some embodiments, the device is used to compare the preliminary binary segmentation mask image with a preset mask image to determine the fault type of the target structure, and the device is specifically used to:

[0112] Determining a problem area in the preliminary binary segmentation mask image, where the problem area indicates an area where a mask error exists;

[0113] Based on the problem area and the preset mask image, a fault type of the target structure is determined.

[0114] In some embodiments, the apparatus is used to determine the problem area in the preliminary binary segmentation mask image, and the apparatus is specifically used to:

[0115] Extracting low-level visual structure information from the optical image, wherein the low-level visual structure information package includes at least: a gradient direction change trend and a significant edge area;

[0116] Calculating the geometric overlap between the edge region of the preliminary binary segmentation mask image and the significant edge region, or calculating a consistency index between the gradient direction change trend of the preliminary binary segmentation mask image and the gradient direction change trend extracted from the optical image;

[0117] If the geometric overlap is less than a first preset value or the consistency index does not meet the standard, the edge area where the geometric overlap is less than the first preset value or the area corresponding to the consistency index not meeting the standard is used as the problem area.

[0118] In some embodiments, before the apparatus is used to compare the preliminary binary segmentation mask image with a preset mask image to determine the fault type of the target structure, the apparatus is specifically used to:

[0119] Extracting the boundary contour point set of the target structure in the auxiliary image;

[0120] Based on the boundary contour point set and the preliminary binary segmentation mask image, calculating the ratio of the area of ​​the boundary contour point set that is not covered by the boundary mask of the target structure in the preliminary binary segmentation mask image;

[0121] If the area ratio is greater than a second preset value, it is determined that there is a problem with the preliminary binary segmentation mask image, and the preliminary binary segmentation mask image is reacquired; otherwise, there is no problem.

[0122] In some embodiments, the auxiliary image is an image of the target structure at a current moment, wherein the auxiliary image is an infrared image or a radar point cloud image;

[0123] The device is also used for:

[0124] Calculating first relevant parameters of the infrared image and second relevant parameters of the radar point cloud image, the first relevant parameters including: Laplace variance of the edge of the target structure, average length of linear structures detected based on Hough transform, and infrared hotspot density; the second relevant parameters including: Laplace variance of the edge of the target structure, average length of linear structures detected based on Hough transform, and laser obstruction ratio;

[0125] Calculating a first comprehensive score of the infrared image based on the first relevant parameter;

[0126] Calculating a second comprehensive score of the radar point cloud image based on the second related parameter;

[0127] If the first comprehensive score is greater than the second comprehensive score, the auxiliary image is the infrared image;

[0128] If the first comprehensive score is not greater than the second comprehensive score, the auxiliary image is the radar point cloud image.

[0129] In some embodiments, when the auxiliary image is the infrared image, the device is configured to extract a boundary contour point set of a target structure in the auxiliary image, and the device is specifically configured to:

[0130] performing enhancement processing on the infrared image to extract a temperature distribution characteristic map of the target structure;

[0131] Based on the temperature distribution feature map of the infrared image, the boundary contour point set of the target structure is extracted.

[0132] The technical solution provided by the present disclosure obtains an optical image of the target structure at the current moment and extracts a multi-scale feature map of the optical image; based on the multi-scale feature map and prompt information, determines a preliminary binary segmentation mask map corresponding to the boundary of the target structure contained in the optical image, wherein the prompt information indicates the boundary information of the target structure contained in the image; compares the preliminary binary segmentation mask map with a preset mask map to determine the fault type of the target structure, wherein different preset mask maps correspond to different fault types. The technical solution provided by each embodiment of the present disclosure determines the fault type of the target structure in the optical image by obtaining a preliminary binary segmentation mask map corresponding to the optical image and comparing it with a preset mask map. This detection method can avoid manual visual detection, improve the recognition accuracy of target structure faults, and improve work efficiency.

[0133] It should be understood that the device embodiments and the method embodiments may correspond to each other, and similar descriptions may refer to the method embodiments. To avoid repetition, they will not be described in detail here. Specifically, the device can perform the above-mentioned method embodiments, and the aforementioned and other operations and / or functions of each module in the device are the corresponding processes in each method in the above-mentioned method embodiments, which will not be described in detail here for the sake of brevity.

[0134] The above describes the apparatus of the embodiment of the present disclosure from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in the form of hardware, can be implemented by instructions in the form of software, or can be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present disclosure can be completed by the hardware integrated logic circuit and / or software instructions in the processor, and the steps of the method disclosed in conjunction with the embodiment of the present disclosure can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above method embodiment in conjunction with its hardware.

[0135] Figure 3 is a schematic block diagram of an electronic device provided by an embodiment of the present disclosure, which may include:

[0136] The memory 301 and the processor 302 are configured to store computer programs and transmit the program code to the processor 302. In other words, the processor 302 can call and run the computer program from the memory 301 to implement the method in the embodiment of the present disclosure.

[0137] For example, the processor 302 may be configured to execute the above method embodiments according to instructions in the computer program.

[0138] In some embodiments of the present disclosure, the processor 302 may include but is not limited to:

[0139] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0140] In some embodiments of the present disclosure, the memory 301 includes but is not limited to:

[0141] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0142] In some embodiments of the present disclosure, the computer program may be divided into one or more modules, which are stored in the memory 301 and executed by the processor 302 to implement the method provided by the present disclosure. The one or more modules may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0143] like Figure 3 As shown, the electronic device may further include:

[0144] The transceiver 303 may be connected to the processor 302 or the memory 301 .

[0145] The processor 302 may control the transceiver 303 to communicate with other devices. Specifically, the processor 302 may send information or data to other devices or receive information or data sent by other devices. The transceiver 303 may include a transmitter and a receiver. The transceiver 303 may further include one or more antennas.

[0146] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0147] The present disclosure also provides a computer storage medium having a computer program stored thereon, which, when executed by a computer, enables the computer to perform the method of the above-mentioned method embodiment. Alternatively, the present disclosure also provides a computer program product containing instructions, which, when executed by a computer, enables the computer to perform the method of the above-mentioned method embodiment.

[0148] When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present disclosure are fully or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disk, hard disk, magnetic tape), optical media (e.g., digital video disc (DVD)), or semiconductor media (e.g., solid-state drive (SSD)).

[0149] Those skilled in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0150] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0151] Modules described as separate components may or may not be physically separate, and components displayed as modules may or may not be physical modules, i.e., they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected based on actual needs to achieve the purpose of the present embodiment. For example, the functional modules in the various embodiments of the present disclosure may be integrated into a single processing module, each module may exist physically separately, or two or more modules may be integrated into a single module.

[0152] The above are only specific embodiments of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A fault detection method, characterized in that: The method comprises: Acquire an optical image of the target structure at a current moment, and extract a multi-scale feature map of the optical image; Determining a preliminary binary segmentation mask corresponding to a boundary of the target structure contained in the optical image based on the multi-scale feature map and the prompt information, wherein the prompt information indicates boundary information of the target structure contained in the image; Comparing the preliminary binary segmentation mask image with a preset mask image to determine the fault type of the target structure, where different preset mask images correspond to different fault types; Before comparing the preliminary binary segmentation mask image with a preset mask image to determine the fault type of the target structure, the method includes: Extracting the boundary contour point set of the target structure in the auxiliary image; Based on the boundary contour point set and the preliminary binary segmentation mask image, calculating the ratio of the area of ​​the boundary contour point set that is not covered by the boundary mask of the target structure in the preliminary binary segmentation mask image; If the area ratio is greater than a second preset value, it is determined that there is a problem with the preliminary binary segmentation mask image, and the preliminary binary segmentation mask image is re-acquired; otherwise, there is no problem; Wherein, the auxiliary image is the image of the target structure at the current moment, and the auxiliary image is an infrared image or a radar point cloud image; The method further comprises: Calculating first relevant parameters of the infrared image and second relevant parameters of the radar point cloud image, the first relevant parameters including: Laplace variance of the edge of the target structure, average length of linear structures detected based on Hough transform, and infrared hotspot density; the second relevant parameters including: Laplace variance of the edge of the target structure, average length of linear structures detected based on Hough transform, and laser obstruction ratio; Calculating a first comprehensive score of the infrared image based on the first relevant parameter; Calculating a second comprehensive score of the radar point cloud image based on the second related parameter; If the first comprehensive score is greater than the second comprehensive score, the auxiliary image is the infrared image; If the second comprehensive score is not greater than the second comprehensive score, the auxiliary image is the radar point cloud image.

2. The method according to claim 1, characterized in that The step of determining a preliminary binary segmentation mask corresponding to a boundary of the target structure contained in the optical image based on the multi-scale feature map and the prompt information includes: Obtaining prompt information of the target structure and converting the prompt information into a prompt vector; Based on the prompt vector, the multi-scale feature map is segmented to obtain a boundary feature map of the target structure, and the boundary feature map is converted into a preliminary binary segmentation mask map.

3. The method according to claim 1, characterized in that The comparing the preliminary binary segmentation mask image with a preset mask image to determine the fault type of the target structure includes: Determining a problem area in the preliminary binary segmentation mask image, where the problem area indicates an area where a mask error exists; Based on the problem area and the preset mask image, a fault type of the target structure is determined.

4. The method according to claim 3, characterized in that Determining the problem area in the preliminary binary segmentation mask image includes: Extracting low-level visual structure information from the optical image, wherein the low-level visual structure information package includes at least: a gradient direction change trend and a significant edge area; Calculating the geometric overlap between the edge region of the preliminary binary segmentation mask image and the significant edge region, or calculating a consistency index between the gradient direction change trend of the preliminary binary segmentation mask image and the gradient direction change trend extracted from the optical image; If the geometric overlap is less than a first preset value or the consistency index does not meet the standard, the edge area where the geometric overlap is less than the first preset value or the area corresponding to the consistency index not meeting the standard is used as the problem area.

5. The method according to claim 1, wherein When the auxiliary image is the infrared image, extracting a boundary contour point set of the target structure in the auxiliary image includes: Performing enhancement processing on the infrared image to extract a temperature distribution characteristic map of the target structure; Based on the temperature distribution feature map of the infrared image, the boundary contour point set of the target structure is extracted.

6. A fault detection device, characterized in that: The device comprises: an acquisition unit, configured to acquire an optical image of the target structure at a current moment and extract a multi-scale feature map of the optical image; a determining unit, configured to determine a preliminary binary segmentation mask corresponding to a boundary of the target structure contained in the optical image based on the multi-scale feature map and prompt information, wherein the prompt information indicates boundary information of the target structure contained in the image; a comparing unit, configured to compare the preliminary binary segmentation mask image with a preset mask image to determine a fault type of the target structure, wherein different preset mask images correspond to different fault types; The device is also used for: Extracting the boundary contour point set of the target structure in the auxiliary image; Based on the boundary contour point set and the preliminary binary segmentation mask image, calculating the ratio of the area of ​​the boundary contour point set that is not covered by the boundary mask of the target structure in the preliminary binary segmentation mask image; If the area ratio is greater than a second preset value, it is determined that there is a problem with the preliminary binary segmentation mask image, and the preliminary binary segmentation mask image is re-acquired; otherwise, there is no problem; Wherein, the auxiliary image is the image of the target structure at the current moment, and the auxiliary image is an infrared image or a radar point cloud image; The device is also used for: Calculating first relevant parameters of the infrared image and second relevant parameters of the radar point cloud image, the first relevant parameters including: Laplace variance of the edge of the target structure, average length of linear structures detected based on Hough transform, and infrared hotspot density; the second relevant parameters including: Laplace variance of the edge of the target structure, average length of linear structures detected based on Hough transform, and laser obstruction ratio; Calculating a first comprehensive score of the infrared image based on the first relevant parameter; Calculating a second comprehensive score of the radar point cloud image based on the second related parameter; If the first comprehensive score is greater than the second comprehensive score, the auxiliary image is the infrared image; If the second comprehensive score is not greater than the second comprehensive score, the auxiliary image is the radar point cloud image.

7. An electronic device, characterized in that: include: processor; as well as a memory for storing executable instructions of the processor; The processor is configured to perform the method according to any one of claims 1 to 5 by executing the executable instructions.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images