Image processing method and device, equipment and medium

Abstract

The invention discloses an image processing method and device, equipment and a medium. The method comprises the following steps: performing feature extraction on a to-be-processed image based on a backbone network of a defect identification model to obtain a first feature image; based on a candidate area generation network of a defect identification model, identifying an area with defects in the first feature image, and generating at least one candidate area; and based on a defect identification network of a defect identification model, identifying defect features in the at least one candidate region, and determining a defect identification result. According to the technical scheme, adaptive segmentation of various types of defects of the petrochemical pipeline is realized, and the defect identification accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of oil and gas exploration technology, and in particular to an image processing method, device, equipment and medium. Background Art

[0002] The petrochemical industry is a crucial strategic component of the national economy, and national development is inseparable from it. However, during service, petrochemical pipelines can develop various types of defects due to factors such as oil and gas corrosion, weather, crustal movement, ocean currents, and external stresses that damage the metal's internal mechanical structure. These defects pose significant risks to the safe transportation of petrochemical pipelines. For example, pipeline leaks and explosions have caused significant economic losses to the nation and society, and threatened human safety. Therefore, regular inspections of petrochemical pipelines are necessary to identify the location and extent of damage and promptly eliminate potential hazards.

[0003] In recent years, with the emergence of new technologies in the field of deep learning, deep learning-based defect detection technology has made significant progress. However, deep learning requires the use of vast amounts of sample data. When sample data is difficult to obtain, neural networks with large numbers of training parameters can suffer from overfitting. Furthermore, since various types of petrochemical pipelines can exhibit a variety of defect types, such as cracks, corrosion, and blockages, and the same defect type often manifests in a variety of different forms in actual image data, existing techniques for segmenting petrochemical pipeline defects based on deep learning theory generally suffer from a lack of training samples and inaccurate segmentation.

[0004] Therefore, how to provide a technical solution that can accurately identify defects in petrochemical pipelines is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the Invention

[0005] The present application provides an image processing method, apparatus, device and medium to achieve adaptive segmentation of various types of defects in petrochemical pipelines and improve the accuracy of defect recognition.

[0006] According to one aspect of the present application, there is provided an image processing method, the method comprising:

[0007] Based on the backbone network of the defect recognition model, feature extraction is performed on the image to be processed to obtain a first feature image;

[0008] Based on the candidate region generation network of the defect recognition model, identifying the region with defects in the first feature image and generating at least one candidate region;

[0009] Based on the defect recognition network of the defect recognition model, defect features in the at least one candidate area are identified to determine a defect recognition result.

[0010] According to another aspect of the present application, there is provided an image processing apparatus, the apparatus comprising:

[0011] An image feature extraction module is used to extract features from the image to be processed based on the backbone network of the defect recognition model to obtain a first feature image;

[0012] a candidate region generating module, configured to identify a region having defects in the first feature image based on a candidate region generating network of the defect recognition model, and generate at least one candidate region;

[0013] The defect recognition module is used to identify defect features in the at least one candidate area based on the defect recognition network of the defect recognition model and determine a defect recognition result.

[0014] According to another aspect of the present application, there is provided an image processing device, the device comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the image processing method described in any embodiment of the present application.

[0018] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the image processing method described in any embodiment of the present application when executed.

[0019] The technical solution provided in this application uses a backbone network based on a defect recognition model to extract features from a processed image to obtain a first feature image. A candidate region generation network based on the defect recognition model identifies defective regions in the first feature image and generates at least one candidate region. Finally, a defect recognition network based on the defect recognition model identifies defect features in the at least one candidate region and determines the defect recognition result. This technical solution enables adaptive segmentation of various types of defects in petrochemical pipelines, improving defect recognition accuracy.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] Figure 1 A flowchart of an image processing method provided in Example 1 of the present application;

[0023] Figure 2 A schematic diagram of the structure of a backbone network provided in Example 1 of the present application;

[0024] Figure 3 A flowchart of an image processing method provided in Example 2 of the present application;

[0025] Figure 4 A schematic diagram of the structure of an image processing device provided in Example 3 of the present application;

[0026] Figure 5 It is a structural diagram of a device for implementing an image processing method according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0028] It should be noted that the terms "first", "second", "to be processed", "candidate", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] Example 1

[0030] Figure 1 This is a flowchart of an image processing method provided in the first embodiment of the present application. This embodiment is applicable to the identification of defects in petrochemical pipelines. The method can be executed by an image processing device, which can be implemented in the form of hardware and / or software. The image processing device can be configured in a device with data processing capabilities. Figure 1 As shown, the method includes:

[0031] S110 , based on the backbone network of the defect recognition model, extract features from the image to be processed to obtain a first feature image.

[0032] The image to be processed can be an image of the interior or exterior of a petrochemical pipeline captured by an image collector. Specifically, the image collector can be deployed at multiple preset locations inside or outside the petrochemical pipeline, and controlled to periodically capture images of the petrochemical pipeline within the visible area and use them as the image to be processed. Alternatively, the image collector can be placed on a movable carrier, and controlled to capture images of the petrochemical pipeline in a target area and use them as the image to be processed.

[0033] The backbone network can be used to extract features from images input into the defect recognition model. A good backbone network can significantly improve the performance of the defect recognition model. The backbone network can be a VGG network, a GoogLeNet network, a ResNet network, an AlexNet network, etc. This embodiment of the present invention is not limited to this, and an appropriate backbone network can be selected according to actual needs.

[0034] Optionally, the backbone network is a ResNet-FPN network; wherein, the ResNet-FPN network includes a Steam layer and at least two Stage layers; accordingly, the backbone network based on the defect recognition model performs feature extraction on the image to be processed to obtain a first feature image, including: based on at least two Stage layers in the ResNet-FPN network, performing feature extraction on the image to be processed to obtain at least two layers of candidate feature images; hierarchically, the high-level candidate feature images in two adjacent levels are up-sampled and fused with the low-level candidate feature images to obtain the first feature image.

[0035] The ResNet-FPN network is a pyramid-shaped ResNet network. Using the FPN-structured ResNet network as the backbone network for feature extraction in processed images effectively increases the accuracy of the entire network for small object detection. Specifically, the ResNet network consists of several Stage layers, each containing several BasicBlock modules or Bottleneck modules.

[0036] For example, the ResNet network can be a ResNet50 network or a ResNet101 network, where the ResNet network consists of 1 Stem layer and 4 Stage layers, each Stage layer contains several Bottleneck modules, each Bottleneck module consists of two 1×1 convolutional layers and one 3×3 convolutional layer, and uses residual connection.

[0037] However, using only the ResNet network as the backbone network will make the entire network unable to effectively detect and segment small targets of petrochemical pipeline defects. Therefore, the pyramid structure FPN is introduced to enhance the backbone network's extraction of small target features, thereby effectively increasing the accuracy of the entire network's small target detection.

[0038] For example, Figure 2 This is a schematic diagram of the structure of a backbone network provided in Example 1 of this application. Figure 2 As shown in the figure, after the image to be processed is input into the defect recognition model, it first undergoes dimensionality increase and size reduction through the Steam layer. Then, through four Stage layers, the high-level candidate feature images of two adjacent layers are upsampled and fused with the low-level candidate feature images. The fused features have both high semantic information and high resolution, which can effectively complete small object detection. The specific implementation steps are to first pass the feature map of the current layer through a 1×1 convolution layer to reduce the number of channels, then add it to the feature map of the previous layer through an upsampling operation, and so on, to obtain the first feature image.

[0039] S120 . Based on the candidate region generation network of the defect recognition model, identify regions with defects in the first feature image and generate at least one candidate region.

[0040] Among them, the candidate region generation network is used to generate candidate regions for defect detection to achieve preliminary positioning of defects.

[0041] In this scheme, the sliding window can be used to perform target detection on the image in each window in turn to determine the candidate area; the image to be processed can also be divided into multiple regions based on the selective search principle, and similar regions can be merged through some heuristic rules to determine the candidate area; the image to be processed can also be processed through the RPN network to determine the candidate area.

[0042] Optionally, a candidate region generation network based on the defect recognition model is used to identify regions with defects in the first feature image and generate at least one candidate region, including: a candidate region generation network based on the defect recognition model is used to use at least one anchor frame of a preset size, and traverse each pixel point of the first feature image as the center of the anchor frame; during the traversal process, each anchor frame is screened according to the existence status of the defects in each anchor frame to generate at least one candidate region.

[0043] The anchor frame may be a rectangular frame of a predefined size. The predefined size may be a specific size or size ratio, such as 32*32, 64*64, 128*128, or an aspect ratio of 1:1, 1:2, 2:1, etc.

[0044] In this solution, each pixel point of the first feature image can be used as the anchor frame center to generate at least one anchor frame area of ​​a preset size, and each anchor frame area is screened and sorted, and the anchor frame area that is most likely to contain defects is selected as the candidate area.

[0045] Optionally, each anchor frame is screened according to the existence status of defects in each anchor frame to generate at least one candidate region, including: determining the candidate anchor frame based on the first screening network in the candidate region generation network; determining the offset of the candidate anchor frame based on the second screening network in the candidate region generation network; and correcting the candidate anchor frame according to the offset to generate at least one candidate region.

[0046] During the training process of the candidate region generation network, the candidate region generation network can be composed of two branches, namely the first screening network and the second screening network. The first screening network can be composed of two convolutional layers for distinguishing positive and negative samples of the anchor box region; the second screening network can be composed of one convolutional layer for calculating the offset of the anchor box region.

[0047] Specifically, the anchor regions generated by the traversal are input into the first screening network to obtain the intersection-over-union (IoU) of the anchor region area and the defect region area. If the IoU is greater than 0.7, the anchor region is marked as a positive sample; if the IoU is less than 0.3, the anchor region is marked as a negative sample. The anchor regions marked as positive samples are input into the second screening network for regression, which outputs the coordinate offset of the anchor region. Finally, the position and size of the anchor region are adjusted based on the output offset to more accurately match the defect.

[0048] During the testing and use of the candidate region generation network, after the first feature image is input into the trained candidate region generation network, the corresponding candidate anchor frame and anchor frame offset can be directly generated, and then the candidate anchor frame can be corrected according to the anchor frame offset to generate at least one candidate region.

[0049] S130 . Based on the defect recognition network of the defect recognition model, identify defect features in the at least one candidate region and determine a defect recognition result.

[0050] Defect characteristics may include defect type, defect location, defect size, etc. Defect types may be cracks, corrosion, blockage, etc. The defect recognition network can be used to accurately predict characteristic parameters of defects in candidate areas.

[0051] In this solution, the feature similarity between the defects in the candidate area and the trained defects can be calculated to classify and identify the defects in the candidate area and determine the defect type; the location of the defects in the candidate area can also be identified to segment the defects and determine the defect location and size.

[0052] An embodiment of the present invention provides an image processing method. This method, based on a backbone network of a defect recognition model, extracts features from an image to be processed to obtain a first feature image. A candidate region generation network based on the defect recognition model identifies defective regions in the first feature image and generates at least one candidate region. Finally, a defect recognition network based on the defect recognition model identifies defect features in the at least one candidate region and determines a defect recognition result. This technical solution enables adaptive segmentation of various types of defects in petrochemical pipelines, improving defect recognition accuracy.

[0053] Example 2

[0054] Figure 3 This is a flow chart of an image processing method provided in Example 2 of this application. This embodiment is optimized based on the above embodiment. Figure 3 As shown, the method of this embodiment specifically includes the following steps:

[0055] S210 , based on the backbone network of the defect recognition model, extract features from the image to be processed to obtain a first feature image.

[0056] S220, based on the candidate region generation network of the defect recognition model, identifies the region with defects in the first feature image and generates at least one candidate region.

[0057] S230 , based on the defect recognition network of the defect recognition model, performing bilinear interpolation on each pixel in the candidate area to obtain image values ​​of the pixel points whose coordinates are floating-point numbers, and determining a second feature image of the candidate area.

[0058] Specifically, the ROI align method can be used to extract features from candidate regions. This method eliminates the quantization operation commonly used in the ROIpooling method and instead uses bilinear interpolation to obtain image values ​​at pixels with floating-point coordinates, thus transforming the entire feature aggregation process into a continuous operation.

[0059] The beneficial effect of the above technical solution is that it solves the regional mismatch problem caused by the quantization process in the ROI pooling algorithm, so as to ensure that the original image and the feature map, and the feature map and the ROI can be aligned at the pixel level, avoiding the image distortion problem, improving the accuracy of target detection, and being more in line with the instance segmentation problem, which can improve the accuracy.

[0060] S240: Identify defect features in the second feature image and determine a defect identification result.

[0061] Optionally, the recognition result includes at least one of the defect type, the defect target frame and the defect segmentation result; accordingly, the defect features in the second feature image are identified to determine the defect recognition result, including at least one of the following: based on the target recognition network, the defect is identified on the second feature image to determine the defect type; based on the target frame regression network, the depth features of the second feature image are obtained, and the defect target frame is determined according to the depth features; based on the semantic segmentation network, the pixel points belonging to the defect in the second feature image are extracted to obtain the defect segmentation result.

[0062] Among them, the target recognition network can be composed of a fully connected layer, which is used to classify the type of defects in the candidate area to determine the defect type.

[0063] The target box regression network can be composed of a fully connected layer to extract the deep features of the defects in the candidate area to generate a more accurate target box. The loss function of the target box regression network can use smooth L1 loss, which is specifically expressed as:

[0064]

[0065]

[0066] Where, L box is the target box regression loss, t i,j is the predicted target box parameter, is the actual target frame parameter, j∈{x,y,w,h} represents the four parameter values ​​of the target frame, N box is the number of target boxes.

[0067] The semantic segmentation network can be composed of two convolutional layers, which are used to extract the pixels belonging to the defects in the candidate area to obtain the defect segmentation results. The loss function of the semantic segmentation network can be specifically expressed as:

[0068]

[0069]

[0070] Where, L mask is the semantic segmentation loss, B i represents the i-th candidate region, M box is the number of candidate regions, M i is the number of pixels in the i-th candidate region, y j Indicates the label of the j-th pixel, y j ∈{1,2,...,k},k≥2,p jl Represents the probability that the j-th pixel output is label l, that is, p jl =p(y j =l|B ij ).

[0071] In this solution, defect recognition results can be defect type; defect target box; defect segmentation result; defect type and defect target box; defect type and defect segmentation result; defect target box and defect segmentation result; defect type, defect target box, and defect segmentation result. Correspondingly, the defect recognition network can include an object recognition network; an object box regression network; a semantic segmentation network; an object recognition network and an object box regression network; an object recognition network and a semantic segmentation network; an object box regression network and a semantic segmentation network; an object recognition network, an object box regression network, and a semantic segmentation network.

[0072] Optionally, if the defect recognition network includes an object recognition network, an object box regression network, and a semantic segmentation network, the loss function of the defect recognition network can be expressed using the following formula:

[0073] L=L cls +L box +Lmask ;

[0074] Where, L is the defect recognition network loss, L cls is the target recognition loss, L box is the target box regression loss, L mask is the semantic segmentation loss.

[0075] An embodiment of the present invention provides an image processing method. This method, based on a backbone network of a defect recognition model, extracts features from an image to be processed to obtain a first feature image. A candidate region generation network based on the defect recognition model identifies regions with defects in the first feature image and generates at least one candidate region. A defect recognition network based on the defect recognition model performs bilinear interpolation on each pixel in the candidate region to obtain image values ​​at the pixel points with floating-point coordinates, thereby determining a second feature image for the candidate region. The defect features in the second feature image are then identified to determine the defect recognition result. This technical solution avoids the region mismatch problem caused by quantization processing and improves target detection accuracy.

[0076] Based on the above embodiments, optionally, the training process of the defect recognition model includes: using the COCO dataset to train the initial model to obtain an intermediate model with the function of detecting and segmenting objects; using a defect dataset containing target defects to train the intermediate model to obtain a defect recognition model.

[0077] All types of petrochemical pipelines can exhibit a variety of defects, including cracks, corrosion, and blockages. Furthermore, even the same defect type often manifests in a variety of different forms in the image data collected. Furthermore, because pipelines undergo regular maintenance, obvious defects are rare, making it difficult to collect diverse and sufficient samples for model training.

[0078] Therefore, this scheme performs a task transfer during the training process of the defect recognition model, so that the defect recognition model can obtain good performance and strong generalization capabilities through a small number of samples, thereby realizing adaptive detection and segmentation of petrochemical pipeline defects.

[0079] The COCO dataset is a publicly available dataset provided by the Microsoft team for image recognition. In this solution, we first used the publicly available COCO dataset to establish a certain level of object detection and segmentation capabilities for the initial model. We then migrated the model to the petrochemical pipeline defect dataset for fine-tuning, ultimately enabling the model to detect and segment petrochemical pipeline defects.

[0080] The beneficial effect of the above technical solution is that it can obtain functions with good performance and strong generalization ability through a small number of samples, thereby realizing adaptive detection and segmentation of petrochemical pipeline defects.

[0081] Furthermore, during the model training phase, the Adam optimizer was used to train and optimize the model, with the optimizer parameters set to β1 = 0.9 and β2 = 0.999. The entire model was trained on an NVIDIA Tesla V100 GPU. During the model training process, the batch size of the images was set to 4, with the input size of each image being 256×256. The training was repeated for 3000 rounds, and the learning rate was set to 2.5×10 -2 , and decays to 1 / 10 of its original value at 2000 and 2750 rounds.

[0082] Example 3

[0083] Figure 4 This is a structural diagram of an image processing device provided in Example 3 of this application. Figure 4 As shown, the device includes:

[0084] An image feature extraction module 310 is configured to extract features from the image to be processed based on a backbone network of a defect recognition model to obtain a first feature image;

[0085] A candidate region generating module 320 is configured to identify defective regions in the first feature image based on a candidate region generating network of the defect recognition model, and generate at least one candidate region;

[0086] The defect recognition module 330 is configured to recognize defect features in the at least one candidate region based on the defect recognition network of the defect recognition model, and determine a defect recognition result.

[0087] An embodiment of the present invention provides an image processing device. Based on a backbone network of a defect recognition model, the device extracts features from an image to be processed to obtain a first feature image. Based on a candidate region generation network of the defect recognition model, the device identifies defective regions in the first feature image and generates at least one candidate region. Furthermore, based on a defect recognition network of the defect recognition model, the device identifies defect features in the at least one candidate region and determines a defect recognition result. This technical solution enables adaptive segmentation of various types of defects in petrochemical pipelines, improving defect recognition accuracy.

[0088] Furthermore, the backbone network is a ResNet-FPN network; wherein the ResNet-FPN network includes a Steam layer and at least two Stage layers;

[0089] Accordingly, the image feature extraction module 310 includes:

[0090] A candidate feature image determination unit is configured to perform feature extraction on the image to be processed based on at least two Stage layers in the ResNet-FPN network to obtain at least two layers of candidate feature images;

[0091] The first feature image determination unit is used to perform feature fusion of the high-level candidate feature images in two adjacent levels with the low-level candidate feature images by upsampling according to the level to obtain the first feature image.

[0092] Furthermore, the candidate region generation module 320 includes:

[0093] A first feature image traversal unit is configured to generate a candidate region network based on the defect recognition model, use at least one anchor frame of a preset size, and traverse each pixel point of the first feature image with each pixel point as the anchor frame center;

[0094] The candidate region generating unit is configured to screen each of the anchor frames according to the presence of defects in each of the anchor frames during the traversal process, and generate at least one candidate region.

[0095] Furthermore, the candidate region generation unit includes:

[0096] a candidate anchor box generation subunit, configured to determine a candidate anchor box based on a first screening network in the candidate region generation network;

[0097] an anchor box offset determination subunit, configured to determine an offset of a candidate anchor box based on a second screening network in the candidate region generation network;

[0098] The candidate region generating subunit is configured to modify the candidate anchor frame according to the offset to generate at least one candidate region.

[0099] Furthermore, the defect identification module 330 includes:

[0100] a second feature image determining unit, configured to perform bilinear interpolation on each pixel point in the candidate area based on the defect recognition network of the defect recognition model, obtain image values ​​at the pixel points whose coordinates are floating-point numbers, and determine a second feature image of the candidate area;

[0101] The defect recognition unit is used to recognize the defect features in the second feature image and determine the defect recognition result.

[0102] Furthermore, the recognition result includes at least one of a defect type, a defect target frame, and a defect segmentation result;

[0103] Accordingly, the defect identification unit includes at least one of the following:

[0104] a defect type determination subunit, configured to perform defect recognition on the second feature image based on a target recognition network to determine the defect type;

[0105] a defect target frame determination subunit, configured to obtain depth features of the second feature image based on a target frame regression network, and determine a defect target frame according to the depth features;

[0106] The defect segmentation result determination subunit is used to extract the pixel points belonging to the defect in the second feature image based on the semantic segmentation network to obtain the defect segmentation result.

[0107] Furthermore, the training process of the defect recognition model includes:

[0108] The COCO dataset is used to train the initial model to obtain an intermediate model with the ability to detect and segment objects;

[0109] The intermediate model is trained using a defect data set containing target defects to obtain a defect recognition model.

[0110] An image processing device provided in an embodiment of the present application can execute an image processing method provided in any embodiment of the present application, and has functional modules and beneficial effects corresponding to the execution method.

[0111] Example 4

[0112] Figure 5 A schematic diagram of the structure of an apparatus 10 that can be used to implement an embodiment of the present application is shown. The apparatus is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The apparatus can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0113] like Figure 5As shown, device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 and a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores a computer program executable by the at least one processor, and processor 11 can perform various appropriate actions and processes based on the computer program stored in the read-only memory (ROM) 12 or loaded from storage unit 18 into the random access memory (RAM) 13. RAM 13 can also store various programs and data required for the operation of device 10. Processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to bus 14.

[0114] Various components in device 10 are connected to I / O interface 15, including an input unit 16, such as a keyboard, mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, optical disk, etc.; and a communication unit 19, such as a network card, modem, wireless communication transceiver, etc. Communication unit 19 allows device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0115] The processor 11 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors for running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the image processing method.

[0116] In some embodiments, the image processing method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed onto device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the image processing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the image processing method in any other suitable manner (e.g., via firmware).

[0117] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0118] Computer programs for implementing the methods of the present application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0119] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0120] To provide interaction with a user, the systems and techniques described herein can be implemented on a device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0121] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0122] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0123] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this application can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of this application can be achieved. This is not limited herein.

[0124] The above specific embodiments do not constitute a limitation on the scope of protection of this application. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application shall be included within the scope of protection of this application.

Claims

1. An image processing method, characterized in that: The method comprises: Based on the backbone network of the defect recognition model, feature extraction is performed on the image to be processed to obtain a first feature image; Based on the candidate region generation network of the defect recognition model, identifying the region with defects in the first feature image and generating at least one candidate region; Based on the defect recognition network of the defect recognition model, defect features in the at least one candidate area are identified to determine a defect recognition result.

2. The method according to claim 1, characterized in that The backbone network is a ResNet-FPN network; wherein the ResNet-FPN network includes a Steam layer and at least two Stage layers; Accordingly, based on the backbone network of the defect recognition model, feature extraction is performed on the image to be processed to obtain a first feature image, including: Based on at least two Stage layers in the ResNet-FPN network, feature extraction is performed on the image to be processed to obtain at least two layers of candidate feature images; According to the hierarchy, the high-level candidate feature images of two adjacent levels are up-sampled and fused with the low-level candidate feature images to obtain the first feature image.

3. The method according to claim 1, characterized in that Based on the candidate region generation network of the defect recognition model, identifying the region where the defect exists in the first feature image and generating at least one candidate region, the method includes: Based on the candidate region generation network of the defect recognition model, at least one anchor frame of a preset size is used, and each pixel point of the first feature image is taken as the center of the anchor frame to traverse the pixel points; During the traversal process, each anchor frame is screened according to the presence status of defects in each anchor frame to generate at least one candidate region.

4. The method according to claim 3, characterized in that Screening each anchor frame according to the presence status of defects in each anchor frame to generate at least one candidate region includes: Determining a candidate anchor box based on a first screening network in the candidate region generation network; Determining an offset of a candidate anchor box based on a second screening network in the candidate region generation network; The candidate anchor frame is modified according to the offset to generate at least one candidate region.

5. The method according to claim 1, wherein Based on the defect recognition network of the defect recognition model, identifying defect features in the at least one candidate area and determining a defect recognition result, including: Based on the defect recognition network of the defect recognition model, bilinear interpolation is performed on each pixel point in the candidate area to obtain image values ​​at the pixel points with floating-point coordinates, and a second feature image of the candidate area is determined; The defect features in the second feature image are identified to determine a defect identification result.

6. The method according to claim 5, characterized in that The recognition result includes at least one of a defect type, a defect target frame, and a defect segmentation result; Accordingly, identifying the defect feature in the second feature image and determining the defect recognition result includes at least one of the following: Based on the target recognition network, performing defect recognition on the second feature image to determine the defect type; Based on the target frame regression network, obtaining the depth features of the second feature image, and determining the defect target frame according to the depth features; Based on the semantic segmentation network, the pixels belonging to defects in the second feature image are extracted to obtain a defect segmentation result.

7. The method according to claim 1, characterized in that The training process of the defect recognition model includes: The COCO dataset is used to train the initial model to obtain an intermediate model with the ability to detect and segment objects; The intermediate model is trained using a defect data set containing target defects to obtain a defect recognition model.

8. An image processing device, characterized in that: The device comprises: An image feature extraction module is used to extract features from the image to be processed based on the backbone network of the defect recognition model to obtain a first feature image; a candidate region generating module, configured to identify a region having defects in the first feature image based on a candidate region generating network of the defect recognition model, and generate at least one candidate region; The defect recognition module is used to identify defect features in the at least one candidate area based on the defect recognition network of the defect recognition model and determine a defect recognition result.

9. An electronic device, characterized in that: The device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the image processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the image processing method according to any one of claims 1 to 7 when executed.

Images