Defect detection method based on sparse convolution redistribution and detection head fusion

Abstract

The invention discloses a defect detection method based on sparse convolution redistribution and detection head fusion. The method comprises the following steps: acquiring discrete point cloud information of a to-be-detected target; the discrete point cloud information is input into a trained PillarNeXt model to obtain a defect detection result, the PillarNeXt model comprises a pillar encoder, a backbone network, a neck network and a detection head, the backbone network comprises multiple groups of sparse convolution modules and is used for extracting point cloud features of the two-dimensional pseudo image to obtain a sparse feature map, the sparse feature map is converted into a dense feature map, and the dense feature map is used for extracting the point cloud features of the two-dimensional pseudo image. The neck network is used for capturing the spatial information of the dense feature map, the detection head is used for predicting the three-dimensional coordinate information, the target size, the target orientation and the target category confidence coefficient thermodynamic diagram of the center point of the target to be detected, and the number of each group of sparse convolution modules is non-uniformly distributed. The number of the sparse convolution modules with the later group number is not greater than the number of the sparse convolution modules with the former group number. According to the invention, rapid and accurate defect detection can be realized.

Description

Technical Field

[0001] The present invention relates to the field of industrial detection technology, and more specifically, to a defect detection method based on sparse convolution redistribution and detection head fusion. Background Art

[0002] With the rapid development of the manufacturing industry and the increase in industrial capacity, the products produced are becoming more and more diversified. Due to the complexity of the manufacturing process of industrial products, defects are inevitable on the product surface, which will have an adverse effect on the quality of the product. In the field of industrial inspection, defect detection needs to quickly and accurately locate defects on the surface or inside of the product, which helps to find unqualified products and reduce the economic losses caused by product quality problems, thereby achieving the purpose of saving manpower and improving product quality.

[0003] At present, in the field of industrial inspection, the use of advanced computer vision technology to detect defects has become an important part of industrial production. However, most defect detection work is mainly achieved by manual means, which has problems such as low efficiency and strong subjectivity, and is very likely to lead to false detection and missed detection. With the rapid development of 3D vision and artificial intelligence technology, point cloud-based deep learning algorithms have good detection effects and generalization in defect detection tasks, but the model complexity is high, and the detection efficiency and accuracy need to be improved.

[0004] In summary, in order to detect defects quickly and accurately, it is necessary to combine 3D vision and artificial intelligence to provide new technical solutions. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a defect detection method based on sparse convolution redistribution and detection head fusion. The method comprises the following steps:

[0006] Obtain discrete point cloud information of the target to be detected;

[0007] Inputting the discrete point cloud information into the trained PillarNeXt model to obtain defect detection results;

[0008] The PillarNeXt model includes a pillar encoder, a backbone network, a neck network and a detection head, wherein the pillar encoder is used to convert the discrete point cloud information into a two-dimensional pseudo image, the backbone network includes multiple groups of sparse convolution modules with a residual connection structure, which are used to extract the point cloud features of the two-dimensional pseudo image, obtain a downsampled sparse feature map, and convert the sparse feature map into a dense feature map, the neck network is used to capture the spatial information of the dense feature map at different scales, and the detection head is used to predict the three-dimensional XY coordinates, three-dimensional Z coordinates, size, orientation and target category confidence heat map of the center point of the target to be detected;

[0009] The number of sparse convolution modules in each group is unevenly distributed, and taking the input of the backbone network as a reference point, the number of sparse convolution modules in the later groups is not greater than the number of sparse convolution modules in the earlier groups.

[0010] Compared with the prior art, the advantage of the present invention is that, in response to the problem of defect detection, the present invention proposes a defect detection method based on sparse convolution redistribution and detection head fusion. This method reallocates multiple groups of sparse convolution modules of the backbone network of the PillarNeXt model, moves the sparse convolution modules forward, improves the detection accuracy of the model, and fuses the detection heads. Under the premise of ensuring the detection accuracy advantage of the model, it accelerates the model inference speed and reduces the model complexity, thereby effectively completing the defect detection task.

[0011] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0013] Figure 1 is a flow chart of a defect detection method based on sparse convolution redistribution and detection head fusion according to an embodiment of the present invention;

[0014] Figure 2 is a schematic diagram of reallocation of sparse convolution modules of a PillarNeXt backbone network according to an embodiment of the present invention;

[0015] Figure 3 is a schematic diagram of PillarNeXt detection head fusion according to an embodiment of the present invention;

[0016] In the accompanying figure, Pseudo Image-pseudo image; Residual Block-residual block; shared-conv-shared convolution module; FuseConvBN-convolution batch normalization fusion module; ReLU-activation module; Conv-convolution module; BN-batch normalization module; deblock-deconvolution activation fusion module; ConvTranspose deconvolution module; FuseConvTransposeBN-deconvolution batch normalization fusion module. DETAILED DESCRIPTION

[0017] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0018] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0019] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0020] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0021] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0022] See also Figure 1 As shown, the defect detection method based on sparse convolution redistribution and detection head fusion includes the following steps:

[0023] Step S110, constructing a PillarNeXt model, which redistributes the sparse convolutional modules of the backbone network and fuses the detection head of the model.

[0024] For the defect detection task, the present invention selects PillarNeXt as the baseline model and improves it, for example, reallocating the sparse convolution modules of the backbone network and the detection head of the fusion model. The improved model is also called ReallocFusePillarNeXt (RFPillarNeXt).

[0025] Typically, the backbone network of PillarNeXt consists of four groups of sparse convolutional modules, and the number of sparse convolutional modules in each group is set to 2. The sparse convolutional modules here are a kind of residual connection structure, which is responsible for extracting the features of two-dimensional pseudo images. Due to the sparsity and irregularity of point clouds, the backbone network with a uniform distribution of sparse convolutional modules may have the defect of insufficient extraction of geometric information of the target. Therefore, in the early stage of extracting cylindrical features, more sparse convolutional modules need to be added to refine the fine-grained point cloud features.

[0026] In order to improve the accuracy of defect detection and take into account the lightweight of the model, in one embodiment of the present invention, the sparse convolution modules of the backbone network are redistributed, and the number of sparse convolution modules with non-uniform distribution is set. For example, taking the input of the backbone network as a reference point, the number of sparse convolution modules in the later group is set to be no greater than the number of sparse convolution modules in the earlier group.

[0027] Figure 2 This is a schematic diagram of the PillarNeXt backbone network sparse convolution module reallocation. Without changing the total number of sparse convolution modules, the number of 4 groups of sparse convolution modules is set to 5, 1, 1, and 1 respectively, which is used to extract the features of the two-dimensional pseudo image multiple times, obtain a sparse feature map that is downsampled 8 times, and convert the generated sparse feature map into a dense feature map. This design can improve the detection accuracy of the model and reduce the size of the model at the same time, making the model more lightweight and occupying less computing resources. It should be noted that according to the requirements of detection accuracy, the number of groups of sparse convolution modules and the number of sparse convolution blocks contained in each group can be flexibly set.

[0028] In order to further improve the speed of defect detection, in another embodiment, a detection head of the PillarNeXt model is integrated, see Figure 3 As shown. PillarNeXt's detection head is composed of Conv layers (or modules), ConvTranspose layers, and BatchNorm2d layers, which are responsible for predicting the three-dimensional XY coordinates of the target center point (or marked as center), the three-dimensional Z coordinates (or marked as center_z), the size of the target (marked as dim), the orientation of the target (marked as rot), and the target category confidence heat map (marked as hm). In the model training stage, the role of the BatchNorm2d layer is to normalize the data, which helps the model converge during the training process. In the model inference stage, the role of the BatchNorm2d layer is to use the trained mean, variance and other parameters to linearly transform the data. The detection head of the fusion model refers to the fusion of the parameters of the BatchNorm2d layer with the parameters of the Conv layer or ConvTranspose layer, making the fused structure equivalent to a new convolutional layer, which can speed up the model inference without reducing the detection accuracy.

[0029] exist Figure 3In the model, the Residual Block is used to improve the learning ability of the model. The shared-conv module contains FuseConvBN (convolution batch normalization fusion module) and ReLU (activation module), which are used to achieve efficient feature extraction and improve model stability and generalization ability. FuseConvBN (convolution batch normalization fusion module) is used to fuse the parameters of the trained Conv module and BN module to speed up the model reasoning efficiency; deblock (deconvolution activation fusion module) contains FuseConvTransposeBN (deconvolution batch normalization fusion module) and ReLU (activation module) to enhance the feature expression ability of the model; FuseConvTransposeBN (deconvolution batch normalization fusion module) fuses the parameters of the trained ConvTranspose module and BN module to improve the model calculation performance; the Conv module contains a two-dimensional convolution module to adjust the number of channels of the output feature.

[0030] Specifically, the RFPillarNeXt model includes a pillar encoder, a backbone network, a neck network, and a detection head. Constructing the improved PillarNeXt model includes the following steps:

[0031] S1, obtain the discrete point cloud information of the target to be detected, and use the pillar encoder to convert the discrete point cloud information into a two-dimensional pseudo image;

[0032] S2, the sparse convolution modules of the backbone network are divided into 4 groups, each group contains 2 sparse convolution modules. Under the premise of ensuring that the total number of sparse convolution modules remains unchanged, the number of sparse convolution modules in the first, second, third and fourth groups is set to 5, 1, 1 and 1 respectively. It is expected that the sparse convolution modules in the early stage will be used to refine the fine-grained point cloud features and fully extract the geometric information of the target. The 4 groups of sparse convolution modules output sparse feature maps that are downsampled 8 times, and the generated sparse feature maps are converted into dense feature maps;

[0033] S3, the neck network is responsible for capturing the spatial information of dense feature maps at different scales and sending the captured information to the detection head;

[0034] S4, the detection head uses a shared_conv module containing convolution operations and a deblock module containing deconvolution operations to convert feature dimensions, and then divides the converted features into 5 branches, each of which outputs the 3D XY coordinates, 3D Z coordinates, target size, target orientation, and target category confidence heat map of the target center point in turn. The shared_conv module, deblock module, and 5 branch modules are composed of Conv or ConvTranspose, BatchNorm2d, and ReLU layers.

[0035] Step S120, training the PillarNeXt model using the public data set.

[0036] The improved PillarNeXt model is trained using a 3D object detection dataset until the set loss target is met. For example, the public dataset KITTI can be used for model training, and the effectiveness of the present invention is demonstrated through comparative experiments and ablation experiments.

[0037] Step S130: applying the trained PillarNeXt model to defect detection of the target to be detected.

[0038] After the model is trained, it can be applied to defect detection of the target to be detected. For example, the model application process includes: obtaining discrete point cloud information of the target to be detected; inputting the discrete point cloud information into the trained PillarNeXt model to obtain detection results such as defect location and defect size.

[0039] In the model application stage, the present invention integrates the trained mean, variance, weight and bias parameters of Conv or ConvTranspose, and makes the Conv or ConvTranspose and BatchNorm2d structures equivalent to a new convolutional layer, so as to achieve faster output of defect location information.

[0040] Specifically, the weights and biases of the trained Conv or ConvTranspose structure are known to be w and b respectively. For the input feature map X, the output of the Conv or ConvTranspose structure is:

[0041] Y1=wX+b (1)

[0042] After passing through the BatchNorm2d structure again, the output is

[0043]

[0044] Where μ is the trained mean in each batch, σ 2 is the trained variance in each batch, ε is a constant used to avoid division by zero, and α and β are the trained parameters. Substituting (1) into (2), we get:

[0045]

[0046] make:

[0047]

[0048] Substituting into formula (3), we can obtain:

[0049] Y2=w′X+b′(6)

[0050] Thus, the fused convolutional layer is obtained. The above formula represents the output of the fused convolutional layer after the input feature map X, with the weight w' and the bias b'.

[0051] The fusion detection head can reduce the amount of calculation of the detection head by making convolution operations and batch normalization operations equivalent to new convolution operations, thereby improving the model detection speed, accelerating model reasoning, reducing the number of model parameters, and achieving model lightweight, which is conducive to the deployment and implementation of the model in the field of industrial defect detection.

[0052] The model training process of the present invention can be performed offline on a server or in the cloud, and real-time defect detection can be achieved by embedding the trained model into an electronic device. The electronic device can be a terminal device or a server, and the terminal device includes any terminal device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a point of sale (POS), a car computer, a smart wearable device (smart watch, virtual reality glasses, virtual reality helmet, etc.). The server includes but is not limited to an application server or a Web server, and can be an independent server or a cluster server or a cloud server, etc.

[0053] In summary, in the prior art, for point cloud 3D target detection tasks, due to the obvious differences in the sizes of different types of targets, the detection algorithm needs to be able to extract rich multi-scale and multi-level feature information to achieve a more accurate detection process. However, due to the needs of actual deployment tasks, it is difficult to use large-size convolution kernels in the backbone network to extract large-scale semantic features. In addition, for distant targets, the number of point clouds contained in them is very small, and after multi-layer downsampling, too many features are lost, thereby reducing the final performance of the detection algorithm. The present invention takes into account the characteristics of point cloud and defect detection, selects the PillarNeXt model as the defect detection model, and redesigns the non-uniform distribution method of the sparse convolution module and the fusion method of the detection head, taking into account the accuracy and efficiency of defect detection, and is particularly suitable for defect detection based on 3D vision, thereby improving the generalization ability of the model, and can be effectively applied to detect targets of different types, sizes, and point cloud features.

[0054] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0055] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0056] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0057] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, Python, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0058] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0059] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0060] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0061] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0062] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A defect detection method based on sparse convolution redistribution and detection head fusion, comprising the following steps: Obtain discrete point cloud information of the target to be detected; Inputting the discrete point cloud information into the trained PillarNeXt model to obtain defect detection results; The PillarNeXt model includes a pillar encoder, a backbone network, a neck network and a detection head, wherein the pillar encoder is used to convert the discrete point cloud information into a two-dimensional pseudo image, the backbone network includes multiple groups of sparse convolution modules with a residual connection structure, which are used to extract the point cloud features of the two-dimensional pseudo image, obtain a downsampled sparse feature map, and convert the sparse feature map into a dense feature map, the neck network is used to capture the spatial information of the dense feature map at different scales, and the detection head is used to predict the three-dimensional XY coordinates, three-dimensional Z coordinates, size, orientation and target category confidence heat map of the center point of the target to be detected; The number of sparse convolution modules in each group is unevenly distributed, and taking the input of the backbone network as a reference point, the number of sparse convolution modules in the later groups is not greater than the number of sparse convolution modules in the earlier groups.

2. The method according to claim 1, characterized in that The number of groups of the multiple groups of sparse convolution modules is 4, the number of the first group of sparse convolution modules is set to 5, the number of the second group of sparse convolution modules is set to 1, the number of the third group of sparse convolution modules is set to 1, and the number of the fourth group of sparse convolution modules is set to 1. After processing by the multiple groups of sparse convolution modules, a sparse feature map downsampled 8 times is obtained, and then the generated sparse feature map is converted into a dense feature map.

3. The method according to claim 2, characterized in that The detection head uses a shared convolution module containing convolution operations and a deconvolution activation fusion module containing deconvolution operations to convert feature dimensions, and divides the converted features into five branches, each of which outputs the three-dimensional XY coordinates, three-dimensional Z coordinates, defect target size, defect target orientation and defect target category confidence heat map of the defect target center point.

4. The method according to claim 3, characterized in that The shared convolution module includes a first convolution module and a first activation module ReLU, the deconvolution activation fusion module includes a second convolution module and a second activation module ReLU, and the five branches all include a convolution module, a BatchNorm2d module and an activation ReLU module.

5. The method according to claim 1, characterized in that The PillarNeXt model is trained using the public dataset KITTI.

6. The method according to claim 3, characterized in that The detection head realizes defect detection by fusing the mean, variance, weight and bias parameters obtained through the training of the shared convolution module, the deconvolution activation fusion module and the five branches of the PillarNeXt model, which is expressed as: The weights and biases of the trained deconvolution module are w and b respectively. For the input feature map X, the output of the shared convolution module structure is: Y1=wX+b After the BatchNorm2d module, the output is: Then, we get: make: Then the fusion output of the detection head is expressed as: Y2=w′X+b′ Among them, μ is the mean obtained by training in each batch, σ 2 is the variance obtained through training within each batch, ε is a constant, and α and β are parameters obtained through training.

7. The method according to claim 1, characterized in that The sparse feature map is 8 times downsampled for the two-dimensional pseudo image features.

8. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images