Workpiece surface defect detection model acquisition method and system

Abstract

The present application relates to a method and system for acquiring a workpiece surface defect detection model, wherein the method comprises: training a defect detection model according to an RGB image and a point cloud image of a target workpiece, and stopping iteration in response to satisfying a preset iteration condition to obtain a defect detection model; the defect detection model comprises an RGB branch, a point cloud branch, a fusion module and an output module, the RGB branch is used to extract RGB image features through a multi-layer network, the point cloud branch is used to extract point cloud image features through a multi-layer network, the fusion module is used to fuse RGB image features and point cloud image features to obtain target image features, and comprises a gradient attention module, the gradient attention module is used to alternately receive current layer extraction features sent by the RGB branch or the point cloud branch, process the current layer features to obtain gradient enhancement features, and send them to another branch for feature fusion to obtain target image features, and the output module is used to detect defect areas according to the target image features and output detection results.

Description

Technical Field

[0001] The present application relates to the field of defect detection, and in particular to a method and system for acquiring a workpiece surface defect detection model. Background Art

[0002] Surface defect detection of workpieces is an important way to ensure the quality of workpiece production. With the development of computer technology and 3D cameras, existing technologies have formed machine vision detection methods based on 2D images and surface defect detection methods based on 3D point clouds.

[0003] Machine vision surface defect detection methods based on 2D images usually use traditional template matching or deep learning methods to detect defects. However, this method is limited by the characteristics of the 2D image itself and can only detect visible defects on the plane image. It has poor performance in detecting three-dimensional defects. The surface defect detection method based on 3D point cloud mainly obtains the three-dimensional point cloud data of the product through 3D imaging technology, captures the geometric information of the workpiece surface according to the point cloud depth information, and realizes defect detection on the surface of the object. Since 3D defect visual detection has high requirements on data quality, it is necessary to perform complex and tedious preprocessing on the acquired 3D data. The preprocessing methods for point clouds acquired in different environments vary greatly, and it is impossible to detect defects on the surface without depth differences, such as dirt.

[0004] The existing workpiece surface defect detection methods have the problem of missed detection, and the defect detection effect needs to be improved. Summary of the invention

[0005] The embodiments of the present application provide a method and system for acquiring a workpiece surface defect detection model, so as to at least solve the problem of missed detection in the workpiece surface defect detection method in the related art.

[0006] In a first aspect, an embodiment of the present application provides a method for acquiring a workpiece surface defect detection model, comprising:

[0007] Obtain RGB images and point cloud images of the target workpiece;

[0008] The defect detection model is trained according to the RGB image and the point cloud image, and when a preset iteration condition is met, the iteration is stopped to obtain the defect detection model;

[0009] The defect detection model includes an RGB branch, a point cloud branch, a fusion module and an output module.

[0010] The RGB branch is used to extract RGB image features of the RGB image through a multi-layer network.

[0011] The point cloud branch is used to extract point cloud image features of the point cloud image through a multi-layer network.

[0012] The fusion module is used to fuse the RGB image features and the point cloud image features to obtain the target image features, and specifically includes a gradient attention module, the gradient attention module is used to alternately receive the current layer extraction features sent by the RGB branch or the point cloud branch, process the current layer features to obtain gradient enhancement features, and send the gradient enhancement features to another branch for feature fusion to obtain the target image features.

[0013] The output module is used to detect defect areas according to the target image features and output detection results.

[0014] In one embodiment, the gradient attention module includes a channel attention module and a spatial attention module.

[0015] In one embodiment, the defect detection model further includes a splicing module, and the splicing module includes an RGB splicing module and a point cloud splicing module.

[0016] The RGB splicing module is connected to each intermediate layer in the RGB branch, and is used to obtain the features of each intermediate layer, splice the features, and transmit the spliced ​​features to the CBR module of the RGB branch;

[0017] The point cloud stitching module is connected to each intermediate layer in the point cloud branch, and is used to obtain the features of each intermediate layer, stitch the features, and transmit the stitched features to the CBR module of the point cloud branch.

[0018] In one embodiment, training a defect detection model according to the RGB image and the point cloud image includes:

[0019] The defect detection model is trained using cross entropy loss. The loss during training includes RGB branch loss, point cloud branch loss, and mixed loss of RGB branch and point cloud branch. The overall loss of the defect detection model is expressed as:

[0020]

[0021] Among them, L is the overall loss, L 3D represents the point cloud branch loss, L 2D represents the RGB branch loss, L main represents the hybrid loss, λ1 and λ2 are loss weights.

[0022] In one embodiment, acquiring the RGB image and the point cloud image of the target workpiece includes:

[0023] Acquire an image of a target workpiece, and crop the image according to a position of the workpiece in the image to obtain a target image, wherein the target image includes an RGB image and a point cloud image;

[0024] Flipping, blurring, and modifying brightness and contrast of the RGB image;

[0025] The point cloud image is flipped and blurred.

[0026] In a second aspect, an embodiment of the present application provides a method for detecting surface defects of a workpiece, comprising:

[0027] Acquire a target workpiece image, the target workpiece image including an RGB image and a point cloud image, and crop the target workpiece image according to the position of the workpiece in the target workpiece image to obtain an image to be detected;

[0028] The image to be detected is processed by the defect detection model described in the first aspect to obtain a defect area.

[0029] In one embodiment, the defect detection model is trained using cross entropy loss during detection, and the loss during detection includes a mixed loss of an RGB branch and a point cloud branch.

[0030] In a third aspect, an embodiment of the present application provides a workpiece surface defect detection model acquisition system, comprising:

[0031] Acquisition module: used to acquire RGB images and point cloud images of the target workpiece;

[0032] Training module: used for training the defect detection model according to the RGB image and the point cloud image, and stopping the iteration in response to meeting the preset iteration condition to obtain the defect detection model;

[0033] The defect detection model includes an RGB branch, a point cloud branch, a fusion module and an output module.

[0034] The RGB branch is used to extract RGB image features of the RGB image through a multi-layer network.

[0035] The point cloud branch is used to extract point cloud image features of the point cloud image through a multi-layer network.

[0036] The fusion module is used to fuse the RGB image features and the point cloud image features to obtain the target image features, and specifically includes a gradient attention submodule, the gradient attention submodule is used to alternately receive the current layer extraction features sent by the RGB branch or the point cloud branch, process the current layer features to obtain gradient enhancement features, and send the gradient enhancement features to another branch for feature fusion to obtain the target image features.

[0037] The output module is used to detect defect areas according to the target image features and output detection results.

[0038] In a fourth aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for acquiring a workpiece surface defect detection model as described in the first aspect and the method for detecting a workpiece surface defect as described in the second aspect are implemented.

[0039] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the workpiece surface defect detection model acquisition method as described in the first aspect and the workpiece surface defect detection method as described in the second aspect.

[0040] The embodiments of the present application provide a method and system for acquiring a workpiece surface defect detection model, which have at least the following technical effects.

[0041] This application adopts a combination of 3D point cloud and 2D-RGB image detection to effectively expand the types of defect detection. Based on 3D point cloud and RGB image, gradient attention module and dual-branch interactive fusion module, it fully combines the characteristics of multimodal data to form a learning model in which multimodal information promotes each other, maximizes information utilization, avoids missed detection problems during defect detection, and improves the accuracy of defect detection models.

[0042] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0044] Figure 1 is a flow chart of a method for acquiring a workpiece surface defect detection model shown in an embodiment of the present application;

[0045] Figure 2 is a structural block diagram of a defect detection model according to an exemplary embodiment;

[0046] Figure 3 is a schematic diagram of the structure of a gradient attention module according to an exemplary embodiment;

[0047] Figure 4 is a structural diagram of a channel attention module according to an exemplary embodiment;

[0048] Figure 5is a schematic diagram of the structure of a spatial attention module according to an exemplary embodiment;

[0049] Figure 6 is a flow chart of a method for detecting surface defects of a workpiece according to an embodiment of the present application;

[0050] Figure 7 It is a structural block diagram of a workpiece surface defect detection model acquisition system provided according to an embodiment of the present application;

[0051] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0053] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0054] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0055] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantity limitation, and may indicate the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0056] In a first aspect, an embodiment of the present application provides a method for acquiring a workpiece surface defect detection model. Figure 1 is a flow chart of a method for acquiring a workpiece surface defect detection model shown in an embodiment of the present application. Figure 1 As shown, the method includes:

[0057] Step S101, acquiring an RGB image and a point cloud image of a target workpiece.

[0058] Optionally, obtain multimodal data of the target workpiece, including 2D-RGB images and 3D point cloud images, to avoid the problem that single 2D image detection or 3D point cloud detection cannot cover various defect types. And perform preprocessing methods such as cropping and data expansion on RGB images and point cloud images.

[0059] In one example, step S101 includes:

[0060] The image of the target workpiece is acquired, and the image is cropped according to the position of the workpiece in the image to obtain the target image, which includes an RGB image and a point cloud image; the RGB image is flipped, blurred, and the brightness and contrast are modified; the point cloud image is flipped and blurred.

[0061] Optionally, since the resolution of the captured image is high and the workpiece area only occupies the middle area of ​​the image, the workpiece area in the image is cut out to reduce the amount of data calculation during subsequent detection. Specifically, a rectangular workpiece area is cut out according to the position of the workpiece in the point cloud, and the workpiece area in the RGB image is cut out accordingly, so that the spatial range of the RGB image corresponds to the spatial range of the point cloud image.

[0062] The cropped images are augmented. The data augmentation methods for RGB images include image flipping, blurring, and adjusting the brightness and contrast of the image. The data augmentation methods for point cloud images include image flipping and blurring. In this way, the quantity and diversity of training data are increased, so that the model can detect surface defects of workpieces at various angles, resolutions, and brightness. This facilitates the subsequent improvement of the accuracy of the defect detection model.

[0063] Step S102, training the defect detection model according to the RGB image and the point cloud image, and stopping the iteration in response to meeting a preset iteration condition to obtain the defect detection model.

[0064] Figure 2 is a structural block diagram of a defect detection model according to an exemplary embodiment. Figure 2 As shown, the defect detection model includes RGB branch, point cloud branch, fusion module and output module.

[0065] The RGB branch is used to extract RGB image features of RGB images through a multi-layer network.

[0066] The point cloud branch is used to extract point cloud image features of point cloud images through a multi-layer network.

[0067] The fusion module is used to fuse RGB image features and point cloud image features to obtain target image features, and specifically includes a gradient attention module. The gradient attention module is used to alternately receive the current layer extraction features sent by the RGB branch or the point cloud branch, process the current layer features to obtain gradient enhanced features, and send the gradient enhanced features to another branch for feature fusion to obtain the target image features.

[0068] The output module is used to detect defect areas according to target image features and output the detection results.

[0069] Optionally, the preset iteration condition includes that the training loss is less than the preset loss or reaches the preset number of iterations. Among them, the preset loss and the preset number of iterations can be set according to actual needs. The present application adopts a dual-branch structure to construct a defect detection model, in which each branch serves as a feature extractor network. The feature extractor network can select a deep learning network such as a U-Net network or a Segformer. The present application uses the Mixformer network, the backbone of the Segformer network, as the feature extractor network of each branch.

[0070] like Figure 2 As shown, the fusion module is used to receive the features extracted by the convolution layer alternately sent by the RGB branch or the point cloud branch through the gradient attention module. The gradient attention module processes the received features to obtain gradient attention features, and fuses the obtained gradient attention features with the features extracted by the convolution layer of the other branch to obtain gradient enhanced features, which are transmitted to the next convolution layer for processing until the convolution ends.

[0071] In this way, the problem of missed detection in the prior art is solved. Specifically, the surface defects of workpieces usually present unevenness in shape, size, distribution and appearance, and may be linear, dot-shaped, block-shaped or irregular in shape, and the size ranges from micrometers to centimeters. A single 2D image detection or 3D point cloud detection cannot cover various defect types. However, whether it is a 2D image or a 3D point cloud image, the defective area always shows gradient changes and differs from the surrounding area. Therefore, the present application adopts a combination of 3D point cloud and 2D-RGB image detection to effectively expand the types of defect detection. A dual-branch network architecture based on mutual supervision of gradient mutation information of 3D point clouds and RGB images is constructed, which fully combines the characteristics of multimodal data to form a learning model in which multimodal information promotes each other, and maximizes information utilization to avoid missed detection problems during defect detection and improve the accuracy of defect detection models.

[0072] In one example, the gradient attention module includes a channel attention module and a spatial attention module.

[0073] Optionally, Figure 3 is a schematic diagram of the structure of a gradient attention module according to an exemplary embodiment. Figure 3 As shown, the gradient attention module combines the channel attention module and the spatial attention module to improve the performance of the model.

[0074] in, Figure 4 is a schematic diagram of the structure of a channel attention module according to an exemplary embodiment. Figure 4As shown in the figure, the channel attention module first transforms the dimension of the input features, then processes the transformed features through a multilayer perceptron (MLP), then transforms the output of the MLP back to the original dimension, and finally maps it through a Sigmoid function to obtain the channel attention weight. The attention weight is multiplied by the channel corresponding to the input feature to obtain the channel attention feature.

[0075] Figure 5 is a schematic diagram of the structure of a spatial attention module according to an exemplary embodiment. Figure 5 As shown in the figure, the spatial attention module first reduces the number of channels through a convolution layer with a convolution kernel of 7 to reduce the amount of calculation. Then, a convolution layer with a convolution kernel of 7 is used to increase the number of channels to keep the number of channels consistent. Finally, the spatial attention weight is obtained by mapping through the Sigmoid function. The spatial attention weight is multiplied by the corresponding pixel of the input feature to obtain the spatial attention feature.

[0076] In this way, the defect area always shows gradient changes and is different from the surrounding areas. By combining the gradient attention module of channel attention and spatial attention, the utilization rate of gradient information is improved, so that the defect detection model pays more attention to the gradient mutation area, thereby improving the accuracy of the defect detection model.

[0077] In one example, if Figure 2 As shown, the defect detection model also includes a stitching module, and the stitching module includes an RGB stitching module and a point cloud stitching module.

[0078] The RGB splicing module is connected to each intermediate layer in the RGB branch, and is used to obtain the features of each intermediate layer, splice the features, and transmit the spliced ​​features to the CBR module of the RGB branch. Optionally, the CBR module is a network structure formed by a combination of a convolutional layer, a batch normalization layer, and a Relu.

[0079] The point cloud stitching module is connected to each intermediate layer in the point cloud branch to obtain the features of each intermediate layer, stitch the features, and transmit the stitched features to the CBR module of the point cloud branch.

[0080] In this way, the multi-scale features of point cloud images and RGB images are obtained and fused to solve the problems of loss and insufficient attention of tiny defects in the process of network information transmission. This makes the defect detection model pay more attention to the location of tiny defects, solves the problem of missed detection of tiny defects, and improves the accuracy of the defect detection model.

[0081] In one example, step S102 includes:

[0082] The defect detection model is trained using cross entropy loss. The losses during training include RGB branch loss, point cloud branch loss, and mixed loss of RGB branch and point cloud branch. The overall loss of the defect detection model is expressed as:

[0083]

[0084] Among them, L is the overall loss, L 3D represents the point cloud branch loss, L 2D represents the RGB branch loss, L main represents the mixed loss, λ1 and λ2 are the loss weights.

[0085] Optionally, loss weights λ1 and λ2 are balanced loss weights, which are determined according to the actual application scenario. In this application, the values ​​of λ1 and λ2 are set to 0.4 through experiments. And during training, the real defect area mask is scaled to one-fourth of the size for supervised training, and the defect area size output by the defect detection model is also one-fourth of the input image. This application adopts a multi-head loss supervision method. On the basis of the dual-branch backbone network loss supervision, it adds detection loss supervision of the two branches to form a mechanism of mutual supervision between 3D point clouds and 2D-RGB images, making full use of training image features, improving the accuracy of the defect detection model, and accelerating the convergence speed of the network.

[0086] In summary, this application adopts a combination of 3D point cloud and 2D-RGB image detection to effectively expand the types of defect detection. Based on 3D point cloud and RGB images, gradient attention module and dual-branch interactive fusion module, the characteristics of multimodal data are fully combined to form a learning model in which multimodal information promotes each other, and maximize information utilization to avoid missed detection problems during defect detection and improve the accuracy of the defect detection model. And based on the multi-scale features of point cloud images and RGB images, the problems of loss and insufficient attention of tiny defects in the process of network information transmission are solved, so that the defect detection model pays more attention to the location of tiny defects, solves the problem of missed detection of tiny defects, and improves the accuracy of the defect detection model.

[0087] In a second aspect, the present application provides a method for detecting surface defects of a workpiece. Figure 6 is a flow chart of a method for detecting surface defects of a workpiece according to an embodiment of the present application. Figure 6 As shown, the method includes:

[0088] Step S201 : acquiring a target workpiece image, the target workpiece image including an RGB image and a point cloud image, and cropping the target workpiece image according to the position of the workpiece in the target workpiece image to obtain an image to be detected.

[0089] Step S202: Process the image to be detected by the defect detection model of the first aspect to obtain a defect area.

[0090] In one example, during detection, the defect detection model is trained using cross entropy loss, and the loss during detection includes a mixed loss of an RGB branch and a point cloud branch.

[0091] Optionally, when training the defect detection model, a multi-head loss supervision method is used, but in the detection application, only the main detection head is used to output the defect area. In addition, during training, the real defect area mask is scaled to one-fourth the size for supervised training, and the defect area size output by the defect detection model is also one-fourth of the input image during output.

[0092] In summary, this application adopts a combination of 3D point cloud and 2D-RGB image detection to effectively expand the types of defect detection. Based on 3D point cloud and RGB images, gradient attention module and dual-branch interactive fusion module, the characteristics of multimodal data are fully combined to form a learning model in which multimodal information promotes each other, and maximize information utilization to avoid missed detection problems during defect detection and improve the accuracy of the defect detection model. And based on the multi-scale features of point cloud images and RGB images, the problems of loss and insufficient attention of tiny defects in the process of network information transmission are solved, so that the defect detection model pays more attention to the location of tiny defects, solves the problem of missed detection of tiny defects, and improves the accuracy of the defect detection model.

[0093] In a third aspect, the present application provides a system for acquiring a workpiece surface defect detection model. Figure 7 is a structural block diagram of a workpiece surface defect detection model acquisition system provided according to an embodiment of the present application, such as Figure 7 As shown, the system includes:

[0094] Acquisition module 100: used to acquire the RGB image and point cloud image of the target workpiece.

[0095] Training module 200: used to train the defect detection model according to the RGB image and the point cloud image, and stop the iteration in response to meeting the preset iteration condition to obtain the defect detection model.

[0096] The defect detection model includes RGB branch, point cloud branch, fusion module and output module.

[0097] The RGB branch is used to extract RGB image features of RGB images through a multi-layer network.

[0098] The point cloud branch is used to extract point cloud image features of point cloud images through a multi-layer network.

[0099] The fusion module is used to fuse RGB image features and point cloud image features to obtain target image features, and specifically includes a gradient attention submodule, which is used to alternately receive the current layer extraction features sent by the RGB branch or the point cloud branch, process the current layer features to obtain gradient enhanced features, and send the gradient enhanced features to another branch for feature fusion to obtain the target image features.

[0100] The output module is used to detect defect areas according to target image features and output the detection results.

[0101] In one example, the gradient attention module in the training module 200 includes a channel attention module and a spatial attention module.

[0102] In one example, the defect detection model in the training module 200 further includes a splicing module, and the splicing module includes an RGB splicing module and a point cloud splicing module.

[0103] The RGB splicing module is connected to each intermediate layer in the RGB branch to obtain the features of each intermediate layer, splice the features, and transmit the spliced ​​features to the CBR module of the RGB branch.

[0104] The point cloud stitching module is connected to each intermediate layer in the point cloud branch to obtain the features of each intermediate layer, stitch the features, and transmit the stitched features to the CBR module of the point cloud branch.

[0105] In one example, the training module 200 includes:

[0106] The defect detection model is trained using cross entropy loss. The losses during training include RGB branch loss, point cloud branch loss, and mixed loss of RGB branch and point cloud branch. The overall loss of the defect detection model is expressed as:

[0107]

[0108] Among them, L is the overall loss, L 3D represents the point cloud branch loss, L 2D represents the RGB branch loss, L main represents the mixed loss, λ1 and λ2 are the loss weights.

[0109] In one example, the acquisition module 100 includes: acquiring an image of a target workpiece, cropping the image according to the position of the workpiece in the image to obtain a target image, wherein the target image includes an RGB image and a point cloud image; flipping, blurring, and modifying brightness and contrast of the RGB image; and flipping and blurring the point cloud image.

[0110] In summary, the present application adopts a combination of 3D point cloud and 2D-RGB image detection to effectively expand the types of defect detection. Based on the gradient detection model in the training module 200, through 3D point cloud and RGB image, gradient attention module and dual-branch interactive fusion module, the characteristics of multimodal data are fully combined to form a learning model in which multimodal information promotes each other, and maximize information utilization to avoid missed detection problems during defect detection and improve the accuracy of the defect detection model. And based on the multi-scale features of point cloud images and RGB images, the problems of loss and insufficient attention of tiny defects in the process of network information transmission are solved, so that the defect detection model pays more attention to the location of tiny defects, solves the problem of missed detection of tiny defects, and improves the accuracy of the defect detection model.

[0111] In a fourth aspect, an embodiment of the present application provides an electronic device, Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for acquiring a workpiece surface defect detection model provided in the first aspect and the method for detecting workpiece surface defects provided in the second aspect are implemented. Figure 8 The electronic device 60 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0112] The electronic device 60 may be in the form of a general-purpose computing device, for example, it may be a server device. The components of the electronic device 60 may include but are not limited to: at least one processor 61, at least one memory 62, and a bus 63 connecting different system components (including the memory 62 and the processor 61).

[0113] The bus 63 includes a data bus, an address bus, and a control bus.

[0114] The memory 62 may include a volatile memory, such as a random access memory (RAM) 621 and / or a cache memory 622 , and may further include a read-only memory (ROM) 623 .

[0115] The memory 62 may also include a program / utility 625 having a set (at least one) of program modules 624, such program modules 624 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0116] The processor 61 executes various functional applications and data processing by running the computer programs stored in the memory 62, such as the workpiece surface defect detection model acquisition method provided in the first aspect of the present application and the workpiece surface defect detection method provided in the second aspect.

[0117] The electronic device 60 may also communicate with one or more external devices 64 (e.g., keyboards, pointing devices, etc.). Such communication may be performed via an input / output (I / O) interface 65. Furthermore, the model-generated device 60 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 66. As shown, the network adapter 66 communicates with other modules of the model-generated device 60 via a bus 63. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the model-generated device 60, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems, etc.

[0118] It should be noted that although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.

[0119] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the workpiece surface defect detection model acquisition method provided in the first aspect and the workpiece surface defect detection method provided in the second aspect.

[0120] The readable storage medium may include but is not limited to: a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device or any suitable combination of the above.

[0121] In a possible embodiment, the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of the workpiece surface defect detection model acquisition method provided by the first aspect and the workpiece surface defect detection method provided by the second aspect.

[0122] The program code for executing the present invention may be written in any combination of one or more programming languages, and may be executed entirely on a user device, partially on a user device, as an independent software package, partially on a user device and partially on a remote device, or entirely on a remote device.

[0123] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for acquiring a workpiece surface defect detection model, characterized in that: include: Obtain RGB image and point cloud image of the target workpiece; The defect detection model is trained according to the RGB image and the point cloud image, and when a preset iteration condition is met, the iteration is stopped to obtain the defect detection model; The defect detection model includes an RGB branch, a point cloud branch, a fusion module and an output module. The RGB branch is used to extract RGB image features of the RGB image through a multi-layer network. The point cloud branch is used to extract the point cloud image features of the point cloud image through a multi-layer network, and the multi-layer network extracts the features of each branch with the backbone Mixformer network of the Segformer network. The fusion module is used to fuse the RGB image features and the point cloud image features to obtain the target image features, and specifically includes a gradient attention module, the gradient attention module is used to alternately receive the current layer extraction features sent by the RGB branch or the point cloud branch, process the current layer features to obtain gradient enhancement features, and send the gradient enhancement features to another branch for feature fusion, and transmit them to the next convolution layer for processing until the convolution ends to obtain the target image features. The output module is used to detect defect areas according to the target image features and output detection results.

2. A method for acquiring a workpiece surface defect detection model according to claim 1, characterized in that: The gradient attention module includes a channel attention module and a spatial attention module.

3. The method for acquiring a workpiece surface defect detection model according to claim 1, characterized in that: The defect detection model also includes a splicing module, which includes an RGB splicing module and a point cloud splicing module. The RGB splicing module is connected to each intermediate layer in the RGB branch, and is used to obtain the features of each intermediate layer, splice the features, and transmit the spliced ​​features to the CBR module of the RGB branch; The point cloud stitching module is connected to each intermediate layer in the point cloud branch, and is used to obtain the features of each intermediate layer, stitch the features, and transmit the stitched features to the CBR module of the point cloud branch.

4. The method for acquiring a workpiece surface defect detection model according to claim 1, characterized in that: The defect detection model is trained according to the RGB image and the point cloud image, including: The defect detection model is trained using cross entropy loss. The loss during training includes RGB branch loss, point cloud branch loss, and mixed loss of RGB branch and point cloud branch. The overall loss of the defect detection model is expressed as: Among them, L is the overall loss, L 3D represents the point cloud branch loss, L 2D represents the RGB branch loss, L main represents the hybrid loss, λ1 and λ2 are loss weights.

5. The method for acquiring a workpiece surface defect detection model according to claim 1, characterized in that: The step of acquiring the RGB image and the point cloud image of the target workpiece includes: Acquire an image of a target workpiece, and crop the image according to a position of the workpiece in the image to obtain a target image, wherein the target image includes an RGB image and a point cloud image; Flipping, blurring, and modifying brightness and contrast of the RGB image; The point cloud image is flipped and blurred.

6. A method for detecting surface defects of a workpiece, characterized in that: include: Acquire a target workpiece image, wherein the target workpiece image includes an RGB image and a point cloud image, and crop the target workpiece image according to the position of the workpiece in the target workpiece image to obtain an image to be detected; The image to be detected is processed by the defect detection model described in any one of claims 1 to 5 to obtain a defect area.

7. A workpiece surface defect detection method according to claim 6, characterized in that: During detection, the defect detection model is trained using cross entropy loss, and the loss during detection includes a mixed loss of an RGB branch and a point cloud branch.

8. A workpiece surface defect detection model acquisition system, characterized in that: include: Acquisition module: used to acquire RGB images and point cloud images of the target workpiece; Training module: used for training the defect detection model according to the RGB image and the point cloud image, and stopping the iteration in response to meeting the preset iteration condition to obtain the defect detection model; The defect detection model includes an RGB branch, a point cloud branch, a fusion module and an output module. The RGB branch is used to extract RGB image features of the RGB image through a multi-layer network. The point cloud branch is used to extract the point cloud image features of the point cloud image through a multi-layer network, and the multi-layer network extracts the features of each branch with the backbone Mixformer network of the Segformer network. The fusion module is used to fuse the RGB image features and the point cloud image features to obtain the target image features, and specifically includes a gradient attention submodule, the gradient attention submodule is used to alternately receive the current layer extraction features sent by the RGB branch or the point cloud branch, process the current layer features to obtain gradient enhancement features, and send the gradient enhancement features to another branch for feature fusion, and transmit them to the next convolution layer for processing until the convolution ends to obtain the target image features. The output module is used to detect defect areas according to the target image features and output detection results.

9. An electronic device, characterized in that: include Memory, processor, and A computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for acquiring a workpiece surface defect detection model as described in any one of claims 1 to 5 and the method for detecting a workpiece surface defect as described in any one of claims 6 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for acquiring a workpiece surface defect detection model according to any one of claims 1 to 5 and the method for detecting a workpiece surface defect according to any one of claims 6 to 7 are implemented.

Images