Defect detection method, device and equipment based on few-sample learning and medium

Abstract

The invention discloses a defect detection method, device and equipment based on few-sample learning and a medium, and relates to the technical field of defect detection.The method comprises the steps that an original workpiece image is preprocessed, and a preprocessed workpiece image is obtained; workpiece key points in the preprocessed workpiece image are detected through a pre-trained first key point detection model; based on the workpiece key point and a preset standard workpiece key point, performing image alignment processing on the preprocessed workpiece image to obtain a calibrated workpiece image; detecting defect part key points and defect type text description information in the calibration workpiece image through a pre-trained second key point detection model; and through a pre-trained image segmentation large model, defect segmentation is carried out on the calibration workpiece image by using the defect part key point and the defect type text description information, and a defect segmentation result is obtained. According to the method, the dependence on training data is small, and the accuracy of workpiece detection is extremely high.

Description

Technical Field

[0001] The present invention relates to the field of defect detection technology, and in particular to a defect detection method, device, equipment and medium based on few-sample learning. Background Art

[0002] In the production and manufacturing of precision components, defects such as holes, scratches, spots, etc. often appear on the surface of steel sheets and PCB boards due to production equipment and human factors. In order to achieve quality control of manufactured products, manual visual quality inspection is often required, which is inefficient and greatly affects the speed of production delivery.

[0003] In order to improve the quality and speed of quality inspection of precision parts such as steel sheets, existing technologies propose to use machine vision to perform edge detection on large steel strip areas, and calculate the edge offset angle through a sliding window to locate the edge crack position of the steel; or use deep learning methods to locate defects, and classify defects by extracting features from the defect ROI (region of interest) through metric learning; or use a solution that combines machine vision and deep learning to extract features of the surface image of the wire rope, and analyze its health status through machine learning methods.

[0004] The above inventions can replace manual visual quality inspection and improve the efficiency of quality inspection. However, machine vision often relies on manual experience to extract the features of the target to be inspected, and its versatility and transferability are relatively poor; the solution of extracting features and locating anomalies using deep learning often requires a large amount of data for training. The existing technology also improves the data dependency problem through transfer learning, data enhancement and other solutions, but there is a certain gap between the pseudo data and the real data of the above solution, which affects the accuracy of quality inspection. Summary of the invention

[0005] The embodiments of the present invention provide a defect detection method, apparatus, computer equipment and storage medium based on few-sample learning, aiming to solve the problem that existing machine vision defect detection solutions rely on large amounts of training data, are inefficient and have low accuracy.

[0006] In a first aspect, an embodiment of the present invention provides a defect detection method based on few-sample learning, which includes: Acquire an original workpiece image of the target workpiece, and preprocess the original workpiece image to obtain a preprocessed workpiece image; Detecting workpiece key points in the preprocessed workpiece image by using a pre-trained first key point detection model; Based on the workpiece key points and preset standard workpiece key points, performing image alignment processing on the pre-processed workpiece image to obtain a calibrated workpiece image; Detecting the key points of defective parts and text description information of defect types in the calibration workpiece image by using a pre-trained second key point detection model; By using the pre-trained image segmentation model and the key points of the defect parts and the text description information of the defect types, defect segmentation is performed on the calibration workpiece image to obtain a defect segmentation result.

[0007] A further technical solution is that the original workpiece image is preprocessed to obtain a preprocessed workpiece image, comprising: The pre-processed workpiece image is obtained by performing histogram equalization processing on the original workpiece image.

[0008] A further technical solution is that, based on the workpiece key points and the preset standard workpiece key points, the pre-processed workpiece image is subjected to image alignment processing to obtain a calibrated workpiece image, including: Based on the workpiece key points and preset standard workpiece key points, obtaining an affine transformation matrix; The pre-processed workpiece image is transformed into a calibration workpiece image based on the affine transformation matrix.

[0009] A further technical solution is that the pre-processed workpiece image is transformed into a calibration workpiece image based on the affine transformation matrix, comprising: All the pixels in the preprocessed workpiece image are affine transformed by the affine transformation matrix to obtain target pixels; The calibration workpiece image is formed by all the target pixel points.

[0010] A further technical solution is that the pre-trained image segmentation model is used to perform defect segmentation on the calibration workpiece image using the key points of the defect location and the text description information of the defect type to obtain a defect segmentation result, including: Inputting the calibration workpiece image into the image encoder of the image segmentation model to obtain an image feature vector; Inputting the key points of the defect part and the text description information of the defect type into the prompt encoder of the image segmentation model to obtain a point prompt feature vector and a text prompt feature vector; The image feature vector, the point prompt feature vector and the text prompt feature vector are input into the mask decoder of the image segmentation model to obtain a defect segmentation mask map and confidence.

[0011] A further technical solution is that the pre-trained image segmentation model is used to perform defect segmentation on the calibration workpiece image using the key points of the defect location and the text description information of the defect type to obtain the defect segmentation result, and further includes: Determining whether the confidence of the defect segmentation mask image is greater than a preset confidence threshold; If the confidence of the defect segmentation mask image is greater than a preset confidence threshold, retaining the defect segmentation mask image; If the confidence of the defect segmentation mask image is not greater than a preset confidence threshold, the defect segmentation mask image is deleted.

[0012] Its further technical solution is that the training data of the second key point detection model and the large image segmentation model are the same, and the second key point detection model and the large image segmentation model are jointly trained in the training stage, and the loss function L during the joint training is set to ;in, is the defect key point regression loss of the second key point detection model, is the sum of the defect pixel-level classification loss and the defect region bounding box regression loss of the image segmentation model, is the weight coefficient of the loss.

[0013] In a second aspect, an embodiment of the present invention further provides a defect detection device based on few-sample learning, which includes a unit for executing the above method.

[0014] In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program can implement the above method when executed by a processor.

[0016] The embodiment of the present invention provides a defect detection method, device, computer equipment and storage medium based on few-sample learning. The method includes: collecting the original workpiece image of the target workpiece, preprocessing the original workpiece image to obtain a preprocessed workpiece image; detecting the workpiece key points in the preprocessed workpiece image through a pre-trained first key point detection model; performing image alignment processing on the preprocessed workpiece image based on the workpiece key points and preset standard workpiece key points to obtain a calibrated workpiece image; detecting the defect part key points and defect type text description information in the calibrated workpiece image through a pre-trained second key point detection model; using the pre-trained image segmentation large model, using the defect part key points and defect type text description information, the calibrated workpiece image is defect segmented to obtain a defect segmentation result. In the present invention, the influence of illumination factors and workpiece position factors in the workpiece quality inspection process is taken into account, and preprocessing and image alignment processing schemes are adopted to obtain a calibrated workpiece image, thereby reducing the influence of illumination factors and workpiece position factors. The key point detection method is innovatively used to perform preliminary detection of defect key points, and combined with text description prompts, it guides the large image segmentation model to accurately segment the defects. Since the key point detection model and the large image segmentation model are less dependent on training data, they can greatly improve the problems of lack of abnormal data, low algorithm generalization ability, and poor anomaly detection performance in the industrial quality inspection process, and effectively improve the work efficiency of similar quality inspection processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0018] Figure 1 A schematic diagram of a process flow of a defect detection method based on few-sample learning provided by an embodiment of the present invention; Figure 2 This is the main network structure diagram of HRNet; Figure 3 A schematic diagram of transforming a pre-processed workpiece image into a calibration workpiece image; Figure 4 It is a structural diagram of the SAM network model; Figure 5 Schematic diagram of accurate segmentation of workpiece scratch defects; Figure 6 A schematic block diagram of a defect detection device based on few-sample learning provided by an embodiment of the present invention; Figure 7A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0021] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0022] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0023] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0024] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a defect detection method based on few-sample learning provided by an embodiment of the present invention. As shown in the figure, the method includes the following steps: S1, collecting an original workpiece image of a target workpiece, and preprocessing the original workpiece image to obtain a preprocessed workpiece image.

[0025] In the specific implementation, first, an image acquisition device (camera) and a supplementary light source are installed on the target production line, so that when the workpiece to be inspected reaches the specified position, the image acquisition device can be controlled to capture the complete image of the workpiece. The original workpiece image captured at the current time t is recorded as .

[0026] Afterwards, in order to eliminate the influence of light variation factors and improve the contrast so that the defects are relatively obvious, the original workpiece image is preprocessed to obtain a preprocessed workpiece image.

[0027] Specifically, in one embodiment, the above step of "preprocessing the original workpiece image to obtain the preprocessed workpiece image" includes: performing histogram equalization processing on the original workpiece image to obtain the preprocessed workpiece image.

[0028] The English meaning of histogram equalization is Histogram Equalization. The preprocessed workpiece image obtained after histogram equalization is recorded as It should be noted that when performing histogram equalization on a color image, operations need to be performed on the three color channels of the image separately (the specific histogram equalization operation is a common process in image processing, and the relevant steps are not repeated here). The three images obtained by histogram equalization are merged to obtain .

[0029] S2: Detect workpiece key points in the preprocessed workpiece image by using a pre-trained first key point detection model.

[0030] In a specific implementation, workpiece key points are pre-set on the workpiece. For example, in this embodiment, the number of workpiece key points on the workpiece is 4, and they are respectively located at the four corners of the workpiece.

[0031] Preprocessed workpiece image after processing in step S1 , often including a certain background area. In order to extract the workpiece key points in the workpiece image and determine the workpiece area, the present invention uses the first key point detection model to detect the workpiece key points of the workpiece. For detection, without loss of generality, the present invention only selects four key points of the workpiece, namely The maximum value is 3. The key point detection algorithm used by the first key point detection model is the HRNet algorithm. It can be understood that other algorithms that can realize key point detection can also achieve the same effect.

[0032] HRNet is an algorithm designed for 2D human posture tasks. It integrates feature information of different scales and has a good feature point extraction effect. Its network structure is as follows Figure 2 .

[0033] The HRNet network consists of 4 stages, each stage has a corresponding number of branch structures, with 1, 2, 3, and 4 branches of different resolutions, and each branch undergoes 4 residual convolutions. Among them, from the previous stage to the next stage, it is necessary to use repeated fusion multi-resolution modules to perform cross-scale information fusion on the feature maps of different scales obtained in the previous stage. In the specific operation, in order to ensure that feature maps of different resolutions can be fused, upsampling is used when merging low resolution into high resolution, and multiple convolution kernels with a step size of 2 are used when merging high resolution into low resolution. The combination of the above stages forms the main network structure of the complete HRNet. When performing key point prediction, the feature maps of different resolutions output by the last stage are upsampled to obtain the original Figure 1 / 4 size features, fuse multiple feature maps, and add corresponding output structures to obtain the key point heat map.

[0034] When the HRNet network is applied to the detection of workpiece key points of the target workpiece, the output structure added to the above-mentioned network main structure after upsampling the feature maps of different resolutions includes 4 layers: convolution layer, batch normalization, ReLU activation function, and convolution layer. The input of the HRNet network is the original workpiece image, and the output is a feature map with a size of 1 / 4 of the original workpiece image and a channel number consistent with the number of workpiece key points, also known as a heat map. Among them, each heat map can predict one workpiece key point, and the prediction method is: .

[0035] in, Indicates the maximum horizontal or vertical coordinate in the heat map. Indicates the horizontal or vertical coordinate of the second largest value.

[0036] It should be noted that the first key point detection model needs to be trained before application. In the training stage, it is necessary to collect target workpiece images, and construct a training set through screening, cleaning, and labeling. The optimal model of the algorithm can be obtained according to the target loss function. The present invention adopts the loss function Mean Square Error Loss (MSE)

[0037] in The horizontal and vertical coordinates of the key points representing the model’s predictions, Indicates the real key point coordinates. In the inference stage, the workpiece image is preprocessed By inputting it into the optimal model obtained in the training phase, the heat map of the key points of the workpiece can be obtained, and then the key points of the workpiece can be calculated. , the connected area formed by the key points of the workpiece is the workpiece area.

[0038] The pre-processed standard workpiece image is input into the above-mentioned pre-trained first key point detection model to obtain the standard workpiece key points , the details are as follows: Get pre-placed and uniformly located The target workpiece image is a target workpiece image, and after the first key point detection model in the above step S2, The target workpiece images of the target workpieces are subjected to key point detection, and then the minimum bounding rectangle of each target workpiece is obtained, and the area image of the minimum bounding rectangle is intercepted and standardized to a uniform size by resizing. , get the standardized workpiece image, where Indicates the standardized workpiece image width, Indicates the height of the standardized workpiece image. The average value of the workpiece key points at each corresponding position in all standardized workpiece images is counted as the standard workpiece key point .

[0039] S3, performing image alignment processing on the pre-processed workpiece image based on the workpiece key points and preset standard workpiece key points to obtain a calibrated workpiece image.

[0040] In the specific implementation, according to the key points of the workpiece obtained in step S2 And preset standard workpiece key points , for the preprocessed workpiece image obtained in step S1 Image alignment is performed to ensure that all artifacts are in the same position in the image, eliminating the influence of different posture differences of different artifacts in the image, thereby improving accuracy.

[0041] For example, in one embodiment, the above step of "performing image alignment processing on the pre-processed workpiece image based on the workpiece key points and the preset standard workpiece key points to obtain a calibrated workpiece image" includes: S31, acquiring an affine transformation matrix based on the workpiece key points and preset standard workpiece key points.

[0042] In the specific implementation, the target workpiece images of 20 target workpieces which are manually placed in advance and have a uniform position are obtained, and the target workpiece images of the 20 target workpieces are subjected to key point detection by the first key point detection model in the above step S2, so as to obtain the minimum bounding rectangle of the target workpiece, intercept the regional image of the minimum bounding rectangle, and standardize it to a uniform size by resizing. , get the standardized workpiece image, where Indicates the standardized workpiece image width, Indicates the height of the standardized workpiece image. The average value of the workpiece key points at each corresponding position in all standardized workpiece images is counted as the standard workpiece key point .

[0043] calculate and The affine transformation matrix between them, the transformation relationship between the two is: .

[0044] Where T represents transpose, Key points of standard workpieces The homogeneous coordinate representation of is the key point of the workpiece detected in step S2 Homogeneous coordinates are expressed, and M is a 3×3 transformation matrix. Solving the above equation (3) yields and The affine transformation matrix M between them.

[0045] S32: transform the pre-processed workpiece image into a calibration workpiece image based on the affine transformation matrix.

[0046] In a specific implementation, the pre-processed workpiece image can be transformed into a calibration workpiece image by using the affine transformation matrix. The calibration workpiece image is consistent with the position and posture of the target workpiece in the standardized workpiece image. Figure 3 , Figure 3 Schematic diagram of transforming a preprocessed workpiece image into a calibration workpiece image.

[0047] For example, in one embodiment, the above step of "transforming the preprocessed workpiece image into a calibration workpiece image based on the affine transformation matrix" includes: obtaining target pixel points after affine transformation of all pixel points in the preprocessed workpiece image through the affine transformation matrix; and constructing the calibration workpiece image through all the target pixel points.

[0048] Specifically, the preprocessed workpiece image All pixels in the image are affine transformed, that is, the aligned calibration workpiece image , calibrate the workpiece image Have the same size .

[0049] S4, detecting the key points of defective parts and text description information of defect types in the calibration workpiece image through a pre-trained second key point detection model.

[0050] In the specific implementation, the calibration workpiece image The key points of the defective parts of the workpiece are input into the pre-trained second key point detection model to preliminarily detect the key points of the defective parts of the workpiece. It should be noted that the defective parts are usually not points, and the key points of the defective parts predicted here are the center points of the connected areas of the defective parts.

[0051] Similar to step S2, the key point detection algorithm used by the second key point detection model can also be the HRNet algorithm. The specific network structure is not described here. The input of the second key point detection model is the calibration workpiece image , the corresponding output is a heat map with multiple channels, the number of channels is equal to the number of abnormal defect types. If only three types of holes, scratches, and spots are predicted, the number of heat map channels is 3. Similarly, the position of the key point of the defect part r can be calculated based on the heat map channel s (s=0,1,2,3), where s represents the number of channels. At the same time, the corresponding defect type text description information is determined based on the type of the heat map channel. For example, if the type of the heat map channel is a hole, then the corresponding defect type text description information is a hole defect. Its loss function L2 is consistent with L1, and its loss can be calculated based on the labels of multiple key points during the training phase.

[0052] Unlike step S2, there are multiple defective parts to be detected. In the inference stage, when there are multiple predicted areas in the same thermal map channel, the second key point detection model only uses the coordinates of the maximum thermal value as the key point of the workpiece defective part corresponding to the thermal map channel, that is, the representative point of the defect type. Like the defect type description text, this representative point is used as the information input of the SAM model in the subsequent step 5. Other key points of defective parts can be detected by Figure 4 The cue encoder in generates a point cue feature vector.

[0053] When there is no predicted area in the heat map channel, it means that there is no defect of the corresponding type. For example, when there are only r0 and r2, it means that there are holes and spots, but no scratches.

[0054] S5, performing defect segmentation on the calibration workpiece image by using the pre-trained image segmentation model and the defect location key points and defect type text description information to obtain a defect segmentation result.

[0055] In the specific implementation, the key point detection algorithm used in step S4 has a relatively small number of algorithm model parameters and calculations, and only requires a relatively small number of samples to complete learning, but its algorithm performance is poor, and only the key points of the defective parts are preliminarily located, and only one key point is retained for each defect type.

[0056] Therefore, in order to obtain accurate segmentation masks of all defects of the target workpiece, the present invention obtains the key points r of the defect part according to step S4. s and the corresponding defect type text description information, and further using a pre-trained image segmentation model, for example, using SAM (Segment Anything Model) to accurately segment the defects of the calibration workpiece image.

[0057] SAM is a network model based on the Transformer structure. Its training data includes about 11 million images and more than 100 million segmentation masks. The huge amount of data gives it a unique advantage in zero-shot learning and few-shot learning. SAM consists of three parts: image encoder, hint encoder, and mask decoder. Figure 4 As shown, Figure 4 This is a structural diagram of the SAM network model.

[0058] Specifically, in one embodiment, the above step of "performing defect segmentation on the calibration workpiece image by using the pre-trained image segmentation model and the defect location key points and defect type text description information to obtain a defect segmentation result" specifically includes: S51, inputting the calibration workpiece image into the image encoder of the image segmentation model to obtain an image feature vector.

[0059] In a specific implementation, the calibration workpiece image After inputting the image encoder of the image segmentation model for feature extraction, the image feature vector is obtained. .

[0060] S52, inputting the key points of the defect parts and the text description information of the defect types into the prompt encoder of the image segmentation model to obtain a point prompt feature vector and a text prompt feature vector.

[0061] In the specific implementation, first, the text descriptions of three defect types, holes, scratches and spots, are preset, such as linear scratches on the surface of the workpiece, etc., which are recorded as (=0, 1, 2), corresponding to the defect type text description information respectively. The present invention does not limit the number of defect types.

[0062] The key points of the defective parts and the text description information of the defect types obtained in step S4 are input into the prompt encoder of the image segmentation model to obtain a point prompt feature vector and a text prompt feature vector.

[0063] For example, in one embodiment, the key point r0 of the defect part and the text description information t0 of the defect type are input to obtain the point prompt feature vector f r0 , text prompt feature vector f t0 .

[0064] S53, inputting the image feature vector, the point prompt feature vector and the text prompt feature vector into the mask decoder of the image segmentation model to obtain a defect segmentation mask map and confidence.

[0065] In a specific implementation, the image feature vector, the point prompt feature vector and the text prompt feature vector are input into the mask decoder of the image segmentation model for prediction analysis and calculation to obtain a defect segmentation mask map and confidence.

[0066] For example, in one embodiment, the image feature vector , point prompt feature vector f r0 and the text prompt feature vector f t0 Input into the mask decoder to obtain the defect segmentation mask map corresponding to the workpiece hole and the corresponding confidence level.

[0067] S54, determining whether the confidence of the defect segmentation mask image is greater than a preset confidence threshold.

[0068] S55: If the confidence of the defect segmentation mask image is greater than a preset confidence threshold, retain the defect segmentation mask image.

[0069] S56: If the confidence of the defect segmentation mask image is not greater than a preset confidence threshold, delete the defect segmentation mask image.

[0070] In the specific implementation, the confidence threshold c is set according to various defect types. s (s=0, 1, 2), the defect segmentation mask map is screened, when the confidence is higher than the threshold, the segmentation result is retained, otherwise, the segmentation result is discarded, and finally an accurate defect segmentation mask map corresponding to multiple defect types of each workpiece is obtained. For example, see Figure 5 , Figure 5 Schematic diagram of accurate segmentation of workpiece scratch defects.

[0071] Furthermore, due to the lack of training data for learning specific fields, the large image segmentation model also needs to be trained before use, just like step S2 and step S4. The difference is that there is no need to train the image encoder and prompt encoder, only fine-tuning training of the lightweight mask decoder is required, which can greatly reduce the dependence on workpiece defect data.

[0072] In one embodiment, the training data of the second key point detection model and the large image segmentation model are the same (the data set used for training consists of image pairs including aligned workpiece images and corresponding workpiece defect real mask images), and the second key point detection model and the large image segmentation model are jointly trained in the training stage, and the loss function L during the joint training is set to ;in, is the defect key point regression loss of the second key point detection model, is the sum of the defect pixel-level classification loss and the defect region bounding box regression loss of the image segmentation model, is the weight coefficient of loss, The value is [0, 1], which can be adjusted according to different training stages and is not specifically limited in the present invention.

[0073] The embodiment of the present invention proposes a defect detection method based on few-sample learning, including: collecting the original workpiece image of the target workpiece, preprocessing the original workpiece image to obtain a preprocessed workpiece image; detecting the workpiece key points in the preprocessed workpiece image through a pre-trained first key point detection model; performing image alignment processing on the preprocessed workpiece image based on the workpiece key points and preset standard workpiece key points to obtain a calibrated workpiece image; detecting the defect part key points and defect type text description information in the calibrated workpiece image through a pre-trained second key point detection model; using the defect part key points and defect type text description information through a pre-trained image segmentation large model, performing defect segmentation on the calibrated workpiece image to obtain a defect segmentation result. In the present invention, the influence of illumination factors and workpiece position factors in the workpiece quality inspection process is taken into account, and preprocessing and image alignment processing schemes are adopted to obtain a calibrated workpiece image, thereby reducing the influence of illumination factors and workpiece position factors. The key point detection method is innovatively used to perform preliminary detection of defect key points, and combined with text description prompts, it guides the large image segmentation model to accurately segment the defects. Since the key point detection model and the large image segmentation model are less dependent on training data, they can greatly improve the problems of lack of abnormal data, low algorithm generalization ability, and poor anomaly detection performance in the industrial quality inspection process, and effectively improve the work efficiency of similar quality inspection processes.

[0074] See also Figure 6 , Figure 6 is a schematic block diagram of a defect detection device based on few-sample learning provided by an embodiment of the present invention. Corresponding to the above defect detection method based on few-sample learning, the present invention also provides a defect detection device 50 based on few-sample learning. The defect detection device 50 based on few-sample learning includes a unit for executing the above defect detection method based on few-sample learning. The defect detection device 50 based on few-sample learning can be configured in a desktop computer, a tablet computer, a laptop, and other terminals. Specifically, the defect detection device 50 based on few-sample learning includes: A preprocessing unit 51 is used to collect an original workpiece image of a target workpiece, and preprocess the original workpiece image to obtain a preprocessed workpiece image; A first detection unit 52, configured to detect workpiece key points in the preprocessed workpiece image by using a pre-trained first key point detection model; An alignment unit 53, configured to perform image alignment processing on the pre-processed workpiece image based on the workpiece key points and preset standard workpiece key points to obtain a calibrated workpiece image; A second detection unit 54 is used to detect the key points of defective parts and text description information of defect types in the calibration workpiece image by using a pre-trained second key point detection model; The segmentation unit 55 is used to perform defect segmentation on the calibration workpiece image by using the pre-trained image segmentation model and the defect location key points and defect type text description information to obtain a defect segmentation result.

[0075] In one embodiment, preprocessing the original workpiece image to obtain a preprocessed workpiece image includes: The pre-processed workpiece image is obtained by performing histogram equalization processing on the original workpiece image.

[0076] In one embodiment, the image alignment processing is performed on the pre-processed workpiece image based on the workpiece key points and the preset standard workpiece key points to obtain the calibrated workpiece image, including: Based on the workpiece key points and preset standard workpiece key points, obtaining an affine transformation matrix; The pre-processed workpiece image is transformed into a calibration workpiece image based on the affine transformation matrix.

[0077] In one embodiment, transforming the pre-processed workpiece image into a calibration workpiece image based on the affine transformation matrix comprises: All the pixels in the preprocessed workpiece image are affine transformed by the affine transformation matrix to obtain target pixels; The calibration workpiece image is formed by all the target pixel points.

[0078] In one embodiment, the pre-trained image segmentation model is used to perform defect segmentation on the calibration workpiece image using the key points of the defect location and the defect type text description information to obtain a defect segmentation result, including: Inputting the calibration workpiece image into the image encoder of the image segmentation model to obtain an image feature vector; Inputting the key points of the defect part and the text description information of the defect type into the prompt encoder of the image segmentation model to obtain a point prompt feature vector and a text prompt feature vector; The image feature vector, the point prompt feature vector and the text prompt feature vector are input into the mask decoder of the image segmentation model to obtain a defect segmentation mask map and confidence.

[0079] In one embodiment, the pre-trained image segmentation model is used to perform defect segmentation on the calibration workpiece image using the key points of the defect location and the text description information of the defect type to obtain the defect segmentation result, and further includes: Determining whether the confidence of the defect segmentation mask image is greater than a preset confidence threshold; If the confidence of the defect segmentation mask image is greater than a preset confidence threshold, retaining the defect segmentation mask image; If the confidence of the defect segmentation mask image is not greater than a preset confidence threshold, the defect segmentation mask image is deleted.

[0080] In one embodiment, the training data of the second key point detection model and the large image segmentation model are the same, and the second key point detection model and the large image segmentation model are jointly trained in the training phase, and the loss function L during the joint training is set to ;in, is the defect key point regression loss of the second key point detection model, is the sum of the defect pixel-level classification loss and the defect region bounding box regression loss of the image segmentation model, is the weight coefficient of the loss.

[0081] It should be noted that technicians in the relevant field can clearly understand that the specific implementation process of the above-mentioned defect detection device 50 based on few-sample learning and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and conciseness of the description, it will not be repeated here.

[0082] The defect detection device 50 based on few-sample learning can be implemented in the form of a computer program. Figure 7 Runs on the computer device shown.

[0083] See also Figure 7 , Figure 7 5 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 may be a terminal or a server, wherein the terminal may be an electronic device with communication functions such as a smart phone, a tablet computer, a laptop computer, a desktop computer, a personal digital assistant, and a wearable device. The server may be an independent server or a server cluster composed of multiple servers.

[0084] The computer device 500 includes a processor 502 , a memory, and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .

[0085] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, the processor 502 may execute a defect detection method based on few-sample learning.

[0086] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .

[0087] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a defect detection method based on few-sample learning.

[0088] The network interface 505 is used to communicate with other devices over the network. Those skilled in the art will appreciate that the above structure is only a block diagram of a portion of the structure related to the present application solution, and does not constitute a limitation on the computer device 500 to which the present application solution is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0089] The processor 502 is used to run a computer program 5032 stored in the memory to implement the following steps: steps of a defect detection method based on few-sample learning provided by any of the above method embodiments.

[0090] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0091] It is understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0092] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program. When the computer program is executed by a processor, the processor executes the steps of a defect detection method based on few-sample learning provided in any of the above method embodiments.

[0093] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which can store program codes. The computer-readable storage medium can be non-volatile or volatile.

[0094] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0095] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0096] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0097] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0098] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0099] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

[0100] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A defect detection method based on few-sample learning, characterized in that: include: Acquire an original workpiece image of the target workpiece, and preprocess the original workpiece image to obtain a preprocessed workpiece image; Detecting workpiece key points in the preprocessed workpiece image by using a pre-trained first key point detection model; Based on the workpiece key points and preset standard workpiece key points, performing image alignment processing on the pre-processed workpiece image to obtain a calibrated workpiece image; Detecting the key points of defective parts and text description information of defect types in the calibration workpiece image by using a pre-trained second key point detection model; By using the pre-trained image segmentation model and the key points of the defect parts and the text description information of the defect types, defect segmentation is performed on the calibration workpiece image to obtain a defect segmentation result.

2. The defect detection method based on few-sample learning according to claim 1, characterized in that: The preprocessing of the original workpiece image to obtain a preprocessed workpiece image includes: The original workpiece image is subjected to histogram equalization processing to obtain the preprocessed workpiece image.

3. The defect detection method based on few-sample learning according to claim 1, characterized in that: The method of performing image alignment processing on the pre-processed workpiece image based on the workpiece key points and the preset standard workpiece key points to obtain a calibrated workpiece image includes: Based on the workpiece key points and preset standard workpiece key points, obtaining an affine transformation matrix; The pre-processed workpiece image is transformed into a calibration workpiece image based on the affine transformation matrix.

4. The defect detection method based on few-sample learning according to claim 3, characterized in that: The step of transforming the preprocessed workpiece image into a calibration workpiece image based on the affine transformation matrix comprises: All the pixels in the preprocessed workpiece image are affine transformed by the affine transformation matrix to obtain target pixels; The calibration workpiece image is formed by all the target pixel points.

5. The defect detection method based on few-sample learning according to claim 1, characterized in that: The pre-trained image segmentation model is used to perform defect segmentation on the calibration workpiece image using the key points of the defect location and the text description information of the defect type to obtain a defect segmentation result, including: Inputting the calibration workpiece image into the image encoder of the image segmentation model to obtain an image feature vector; Inputting the key points of the defect part and the text description information of the defect type into the prompt encoder of the image segmentation model to obtain a point prompt feature vector and a text prompt feature vector; The image feature vector, the point prompt feature vector and the text prompt feature vector are input into the mask decoder of the image segmentation model to obtain a defect segmentation mask map and confidence.

6. The defect detection method based on few-sample learning according to claim 5, characterized in that: The method further comprises: performing defect segmentation on the calibration workpiece image by using the pre-trained image segmentation model and the defect location key points and defect type text description information to obtain a defect segmentation result; Determining whether the confidence of the defect segmentation mask image is greater than a preset confidence threshold; If the confidence of the defect segmentation mask image is greater than a preset confidence threshold, retaining the defect segmentation mask image; If the confidence of the defect segmentation mask image is not greater than a preset confidence threshold, the defect segmentation mask image is deleted.

7. The defect detection method based on few-sample learning according to claim 1, characterized in that: The training data of the second key point detection model and the large image segmentation model are the same. In the training stage, the second key point detection model and the large image segmentation model are jointly trained, and the loss function L during the joint training is set to be ;in, is the defect key point regression loss of the second key point detection model, is the sum of the defect pixel-level classification loss and the defect region bounding box regression loss of the image segmentation model, is the weight coefficient of the loss.

8. A defect detection device based on few-sample learning, characterized in that: The method comprises a unit for executing the method according to any one of claims 1 to 7.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 can be implemented.

Images