A method and apparatus for detecting surface defects in products

By using downsampling and upsampling calculations in the region segmentation model, the problem of difficulty in efficiently and accurately detecting minute defects on the product surface in existing technologies has been solved, achieving efficient and accurate defect region segmentation and detection.

CN118696339BActive Publication Date: 2026-11-10CONTEMPORARY AMPEREX TECHNOLOGY CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202280091450.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-07-29
Publication Date
2026-11-10
Estimated Expiration
2042-07-29

AI Technical Summary

Technical Problem

Existing technologies struggle to efficiently and accurately detect minute and poorly defined defects on product surfaces, especially in areas with complex shapes where it is difficult to distinguish them from other areas.

Method used

A region segmentation model is used to segment defect areas on the product surface. By connecting sampling calculations through multiple data processing layers, the amount of computation is reduced and the detection efficiency is improved. This includes downsampling and upsampling calculations to achieve high-precision segmentation.

Benefits of technology

It achieves high-precision segmentation of defect areas on product surfaces, reduces computational load and memory usage, improves detection efficiency and segmentation speed, and can accurately locate the position and shape of defect areas.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118696339B_ABST
    Figure CN118696339B_ABST
Patent Text Reader

Abstract

A method and apparatus for detecting surface defects in a product are disclosed, which can reduce computational load and improve detection efficiency while accurately segmenting the defective areas on the product surface. The method for detecting surface defects includes: acquiring a first image of the product (S210); and obtaining the defective areas of the product based on the first image and a region segmentation model (S220). The region segmentation model includes multiple data processing layers, which are connected by sampling calculations of data to be sampled. The data to be sampled in at least one data processing layer is obtained by concatenating first intermediate data generated from the input data of the data processing layer with second intermediate data generated from the first intermediate data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method and apparatus for detecting surface defects of a product. Background Technology

[0002] In modern industrial manufacturing, due to processes, equipment, and multiple production steps, industrial products may have one or more defects. Defect detection is a key link in controlling the quality of industrial products.

[0003] However, currently used detection methods often only identify relatively obvious defects, making it difficult to detect smaller defects with unclear boundaries. For defects with high morphological complexity, it is also difficult to accurately distinguish the defective area from other areas. Therefore, how to efficiently and accurately detect defects on product surfaces remains a problem that urgently needs to be solved. Summary of the Invention

[0004] This application provides a method and apparatus for detecting defects on the surface of a product, which can reduce the amount of computation and improve detection efficiency while accurately segmenting the defect area on the product surface.

[0005] In a first aspect, a method for detecting surface defects of a product is provided, comprising: acquiring a first image of the product; obtaining a defective region of the product based on the first image and a region segmentation model, wherein the region segmentation model includes multiple data processing layers, the multiple data processing layers are connected by sampling calculation of data to be sampled, and the data to be sampled for at least one data processing layer is obtained by concatenating first intermediate data generated from the input data of the data processing layer with second intermediate data generated from the first intermediate data.

[0006] By segmenting the defective regions in the first image using a region segmentation model, the analysis speed of defective regions can be improved. Simultaneously, the output feature map has pixel labels, enabling high-precision segmentation. Using the processing method provided in this application within the region segmentation model significantly reduces the computational load when analyzing the first image, further improving the segmentation speed of defective regions in the first image. It also reduces the GPU memory usage for model inference, thereby improving data processing efficiency.

[0007] In some embodiments, segmenting the defective region of the product based on the first image and the region segmentation model includes: segmenting a continuous region of the product based on the first image and the region segmentation model, wherein the continuous region includes the defective region and the non-defective region.

[0008] When there are many product features in the first image, the region segmentation model can separate different continuous regions with high accuracy, avoiding the analysis of non-defective regions as defective regions, which is conducive to accurately judging product quality based on defective regions.

[0009] In some embodiments, segmenting the defective region of the product based on the first image and the region segmentation model includes: obtaining a second image based on the first image and the region segmentation model, wherein the defective region is marked on the second image.

[0010] By outputting a second image that identifies the defective area, the defective area can be correlated with the first image, making it easier to see the location, shape, and size of the defective area on the first image, which is beneficial for analyzing defective areas on the product.

[0011] In some embodiments, the plurality of data processing layers includes n data processing layers, where n is a positive integer greater than 1; obtaining the second image based on the first image and the region segmentation model includes: performing n-1 downsampling calculations on the first image to obtain input data for the nth layer; concatenating the first intermediate data of the nth layer generated from the input data of the nth layer with the second intermediate data of the nth layer generated from the first intermediate data of the nth layer to obtain the sampled data of the nth layer; and performing n-1 upsampling calculations on the sampled data of the nth layer to obtain the second image.

[0012] By processing the first image through upsampling and downsampling calculations, more information can be obtained, resulting in better segmentation of small targets and target edges, which is beneficial for accurate segmentation of different regions in the first image. It also helps prevent overfitting and reduces computational load.

[0013] In some embodiments, performing n-1 downsampling calculations on the first image includes: concatenating the first intermediate data of the m-th layer generated from the input data of the m-th layer with the second intermediate data of the m-th layer generated from the first intermediate data of the m-th layer to obtain the sampled data of the m-th layer, where m is a positive integer greater than 0 and less than n; and performing downsampling calculations on the sampled data of the m-th layer to obtain the input data of the (m+1)-th layer.

[0014] By processing the input data of the data processing layer to obtain the data to be sampled, and then performing downsampling calculations on the data to be sampled to obtain the input data of the next layer, the number of input data channels can be increased while the resolution is reduced. This enables the full extraction of defect features in the first image or the feature image obtained by processing the first image, thereby effectively detecting complex defects in the first image and facilitating accurate segmentation of defect regions.

[0015] In some embodiments, performing n-1 upsampling calculations on the sampled data of the nth layer includes: concatenating the input data of the kth layer with the sampled data in the downsampling calculation of the kth layer to obtain the concatenated data of the kth layer, where k is a positive integer greater than 1 and less than n; concatenating the first intermediate data of the kth layer generated by the concatenated data with the second intermediate data of the kth layer generated by the first intermediate data of the kth layer to obtain the sampled data of the kth layer; and performing upsampling calculations on the sampled data of the kth layer to obtain the input data of the (k-1)th layer.

[0016] Upsampling calculations reconstruct a feature map with pixel labels, i.e., a second image annotated with defective regions, enabling pixel-level defect segmentation of the first image. The output second image has the same resolution as the input first image, thus allowing for accurate location of defective regions within the first image from the annotated second image, facilitating quality control and adjustment of the product. Furthermore, the upsampling calculation incorporates data obtained during downsampling, supplementing image details and improving segmentation accuracy.

[0017] In some embodiments, the region segmentation model is trained using sample images of the product and target images corresponding to the sample images.

[0018] By training the region segmentation model with sample images and their corresponding target images, the model can more accurately detect surface defects in the product being inspected, thus obtaining precisely segmented defect regions.

[0019] In some embodiments, the method further includes: acquiring an original sample image and an original target image corresponding to the original sample image; performing the same elastic transformation on the original sample image and the original target image to obtain the sample image and the target image.

[0020] By applying the same elastic transform to both the original sample images and the original target images, the database used to train the region segmentation model can be expanded, thereby improving the accuracy of the trained model and facilitating the efficient and accurate detection of defects on product surfaces. Simultaneously, it can also mitigate the imaging effects of lens distortion to some extent, enhancing the algorithm's robustness.

[0021] In some embodiments, the region segmentation model is trained using the sample image, the target image, the original sample image, and the original target image.

[0022] This ensures that there is a sufficiently large amount of data during the training of the region segmentation model, so that when the trained region segmentation model is used to detect the first image, it can output high-precision detection results.

[0023] In some embodiments, the product is a battery, and the first image is the surface of the battery to be inspected.

[0024] This allows for efficient and accurate detection of surface defects in batteries during the production process, enabling timely adjustments to the battery manufacturing process and ensuring battery quality.

[0025] Secondly, an apparatus for detecting surface defects of a product is provided, comprising: a processing module for acquiring a first image of the product; the processing module for obtaining a defective region of the product based on the first image and a region segmentation model; wherein the region segmentation model includes multiple data processing layers, the multiple data processing layers are connected by sampling calculation of data to be sampled, and the data to be sampled for at least one data processing layer is obtained by concatenating first intermediate data generated from the input data of the data processing layer with second intermediate data generated from the first intermediate data.

[0026] In some embodiments, the processing module is configured to segment a continuous region of the product based on the first image and a region segmentation model, the continuous region including the defective region and the non-defective region.

[0027] In some embodiments, the processing module is configured to obtain a second image based on the first image and the region segmentation model, wherein the defect region is marked on the second image.

[0028] In some embodiments, the plurality of data processing layers include n data processing layers, where n is a positive integer greater than 1; the processing module is used to perform n-1 downsampling calculations on the first image to obtain the input data of the nth layer; the processing module is used to concatenate the first intermediate data of the nth layer generated from the input data of the nth layer with the second intermediate data of the nth layer generated from the first intermediate data of the nth layer to obtain the sampled data of the nth layer; the processing module is used to perform n-1 upsampling calculations on the sampled data of the nth layer to obtain the second image.

[0029] In some embodiments, the processing module is used to concatenate the first intermediate data of the m-th layer generated from the input data of the m-th layer with the second intermediate data of the m-th layer generated from the first intermediate data of the m-th layer to obtain the sampled data of the m-th layer, where m is a positive integer greater than 0 and less than n; the processing module is used to perform downsampling calculation on the sampled data of the m-th layer to obtain the input data of the (m+1)-th layer.

[0030] In some embodiments, the processing module is used to concatenate the input data of the k-th layer with the data to be sampled in the downsampling calculation of the k-th layer to obtain the concatenated data of the k-th layer, where k is a positive integer greater than 1 and less than n; the processing module is used to concatenate the first intermediate data of the k-th layer generated by the concatenated data with the second intermediate data of the k-th layer generated by the first intermediate data of the k-th layer to obtain the data to be sampled of the k-th layer; the processing module is used to perform upsampling calculation on the data to be sampled of the k-th layer to obtain the input data of the (k-1)-th layer.

[0031] In some embodiments, the region segmentation model is trained using sample images of the product and target images corresponding to the sample images.

[0032] In some embodiments, the processing module is used to acquire an original sample image and an original target image corresponding to the original sample image; and to perform the same elastic transformation on the original sample image and the original target image to obtain the sample image and the target image.

[0033] In some embodiments, the region segmentation model is trained using the sample image, the target image, the original sample image, and the original target image.

[0034] In some embodiments, the product is a battery, and the first image is the surface of the battery to be inspected.

[0035] Thirdly, an apparatus for detecting surface defects in a product is provided, comprising: a processor and a memory, the memory storing instructions which, when executed by the processor, cause the apparatus to perform the method as described in any of the embodiments of the first aspect above.

[0036] Fourthly, an electronic device is provided, comprising: an apparatus for detecting surface defects of a product as described in any embodiment of the second aspect above.

[0037] Fifthly, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, which, when executed, performs the method as described in any of the embodiments of the first aspect above. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on the drawings without creative effort.

[0039] Figure 1 This is a schematic diagram of a system architecture provided in an embodiment of this application.

[0040] Figure 2 This is a schematic flowchart illustrating a method for detecting surface defects in a product, as provided in an embodiment of this application.

[0041] Figure 3 This is a schematic diagram of the structure of a region segmentation model provided in an embodiment of this application;

[0042] Figure 4 This is a schematic diagram of a processing method provided in an embodiment of this application;

[0043] Figure 5 These are example images of the first and second images provided in the embodiments of this application;

[0044] Figure 6 This is a schematic diagram of a device for detecting surface defects of a product provided in an embodiment of this application.

[0045] The accompanying drawings are not drawn to scale. Detailed Implementation

[0046] The embodiments of this application will be described in further detail below with reference to the accompanying drawings and examples. The detailed description of the following embodiments and the accompanying drawings are used to illustrate the principles of this application by way of example, but should not be used to limit the scope of this application, that is, this application is not limited to the described embodiments.

[0047] In the description of this application, it should be noted that, unless otherwise stated, "a plurality of" means two or more; the terms "upper," "lower," "left," "right," "inner," and "outer," etc., indicating orientation or positional relationships, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this application. Furthermore, the terms "first," "second," and "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. "Vertical" is not vertical in the strict sense, but within the allowable tolerance range. "Parallel" is not parallel in the strict sense, but within the allowable tolerance range.

[0048] The directional terms used in the following description refer to the directions shown in the figures and are not intended to limit the specific structure of this application. It should also be noted in the description of this application that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0049] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0050] In the manufacturing process of industrial products, defects are inevitable, affecting product quality. Therefore, defect detection is an indispensable step. Current defect detection methods often struggle to identify minute surface defects, and defects with unclear boundaries and variable shapes are difficult to describe and extract using manually designed operators. Furthermore, high-accuracy algorithms often have complex network structures and involve huge computational loads, making them unsuitable for the demands of high-speed production cycles.

[0051] In view of this, this application provides a method for detecting surface defects in products. By segmenting defective regions in the product using a region segmentation model, high-precision region segmentation can be achieved. Furthermore, in the region segmentation model of this application, multiple data processing layers are connected through sampling calculations. The sampled data for at least one data processing layer is obtained by concatenating first intermediate data generated from the input data of the data processing layer with second intermediate data generated from the first intermediate data. This replaces the commonly used convolution calculation, reducing computational load and memory usage for model inference, thus achieving better results in efficiently and accurately segmenting defective regions.

[0052] The embodiments of this application can be applied to defect detection, for example, in the battery manufacturing process. Specifically, it can be used for defect detection of weld seams during the welding process, weld marks generated during the welding of connecting components that connect the tabs and electrode terminals, and surface defect detection generated during the die-cutting process. In addition, it can also effectively detect dirty areas generated during battery manufacturing. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0053] Figure 1A schematic diagram of a system architecture applicable to an embodiment of this application is shown. Figure 1 In this process, the data acquisition device 160 is used to collect training data. After collecting the training data, the data acquisition device 160 stores the training data in the database 130, and the training device 120 trains the target model / rule 101 based on the training data maintained in the database 130.

[0054] The aforementioned target model / rule 101 can be used to implement the solder mark detection method of this application embodiment, specifically a neural network. It should be noted that in practical applications, the training data maintained in the database 130 may not all come from the data acquisition device 160; it may also be received from other devices. Furthermore, it should be noted that the training device 120 may not necessarily train the target model / rule 101 entirely based on the training data maintained in the database 130; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0055] The target model / rule 101 trained using training device 120 can be applied to different systems or devices, such as... Figure 1 The execution device 110 shown can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, etc., or it can be a server or cloud computing platform acting as a host computer. Figure 1 In this embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. Users can input data to the I / O interface 112 through the industrial camera 140. The input data may include the video or image to be processed input from the industrial camera 140.

[0056] In some embodiments, the industrial camera 140 may be the same device as the aforementioned execution device 110. For example, the industrial camera 140 and the aforementioned execution device 110 may both be terminal devices.

[0057] In other embodiments, the industrial camera 140 may be a different device from the execution device 110. For example, the industrial camera 140 may be a terminal device, while the execution device 110 may be a cloud device, server, or other such device. The industrial camera 140 may interact with the execution device 310 through a communication network of any communication mechanism / standard. The communication network may be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0058] The calculation module 111 of the execution device 110 is used to process the input data (such as the image to be processed) received by the I / O interface 112. During the calculation and other related processing performed by the calculation module 111 of the execution device 110, the execution device 110 can call data, code, etc. in the data storage system 150 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0059] Finally, the I / O interface 112 returns the processing results, such as the obtained solder mark classification results, to the industrial camera 140, thereby providing them to the user.

[0060] It is worth noting that, Figure 1 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 1 In this context, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed within the execution device 110.

[0061] like Figure 1 As shown, the target model / rule 101 is trained by the training device 120. In this embodiment of the application, the target model / rule 101 can be a neural network. Specifically, the neural network in this embodiment of the application can be a CNN, a region convolutional neural network (RCNN), a faster region convolutional neural network (faster RCNN), or other types of neural networks, etc. This application does not make any specific limitations on this.

[0062] The following is combined Figure 2 The method 200 for detecting solder marks provided in this application will be described in detail. Figure 2 This is a schematic flowchart illustrating a method for detecting surface defects in a product according to an embodiment of this application. This method 200 can be performed by a device for detecting solder marks, which can be... Figure 1 The execution device 110 in the embodiment. For ease of explanation, the product in this application embodiment is a battery, and the execution device 110 is an example of a host computer. Method 200 may include at least some of the following.

[0063] S210: Obtain the first image of the product.

[0064] S220: Based on the first image and the region segmentation model, the defective areas of the product are obtained.

[0065] The region segmentation model includes multiple data processing layers, which are connected by sampling calculations on the data to be sampled. The data to be sampled in at least one data processing layer is obtained by concatenating the first intermediate data generated from the input data of the data processing layer with the second intermediate data generated from the first intermediate data.

[0066] The first image can be obtained by taking pictures of the product during the actual production process, for example, using image acquisition equipment such as an industrial camera. This image may include, but is not limited to, the surface of the product to be inspected. For instance, if a certain surface of a product needs to be inspected in a particular process, an industrial camera can be positioned above the surface to be inspected and take a picture of the product as it passes by, thus obtaining the first image of the product. The industrial camera then sends the first image of the product to a host computer for analysis.

[0067] To analyze the defective areas in the first image, it is necessary to first segment the defective areas. In the embodiments provided in this application, the defective areas in the first image can be segmented based on the first image and the region segmentation model. The defective areas may include surface contaminants from the product during the manufacturing process, or defective weld marks generated in the welding area, such as weld porosity, weld undercut, weld cracks, and other welding defects.

[0068] The defective region can refer to the area after removing other parts of the first image, or it can refer to the area clearly marked on top of the first image. Understandably, the segmented defective region should be clearly distinguishable from other areas of the first image.

[0069] A region segmentation model refers to a model that segments regions within a first image. This model can be, for example, a neural network. Specifically, it can be, for example, a U-Net network. The structure of a region segmentation model can be as follows: Figure 3 As shown, it includes multiple data processing layers, which are connected through sampling calculations. The sampling calculations include downsampling calculations, i.e. Figure 3 The processing method 302 shown in the figure, and the upsampling calculation, namely Figure 3The processing method 303 is shown in the figure. Downsampling calculation can reduce the resolution of the feature map by increasing the number of channels in the feature image, thereby increasing the receptive field of the first image; upsampling calculation gradually restores the number of channels and resolution of the feature map obtained in the last layer to be consistent with the original input image, and finally obtains a segmented image with the same resolution as the original image. In one possible implementation, at least one of processing methods 302 and 303 can be a convolution calculation; in another possible implementation, at least one of processing methods 302 and 303 can be generating a part of intermediate data based on the data to be sampled, generating another part of intermediate data based on the part of intermediate data, and concatenating the two parts of intermediate data as the input data for the next layer.

[0070] During the upsampling calculation process, the data processing layer can fuse the sampled data output from the downsampling calculation in the same layer, i.e. Figure 3 The processing method 304 shown in the figure is to gradually restore the channels and resolution of the feature map obtained during the data processing to be consistent with the original input image.

[0071] Figure 3 The structure of the region segmentation model shown is such that the data processing layer of the input first image is used as the first layer L1. The input data of the first layer L1 during the upsampling process is the data to be sampled in the second layer L2, obtained through upsampling calculation. After processing by processing method 301, the input data of the first layer L1 outputs a feature map with pixel labels. In other words, each pixel in the final feature map has a label, and defects in the feature map can be segmented based on these labels.

[0072] In one possible implementation, the data after one downsampling calculation can be used as the input data of the first layer. In the upsampling calculation process, the output data of the first layer needs to be upsampled again. After processing by processing method 301, a feature map with pixel labels is output. This feature map is the final output result and no further sampling calculation is needed.

[0073] During the process of processing the input data of this layer in the data processing layer, it is possible to use Figure 3The processing method 301 shown in the figure processes the input data. In one possible implementation, processing method 301 can be a convolution calculation. In another possible implementation, processing method 301 can generate first intermediate data based on the input data of the data processing layer, generate second intermediate data based on the first intermediate data, and concatenate the first intermediate data and the second intermediate data to obtain the sampling data of the layer. The input data can generate the first intermediate data through convolution calculation, and the first intermediate data can also generate the second intermediate data based on convolution calculation. It should be understood that in the embodiments provided in this application, convolution calculation can be replaced by a calculation method that generates first intermediate data from the input data, then generates second intermediate data from the first intermediate data, and concatenates the first intermediate data and the second intermediate data to obtain the output data. Within the same layer, the input data can be processed multiple times using processing method 301. For example, in the first layer L1, the first image is processed once using processing method 301 to obtain first processed data, and the first processed data is processed again using processing method 301 to obtain the sampling data of the first layer. Each of the multiple data processing layers can process the input data of that layer according to the same processing method, and different data processing layers in the multiple data processing layers can also process the input data according to different processing methods.

[0074] Before obtaining the final output, the output data obtained by each data processing layer is the sampled data. The sampled data is then processed by sampling calculations and enters the next or previous layer. Specifically, the sampled data is processed by downsampling calculations and enters the next layer, and by upsampling calculations and enters the previous layer.

[0075] In each data processing layer, the input data can be processed in the following ways: Figure 4 As shown, taking one of the multiple data processing layers as an example, images 1 to 32 are the input data for this layer, and each image represents a feature map of one channel. Figure 4 The input data shown includes feature maps with 32 channels. Alternatively, images 1 to 32 can also be intermediate feature maps obtained by processing the original input data, such as the intermediate feature maps obtained by convolutional calculation on the first image, where each image also represents a feature map of one channel. In the same layer, the number of channels in the input data and the output target data is the same. Therefore, the number of channels in the first intermediate data generated from the input data can be half the number of channels in the target data to be output. The number of channels in the second intermediate data generated from the first intermediate data is the same as the number of channels in the first intermediate data, i.e., half the number of channels in the target data. The first intermediate data and the second intermediate data are concatenated to obtain the target data with the same number of channels as the input data. In the process of processing data in the same layer, the target data is the data to be sampled. Figure 4For example, specifically, if images 1 to 32 are the input data, then images 1' to 16', i.e., the first intermediate data, can be generated based on images 1' to 32; then images 17' to 32', i.e., the second intermediate data, can be generated based on images 1' to 16'; and images 1' to 16' and images 17' to 32' can be stitched together to obtain images 1' to 32', which are the target data output by this layer, also known as the data to be sampled.

[0076] During the sampling calculation process at different layers, it is also possible to... Figure 4 The processing method shown processes the data to be sampled. Specifically, taking downsampling as an example, downsampling is a process of increasing the number of channels. In the process of generating the first intermediate data based on the data to be sampled, the number of channels in the first intermediate data is half the number of channels in the target data to be output. For example, if the data to be sampled is a 32-channel feature map, and the target data to be output is a 64-channel feature map, then the first intermediate data is 32-channel data generated from the 32-channel data to be sampled, and the second intermediate data is 32-channel data generated from the 32-channel first intermediate data. Concatenating the first and second intermediate data yields the 64-channel target data. In this case, the output target data is the input data for the next layer.

[0077] It should be understood that, Figure 4 The image numbers 1 to 32 are used only to distinguish different images and do not restrict the order or position of the images, nor do they limit the number of images. Figure 4 The data processing algorithm shown can be, for example, the Ghost Module in GhostNet.

[0078] By segmenting the defective regions in the first image using a region segmentation model, the analysis speed of defective regions can be improved. Simultaneously, the output feature map has pixel labels, enabling high-precision segmentation. In the region segmentation model, processing the input data to obtain the data to be sampled or the input data for the next layer significantly reduces the computational load when analyzing the first image, further improving the segmentation speed of defective regions in the first image. It also reduces the GPU memory usage for model inference, improving data processing efficiency.

[0079] According to some embodiments of this application, step S220 may optionally include: segmenting a continuous region of the product based on a first image and a region segmentation model, wherein the continuous region includes defective regions and non-defective regions.

[0080] Given that the first image can be a photograph taken in reality, when photographing the surface of the product to be inspected, all features of the entire surface are captured. In other words, the first image may include not only defective areas on the surface to be inspected, but also some structural features of the product itself; the areas containing these structural features are the non-defective areas.

[0081] Taking battery manufacturing as an example, in one possible implementation, the surface to be inspected is the end cap surface of the battery. The first image may include areas such as electrode terminals, pressure relief mechanisms, and battery labels. These areas have certain boundaries on the first image and are continuous areas that can be distinguished from other areas. When processing the first image using a region segmentation model, the region segmentation model can segment these continuous areas to obtain different continuous areas on the surface to be inspected.

[0082] In another possible implementation, the surface to be tested includes electrode terminals and solder marks on the electrode terminals. The electrode terminals and solder marks on the electrode terminals can be segmented by setting different parameters or operators.

[0083] After processing the first image, the region segmentation model obtains a feature map with pixel labels. Each pixel in the final output feature map has a label, allowing defects to be segmented based on these labels. In one possible implementation, the pixel labels are only used to distinguish different regions, and the labeled regions are manually judged to determine whether they are defective or non-defective regions. In another possible implementation, the region segmentation model can directly identify whether a continuous region is defective or non-defective, in which case defective regions can be labeled with one type of label, and non-defective regions with another type of label.

[0084] When there are many product features in the first image, the region segmentation model can separate different continuous regions with high accuracy, avoiding the analysis of non-defective regions as defective regions, which is conducive to accurately judging product quality based on defective regions.

[0085] According to some embodiments of this application, step S220 may optionally include: obtaining a second image based on the first image and the region segmentation model, wherein the defective region is marked on the second image.

[0086] The first image is input into the region segmentation model, and its output can be the second image, which identifies the defective regions. For example... Figure 5 As shown, Figure 5 In the image, (a) is the first image, or a portion thereof, and (b) is the second image obtained by inputting (a) into the region segmentation model. It can be seen from (b) that the black dots in the first image, i.e., the defect areas, are clearly marked in the second image.

[0087] Similarly, Figure 5 In the diagram, (c) is the first image, or a portion of the first image, and (d) is the second image obtained after inputting (c) into the region segmentation model. From (d), it can be seen that the dark areas in the first image, i.e., the defect areas, are clearly marked in the second image. Furthermore, from... Figure 5 As can be seen, the region segmentation model can also achieve good segmentation results for minor defects and defects with unclear boundaries.

[0088] By outputting a second image that identifies the defective area, the defective area can be correlated with the first image, making it easier to see the location, shape, and size of the defective area on the first image, which is beneficial for analyzing defective areas on the product.

[0089] According to some embodiments of this application, optionally, the multiple data processing layers include n data processing layers, where n is a positive integer. The first image is downsampled n-1 times to obtain the input data for the nth layer; the first intermediate data of the nth layer generated from the input data of the nth layer is concatenated with the second intermediate data of the nth layer generated from the first intermediate data of the nth layer to obtain the data to be sampled in the nth layer; the data to be sampled in the nth layer is upsampled n-1 times to obtain the second image.

[0090] by Figure 3 For example, Figure 3 The diagram illustrates n data processing layers L1 to Ln. In the first layer L1, the input first image is processed using processing method 301, and the resulting sampled data undergoes a downsampling calculation, which serves as the input data for the second layer L2. The input data for the second layer L2 is then processed using processing method 301, and the resulting sampled data undergoes a second downsampling calculation, which serves as the input data for the third layer L3. This process continues, performing n-1 downsampling calculations on the first image to obtain the input data for the nth layer Ln.

[0091] The nth layer Ln is the last layer of multiple data processing layers in the region segmentation model. The input data of the nth layer Ln is processed by processing method 301 to obtain the sampled data of the nth layer Ln, and then upsampling calculation begins.

[0092] Upsampling the data to be sampled in layer n yields the input data for layer n-1. This process continues until n-1 upsampling calculations are performed, at which point the input data for layer L1 in the upsampling process is obtained. After processing this input data, the second image is obtained.

[0093] exist Figure 3In the above, processing method 301 is to process the input data of the corresponding layer, that is, to generate the first intermediate data of the layer based on the input data of the layer, generate the second intermediate data of the layer based on the first intermediate data of the layer, and concatenate the two to obtain the sampling data of the layer. Figure 3 Only the structure of the region segmentation model with the data processing layer of the first input image as the first layer L1 is shown. In one possible implementation, the data after one downsampling calculation can also be used as the input data of the first layer. In this case, the first image is downsampled n times to obtain the input data of the nth layer; similarly, the data to be sampled in the nth layer is upsampled n times to obtain the second image.

[0094] By processing the first image through upsampling and downsampling calculations, more information can be obtained, resulting in better segmentation of small targets and target edges, which is beneficial for accurate segmentation of different regions in the first image. It also helps prevent overfitting and reduces computational load.

[0095] According to some embodiments of this application, optionally, the first image is subjected to n-1 downsampling calculations, including: concatenating the first intermediate data of the m-th layer generated from the input data of the m-th layer with the second intermediate data of the m-th layer generated from the first intermediate data of the m-th layer to obtain the sampled data of the m-th layer, where m is a positive integer greater than 0 and less than n; and performing downsampling calculations on the sampled data of the m-th layer to obtain the input data of the (m+1)-th layer.

[0096] The downsampling calculation for the first image (n-1 times) includes downsampling the data to be sampled in the m-th layer, where the m-th layer is one of the n data processing layers. Specifically, input data is obtained in the m-th layer, and processed. This involves generating first intermediate data for the m-th layer based on the input data, generating second intermediate data for the m-th layer based on the first intermediate data, and concatenating the two to obtain the data to be sampled in the m-th layer. Downsampling the data to be sampled in the m-th layer yields the input data for the (m+1)-th layer. When m = 1, the input data for the m-th layer can be the first image; when m ≠ 1, the input data for the m-th layer can be obtained by downsampling the data to be sampled in the (m-1)-th layer. In other words, during the downsampling process, the input data for each layer can be obtained and processed in the above manner.

[0097] By processing the input data of the data processing layer to obtain the data to be sampled, and then performing downsampling calculations on the data to be sampled to obtain the input data of the next layer, the number of input data channels can be increased while the resolution is reduced. This enables the full extraction of defect features in the first image or the feature image obtained by processing the first image, thereby effectively detecting complex defects in the first image and facilitating accurate segmentation of defect regions.

[0098] According to some embodiments of this application, optionally, the upsampling calculation is performed n-1 times on the data to be sampled in the nth layer, including concatenating the input data of the kth layer with the data to be sampled in the downsampling calculation of the kth layer to obtain the concatenated data of the kth layer, where k is a positive integer greater than 1 and less than n; concatenating the first intermediate data of the kth layer generated by the concatenated data with the second intermediate data of the kth layer generated by the first intermediate data of the kth layer to obtain the data to be sampled in the kth layer; and performing upsampling calculation on the data to be sampled in the kth layer to obtain the input data of the (k-1)th layer.

[0099] Performing n-1 upsampling calculations on the data to be sampled in layer n includes upsampling calculations on the data to be sampled in layer k, where layer k is one of the n data processing layers. Specifically, the input data of layer k can be obtained by sampling the data to be sampled in layer k+1. In order to restore the channels and resolution of the multi-channel and low-resolution feature maps obtained during the downsampling calculation to be consistent with the channels and resolution of the original input image, it is necessary to concatenate the input data of layer k with another part of the data. The data used for concatenation with the input data of layer k can be the data to be sampled in layer k during the downsampling calculation. Specifically, the data to be sampled in layer k during the downsampling calculation is used for downsampling to obtain the input data of layer k+1, and also for concatenation with the input data during the upsampling calculation. In other words, compared with the input data, the concatenated data of layer k can obtain more details, thereby achieving pixel-level defect region segmentation.

[0100] During the upsampling process, the concatenated data is processed to obtain the sampled data for the k-th layer. Specifically, the first intermediate data for the k-th layer is generated from the concatenated data, and the second intermediate data for the k-th layer is generated from the first intermediate data. These two intermediate data are then concatenated to obtain the sampled data for the k-th layer. Upsampling is then performed on the sampled data for the k-th layer to obtain the input data for the (k-1)-th layer.

[0101] Considering that the first layer is the last layer in the upsampling calculation process, the data to be sampled obtained in this layer is the output data of the region segmentation model, so there is no need to perform upsampling calculation on the output data again.

[0102] Considering that the nth layer is the first layer in the upsampling calculation process and the last layer among the n data processing layers, the input data of the nth layer is both the input data in the downsampling process and the input data in the upsampling process. The data to be sampled obtained in the downsampling process is also the data to be sampled in the upsampling process. Therefore, the input data of the nth layer can be processed directly to obtain the data to be sampled, and the upsampling calculation can be performed on the data to be sampled, without needing to concatenate the input data.

[0103] Upsampling calculations reconstruct a feature map with pixel labels, i.e., a second image annotated with defective regions, enabling pixel-level defect segmentation of the first image. The output second image has the same resolution as the input first image, thus allowing for accurate location of defective regions within the first image from the annotated second image, facilitating quality control and adjustment of the product. Furthermore, the upsampling calculation incorporates data obtained during downsampling, supplementing image details and improving segmentation accuracy.

[0104] According to some embodiments of this application, optionally, the region segmentation model is trained from sample images of the product and target images corresponding to the sample images.

[0105] Training a region segmentation model requires a large number of sample images and corresponding target images for each sample image to adjust the parameters of the model. A sample image is the first image with known region segmentation results, while a target image is the image corresponding to the sample image, with the regions on the sample image already labeled.

[0106] During the training of a region segmentation model, sample images can be input into the model, which then analyzes and processes them to obtain reference regions. These reference regions are then compared with regions in the target image, allowing for adjustments to the model, such as adjusting parameters or operators, to ensure a proper match between the reference and target regions. In one possible implementation, the region segmentation model outputs an image with marked defective regions. During training, after analyzing the sample images, the model obtains a reference image, which is then compared with the target image to adjust the model accordingly.

[0107] By training with a large number of sample images and target images, when a first image is input into the region segmentation model, the defect region output by the model can match the actual defect region represented in the first image. That is, within a certain error range, the defect region output by the region segmentation model can be considered accurate.

[0108] By training the region segmentation model with sample images and their corresponding target images, the model can more accurately detect surface defects in the product being inspected, thus obtaining precisely segmented defect regions.

[0109] According to some embodiments of this application, method 200 may optionally further include: obtaining an original sample image and an original target image corresponding to the original sample image; performing the same elastic transformation on the original sample image and the original target image to obtain a sample image and a target image.

[0110] Considering that products with defective areas are extremely rare in industrial production, the number of sample and target images available for training the region segmentation model is also limited. To ensure the accuracy of the trained region segmentation model, the directly obtained original sample and target images can be transformed to generate multiple corresponding sample and target images for training the region segmentation model.

[0111] Specifically, the same elastic transformation can be applied to both the original sample image and the original target image. An elastic transformation refers to randomly changing each pixel in an image using a specific algorithm. For example, pixels can be moved a random distance in a random direction within a certain range; another example is generating the transformed image through convolution calculations.

[0112] By applying the same elastic transformation to the original sample images and the original target images, the database used to train the region segmentation model can be expanded, thereby improving the accuracy of the trained region segmentation model and facilitating the efficient and accurate detection of defects on the product surface.

[0113] According to some embodiments of this application, optionally, the region segmentation model is trained from sample images, target images, original sample images, and original target images.

[0114] To ensure a sufficient number of samples for training the region segmentation model, the original sample images, the original target images, and images derived from both through elastic transformation can all be used. During training, the parameters of the region segmentation model can be adjusted based on the sample images, target images, original sample images, and original target images.

[0115] This ensures a sufficiently large amount of data during the training of the region segmentation model, enabling it to output high-accuracy detection results when used to detect images in the first image. It also helps to mitigate the imaging effects of lens distortion to some extent, improving the algorithm's robustness.

[0116] According to some embodiments of this application, optionally, the product is a battery, and the first image is the surface of the battery to be tested.

[0117] In other words, by inputting a first image of the surface to be inspected, including the battery, into the region segmentation model, the defective regions of the battery's surface to be inspected can be segmented out.

[0118] This allows for efficient and accurate detection of surface defects in batteries during the production process, enabling timely adjustments to the battery manufacturing process and ensuring battery quality.

[0119] This application also provides an apparatus for detecting surface defects in a product, including a processing module, which may be a processor. The processing module is used to acquire a first image of the product; the processing module is used to obtain defective regions of the product based on the first image and a region segmentation model; wherein the region segmentation model includes multiple data processing layers, which are connected by sampling calculations of data to be sampled, and the data to be sampled for at least one data processing layer is obtained by concatenating first intermediate data generated from the input data of the data processing layer with second intermediate data generated from the first intermediate data.

[0120] According to some embodiments of this application, optionally, the processing module is used to segment a continuous region of the product based on a first image and a region segmentation model, the continuous region including defective regions and non-defective regions.

[0121] According to some embodiments of this application, optionally, the processing module is used to obtain a second image based on the first image and the region segmentation model, wherein the defective region is marked on the second image.

[0122] According to some embodiments of this application, optionally, the multiple data processing layers include n data processing layers, where n is a positive integer greater than 1; the processing module is used to perform n-1 downsampling calculations on the first image to obtain the input data of the nth layer; the processing module is used to concatenate the first intermediate data of the nth layer generated from the input data of the nth layer with the second intermediate data of the nth layer generated from the first intermediate data of the nth layer to obtain the sampled data of the nth layer; the processing module is used to perform n-1 upsampling calculations on the sampled data of the nth layer to obtain the second image.

[0123] According to some embodiments of this application, optionally, the processing module is used to concatenate the first intermediate data of the m-th layer generated from the input data of the m-th layer with the second intermediate data of the m-th layer generated from the first intermediate data of the m-th layer to obtain the sampled data of the m-th layer, where m is a positive integer greater than 0 and less than n; the processing module is used to perform downsampling calculation on the sampled data of the m-th layer to obtain the input data of the (m+1)-th layer.

[0124] According to some embodiments of this application, optionally, the processing module is used to concatenate the input data of the k-th layer with the data to be sampled in the downsampling calculation of the k-th layer to obtain the concatenated data of the k-th layer, where k is a positive integer greater than 1 and less than n; the processing module is used to concatenate the first intermediate data of the k-th layer generated by the concatenated data with the second intermediate data of the k-th layer generated by the first intermediate data of the k-th layer to obtain the data to be sampled of the k-th layer; the processing module is used to perform upsampling calculation on the data to be sampled of the k-th layer to obtain the input data of the (k-1)-th layer.

[0125] According to some embodiments of this application, optionally, the region segmentation model is trained from sample images of the product and target images corresponding to the sample images.

[0126] According to some embodiments of this application, optionally, the processing module is used to obtain the original sample image and the original target image corresponding to the original sample image; and to perform the same elastic transformation on the original sample image and the original target image to obtain the sample image and the target image.

[0127] According to some embodiments of this application, optionally, the region segmentation model is trained from sample images, target images, original sample images, and original target images.

[0128] According to some embodiments of this application, optionally, the product is a battery, and the first image is the surface of the battery to be tested.

[0129] This application also provides a device 600 for detecting surface defects in a product, such as... Figure 6 As shown, the device includes a processor 601 and a memory 602. The memory 602 stores instructions that, when executed by the processor 601, cause the device 600 to perform the method described in any of the above embodiments.

[0130] This application also provides an electronic device, including the apparatus for detecting surface defects of a product as described in any of the above embodiments.

[0131] This application also provides a computer-readable storage medium storing a computer program that, when run, performs the method described in any of the above embodiments.

[0132] Although this application has been described with reference to preferred embodiments, various modifications can be made thereto and components can be replaced with equivalents without departing from the scope of this application. In particular, the technical features mentioned in the various embodiments can be combined in any manner, provided there is no structural conflict. This application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A method for detecting surface defects in a product, characterized in that, include: Obtain the first image of the product; Based on the first image and the region segmentation model, the defect area of ​​the product is obtained; The region segmentation model includes multiple data processing layers, which are connected by sampling calculations on the data to be sampled. The data to be sampled in at least one of the data processing layers is obtained by concatenating first intermediate data generated from the input data of the data processing layer with second intermediate data generated from the first intermediate data. The step of segmenting the defective region of the product based on the first image and the region segmentation model includes: A second image is obtained based on the first image and the region segmentation model, and the defective region is marked on the second image; The multiple data processing layers include n data processing layers, where n is a positive integer greater than 1; obtaining the second image based on the first image and the region segmentation model includes: Perform n-1 downsampling calculations on the first image to obtain the input data for the nth layer; The first intermediate data of the nth layer generated from the input data of the nth layer is concatenated with the second intermediate data of the nth layer generated from the first intermediate data of the nth layer to obtain the sampling data of the nth layer; The second image is obtained by performing n-1 upsampling calculations on the data to be sampled in the nth layer; The step of performing n-1 downsampling calculations on the first image includes: The first intermediate data of the m-th layer generated from the input data of the m-th layer is concatenated with the second intermediate data of the m-th layer generated from the first intermediate data of the m-th layer to obtain the sampling data of the m-th layer, where m is a positive integer greater than 0 and less than n; The data to be sampled in the m-th layer is downsampled to obtain the input data of the (m+1)-th layer; The step of performing n-1 upsampling calculations on the data to be sampled in the nth layer includes: The input data of the k-th layer is concatenated with the data to be sampled in the downsampling calculation of the k-th layer to obtain the concatenated data of the k-th layer, where k is a positive integer greater than 1 and less than n; The first intermediate data of the kth layer generated by the spliced ​​data is spliced ​​together with the second intermediate data of the kth layer generated by the first intermediate data of the kth layer to obtain the sampling data of the kth layer; Upsampling is performed on the data to be sampled in the k-th layer to obtain the input data in the (k-1)-th layer.

2. The method according to claim 1, characterized in that, The step of segmenting the defective region of the product based on the first image and the region segmentation model includes: Based on the first image and the region segmentation model, the continuous region of the product is segmented, and the continuous region includes the defective region and the non-defective region.

3. The method according to any one of claims 1 to 2, characterized in that, The region segmentation model is trained using sample images of the product and target images corresponding to the sample images.

4. The method according to claim 3, characterized in that, The method further includes: Obtain the original sample image and the original target image corresponding to the original sample image; The original sample image and the original target image are subjected to the same elastic transformation to obtain the sample image and the target image.

5. The method according to claim 4, characterized in that, The region segmentation model is trained using the sample image, the target image, the original sample image, and the original target image.

6. The method according to any one of claims 1 to 5, characterized in that, The product is a battery, and the first image shows the surface of the battery to be tested.

7. A device for detecting surface defects in a product, characterized in that, include: A processor and a memory, the memory storing instructions that, when executed by the processor, cause the apparatus to perform the method as described in any one of claims 1 to 6.

8. An electronic device, characterized in that, include: The apparatus for detecting surface defects of a product as described in claim 7.

9. A computer-readable storage medium storing a computer program that, when executed, performs the method as described in any one of claims 1 to 6.

Citation Information

Patent Citations

  • Bearing surface deformation and character mixed defect image detection method

    CN112001396A

  • Mobile phone screen defect visual detection method and system

    CN114549416A