Image processing method and device, computer device, storage medium and program product

Abstract

The application provides an image processing method and device, computer equipment, a storage medium and a program product, and relates to the technical field of machine vision. The image processing method comprises the following steps: acquiring a to-be-detected image; performing feature extraction on the to-be-detected image based on at least two first convolutional layers to obtain at least two element image features corresponding to image elements respectively; the image elements comprise pixels, superpixels or subpixels; at least two element image features are synthesized through a second convolutional layer to generate a synthesized image feature; and a defect saliency image is generated based on the synthesized image feature, and defect detection is performed on the defect saliency image to obtain defect features in the to-be-detected image. The image processing method can effectively improve the accuracy of defect detection.

Description

Technical Field

[0001] This application relates to the field of machine vision technology, specifically to an image processing method, apparatus, computer equipment, storage medium, and program product. Background Technology

[0002] With the development of artificial intelligence and computer vision technologies, machine vision technology is being used more and more in industrial settings, and its importance is growing. For example, machine vision technology can be applied to quality control in industrial production processes, leading to the development of image recognition methods for detecting defects in product images.

[0003] Currently, defect detection can be performed by identifying regions of interest in an image and then based on those regions, but the accuracy of the final detection results still needs to be improved. Summary of the Invention

[0004] This application addresses the shortcomings of existing methods by proposing an image processing method, apparatus, electronic device, and storage medium to solve the technical problem of low accuracy in defect detection in existing technologies.

[0005] On one hand, embodiments of this application provide an image processing method, including:

[0006] Acquire the image to be detected;

[0007] Based on at least two first convolutional layers, feature extraction is performed on the image to be detected to obtain element image features corresponding to at least two types of image elements; the image elements include pixels, superpixels, or subpixels.

[0008] The second convolutional layer synthesizes at least two types of element image features to generate synthetic image features.

[0009] Based on the features of the synthesized image, a defect saliency image is generated, and defect detection is performed on the defect saliency image to obtain the defect features in the image to be detected.

[0010] In some possible implementations, acquiring the image to be detected includes:

[0011] Obtain the initial image to be detected, and extract the noise image from the initial image;

[0012] The initial image is denoised based on the noisy image to obtain the image to be detected.

[0013] In some possible implementations, at least two types of image elements include subpixels, pixels, and superpixels; the element image features include first image features, second image features, and third image features;

[0014] Based on at least two first convolutional layers, feature extraction is performed on the image to be detected to obtain element image features corresponding to at least two types of image elements, including:

[0015] Determine the convolution parameters corresponding to subpixels, pixels, and superpixels respectively;

[0016] Based on the determined convolution parameters, the corresponding first convolutional layers are used to extract features from the images to be detected, resulting in first image features corresponding to subpixels, second image features corresponding to pixels, and third image features corresponding to superpixels.

[0017] In some possible implementations, at least two types of elemental image features are synthesized through a second convolutional layer to generate synthesized image features, including:

[0018] The second convolutional layer determines the global image features corresponding to the first image features, the second image features, and the third image features, respectively.

[0019] The third convolutional layer synthesizes the global image features corresponding to the first, second, and third image features respectively, to obtain the synthesized image features.

[0020] In some possible implementations, a defect saliency image is generated based on synthetic image features, including:

[0021] Adjust the spatial array arrangement of features in the synthesized image to generate spatial arrangement features of the receptive field of view;

[0022] Based on the spatial arrangement characteristics of the receptive field of view, defect saliency images are generated.

[0023] In some possible implementations, the spatial array arrangement of the synthesized image features is adjusted to generate spatial arrangement features of the receptive field of view, including:

[0024] The receptive field constraint information for element image features, global image features, and synthetic image features is determined sequentially;

[0025] Based on the determined receptive field constraint information, the spatial array arrangement of the synthetic image features is adjusted to generate the spatial arrangement features of the receptive field of view.

[0026] In some possible implementations, the spatial arrangement features of the receptive field of view include the receptive field size and receptive field center coordinates of the synthetic image features;

[0027] Based on the determined receptive field constraint information, the spatial array arrangement of the synthetic image features is adjusted to generate spatial arrangement features of the receptive field of view, including:

[0028] The first relationship between the receptive fields of adjacent image features is established based on receptive field constraint information;

[0029] The receptive field size of the synthesized image features is determined based on the first relation;

[0030] A second relationship is established between the receptive field centers of adjacent image features based on receptive field constraint information;

[0031] The receptive field center coordinates of the synthetic image features are determined based on the second relation.

[0032] In some possible implementations, the spatial array arrangement of synthetic image features is adjusted based on the determined receptive field constraint information to generate spatial arrangement features of the receptive field of view, including:

[0033] Based on the determined receptive field constraint information, the spatial array arrangement of the synthetic image features is adjusted, and the holes in the synthetic image features are filled by convolution to generate the spatial arrangement features of the receptive field.

[0034] In some possible implementations, the receptive field constraint information includes the order of magnitude of the image features, the distance between any two adjacent image elements in the image features, the size of the receptive field, and the coordinates of the receptive field center.

[0035] On the other hand, embodiments of this application provide an image processing apparatus, including:

[0036] The acquisition module is used to acquire the image to be detected;

[0037] The extraction module is used to extract features from the image to be detected based on at least two first convolutional layers to obtain element image features corresponding to at least two types of image elements; the image elements include pixels, superpixels, or subpixels;

[0038] The synthesis module is used to synthesize at least two types of element image features through a second convolutional layer to generate synthesized image features;

[0039] The detection module is used to generate a defect saliency image based on the features of the synthesized image, perform defect detection on the defect saliency image, and obtain the defect features in the image to be detected.

[0040] In some possible implementations, the acquisition module, when acquiring the image to be detected, is specifically used for:

[0041] Obtain the initial image to be detected, and extract the noise image from the initial image;

[0042] The initial image is denoised based on the noisy image to obtain the image to be detected.

[0043] In some possible implementations, at least two types of image elements include subpixels, pixels, and superpixels; the element image features include first image features, second image features, and third image features;

[0044] When the extraction module extracts features from the image to be detected based on at least two first convolutional layers to obtain element image features corresponding to at least two types of image elements, it is specifically used for:

[0045] Determine the convolution parameters corresponding to subpixels, pixels, and superpixels respectively;

[0046] Based on the determined convolution parameters, the corresponding first convolutional layers are used to extract features from the images to be detected, resulting in first image features corresponding to subpixels, second image features corresponding to pixels, and third image features corresponding to superpixels.

[0047] In some possible implementations, when the synthesis module synthesizes at least two types of elemental image features through a second convolutional layer to generate synthesized image features, it is specifically used for:

[0048] The second convolutional layer determines the global image features corresponding to the first image features, the second image features, and the third image features, respectively.

[0049] The third convolutional layer synthesizes the global image features corresponding to the first, second, and third image features respectively, to obtain the synthesized image features.

[0050] In some possible implementations, when generating a defect saliency image based on synthetic image features, the detection module is specifically used for:

[0051] Adjust the spatial array arrangement of features in the synthesized image to generate spatial arrangement features of the receptive field of view;

[0052] Based on the spatial arrangement characteristics of the receptive field of view, defect saliency images are generated.

[0053] In some possible implementations, when the detection module adjusts the spatial array arrangement of the synthetic image features to generate spatial arrangement features of the receptive field of view, it is specifically used for:

[0054] The receptive field constraint information for element image features, global image features, and synthetic image features is determined sequentially;

[0055] Based on the determined receptive field constraint information, the spatial array arrangement of the synthetic image features is adjusted to generate the spatial arrangement features of the receptive field of view.

[0056] In some possible implementations, the spatial arrangement features of the receptive field include the receptive field size and receptive field center coordinates of the synthetic image features; when the detection module adjusts the spatial array arrangement of the synthetic image features based on the determined receptive field constraint information to generate the spatial arrangement features of the receptive field, it is specifically used for:

[0057] The first relationship between the receptive fields of adjacent image features is established based on receptive field constraint information;

[0058] The receptive field size of the synthesized image features is determined based on the first relation;

[0059] A second relationship is established between the receptive field centers of adjacent image features based on receptive field constraint information;

[0060] The receptive field center coordinates of the synthetic image features are determined based on the second relation.

[0061] In some possible implementations, when the detection module adjusts the spatial array arrangement of the synthetic image features based on the determined receptive field constraint information to generate spatial arrangement features of the receptive field, it specifically includes:

[0062] Based on the determined receptive field constraint information, the spatial array arrangement of the synthetic image features is adjusted, and the holes in the synthetic image features are filled by convolution to generate the spatial arrangement features of the receptive field.

[0063] In some possible implementations, the receptive field constraint information includes the order of magnitude of the image features, the distance between any two adjacent image elements in the image features, the size of the receptive field, and the coordinates of the receptive field center.

[0064] On the other hand, a computer device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the above-described image processing method.

[0065] On the other hand, a computer-readable storage medium is provided on which a computer program is stored, which, when executed by a processor, implements the above-described image processing method.

[0066] On the other hand, a computer program product is provided, including a computer program that, when executed by a processor, implements the image processing method described above.

[0067] The beneficial effects of the technical solution provided in this application are:

[0068] Synthetic image features, obtained by combining image features of at least two elements, can be combined with complementary information from at least two image elements in pixels, subpixels, or superpixels to make decisions. Defect detection can then be performed based on the synthetic image features, which can effectively improve the accuracy of defect detection.

[0069] Furthermore, by adjusting the spatial array arrangement of the synthetic image features, spatial arrangement features of the receptive field of view are generated, improving the consistency of spatial array density, brightness, and chromaticity of different image elements, enhancing the display effect of the synthetic image, and avoiding image grids that appear when the spatial arrays of different image elements have inconsistent densities, thereby improving the accuracy of defect detection.

[0070] Furthermore, as the convolutional hierarchy increases, the receptive field also increases. Through the shift in span and the increase in convolutional hierarchy, the receptive field can cover the entire image to be detected, as well as the input and output of features at different levels of the entire image. In the final defect saliency image, each sub-pixel, pixel, and superpixel will use all the contextual information of the entire image to be detected, thereby effectively improving the accuracy of defect detection.

[0071] Additional aspects and advantages of this application will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of this application. Attached Figure Description

[0072] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0073] Figure 1 A schematic diagram illustrating the implementation environment of an image processing method provided in this application embodiment;

[0074] Figure 2 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0075] Figure 3 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0076] Figure 4 This is a schematic diagram of an image processing scheme in one example of this application;

[0077] Figure 5 A schematic diagram illustrating an example of a void-filling scheme for this application;

[0078] Figure 6 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application;

[0079] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0080] The embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions of the embodiments of this application.

[0081] Those skilled in the art will understand that, unless otherwise stated, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in embodiments of this application mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element establish a connection relationship through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “and / or” as used herein indicates at least one of the items defined by the term; for example, “A and / or B” indicates implementation as “A,” or implementation as “A,” or implementation as “A and B.”

[0082] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. It should be noted that the following embodiments can be referenced, learned from or combined with each other, and the same terms, similar features and similar implementation steps in different embodiments will not be described again.

[0083] Figure 1 This is a schematic diagram of the application environment of an image processing method provided in an embodiment of the present invention. See also... Figure 1 The application environment may include an image acquisition device 101 and a terminal 102. Specifically, the terminal 102 acquires an image to be detected, extracts features from the image based on at least two first convolutional layers to obtain element image features corresponding to at least two types of image elements, synthesizes the at least two types of element image features through a second convolutional layer to generate a synthesized image feature, generates a defect saliency image based on the synthesized image feature, and performs defect detection on the defect saliency image to obtain defect features in the image to be detected.

[0084] Understandable, Figure 1 This illustrates an example application scenario and does not limit the application scenarios of the image processing method in this application. In other application scenarios, the server may process the image to be detected, or the terminal may have image acquisition capabilities.

[0085] Those skilled in the art will understand that a server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server or server cluster that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. A terminal can be a smartphone (such as an Android phone, iOS phone, etc.), tablet computer, laptop computer, digital radio receiver, MID (Mobile Internet Device), PDA (Personal Digital Assistant), desktop computer, smart home appliance, in-vehicle terminal (such as in-vehicle navigation terminal, in-vehicle computer, etc.), smart speaker, smartwatch, etc. Terminals and servers can be connected directly or indirectly through wired or wireless communication, but are not limited to these methods. Specific details can be determined based on the actual application scenario requirements and are not limited here.

[0086] The image processing method provided in this application embodiment can be applied to a terminal or a server.

[0087] Figure 2 This is a flowchart illustrating an image processing method provided in an embodiment of this application, taking a terminal as the executing entity as an example. Figure 2 As shown, the method may include the following steps:

[0088] Step S201: Obtain the image to be detected.

[0089] Specifically, an image acquisition device can acquire the image to be detected and send it to the terminal. Alternatively, the terminal device can acquire the image to be detected using its own image acquisition function, or the terminal can obtain the image to be detected from a server or other terminals.

[0090] Step S202: Based on at least two first convolutional layers, feature extraction is performed on the image to be detected to obtain element image features corresponding to at least two types of image elements respectively.

[0091] Image elements can be units used to measure images.

[0092] Specifically, image elements can include subpixels, pixels, or superpixels. A pixel is the basic unit that makes up a digital image. Pixels are 5.2µm apart and can be seen as connected macroscopically, but microscopically, there are infinitely smaller elements between them—the "pixels" between two physical pixels. These smaller elements are called subpixels, meaning they are even finer than pixels. Subpixels are the smallest unit of measurement in an image. Superpixels, most intuitively, are created by grouping pixels with similar characteristics together to form a larger, more representative element. Superpixels are the largest unit of measurement in an image.

[0093] Step S203: At least two types of element image features are synthesized through the second convolutional layer to generate synthesized image features.

[0094] In this process, synthesis can be achieved by converting the element image features corresponding to at least two types of image elements into global images of the same size, and then synthesizing them.

[0095] Specifically, at least two types of element image features can be determined separately, namely local image features and global image features. Then, the corresponding global image features are synthesized to obtain the synthesized image features.

[0096] Step S204: Based on the features of the synthesized image, generate a defect saliency image, perform defect detection on the defect saliency image, and obtain the defect features in the image to be detected.

[0097] Specifically, spatial arrays can be arranged for the features of the synthesized image to avoid image grids caused by inconsistent density of spatial arrays of different image elements. The specific processing procedure for the features of the synthesized image will be further explained in detail below.

[0098] In the above embodiments, the synthesized image features obtained by combining at least two types of image features can combine at least two types of image elements in pixels, subpixels or superpixels to make complementary information decisions, and then perform defect detection based on the synthesized image features, which can effectively improve the accuracy of defect detection.

[0099] In some possible implementations, step S201, acquiring the image to be detected, may include:

[0100] (1) Obtain the initial image to be detected, and extract the noise image from the initial image;

[0101] (2) The initial image is denoised based on the noise image to obtain the image to be detected.

[0102] Specifically, a pre-trained noise generation model can be used to extract noise images from the initial image.

[0103] In the specific implementation process, for different products to be inspected, corresponding noise generation models can be pre-trained based on product parameters such as texture, size and type of the products to be inspected, and then the corresponding noise images can be extracted.

[0104] Specifically, the noisy image can be subtracted from the initial image to obtain a clear image, and then the clear image can be used as the image to be detected.

[0105] In some possible implementations, at least two types of image elements include subpixels, pixels, and superpixels; the element image features include first image features, second image features, and third image features;

[0106] Step S202 extracts features from the image to be detected based on at least two first convolutional layers to obtain element image features corresponding to at least two types of image elements, which may include:

[0107] (1) Determine the convolution parameters corresponding to sub-pixels, pixels and superpixels respectively;

[0108] (2) Based on the determined convolution parameters, the corresponding first convolution layers are used to extract features from the images to be detected, so as to obtain the first image features corresponding to subpixels, the second image features corresponding to pixels and the third image features corresponding to superpixels.

[0109] The convolution parameters can include the kernel size and stride. For example, the first convolutional layer corresponding to a subpixel can be a 1×1 convolution with a stride of 1; the first convolutional layer corresponding to a pixel can be a 3×3 convolution with a stride of 3; and the first convolutional layer corresponding to a superpixel can be a 5×5 convolution with a stride of 5.

[0110] In some possible implementations, step S203, by synthesizing at least two types of elemental image features through a second convolutional layer to generate synthesized image features, may include:

[0111] (1) The global image features corresponding to the first image features, the second image features and the third image features are determined by the second convolutional layer respectively;

[0112] (2) The global image features corresponding to the first image features, the second image features and the third image features are synthesized by the third convolutional layer to obtain the synthesized image features.

[0113] Specifically, the size of global image features and the size of local image features can be set. For example, the size of subpixel local features can be set to W×H×1; the size of pixel local features can be set to W×H×3; and the size of superpixel local features can be set to W×H×5. Here, H is the height of the local or global feature, and W is the width of the local or global feature. The parameters of the pixel, subpixel, and superpixel convolutional autoencoder algorithms for local or global features can be set, and convolutional autoencoder operations can be performed on the three types of local or global features respectively.

[0114] In some possible implementations, such as Figure 3 As shown, step S204, which generates a defect saliency image based on the features of the synthesized image, may include:

[0115] Step S410: Adjust the spatial array arrangement of the synthesized image features to generate spatial arrangement features of the receptive field of view.

[0116] Specifically, the spatial array arrangement of synthetic image features can be adjusted based on receptive field constraint information to generate spatial arrangement features of the receptive field of view.

[0117] The receptive field constraint information may include the order of magnitude N of the image features, the distance J between any two adjacent image elements in the image features, the size of the receptive field F, and the coordinates S of the receptive field center.

[0118] Here, the distance J between adjacent image elements can be understood as the alignment of the feature element with the center of the receptive field on Layer 0 of the input image, i.e., the image to be detected. The distance between adjacent elements on the input image can also be understood as how many pixels, subpixels, or superpixels are equivalent to moving forward one step on the feature in the input image.

[0119] Specifically, step S410, adjusting the spatial array arrangement of the synthesized image features to generate spatial arrangement features of the receptive field of view, may include:

[0120] (1) Determine the receptive field constraint information for element image features, global image features and synthetic image features in sequence;

[0121] (2) Adjust the spatial array arrangement of the synthetic image features based on the determined receptive field constraint information to generate the spatial arrangement features of the receptive field of view.

[0122] Specifically, during the processing of the image to be detected based on elemental image feature Layer1, global image feature Layer2, and synthetic image feature Layer3, each element in elemental image feature Layer1 can be seen in global image feature Layer2. For example: Layer1, sub-pixel 1×1 Conv (convolution), Rate (step) = 1, W×H×1; pixel 3×3 Conv, Rate = 3, W×H×3; superpixel 5×5 Conv, Rate = 5, W×H×5. Similarly, each element in Layer1 can be seen in Layer3.

[0123] The receptive field is the region in the image that the input parameters of a feature element are influenced by. The receptive field is relative to the region in the previous layer that a feature element in a given layer sees; unless otherwise specified, the receptive field is the region in the image where the input self-convolution parameters are seen.

[0124] Specifically, the receptive field constraint information can be calculated based on the following formula:

[0125] N_output = [(N_input + 2P - K) / R] + 1 (1)

[0126] Where N_output is the order of magnitude of the output of the convolutional layer; N_input is the order of magnitude of the input to the convolutional layer; P is the hole filling size; K is the convolutional kernel size; and R is the stride.

[0127] Joutput = Jinput × R (2)

[0128] Where J_output is the distance between any two adjacent image elements in the image features output by the convolutional layer; J_in is the distance between any two adjacent image elements in the image features input to the convolutional layer; and R is the stride.

[0129] F_output = F_input + (K-1) × J_input (3)

[0130] Where F_output is the receptive field of the convolutional layer output; F_input is the receptive field of the convolutional layer input; J_input is the distance between any two adjacent image elements in the image features input to the convolutional layer; and K is the convolutional kernel size.

[0131] S_output = S_input + [(K-1) / 2-P] × J_input (4)

[0132] Where S_output is the receptive field center coordinate of the image features output by the convolutional layer; S_input is the receptive field center coordinate of the image features input to the convolutional layer; J_input is the distance between every two adjacent image elements in the image features input to the convolutional layer; K is the convolutional kernel size; and P is the hole filling size.

[0133] Step S420: Based on the spatial arrangement features of the receptive field of view, generate a defect saliency image.

[0134] Among them, the spatial arrangement features of the receptive field include the size of the receptive field and the coordinates of the receptive field center of the synthetic image features.

[0135] Specifically, step S420, based on the determined receptive field constraint information, adjusts the spatial array arrangement of the synthesized image features to generate spatial arrangement features of the receptive field of view, which may include:

[0136] (1) Establish the first relationship between the receptive fields of adjacent image features based on receptive field constraint information;

[0137] (2) Determine the receptive field size of the synthesized image features based on the first relation;

[0138] (3) Establish a second relationship between the receptive field centers of adjacent image features based on receptive field constraint information;

[0139] (4) Determine the receptive field center coordinates of the synthetic image features based on the second relation.

[0140] In one example, assuming all layers are convolutional layers, if an element A on the global image feature Layer2 sees a pixel range of 3×3 on the element image feature Layer1, which is equal to the size of the convolution kernel K2, then the receptive field F2 seen by A is equivalent to the pixel range of 3×3 on the element image feature Layer1 and the range of the image to be detected Layer0 seen by A.

[0141] Establish the receptive field relationship between adjacent layers. Based on the fact that the receptive field F2 seen by layer A is equivalent to a 3×3 pixel range on layer 1, and the range seen by layer 0, establish the receptive field relationship between adjacent layers. The formula for calculating the receptive field relationship between adjacent layers, i.e., the first relationship, is as follows:

[0142] FL=(FL-1)+(K1-1)×(JL-1) (5)

[0143] Where FL is the receptive field of Layer L; (FL-1) is the receptive field of (Layer L-1); the receptive field FL of an element in Layer L is equivalent to the superposition of K×K receptive fields on (Layer L-1); the receptive field of an element on (Layer L-1) is (FL-1).

[0144] The receptive field of K consecutive elements in Layer L can be understood as the receptive field seen by the first element plus the range swept by the remaining (K-1) steps; each step forward in (Layer L-1) is equivalent to moving forward (JL-1) elements in the input image, and the result is equivalent to (FL-1)+(K-1)×(JL-1), where Conv1: K1=3, P=1, R1=2, J1=(K2-1)×J1).

[0145] Moving forward one element in LayerL is equivalent to moving forward RL elements in (LayerL-1). Converted to pixels, the calculation formula is:

[0146] JL=(JL-1)×RL (6)

[0147] Where RL is the convolution stride of the kernel of ConvL when it moves forward on (Layer L-1), and RL = 1 for the input image.

[0148] According to the recursive formula:

[0149]

[0150] Where RI represents the number of pixels that move forward one element on the feature map Layer L, which corresponds to moving forward one pixel on the input image. If the elements on the feature map are aligned with the center of the receptive field, then JL is the pixel distance between the centers of the receptive fields.

[0151] In this process, the K, P, and R hyperparameters of each layer are marked in the image, and hyperparameter hiding is performed when it is not necessary to mark them.

[0152] Moving forward one element in LayerL is equivalent to moving forward one element in the input image. Each pixel represents a large product of all preceding layers.

[0153] Furthermore, we learned that the receptive field size of Layer L is:

[0154]

[0155] The relationship between the receptive field centers of adjacent layers, i.e., the second relationship, is as follows:

[0156] Start L = Start (L-1) + {(KL–1) / 2 - PL} × (JL–1) (9)

[0157] All starting coordinates are relative to the input image coordinate system; starting 0 = (0.5, 0.5) is the center coordinate of the top left pixel of the input image; starting L-1 represents the center coordinate of the receptive field of the top left element of (Layer L-1); (2KL-1-PL) is the deviation between the receptive field center of Layer L and (Layer L-1) relative to the (Layer L-1) coordinate system. This deviation needs to be converted before being input into the image coordinate system.

[0158] Its value needs to be multiplied by (JL–1), which is the subpixel distance, pixel distance, and superpixel distance between adjacent elements (LayerL–1). The result of the multiplication is (2KL–1–PL)*(JL–1), which means that the subpixel distance, pixel distance, and superpixel distance between the centers of the receptive field are equal to the coordinate system relative to the input image.

[0159] Wherein, Conv2: K2 = 3; P2 = 1; R2 = 2.

[0160] In the above example, the receptive field size of Layer L is not related to PL and RL. The receptive field size of the current feature map element is not related to the receptive field size of the current convolutional layer or the pixel distance between adjacent elements of the current convolutional layer. The padding size is set to the radius of K, for example; P = 2(K-1), resulting in start L = (start-1); making the receptive field center coordinates of the element at the (X,Y) coordinate position on the feature map L (XJL,YJL). Dilated convolution is equivalent to changing the size of the convolution kernel. If a threshold parameter for dilated convolution is included, simply replace KL with the threshold parameter and multiply by (KL-1)+1; when the threshold parameter for dilated convolution is equal to 1, it is a normal convolution. A merged layer is defined as a special convolutional layer; similarly, K, P, and R are also convolutional layer hyperparameters; when performing non-linear convolution, it is a stepwise, point-by-point, element-by-element operation without changing the receptive field. A certain position in a feature map of a certain layer is calculated from the input of the fixed region of the previous convolution layer. The (XJL,YJL) coordinates of the fixed region of the convolution are the center of the receptive field; the element at the (X,Y) coordinate position is the receptive field of the fixed region of the convolution.

[0161] like Figure 4As shown in one example, the image to be detected, i.e., the clear image, is obtained. The corresponding first convolutional layers are used to extract features from the image to be detected, resulting in first image features corresponding to subpixels, second image features corresponding to pixels, and third image features corresponding to superpixels. A second convolutional layer is used to determine global image features corresponding to the first, second, and third image features, respectively. A third convolutional layer is used to synthesize the global image features corresponding to the first, second, and third image features, respectively, to obtain synthesized image features. The spatial array arrangement of the synthesized image features is adjusted to generate spatial arrangement features of the receptive field of view. Based on the spatial arrangement features of the receptive field of view, a defect saliency image is generated.

[0162] In some possible implementations, the spatial array arrangement of synthetic image features is adjusted based on the determined receptive field constraint information to generate spatial arrangement features of the receptive field of view, including:

[0163] Based on the determined receptive field constraint information, the spatial array arrangement of the synthetic image features is adjusted, and the holes in the synthetic image features are filled by convolution to generate the spatial arrangement features of the receptive field.

[0164] The process of receptive field calculation and hole filling will be further explained below with reference to the accompanying figures and examples.

[0165] like Figure 5 As shown in the example, each convolutional kernel has a size of 3 and a span of 1. The first layer has a receptive field of 3, i.e., 3 = 1 + (3-1) × 1, which is visually represented as a 20% trapezoid. The second layer has a receptive field of 5, i.e., 5 = 1 + (5-1) × 1, which is visually represented as a 40% trapezoid. The third layer has a receptive field of 7, i.e., 7 = 1 + (7-1) × 1, which is visually represented as a 60% trapezoid. The receptive field features are represented by rectangles. When the first layer is input to the convolutional kernel and its span, it changes the features of the first layer; all the output features of the first layer change the features of the second layer; all the output features of the second layer change the features of the third layer; and all the output features of the third layer change the features of the fourth layer.

[0166] In practical implementation, hyperparameters such as input convolution and span in convolutional autoencoders alter elements in feature maps across all levels. Elements closer to the center of the receptive field contribute more to the features, meaning more information is concentrated in the center of the receptive field, while less is processed at the edges. In practical applications, a span of 1 is used to move the span, causing the receptive fields in the feature maps to converge. If the span is greater than 1, it will change the features of subsequent layers exponentially. As the convolutional layers increase, the receptive field increases accordingly. Through span movement and increasing convolutional layers, the receptive field can cover the entire image and the input and output of features at different levels. Ultimately, each subpixel, pixel, and superpixel in the output feature map or saliency map utilizes all the contextual information of the entire image.

[0167] During hole filling, rules can be followed such as filling from left to right, right to left, top to bottom, and bottom to top, using hierarchical convolution and network convolution to calculate and fill the receptive field. Depending on the visualized shape of the receptive field, it can be pyramidal, inverted pyramidal, trapezoidal, inverted trapezoidal, linear pyramidal, linear inverted pyramidal, linear trapezoidal, linear inverted trapezoidal, or a combination thereof. A linear transformation is applied to the input signal to minimize errors in image reconstruction, and the linear transformation weights are shared; this is to hide the linear combination of encoded basic image blocks during reconstruction. In image network convolution, there can be a single network convolution, multiple network convolutions, or a combination of single or multiple network convolutions.

[0168] When an image has holes, the convolutional autoencoder performs convolution to fill the holes, with the filling content being the convolution kernel weights, and automatically processes the input and output images. Through the dilation rate α, after dilated convolution, the convolution kernel size K above is transformed into α×(K-1)+1. Obviously, K in the above receptive field calculation formula (8) can be replaced.

[0169] The image processing method described above, by synthesizing image features from at least two types of image features, can combine complementary information from at least two types of image elements in pixels, subpixels, or superpixels to make decisions, and then perform defect detection based on the synthesized image features, which can effectively improve the accuracy of defect detection.

[0170] Furthermore, by adjusting the spatial array arrangement of the synthetic image features, spatial arrangement features of the receptive field of view are generated, improving the consistency of spatial array density, brightness, and chromaticity of different image elements, enhancing the display effect of the synthetic image, and avoiding image grids that appear when the spatial arrays of different image elements have inconsistent densities, thereby improving the accuracy of defect detection.

[0171] Furthermore, as the convolutional hierarchy increases, the receptive field also increases. Through the shift in span and the increase in convolutional hierarchy, the receptive field can cover the entire image to be detected, as well as the input and output of features at different levels of the entire image. In the final defect saliency image, each sub-pixel, pixel, and superpixel will use all the contextual information of the entire image to be detected, thereby effectively improving the accuracy of defect detection.

[0172] Figure 6 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application. Figure 6 As shown, the device includes:

[0173] The acquisition module 601 is used to acquire the image to be detected;

[0174] The extraction module 602 is used to extract features from the image to be detected based on at least two first convolutional layers to obtain element image features corresponding to at least two types of image elements; the image elements include pixels, superpixels, or subpixels;

[0175] The synthesis module 603 is used to synthesize at least two types of element image features through a second convolutional layer to generate synthesized image features;

[0176] The detection module 604 is used to generate a defect saliency image based on the features of the synthesized image, perform defect detection on the defect saliency image, and obtain the defect features in the image to be detected.

[0177] In some possible implementations, when acquiring the image to be detected, the acquisition module 601 is specifically used for:

[0178] Obtain the initial image to be detected, and extract the noise image from the initial image;

[0179] The initial image is denoised based on the noisy image to obtain the image to be detected.

[0180] In some possible implementations, at least two types of image elements include subpixels, pixels, and superpixels; the element image features include first image features, second image features, and third image features;

[0181] When the extraction module 602 extracts features from the image to be detected based on at least two first convolutional layers to obtain element image features corresponding to at least two types of image elements, it is specifically used for:

[0182] Determine the convolution parameters corresponding to subpixels, pixels, and superpixels respectively;

[0183] Based on the determined convolution parameters, the corresponding first convolutional layers are used to extract features from the images to be detected, resulting in first image features corresponding to subpixels, second image features corresponding to pixels, and third image features corresponding to superpixels.

[0184] In some possible implementations, when the synthesis module 603 synthesizes at least two types of element image features through the second convolutional layer to generate synthesized image features, it is specifically used for:

[0185] The second convolutional layer determines the global image features corresponding to the first image features, the second image features, and the third image features, respectively.

[0186] The third convolutional layer synthesizes the global image features corresponding to the first, second, and third image features respectively, to obtain the synthesized image features.

[0187] In some possible implementations, when generating a defect saliency image based on synthetic image features, the detection module 604 is specifically used for:

[0188] Adjust the spatial array arrangement of features in the synthesized image to generate spatial arrangement features of the receptive field of view;

[0189] Based on the spatial arrangement characteristics of the receptive field of view, defect saliency images are generated.

[0190] In some possible implementations, the spatial arrangement features of the receptive field of view include the receptive field size and receptive field center coordinates of the synthetic image features;

[0191] When the detection module 604 adjusts the spatial array arrangement of the synthetic image features based on the determined receptive field constraint information to generate the spatial arrangement features of the receptive field of view, it is specifically used for:

[0192] The first relationship between the receptive fields of adjacent image features is established based on receptive field constraint information;

[0193] The receptive field size of the synthesized image features is determined based on the first relation;

[0194] A second relationship is established between the receptive field centers of adjacent image features based on receptive field constraint information;

[0195] The receptive field center coordinates of the synthetic image features are determined based on the second relation.

[0196] In some possible implementations, the spatial arrangement features of the receptive field of view include the receptive field size and receptive field center coordinates of the synthetic image features;

[0197] When the detection module adjusts the spatial array arrangement of the synthetic image features based on the determined receptive field constraint information to generate the spatial arrangement features of the receptive field of view, it is specifically used for:

[0198] The first relationship between the receptive fields of adjacent image features is established based on receptive field constraint information;

[0199] The receptive field size of the synthesized image features is determined based on the first relation;

[0200] A second relationship is established between the receptive field centers of adjacent image features based on receptive field constraint information;

[0201] The receptive field center coordinates of the synthetic image features are determined based on the second relation.

[0202] In some possible implementations, when the detection module 604 adjusts the spatial array arrangement of the synthetic image features based on the determined receptive field constraint information to generate spatial arrangement features of the receptive field of view, it specifically includes:

[0203] Based on the determined receptive field constraint information, the spatial array arrangement of the synthetic image features is adjusted, and the holes in the synthetic image features are filled by convolution to generate the spatial arrangement features of the receptive field.

[0204] In some possible implementations, the receptive field constraint information includes the order of magnitude of the image features, the distance between any two adjacent image elements in the image features, the size of the receptive field, and the coordinates of the receptive field center.

[0205] The image processing device described above can combine at least two types of image features to obtain synthetic image features, which can make complementary information decisions based on at least two types of image elements in pixels, subpixels, or superpixels, and then perform defect detection based on the synthetic image features, thereby effectively improving the accuracy of defect detection.

[0206] Furthermore, by adjusting the spatial array arrangement of the synthetic image features, spatial arrangement features of the receptive field of view are generated, improving the consistency of spatial array density, brightness, and chromaticity of different image elements, enhancing the display effect of the synthetic image, and avoiding image grids that appear when the spatial arrays of different image elements have inconsistent densities, thereby improving the accuracy of defect detection.

[0207] Furthermore, as the convolutional hierarchy increases, the receptive field also increases. Through the shift in span and the increase in convolutional hierarchy, the receptive field can cover the entire image to be detected, as well as the input and output of features at different levels of the entire image. In the final defect saliency image, each sub-pixel, pixel, and superpixel will use all the contextual information of the entire image to be detected, thereby effectively improving the accuracy of defect detection.

[0208] In one alternative embodiment, a computer device is provided, such as Figure 7 As shown, Figure 7 The computer device 700 shown includes a processor 701 and a memory 703. The processor 701 and the memory 703 are connected, for example, via a bus 702. Optionally, the computer device 700 may further include a transceiver 704, which can be used for data interaction between the computer device and other computer devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 704 is not limited to one type, and the structure of the computer device 700 does not constitute a limitation on the embodiments of this application.

[0209] Processor 701 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 701 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0210] Bus 702 may include a pathway for transmitting information between the aforementioned components. Bus 702 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 702 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0211] The memory 703 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0212] The memory 703 stores application code (computer program) that executes the solution of this application, and its execution is controlled by the processor 701. The processor 701 executes the application code stored in the memory 703 to implement the content shown in the foregoing method embodiments.

[0213] Computer equipment includes, but is not limited to: virtualized computer equipment, virtual machines, servers, service clusters, and user terminals.

[0214] This application provides a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to execute the corresponding content of the image processing method described in the foregoing method embodiments.

[0215] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image processing method described above.

[0216] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0217] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0218] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0219] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0220] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0221] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0222] The modules described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a module does not necessarily limit the module itself; for example, an extraction module can also be described as "a module for extracting user key point features".

[0223] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

Claims

1. An image processing method, characterized in that, include: Acquire the image to be detected; The image to be detected is subjected to feature extraction based on at least two first convolutional layers to obtain element image features corresponding to at least two types of image elements; the image elements include pixels, superpixels, or subpixels. The image features of the at least two elements are synthesized through a second convolutional layer to generate a synthetic image feature; Based on the synthesized image features, a defect saliency image is generated, and defect detection is performed on the defect saliency image to obtain the defect features in the image to be detected; The step of generating a defect saliency image based on the synthesized image features includes: Adjust the spatial array arrangement of the synthesized image features to generate spatial arrangement features of the receptive field of view; Based on the spatial arrangement characteristics of the receptive field of view, a defect salience image is generated; The step of adjusting the spatial array arrangement of the synthesized image features to generate spatial arrangement features of the receptive field of view includes: The receptive field constraint information for the element image features, global image features and the synthesized image features is determined sequentially, wherein each element in the element image features can be seen in the global image features; Based on the determined receptive field constraint information, the spatial array arrangement of the synthetic image features is adjusted to generate the spatial arrangement features of the receptive field of view, which include the receptive field size and receptive field center coordinates of the synthetic image features.

2. The image processing method according to claim 1, characterized in that, The acquisition of the image to be detected includes: Obtain the initial image to be detected, and extract the noise image from the initial image; The initial image is denoised based on the noise image to obtain the image to be detected.

3. The image processing method according to claim 1, characterized in that, The at least two types of image elements include subpixels, pixels, and superpixels; the element image features include first image features, second image features, and third image features; The step of extracting features from the image to be detected based on at least two first convolutional layers to obtain element image features corresponding to at least two types of image elements includes: Determine the convolution parameters corresponding to subpixels, pixels, and superpixels respectively; Based on the determined convolution parameters, the corresponding first convolutional layers are used to extract features from the image to be detected, resulting in first image features corresponding to subpixels, second image features corresponding to pixels, and third image features corresponding to superpixels.

4. The image processing method according to claim 3, characterized in that, The step of synthesizing the at least two types of element image features through a second convolutional layer to generate synthesized image features includes: The second convolutional layer determines global image features corresponding to the first image feature, the second image feature, and the third image feature, respectively. The global image features corresponding to the first image feature, the second image feature, and the third image feature are synthesized by the third convolutional layer to obtain the synthesized image features.

5. The image processing method according to claim 1, characterized in that, The step of adjusting the spatial array arrangement of the synthetic image features based on the determined receptive field constraint information to generate spatial arrangement features of the receptive field of view includes: A first relationship between the receptive fields of adjacent image features is established based on the receptive field constraint information; The receptive field size of the synthesized image features is determined based on the first relationship; A second relationship is established between the receptive field centers of adjacent image features based on the receptive field constraint information; The receptive field center coordinates of the synthesized image features are determined based on the second relationship.

6. The image processing method according to claim 1, characterized in that, The step of adjusting the spatial array arrangement of the synthetic image features based on the determined receptive field constraint information to generate spatial arrangement features of the receptive field of view includes: Based on the determined receptive field constraint information, the spatial array arrangement of the synthetic image features is adjusted, and the holes in the synthetic image features are filled by convolution to generate the spatial arrangement features of the receptive field of view.

7. The image processing method according to claim 1, characterized in that, The receptive field constraint information includes the order of magnitude of the image features, the distance between any two adjacent image elements in the image features, the size of the receptive field, and the coordinates of the receptive field center.

8. An image processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire the image to be detected; An extraction module is used to extract features from the image to be detected based on at least two first convolutional layers to obtain element image features corresponding to at least two types of image elements; the image elements include pixels, superpixels, or subpixels. The synthesis module is used to synthesize the image features of the at least two elements through a second convolutional layer to generate synthesized image features; The detection module is used to generate a defect saliency image based on the features of the synthesized image, perform defect detection on the defect saliency image, and obtain the defect features in the image to be detected; The detection module is specifically used to adjust the spatial array arrangement of the synthetic image features to generate spatial arrangement features of the receptive field of view; and to generate a defect saliency image based on the spatial arrangement features of the receptive field of view. The detection module is specifically used to sequentially determine the receptive field constraint information for the element image features, global image features and the synthetic image features, wherein each element in the element image features can be seen in the global image features; Based on the determined receptive field constraint information, the spatial array arrangement of the synthetic image features is adjusted to generate the spatial arrangement features of the receptive field of view, which include the receptive field size and receptive field center coordinates of the synthetic image features.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the image processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image processing method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image processing method according to any one of claims 1 to 7.

Images