Image processing method, device, equipment, medium and program product

Abstract

The present application discloses an image processing method, device, equipment, medium and program product, which relate to the field of machine learning. The method includes: obtaining an image denoising result corresponding to a first image; performing semantic analysis on the first image to obtain semantic analysis data; performing image denoising on the image denoising result based on the semantic analysis data to obtain a second image; and determining a defect recognition result corresponding to the first image based on the feature similarity between the first image and the second image. That is, the semantic analysis data guides the image denoising to ensure that the image object in the denoised second image is consistent with the image object in the first image, and then compares the feature similarity to determine the defect recognition result, which can improve the accuracy and efficiency of image object recognition.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of machine learning, and in particular to an image processing method, apparatus, device, medium and program product. Background Art

[0002] Defect detection is often used in industrial production processes to identify manufacturing defects in products by inspecting product images.

[0003] In related technologies, defect detection methods usually involve pre-training a denoising model using normal sample images, performing denoising on the product image to be identified and then inputting the denoising model for denoising to obtain a denoised reconstructed image. By comparing the difference between the reconstructed image and the product image, the image object with defects in the product image is identified.

[0004] However, in the related art, denoising errors may occur in the process of denoising the noisy product image through the denoising model, resulting in inconsistency between the image objects in the reconstructed image and the product image, making it impossible to determine the defective image object by comparing the reconstructed image and the product image, thereby reducing the recognition accuracy of the image object. Summary of the invention

[0005] The embodiments of the present application provide an image processing method, apparatus, device, medium and program product, which can improve the recognition accuracy of image objects. The technical solution is as follows:

[0006] In one aspect, an image processing method is provided, the method comprising:

[0007] Acquire an image noise addition result corresponding to a first image, where the image noise addition result is a result of adding first noise data to the first image, where the first image includes an image object;

[0008] Performing semantic analysis on the first image to obtain semantic analysis data, where the semantic analysis data is used to represent information of the image object in the first image;

[0009] Performing image denoising processing on the image denoising result based on the semantic analysis data to obtain a second image;

[0010] Based on the feature similarity between the first image and the second image, a defect recognition result corresponding to the first image is determined, and the defect recognition result includes the defect situation of the image object.

[0011] In another aspect, an image processing device is provided, the device comprising:

[0012] An acquisition module, configured to acquire an image noise addition result corresponding to a first image, wherein the image noise addition result is a result obtained by adding first noise data to the first image, wherein the first image includes an image object;

[0013] An analysis module, configured to perform semantic analysis on the first image to obtain semantic analysis data, wherein the semantic analysis data is used to characterize information of the image object in the first image;

[0014] A denoising module, configured to perform image denoising processing on the image denoising result based on the semantic analysis data to obtain a second image;

[0015] A determination module is used to determine a defect recognition result corresponding to the first image based on feature similarity between the first image and the second image, wherein the defect recognition result includes a defect condition of the image object.

[0016] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the image processing method as described in any of the above embodiments.

[0017] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, a code set or an instruction set is stored in the storage medium, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the image processing method as described in any of the above embodiments.

[0018] On the other hand, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the image processing method described in any of the above embodiments.

[0019] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0020] After adding the first noise data to the first image to obtain the image noise result, the first image is subjected to semantic analysis to obtain semantic analysis data corresponding to the information used to characterize the image object in the first image, and then the image noise result is subjected to image denoising based on the semantic analysis data to obtain the second image, and finally the defect corresponding to the first image is determined by comparing the feature similarity between the first image and the second image. That is, by performing semantic analysis on the first image to obtain the semantic analysis data, the semantic analysis data is used to guide the image denoising, ensuring that the image object in the denoised second image is consistent with the image object in the first image, and then comparing the feature similarity to determine the defect of the image object in the first image, which can improve the accuracy and efficiency of image object defect recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 It is a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application;

[0023] Figure 2 is a flow chart of an image processing method provided by an exemplary embodiment of the present application;

[0024] Figure 3 is a flowchart of an image processing method provided by another exemplary embodiment of the present application;

[0025] Figure 4 is a schematic diagram of a feature fusion process provided by another exemplary embodiment of the present application;

[0026] Figure 5 is a flowchart of an image processing method provided by another exemplary embodiment of the present application;

[0027] Figure 6 is a flow chart of a training process of an image processing method provided by another exemplary embodiment of the present application;

[0028] Figure 7 is a schematic diagram of an image processing method provided by another exemplary embodiment of the present application;

[0029] Figure 8 is a schematic diagram of the effect of an image processing method provided by another exemplary embodiment of the present application;

[0030] Fig. 9is a structural block diagram of an image processing device provided by an exemplary embodiment of the present application;

[0031] Fig.10 is a structural block diagram of an image processing device provided by another exemplary embodiment of the present application;

[0032] Fig.11 It is a structural block diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the present application clearer, the implementation mode of the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0034] In this application, the terms "first", "second", etc. are used to distinguish identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first" and "second", nor is there any limitation on quantity and execution order.

[0035] Figure 1 is a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application, such as Figure 1 As shown, the implementation environment includes a terminal 110, a server 120 and a communication network 130, wherein the terminal 110 and the server 120 are connected via the communication network 130. Optionally, the communication network 130 may be a wired network or a wireless network, which is not limited here.

[0036] In the embodiment of the present application, the terminal 110 is used to send data to the server 120. Optionally, a target application with an image recognition function is installed in the terminal 110, which is not limited in this embodiment. Schematically, the target application can be a traditional application, a cloud application, a mini-program or an application module in a host application, or a web platform, which is not limited in this embodiment.

[0037] After receiving the first image, the terminal 110 generates an object recognition request based on the first image and sends it to the server 120 . The object recognition request is used to request recognition of an image object in the first image.

[0038] After receiving the object recognition request, the server 120 adds the first noise data to the first image, thereby obtaining an image noise addition result corresponding to the first image, and performs semantic analysis on the first image to obtain semantic analysis data. The image noise addition result is subjected to image denoising based on the semantic analysis data to obtain a second image, thereby determining a defect recognition result corresponding to the first image based on the feature similarity between the first image and the second image. The defect recognition result is fed back to the terminal 110 for display.

[0039] In some optional embodiments, the terminal 110 is a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart home appliance, a smart car terminal, a smart speaker, an intelligent voice interaction device, an aircraft, etc., but is not limited thereto.

[0040] It is worth noting that server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network) and big data and artificial intelligence platforms.

[0041] Among them, cloud technology refers to a hosting technology that unifies hardware, software, network and other series of resources in a wide area network or local area network to realize data calculation, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model application, which can form a resource pool, which is used on demand and flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, each item may have its own identification mark in the future, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately, and all kinds of industry data require strong system backing support, which can only be achieved through cloud computing. Optionally, the server 120 can also be implemented as a node in the blockchain system.

[0042] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards in the relevant regions.

[0043] Combined with the above introduction and implementation environment, Figure 2 : is a flowchart of an image processing method provided in an embodiment of the present application. The method can be applied to a terminal, a server, or both. The present application embodiment takes the method applied to a server as an example for explanation. The method includes:

[0044] Step 210: Obtain an image noise addition result corresponding to the first image.

[0045] The image noise addition result is a result obtained by adding first noise data to the first image, and the first image includes an image object.

[0046] Illustratively, the first image is an image containing at least one image object.

[0047] Optionally, the first image is an R (Red) G (Green) B (Blue) image; or, the first image is a grayscale image.

[0048] In some embodiments, the image object in the first image is used for defect recognition, wherein defect recognition refers to identifying whether there is a defective part in the image object, for example, when the first image is a screw, identifying whether there is a concave part on the screw cap.

[0049] Optionally, the image object in the first image is displayed in a two-dimensional form; or, the image in the first image is displayed in a three-dimensional form.

[0050] In some embodiments, noise data refers to data that interferes with image content in an image. For example, noise may cause the image to become blurred, distorted, or contain random brightness or color changes.

[0051] Optionally, the noise data includes at least one of salt and pepper noise (black and white pixels that appear randomly in an image), Gaussian noise (pixel values ​​in an image are affected by normally distributed random noise), analog signal noise (noise introduced due to interference such as noise from electronic components during image transmission and acquisition), compression noise (noise introduced during image compression), blur noise (noise corresponding to image blur caused by vibration, motion blur, optical blur, etc. during shooting or transmission), color noise (noise differences between different color channels in an RGB image, such as color aberration), or environmental noise (noise data generated from the environment in the image, such as light changes, shadows, and reflections).

[0052] Optionally, the first noise data is obtained in at least one of the following ways:

[0053] First, a noise data set is acquired in advance, and at least one noise data is randomly adjusted from the noise data set as the first noise data;

[0054] Second, perform noise analysis on the first image. If there is noise data in the first image itself, after obtaining the noise data, copy the noise data multiple times and use the copy result as the first noise data. For example, the first image contains 10 pixels, of which pixel 2 corresponds to Gaussian noise. The high tearing noise is introduced as the first noise data into other pixels to change their respective pixel values.

[0055] It is worth noting that the above-mentioned method for obtaining the first noise data is only an example for adaptation and is not limited to this in the embodiment of the present application.

[0056] Optionally, the first noise data includes at least one type of noise data. When the first noise data includes multiple types of noise data, the multiple types of noise data are noise data of the same noise type, or are noise data of different noise types.

[0057] In some embodiments, the process of adding the first noise data to the first image is called an image noise adding process.

[0058] Optionally, the image noise addition process is performed once or multiple times. When the image noise addition process is performed multiple times, the noise addition result of the previous adding of the first noise data is used as the data to be processed for the current noise addition process. For example, the first noise data is added to image a to obtain image noise addition result 1, which is used as the first image noise addition process; the first noise data is added to image noise addition result 2 to obtain image noise addition result 2, which is used as the second image noise addition process, until the image noise addition process is completed, and the image noise addition result obtained by the last noise addition process is used as the image noise addition result corresponding to the first image.

[0059] The noise data applied in each execution of the image noise addition process may be the same or different, and this is not limited.

[0060] In an illustrative manner, a specified number of steps is obtained in advance, and noise processing is performed on the first image according to the specified number of steps.

[0061] Step 220: perform semantic analysis on the first image to obtain semantic analysis data.

[0062] The semantic analysis data is used to represent information of the image object in the first image.

[0063] Optionally, the information of the image object includes at least one of information types such as object direction, position distribution, placement posture, object category, etc. of the image object in the first image.

[0064] The object direction refers to the direction of the image object in the first image. For example, the first image includes a screw, and the screw cap faces the left side of the first image.

[0065] The position distribution refers to the position of the image object in the first image. For example, the first image includes a gear, and the gear is located in the upper left area of ​​the first image.

[0066] The placement posture refers to the posture angle of the image object in the first image. For example, the first image includes a gear, wherein the lower edge line of the first image is taken as the horizontal line, and the gear is displayed in the first image at an angle of 45 degrees rotated counterclockwise from the horizontal line.

[0067] The object category refers to the classification corresponding to the image object. For example, image a includes gears, and image b includes nuts. Gears and nuts correspond to different workpiece categories.

[0068] Schematically, the semantic analysis method includes at least one of the following methods:

[0069] First, pre-train a semantic analysis model, input the first image into the semantic analysis model, and output semantic analysis data corresponding to the first image;

[0070] Second, the pixel values ​​corresponding to each pixel point in the first image are obtained, and a pixel value threshold is set. The pixel points that reach the pixel value threshold are used as the pixel points corresponding to the image object in the first image, thereby obtaining the pixel point distribution of the image object in the first image. The position distribution of the image object in the first image can be obtained based on the pixel point distribution. The object contour of the image object can also be obtained based on the pixel point distribution, and the object category, object direction and placement posture corresponding to the image object can be obtained in turn.

[0071] It is worth noting that the above semantic analysis method is only an illustrative example and is not limited to this in the embodiments of the present application.

[0072] In the case of obtaining the semantic analysis model through pre-training as mentioned above, optionally, the semantic analysis model includes at least one of the model types such as a word vector model, a bag-of-words model, a document vector model, a recurrent neural network (RNN) model (for example, a long short-term memory network, a gated recurrent unit, etc.), a convolutional neural network (CNN) model, an attention mechanism model (for example, a Transformer model), and a pre-trained language model (for example, a BERT model).

[0073] Optionally, the semantic analysis data refers to state information expressed by a single image object; or, the semantic analysis data includes state information corresponding to a plurality of image objects.

[0074] Step 230: performing image denoising processing on the image denoising result based on the semantic analysis data to obtain a second image.

[0075] In some embodiments, image denoising processing refers to suppressing or eliminating noise data in the first image.

[0076] Optionally, the image denoising process includes at least one of the following methods:

[0077] First, mean filtering: replace each pixel in the image with the mean of its surrounding pixels;

[0078] Second, median filtering: replace each pixel in the image with the median value of its surrounding pixels;

[0079] Third, Gaussian filtering: use Gaussian function to blur the image;

[0080] Fourth, wavelet transform: convert the image to the wavelet domain and remove noise by thresholding the wavelet coefficients;

[0081] Fifth, total variation denoising: The total variation regularization model is used to optimize the image and reduce noise by minimizing the total variation of the image;

[0082] Sixth, non-local mean denoising: using non-local similarity in the image to estimate the noise and perform denoising;

[0083] Seventh, the method based on deep learning: pre-train a neural network model, and use the neural network model to learn and denoise the image.

[0084] It is worth noting that the above-mentioned method for image denoising is only an illustrative example and is not limited to this in the embodiments of the present application.

[0085] In some embodiments, the second image refers to an image generated by performing image denoising processing on the image denoising result.

[0086] Optionally, the first image and the second image are the same, or there is a certain difference between the first image and the second image.

[0087] Illustratively, when the first image is an image in which a defective portion exists in the image object, the second image is a corresponding image in which the defective portion does not exist in the image object in the first image.

[0088] In some embodiments, the image denoising result is guided by the semantic analysis data to perform image denoising processing, thereby ensuring that the state information corresponding to the image object in the denoised second image is consistent with the state information in the first image.

[0089] Step 240: Determine a defect recognition result corresponding to the first image based on the feature similarity between the first image and the second image.

[0090] The defect recognition result includes the defect condition of the image object.

[0091] Schematically, the image similarity between the first image and the second image is determined by comparing the feature similarity between the first image and the second image. The higher the similarity, the more likely it is that the second image is a defect-free image, and therefore the image object in the first image is also considered to have no defects. Conversely, if the similarity is lower than a preset similarity threshold, it indicates that there are defective parts in the image object in the first image.

[0092] In some embodiments, a first image feature representation corresponding to the first image is extracted, and a second feature representation corresponding to the second image is extracted, so that the defect status of the image object in the first image is obtained by calculating the feature similarity between the first feature representation and the second feature representation.

[0093] Optionally, feature similarity is calculated by calculating Euclidean distance, cosine similarity, correlation coefficient (calculating the correlation coefficient of two feature representations, that is, the oblique variance of the two feature representations divided by the product of their standard deviations. The correlation coefficient ranges from [-1, 1], and the closer to 1, the higher the similarity), Hamming distance, and Pearson correlation coefficient (used to measure the linear relationship between two feature representations, calculating the covariance of two feature vectors divided by the product of their standard deviations).

[0094] In some embodiments, the defect condition includes the position distribution of defective parts of the image object in the first image, and the defect category corresponding to the defective parts (eg, dents, cracks, flaws, etc.).

[0095] Optionally, the defect situation is represented by coordinate points and numerical results, wherein the coordinate points are used to indicate the coordinate points corresponding to the defective parts, and when different defect categories are pre-defined as different numerical values, the numerical results are used to indicate the defect categories corresponding to the defective parts; or, the defect situation is represented by a score map, wherein the score map is an image with the same pixel size as the first image, and the pixel channel value range is between 0 and 1, wherein 0 indicates that the pixel value of the first image and the second image at the pixel point is the same, and 1 indicates that the pixel values ​​of the first image and the second image at the pixel point are completely different, therefore, the position corresponding to the pixel channel value 1 is the defective part.

[0096] In summary, the image processing method provided in the embodiment of the present application, after adding the first noise data to the first image to obtain the image noise addition result, performs semantic analysis on the first image to obtain the semantic analysis data corresponding to the information used to characterize the image object in the first image, and then performs image denoising on the image noise addition result based on the semantic analysis data to obtain the second image, and finally determines the defect corresponding to the first image by comparing the feature similarity between the first image and the second image. That is, by performing semantic analysis on the first image to obtain the semantic analysis data, the semantic analysis data guides the image denoising, ensuring that the image object in the denoised second image is consistent with the image object in the first image, and then comparing the feature similarity to determine the defect of the image object in the first image, which can improve the accuracy and efficiency of image object defect recognition.

[0097] In an optional embodiment, the semantic analysis process includes encoding processing and decoding processing. For example, please refer to Figure 3 , which shows a flowchart of an image processing method provided by an exemplary embodiment of the present application, that is, step 220 also includes steps 221 to 224, and step 230 also includes steps 231 and 232. Schematically, as shown in Figure 4 As shown, the method includes the following steps.

[0098] Step 221, down-sampling the first image to obtain a down-sampling result.

[0099] Illustratively, image downsampling refers to reducing the image resolution or image size.

[0100] In this embodiment, the first image is in, represents channel, H represents height, and W represents width.

[0101] Step 222, performing iterative feature encoding processing on the downsampling result to obtain a plurality of semantic feature encoding data.

[0102] Among them, the i+1th semantic feature coding data is obtained by performing coding feature processing on the i-th semantic feature coding data, and multiple semantic feature coding data correspond to different feature dimensions respectively, and i is a positive integer.

[0103] Schematically, feature encoding is an image processing method used to extract key point information in an image.

[0104] In some embodiments, feature encoding is performed on the downsampling result to output semantic feature encoding data, and the semantic feature encoding data is used as input data for the next feature encoding process, thereby treating multiple feature encoding processes as iterative feature encoding processes.

[0105] In schematic form, as the number of feature encoding processing times increases, the feature dimension corresponding to the semantic feature encoding data gradually decreases. For example, the feature dimension of the first layer of semantic feature encoding data obtained after the first feature encoding processing is 32×32, and the feature dimension of the second layer of semantic feature encoding data obtained after feature encoding processing of the first layer of semantic feature encoding data is 16×16, and so on, until the last semantic feature encoding data is obtained, and its corresponding feature dimension is the smallest.

[0106] In this embodiment, the iterative feature encoding process is described by taking the execution of feature encoding process four times as an example.

[0107] In some embodiments, after the fourth semantic feature encoding data is obtained, it is stored in a semantic data storage.

[0108] Step 223: Fusing at least two semantic feature encoding data among the plurality of semantic encoding feature data to obtain a first fused feature representation.

[0109] Illustratively, after obtaining a plurality of semantic feature encoding data, at least two semantic feature encodings in the plurality of semantic feature encoding data are subjected to feature fusion, thereby obtaining a first fused feature representation.

[0110] Optionally, the at least two semantic feature encoding data are at least two adjacent semantic feature encoding data; or, at least two non-adjacent semantic feature encoding feature data.

[0111] In some embodiments, at least two semantic feature encoding data include the nth semantic feature encoding data and the mth semantic feature encoding data, the nth semantic feature encoding data includes the kth layer of first semantic sub-data, and the mth semantic feature encoding data includes the kth layer of second semantic sub-data, where n, m and k are positive integers; the kth layers of second semantic sub-data are respectively subjected to convolution processing to obtain k convolution processing results; the k convolution processing results are feature fused with the jth layer of first semantic sub-data in the kth layer of first semantic sub-data to obtain the jth layer first fused sub-feature, where 0<j≤k and j is an integer; and a first fused feature representation is obtained based on the kth layer first fused sub-feature.

[0112] Illustratively, for a single semantic feature encoding data, the single semantic feature encoding includes multiple layers of semantic sub-data, and the first fused feature representation is obtained by performing feature fusion on the semantic sub-data corresponding to at least two semantic feature encoding data.

[0113] In this embodiment, the at least two semantic feature encoding data are implemented as the third semantic feature encoding data and the fourth semantic feature encoding data as an example for description, that is, n=3, m=4.

[0114] In this embodiment, an example is taken in which the third semantic feature encoding data includes three layers of first semantic sub-data and the fourth semantic feature encoding data includes three layers of second semantic sub-data for description.

[0115] In the process of feature fusion of the third semantic feature coding data and the fourth semantic feature coding data, the three layers of second semantic sub-data in the fourth semantic feature coding data are convolved to obtain convolution processing results corresponding to the three layers of second semantic sub-data respectively.

[0116] After obtaining the three second semantic sub-data, the first second semantic sub-data is feature concatenated with the fourth semantic feature encoding data to obtain the first-layer first fusion sub-feature, and the second second semantic sub-data is feature concatenated with the first-layer first fusion sub-feature to obtain the second-layer first fusion sub-feature, until the feature fusion is completed, and the three-layer first fusion sub-feature is obtained. The three-layer first fusion sub-feature is represented as the first fusion feature, which is recorded as J=3 means that there are three layers in the third semantic feature encoding data for use. Represents a convolutional module consisting of a 3×3 convolutional layer, a normalization layer, and an activation layer.

[0117] For illustration, please refer to Figure 4 , which shows a schematic diagram of a feature fusion process provided by an exemplary embodiment of the present application, such as Figure 4 As shown, in the process of feature fusion of the third semantic feature encoding data 410 and the fourth semantic feature encoding data 420, taking the first-layer second semantic sub-data 421 in the fourth semantic feature encoding data 420 as an example, the first-layer first semantic sub-data 411 in the third semantic feature encoding data 410 is convolved, and the convolution result obtained is feature spliced ​​with the first-layer second semantic sub-data 421 to obtain a first fusion result, the convolution result obtained by convolving the second-layer first semantic sub-data 412 is feature spliced ​​with the first fusion result to obtain a second fusion result, the convolution result obtained by convolving the third-layer first semantic sub-data 412 is feature spliced ​​with the second fusion result to finally obtain a first-layer first fusion sub-feature 431, and finally the first-layer first fusion sub-feature 431, the second-layer first fusion sub-feature 432 and the third-layer first fusion sub-feature 433 corresponding to the three layers of second semantic sub-data are respectively represented as the first fusion feature.

[0118] In some embodiments, semantic segmentation is performed on target semantic feature encoding data among a plurality of semantic feature encoding data to obtain a first semantic segmentation feature, and the first semantic segmentation feature is used to characterize a classification result of pixels in the first image.

[0119] Schematically, semantic segmentation refers to classifying pixels in an input image.

[0120] In this embodiment, after obtaining four semantic feature encoding data, the fourth semantic feature encoding data is input into the pre-trained first semantic segmentation model for semantic segmentation, thereby outputting the first semantic segmentation feature, which is used to characterize the classification results of each pixel in the first image, for example: pixel a and pixel b belong to the pixel points corresponding to object 1 in the first image, and pixel c, pixel d and pixel e belong to the pixel points corresponding to object 2 in the first image.

[0121] In this embodiment, the first semantic segmentation model is implemented as a spatial gridding module (Spatial Gridding Module, SGM), and its corresponding module structure has three layers, consisting of a residual module (ResnetBlock), a spatial transformation module (Spatial Transformer) and a residual module. Among them, ResnetBlock is the basic building block in the residual network for learning image features. It consists of two convolutional layers, which contain residual connections. SpatialTransformer is a module for learning image geometric transformations. It improves the robustness of the network to geometric changes in the image by learning how to perform geometric transformations such as translation, rotation, and scaling on the input image.

[0122] Step 224 , performing feature decoding processing on at least one semantic feature encoding data among the plurality of semantic feature encoding data to obtain at least one semantic feature decoding data.

[0123] Schematically, feature decoding processing refers to an image processing process of restoring data obtained after feature encoding processing to original data.

[0124] In this embodiment, after a plurality of semantic feature coded data are obtained through semantic feature coding processing, iterative feature decoding processing is performed on one or more of the semantic feature coded data to finally obtain one or more semantic feature decoded data.

[0125] In some embodiments, feature decoding processing is performed on the first fused feature representation to obtain semantic feature decoding data.

[0126] In this embodiment, after the first fused feature representation is obtained by feature fusion, feature decoding processing is performed on the first fused feature representation to obtain semantic feature decoding data corresponding to the first feature representation.

[0127] Step 231, performing iterative feature coding processing on the image denoising result to obtain a plurality of denoising feature coding data.

[0128] Schematically, the process of performing image denoising on the image noise added result also includes two processes: feature encoding processing and feature decoding processing.

[0129] In this embodiment, the feature coding process is iterated four times for illustration. First, the image denoising result is subjected to feature coding to obtain the first denoised feature coding data. Secondly, the first denoised feature coding data is subjected to feature coding to obtain the second denoised feature coding data. Then, the second denoised feature coding data is subjected to feature coding to obtain the third denoised feature coding data. Finally, the third denoised feature coding data is subjected to feature coding to obtain the fourth denoised feature coding data.

[0130] In some embodiments, the qth denoised feature coding data is feature fused with the downsampling result to obtain a second fused feature representation; the second fused feature representation is iteratively feature coded to obtain multiple semantic feature coding data, where q<p and p, q are positive integers.

[0131] In this embodiment, after performing feature encoding processing on the image denoising result to obtain the first denoising feature encoding data, the first denoising feature encoding data is feature fused with the downsampling result corresponding to the first image to obtain a second fused feature representation, and the above-mentioned iterative feature encoding processing is performed on the second fused feature representation to obtain the above-mentioned multiple semantic feature encoding data.

[0132] Step 232 , performing iterative feature decoding processing on the pth denoised feature coded data among the multiple denoised feature coded data based on the semantic feature decoded data and the semantic feature coded data to obtain a second image.

[0133] Illustratively, after obtaining the pth denoised feature coding data, the semantic feature coding data and / or semantic feature decoding data obtained in the above steps are iteratively feature decoded with the pth denoised feature coding data to obtain denoised feature decoded data. After iterative feature decoding of the denoised feature decoded data, a second image is finally obtained.

[0134] In this embodiment, after obtaining the last denoised feature coding data, it is stored together with the fourth semantic feature coding data to obtain fused data, and the semantic feature decoding data and the de-fused data are feature decoded to obtain the first denoised feature decoding data, and the semantic feature decoding data, the fourth denoised feature coding data and the first denoised feature decoding data are feature decoded to obtain the second denoised feature decoding data, and iterative feature decoding is performed until the second image is obtained.

[0135] In some embodiments, based on the semantic feature decoding data, the pth denoising feature encoding data is iterated feature decoding processing p times to obtain second noise data, and the second noise data is used to indicate the prediction result of the first noise data; based on the second noise data, the image denoising result is subjected to data denoising processing to obtain data denoising features, and the data denoising features are used to characterize the image feature representation corresponding to the image after the image denoising result removes the second noise data; the data denoising features are subjected to image decoding processing to obtain a second image.

[0136] In some embodiments, semantic segmentation is performed on the pth denoising feature encoding data to obtain a second semantic segmentation feature, and the second semantic segmentation feature is used to characterize the pixel classification result in the image noise addition result; the first semantic segmentation feature and the second semantic segmentation feature are feature spliced ​​to obtain a semantic splicing feature; and the semantic splicing feature is iteratively feature decoded based on the semantic feature decoding data and the pth denoising feature encoding data to obtain a second noise data.

[0137] In this embodiment, taking p as 4 as an example, after obtaining the fourth denoising feature encoding data, the fourth denoising feature encoding data is input into the second semantic segmentation model obtained by pre-training for semantic segmentation, and the second semantic segmentation feature is output, wherein the second semantic segmentation feature is used to characterize the classification result corresponding to each pixel in the image noise addition result. Among them, the second semantic segmentation model has the same model structure as the first semantic segmentation model, that is, the second semantic segmentation model (Spatial Gridding Module, SDM) also includes a layer of residual module (ResnetBlock), a layer of spatial transformation module (Spatial Transformer) and a layer of residual module (ResnetBlock).

[0138] In this embodiment, after the second semantic segmentation feature is obtained, the first semantic segmentation feature and the second semantic segmentation feature are concatenated to obtain a semantic concatenation feature.

[0139] In this embodiment, the semantic concatenation feature is subjected to four iterative feature decoding processes in combination with the semantic feature decoding data and the fourth denoising feature encoding data, thereby obtaining the second noise data.

[0140] In this embodiment, taking the feature decoding process iteratively executed four times as an example, after performing four iterative feature decoding processes on the fourth denoised feature encoded data, the second noise data ε is obtained. θ , perform data denoising on it, and thus obtain data denoising features

[0141] Optionally, the first noise data is the same as the second noise data; or, the first noise data is different from the second noise data.

[0142] In this embodiment, the image denoising result is subjected to data denoising based on the second noise data by using the denoising formula obtained in advance to obtain a data denoising feature, whose formula is p θ (x0|x t-1 ), where x0 represents the first image, x t-1 It represents the noisy sub-image obtained after the image is denoised in step t-1.

[0143] In this embodiment, after the data denoising feature is obtained, it is output to a pre-trained image decoder for image decoding processing, and the second image is output. The second image is recorded as

[0144] Step 241 , extracting a first image feature representation corresponding to the first image, and extracting a second image feature representation corresponding to the second image.

[0145] Illustratively, the defect condition of the image object includes the position distribution of defects in the image object and the defect category.

[0146] Schematically, the first image and the second image are input into a same pre-trained feature extraction model in a common feature space to extract a first image feature representation and a second image feature representation corresponding to the first image.

[0147] In some embodiments, the first image feature representation includes feature representations of multiple different feature dimensions, and the second image feature representation also includes feature representations of multiple different feature dimensions, wherein the feature dimension distribution in the first image feature representation is consistent with the feature dimension distribution in the second image feature representation.

[0148] In this embodiment, the pre-obtained feature extraction model is implemented as a convolutional neural network resnet50.

[0149] Step 242: obtaining a defect score distribution image based on the cosine similarity between the first image feature representation and the second image feature representation.

[0150] The pixel values ​​in the defect score distribution image are used to indicate the difference between the pixel values ​​in the first image and the pixel values ​​in the second image.

[0151] In this embodiment, the defect scores of different feature dimensions in the first image feature representation and the second image feature representation are calculated according to the cosine similarity formula, thereby obtaining a defect score distribution image. The cosine similarity formula can refer to the following formula 1.

[0152] Formula 1:

[0153]

[0154] Where n represents the image feature representation corresponding to the nth feature dimension.

[0155] Schematically, the defect score partial image is a grayscale image with the same pixel size as the first image, and the corresponding pixel channel value range includes 0 to 1. When the channel value is 0, it means that the pixel value in the first image and the pixel value at the same position in the second image are the same. When the channel value is 1, it means that the pixel value in the first image is different from the pixel value at the same position in the second image.

[0156] Step 243, perform feature upsampling on the defect score distribution image to obtain a defect position distribution image.

[0157] The defect location distribution image is used to indicate the location distribution of defects in the image object.

[0158] Schematically, after obtaining the defect score distribution image, feature upsampling is performed on it, and the number of feature dimensions to be applied is selected to obtain the defect position distribution image. The feature upsampling process can be referred to the following formula 2.

[0159] Formula 2:

[0160] Among them, σ n represents the upsampling rate and N represents the number of feature dimensions used.

[0161] Step 244, performing average pooling processing on the defect position distribution image to obtain the defect category.

[0162] Schematically, after the defect position distribution image is obtained, a global average pooling process is performed on it, and the maximum value obtained by the process is used as the defect category result.

[0163] Step 245, taking the defect location distribution image and the defect category result as the defect situation.

[0164] Finally, the defect location distribution image and defect category results are taken as the defect recognition results.

[0165] In some embodiments, the defect scores of the first image and the second image at the RGB level may also be calculated and combined with the defect location distribution image and the defect category results as the defect situation.

[0166] In summary, the image processing method provided in the embodiment of the present application, after adding the first noise data to the first image to obtain the image noise addition result, performs semantic analysis on the first image to obtain the semantic analysis data corresponding to the information used to characterize the image object in the first image, and then performs image denoising on the image noise addition result based on the semantic analysis data to obtain the second image, and finally determines the defect corresponding to the first image by comparing the feature similarity between the first image and the second image. That is, by performing semantic analysis on the first image to obtain the semantic analysis data, the semantic analysis data guides the image denoising, ensuring that the image object in the denoised second image is consistent with the image object in the first image, and then comparing the feature similarity to determine the defect of the image object in the first image, which can improve the accuracy and efficiency of image object defect recognition.

[0167] In this embodiment, by performing feature encoding processing and feature decoding processing on the first image, the output result can include state information of the image object in the first image, thereby improving the accuracy of semantic analysis.

[0168] In this embodiment, after feature fusion of at least two semantic feature encoding data, feature decoding processing is performed on them, which can improve the information richness of the semantic feature decoding data finally obtained.

[0169] In this embodiment, after the two semantic feature encoding data are layered, feature fusion is performed through convolution processing, which can improve the decoding accuracy of the semantic feature decoding data.

[0170] In this embodiment, the second image obtained by iteratively encoding the image loading result and iteratively decoding the denoised feature encoding data based on the semantic feature decoding data can improve the consistency of the second image and the first image.

[0171] In this embodiment, feature encoding processing is performed on the denoised feature coded data after feature fusion with the down-sampling result, thereby improving the accuracy of semantic analysis.

[0172] In an optional embodiment, the above-mentioned image object recognition process is implemented by a deep learning-based approach. For example, please refer to Figure 5 , which shows a flow chart of an image processing method provided by an exemplary embodiment of the present application, such as Figure 5 As shown, the method includes the following steps.

[0173] Step 2201, input the first image into a pre-trained semantic analysis model, and output semantic analysis data.

[0174] Schematically, the semantic analysis model includes an encoder and a decoder, which are respectively used to perform the above-mentioned feature encoding processing and feature decoding processing, so as to obtain semantic analysis data.

[0175] Among them, the semantic analysis model includes multiple semantic encoding modules arranged in sequence, a first semantic analysis model and a semantic decoding module. The image denoising data is input into the first semantic encoding module, and the first semantic feature encoding data is output. The first semantic feature encoding data is input into the second semantic encoding module, and the second semantic feature encoding data is output. In this way, iterative feature encoding processing is performed to finally obtain multiple semantic feature encoding data.

[0176] The fourth semantic feature encoding is input into the pre-trained first semantic segmentation model, and the first semantic segmentation feature is output.

[0177] A first fused feature representation obtained by performing feature fusion on at least two semantic feature encoding data is input into a decoding module, and the semantic feature decoded data is output.

[0178] Among them, the semantic decoding module is implemented as a semantic guided decoding module (Semantic Guided EncoderBlocks, SGEB), and its network structure includes a three-layer residual module (ResnetBlock).

[0179] The semantic feature encoding data and the semantic feature decoding data are used as semantic analysis data.

[0180] Step 2301, input the semantic analysis data and the image denoising result into a pre-trained image denoising model to obtain a second image.

[0181] Illustratively, the image denoising model includes a plurality of denoising encoding modules arranged in sequence and a plurality of denoising decoding modules arranged in sequence, which are respectively used to perform the above-mentioned denoising feature encoding processing and denoising feature decoding processing, so as to obtain a second image.

[0182] The image denoising result is input into the first denoising coding module, and the first denoising feature coding data is output. The first denoising feature coding data is input into the second denoising coding module, and the second denoising feature coding data is output. This is done by iterative coding processing to obtain the last denoising feature coding data.

[0183] The last denoised feature encoding data, the fourth semantic feature encoding data and the semantic feature decoding data are input into the first denoising decoding module, and the first denoised feature decoding data is output. The first denoised feature decoding data is input into the second denoising decoding module, and the second denoised feature decoding data is output. The iterative feature decoding process is performed to obtain the second noise data. The data denoising process is performed based on the second noise data to finally obtain the second image. The second noise data can refer to Formula 3.

[0184] Formula 3:

[0185] in, represents the decoding module in the image denoising model, represents the denoising data storage module in the image denoising model, ε SD represents the encoding module in the image denoising model, A semantic data storage module representing a semantic analysis model, represents the convolutional neural network layer, Represents the decoding module of the semantic analysis model.

[0186] Among them, the semantic encoding module is implemented as a semantic guided decoding module (Semantic Guided DecoderBlocks, SGDB), which also includes three layers of ResnetBlock. The denoising decoding module includes a residual module (ResnetBlock), a spatial transformation module (SpatialTransformer), a residual module (ResnetBlock) and an upsampling module (Upsample). Therefore, the three-layer output result of the fourth semantic encoding module (the fourth semantic feature encoding data) is added to the three-layer output result of the semantic decoding module (semantic feature decoding data), and then connected to the corresponding residual module in the first denoising decoding module, and the fourth denoising feature encoding data is input into the first denoising decoding module, so as to perform denoising decoding processing and obtain the first denoising feature decoding data.

[0187] Next, the training process of the semantic analysis model and the image denoising model is described. Figure 6 As shown, a flowchart of the image recognition method training process provided by an exemplary embodiment of the present application is currently shown. Figure 6 As shown, the method includes the following steps.

[0188] Step 610: Obtain an image noise addition result corresponding to the first sample image.

[0189] The image noise addition result is a result obtained by adding the first sample noise data to the first sample image.

[0190] Illustratively, firstly, a first sample image for training is obtained, the first sample image is input into a pre-trained encoder to obtain a latent variable representation, and noise data is randomly added to the latent variable representation to obtain an image noise addition result.

[0191] Illustratively, the first sample image is a sample image without defects.

[0192] Step 620: Input the first sample image into the sample analysis model, and output semantic analysis data.

[0193] The first sample image is input into the sample analysis model for feature encoding processing, feature decoding processing and feature fusion to obtain semantic analysis data.

[0194] Step 630: input the semantic analysis data and the image noise addition result into a sample denoising model, and output the first predicted noise data.

[0195] In the process of inputting the image denoising result into the sample denoising model for image denoising processing, the semantic analysis data is input into the sample denoising model, and the first predicted noise data is outputted.

[0196] Step 640: Based on the difference between the first sample noise data and the first predicted noise data, the sample analysis model and the sample denoising model are trained to obtain a semantic analysis model and an image denoising model.

[0197] By calculating the L2 distance loss value between the first sample noise data and the first predicted noise data, the sample analysis model and the sample denoising model are trained to obtain the semantic analysis model and the image denoising model.

[0198] In summary, the image processing method provided in the embodiment of the present application, after adding the first noise data to the first image to obtain the image noise addition result, performs semantic analysis on the first image to obtain the semantic analysis data corresponding to the information used to characterize the image object in the first image, and then performs image denoising on the image noise addition result based on the semantic analysis data to obtain the second image, and finally determines the defect corresponding to the first image by comparing the feature similarity between the first image and the second image. That is, by performing semantic analysis on the first image to obtain the semantic analysis data, the semantic analysis data guides the image denoising, ensuring that the image object in the denoised second image is consistent with the image object in the first image, and then comparing the feature similarity to determine the defect of the image object in the first image, which can improve the accuracy and efficiency of image object defect recognition.

[0199] For illustration, please refer to Figure 7 , which shows a schematic diagram of an image processing method provided by an exemplary embodiment of the present application, such as Figure 7As shown, the method includes the following process.

[0200] A first image 701 is obtained, and the first image 701 is input into an encoder 702 to obtain a latent variable representation 703 , and image denoising processing is performed on the latent variable representation 703 to obtain an image denoising result 704 .

[0201] The image denoising result 704 is input into the image denoising model 710 for image denoising processing. Meanwhile, the first image 701 is input into the semantic analysis model 720 for semantic analysis.

[0202] During the semantic analysis process, the first image 701 is feature downsampled to obtain a downsampled result. At this time, during the image denoising process, the first denoising feature coding data obtained by feature coding processing by the coding module 1 in the encoder in the image denoising model 710 is input together with the downsampled result into the coding module a in the semantic analysis model 720 for feature coding processing, and then the semantic feature coding data output by the coding module c and the coding module d are respectively input into the feature fusion module 721 for feature fusion after passing through the coding module b, the coding module c, and the coding module d to obtain a first fused feature representation, and the first fused feature representation is input into the decoding module a to obtain semantic feature decoding data. In addition, the fourth semantic feature coding data output by the coding module d is input into the semantic segmentation model a to obtain a first semantic segmentation feature.

[0203] Among them, encoding module a, encoding module b, encoding module c and encoding module d are implemented as four semantically guided encoding modules (SGEB), so encoding module a corresponds to SGEB1, encoding module b corresponds to SGEB2, encoding module c corresponds to SGEB3, and encoding module d corresponds to SGEB4. Among them, the network structure of SGEB includes a residual module (ResnetBlock), a spatial transformer module (Spatial Transformer), a layer (ResnetBlock) and a downsampling module (Downsample).

[0204] Among them, the semantic segmentation model a is implemented as an SGM model.

[0205] Among them, the decoding module a is implemented as a semantic guided decoding module (Semantic Guided Decoder Blocks, SGDB).

[0206] During the image denoising process, iterative feature encoding processing is performed through encoding module 1, encoding module 2, encoding module 3 and encoding module 4, the fourth denoising feature encoding data obtained by encoding module 4 is input into the semantic segmentation model b to obtain the second semantic segmentation feature, the first semantic segmentation feature and the second semantic segmentation feature are feature fused and input into the decoding module 1 for decoding to obtain the first denoising feature encoding data, and the first denoising feature encoding data and the first fused feature representation are jointly input into the decoding module 2 for feature decoding, and then pass through the decoding module 3 and the decoding module 4 in turn to obtain the second noise data.

[0207] Among them, encoding module 1, encoding module 2, encoding module 3 and encoding module 4 are implemented as four stable diffusion encoding modules (Stable Diffusion Encoder Blocks, SDEB), therefore, encoding module 1 corresponds to SDEB1, encoding module 2 corresponds to SDEB2, encoding module 3 corresponds to SDEB3, and encoding module 4 corresponds to SDEB4.

[0208] Among them, decoding module 1, decoding module 2, decoding module 3 and decoding module 4 are implemented as four stable diffusion decoding modules (Stable Diffusion Decoder Blocks, SDDB). Therefore, decoding module 1 corresponds to SDDB1, decoding module 2 corresponds to SDDB2, decoding module 3 corresponds to SDDB3, and decoding module 4 corresponds to SSDDB4.

[0209] After performing data denoising on the second noise data, a denoised feature representation 705 is obtained. After the denoised feature representation 705 is input into the decoder 706, a second image 707 is obtained. The first image 701 and the second image 707 are input into the feature space 730, and a first image feature representation 731 and a second image feature representation 732 are extracted. Finally, the defect situation is obtained based on the cosine similarity between the first image feature representation 731 and the second image feature representation 732.

[0210] For illustration, please refer to Figure 8 , which shows a comparison diagram of the effects of an image processing method provided by an exemplary embodiment of the present application, such as Figure 8 As shown, the first image 801 is the input image to be detected, the image 802 is the image obtained by performing denoising after adding noise by the related technology, the image 803 is the image obtained by the solution of the present application, and the image 804 is the contour reference image. Figure 8 It can be seen that the accuracy of the image processing results of the image processing solution provided by the present application is higher than the accuracy of the image corresponding to the image processed by the related technology.

[0211] In summary, the image processing method provided in the embodiment of the present application, after adding the first noise data to the first image to obtain the image noise addition result, performs semantic analysis on the first image to obtain the semantic analysis data corresponding to the information used to characterize the image object in the first image, and then performs image denoising on the image noise addition result based on the semantic analysis data to obtain the second image, and finally determines the defect corresponding to the first image by comparing the feature similarity between the first image and the second image. That is, by performing semantic analysis on the first image to obtain the semantic analysis data, the semantic analysis data guides the image denoising, ensuring that the image object in the denoised second image is consistent with the image object in the first image, and then comparing the feature similarity to determine the defect of the image object in the first image, which can improve the accuracy and efficiency of image object defect recognition.

[0212] This technical solution proposes a multi-class defect detection method based on a diffusion model framework, improves the existing diffusion model denoising network framework, adds a semantic guidance network, improves the category error and semantic error problems of the existing diffusion model when dealing with multi-class defect detection tasks, maintains the consistency of the semantic information of the input image and the reconstructed image while reconstructing a large area of ​​defect areas, can effectively reconstruct defects of different categories and types into normal samples, and extracts the input image and the reconstructed image through the feature extraction network to effectively detect and locate defects, and can cope with the detection and positioning of multi-category defects in actual industrial scenarios.

[0213] Fig. 9 is a structural block diagram of an image processing device provided by an exemplary embodiment of the present application, such as Fig. 9 As shown, the device includes the following parts:

[0214] An acquisition module 910 is used to acquire an image noise addition result corresponding to a first image, where the image noise addition result is a result obtained by adding first noise data to the first image, and the first image includes an image object;

[0215] An analysis module 920, configured to perform semantic analysis on the first image to obtain semantic analysis data, wherein the semantic analysis data is used to represent information of the image object in the first image;

[0216] A denoising module 930, configured to perform image denoising processing on the image denoising result based on the semantic analysis data to obtain a second image;

[0217] The determination module 940 is used to determine a defect recognition result corresponding to the first image based on the feature similarity between the first image and the second image, wherein the defect recognition result includes the defect condition of the image object.

[0218] In some embodiments, Fig.10 As shown, the analysis module 920 includes:

[0219] The sampling unit 921 is used to perform image downsampling on the first image to obtain a downsampling result;

[0220] The encoding unit 922 is used to perform iterative feature encoding processing on the downsampling result to obtain multiple semantic feature encoding data, wherein the i+1th semantic feature encoding data is obtained by performing feature encoding processing on the ith semantic feature encoding data, and the multiple semantic feature encoding data correspond to different feature dimensions respectively, and i is a positive integer;

[0221] The decoding unit 923 is used to perform feature decoding processing on at least one semantic feature encoding data among the multiple semantic feature encoding data to obtain at least one semantic feature decoding data.

[0222] In some embodiments, the analysis module 920 further includes:

[0223] A fusion unit 924 is used to perform feature fusion on at least two semantic feature encoding data among the plurality of semantic feature encoding data to obtain a first fused feature representation;

[0224] The decoding unit 923 is used to perform feature decoding processing on the first fusion feature representation to obtain the semantic feature decoding data.

[0225] In some embodiments, the at least two semantic feature encoding data include nth semantic feature encoding data and mth semantic feature encoding data, the nth semantic feature encoding data includes k layers of first semantic sub-data, the mth semantic feature encoding data includes k layers of second semantic sub-data, and n, m and k are positive integers;

[0226] The fusion unit 924 is used to perform convolution processing on the k layers of second semantic sub-data respectively to obtain k convolution processing results; perform feature fusion on the k convolution processing results and the j-th layer of first semantic sub-data to obtain the j-th layer first fused sub-feature, 0<j≤k and j is an integer; and obtain the first fused feature representation based on the k-th layer first fused sub-feature.

[0227] In some embodiments, the denoising module 930 is also used to perform iterative feature coding processing on the image denoising result to obtain multiple denoised feature coding data; based on the semantic feature decoding data and the semantic feature coding data, iterative feature decoding processing is performed on the pth denoised feature coding data among the multiple denoised feature coding data to obtain the second image, where p is a positive integer.

[0228] In some embodiments, the denoising module 930 is further used to perform feature fusion on the qth denoised feature coding data and the downsampling result to obtain a second fused feature representation, where q<p and q is a positive integer; and perform iterative feature coding processing on the second fused feature representation to obtain multiple semantic feature coding data.

[0229] In some embodiments, the denoising module 930 is also used to perform feature decoding processing on the pth denoising feature encoding data based on the semantic feature decoding data and the semantic feature encoding data to obtain second noise data, and the second noise data is used to indicate the prediction result of the first noise data; based on the second noise data, perform data denoising processing on the image denoising result to obtain data denoising features, and the data denoising features are used to characterize the image feature representation corresponding to the image after the image denoising result removes the second noise data; perform image decoding processing on the data denoising features to obtain the second image.

[0230] In some embodiments, the denoising module 930 is also used to perform semantic segmentation on the target semantic feature encoding data among the multiple semantic feature encoding data to obtain a first semantic segmentation feature, and the first semantic segmentation feature is used to characterize the pixel classification result in the first image; perform semantic segmentation on the pth denoised feature encoding data to obtain a second semantic segmentation feature, and the second semantic segmentation feature is used to characterize the pixel classification result in the image denoising result; perform feature splicing on the first semantic segmentation feature and the second semantic segmentation feature to obtain a semantic splicing feature; perform iterative feature decoding processing on the semantic splicing feature based on the semantic feature decoding data and the pth denoised feature encoding data to obtain the second noise data.

[0231] In some embodiments, the defect condition of the image object includes the position distribution of defects of the image object and the defect category;

[0232] The determination module 940 is also used to extract a first image feature representation corresponding to the first image, and to extract a second image feature representation corresponding to the second image; based on the cosine similarity between the first image feature representation and the second image feature representation, a defect score distribution image is obtained, and the pixel values ​​in the defect score distribution image are used to indicate the difference between the pixel values ​​in the first image and the pixel values ​​in the second image; feature upsampling is performed on the defect score distribution image to obtain a defect position distribution image, and the defect position distribution image is used to indicate the position distribution of defects in the image object; average pooling is performed on the defect position distribution image to obtain a defect category result; and the defect position distribution image and the defect category result are used as the defect recognition result.

[0233] In some embodiments, the analysis module 920 is further used to input the first image into a pre-trained semantic analysis model, and output the semantic analysis data;

[0234] The denoising module 930 is further configured to input the semantic analysis data and the image denoising result into a pre-trained image denoising model to obtain the second image.

[0235] In some embodiments, the apparatus further comprises:

[0236] A training module 950 is used to obtain an image denoising result corresponding to a first sample image, wherein the image denoising result is a result obtained by adding first sample noise data to the first sample image; input the first sample image into a sample analysis model, and output semantic analysis data; input the semantic analysis data and the image denoising result into a sample denoising model, and output first predicted noise data; and train the sample analysis model and the sample denoising model based on the difference between the first sample noise data and the first predicted noise data to obtain the semantic analysis model and the image denoising model.

[0237] In summary, the image processing device provided in the embodiment of the present application, after adding the first noise data to the first image to obtain the image noise addition result, performs semantic analysis on the first image to obtain the semantic analysis data corresponding to the information used to characterize the image object in the first image, and then performs image denoising on the image noise addition result based on the semantic analysis data to obtain the second image, and finally determines the defect corresponding to the first image by comparing the feature similarity between the first image and the second image. That is, by performing semantic analysis on the first image to obtain the semantic analysis data, the semantic analysis data guides the image denoising, ensuring that the image object in the denoised second image is consistent with the image object in the first image, and then comparing the feature similarity to determine the defect of the image object in the first image, which can improve the accuracy and efficiency of image object defect recognition.

[0238] It should be noted that the image processing device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the image processing device provided in the above embodiment and the image processing method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0239] Fig.11The block diagram of the structure of a computer device 1100 provided by an exemplary embodiment of the present application is shown. The computer device 1100 may be a portable mobile terminal, such as a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer or a desktop computer. The computer device 1100 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0240] Typically, the computer device 1100 includes a processor 1101 and a memory 1102 .

[0241] The processor 1101 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1101 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1101 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1101 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1101 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0242] The memory 1102 may include one or more computer-readable storage media, which may be non-transitory. The memory 1102 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1102 is used to store at least one instruction, which is used to be executed by the processor 1101 to implement the training method and data classification method of the classification model provided in the method embodiment of the present application.

[0243] In some embodiments, the computer device 1100 may further include other components, which can be understood by those skilled in the art. Fig.11 The structure shown in the figure does not constitute a limitation on the computer device 1100, and the computer device 1100 may include more or less components than shown in the figure, or combine some components, or adopt a different component arrangement.

[0244] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which can be a computer-readable storage medium contained in the memory in the above embodiments; or it can be a computer-readable storage medium that exists independently and is not assembled in the terminal. The computer-readable storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by the processor to implement the training method of the classification model and the data classification method described in any of the above embodiments.

[0245] Optionally, the computer readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid state drive (SSD), or an optical disk. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0246] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0247] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire an image noise addition result corresponding to a first image, where the image noise addition result is a result of adding first noise data to the first image, where the first image includes an image object; Performing semantic analysis on the first image to obtain semantic analysis data, where the semantic analysis data is used to represent information of the image object in the first image; Performing image denoising processing on the image denoising result based on the semantic analysis data to obtain a second image; Based on the feature similarity between the first image and the second image, a defect recognition result corresponding to the first image is determined, and the defect recognition result includes the defect situation of the image object.

2. The method according to claim 1, characterized in that The performing semantic analysis on the first image to obtain semantic analysis data includes: Performing image downsampling on the first image to obtain a downsampling result; Performing iterative feature coding processing on the downsampling result to obtain a plurality of semantic feature coding data, wherein the i+1th semantic feature coding data is obtained by performing feature coding processing on the ith semantic feature coding data, the plurality of semantic feature coding data respectively correspond to different feature dimensions, and i is a positive integer; A feature decoding process is performed on at least one semantic feature encoding data among the plurality of semantic feature encoding data to obtain at least one semantic feature decoding data.

3. The method according to claim 2, characterized in that The step of performing feature decoding processing on at least one semantic feature encoding data among the plurality of semantic feature encoding data to obtain at least one semantic feature decoding data comprises: Performing feature fusion on at least two of the plurality of semantic feature coded data to obtain a first fused feature representation; Perform feature decoding processing on the first fused feature representation to obtain the semantic feature decoding data.

4. The method according to claim 3, characterized in that The at least two semantic feature encoding data include nth semantic feature encoding data and mth semantic feature encoding data, the nth semantic feature encoding data includes k layers of first semantic sub-data, the mth semantic feature encoding data includes k layers of second semantic sub-data, and n, m and k are positive integers; The step of fusing at least two semantic feature encoding data among the plurality of semantic feature encoding data to obtain a first fused feature representation includes: The k layers of second semantic sub-data are respectively subjected to convolution processing to obtain k convolution processing results; Perform feature fusion on the k convolution processing results and the first semantic sub-data of the j-th layer to obtain the first fused sub-feature of the j-th layer, where 0<j≤k and j is an integer; The first fused feature representation is obtained based on the k-layer first fused sub-features.

5. The method according to claim 2, characterized in that: The performing image denoising processing on the image denoising result based on the semantic analysis data to obtain a second image includes: Performing iterative feature coding processing on the image noise addition result to obtain a plurality of denoising feature coding data; An iterative feature decoding process is performed on the pth denoised feature coded data among the multiple denoised feature coded data based on the semantic feature decoding data and the semantic feature coded data to obtain the second image, where p is a positive integer.

6. The method according to claim 5, characterized in that The iterative feature encoding process is performed on the downsampling result to obtain a plurality of semantic feature encoding data, including: Perform feature fusion on the qth denoised feature encoding data and the downsampling result to obtain a second fused feature representation, wherein q<p and q is a positive integer; An iterative feature encoding process is performed on the second fused feature representation to obtain a plurality of semantic feature encoding data.

7. The method according to claim 5, characterized in that The iterative feature decoding process is performed on the pth denoised feature coded data among the plurality of denoised feature coded data based on the semantic feature decoded data and the semantic feature coded data to obtain the second image, comprising: Performing iterative feature decoding processing on the p-th denoised feature encoding data based on the semantic feature decoding data and the semantic feature encoding data to obtain second noise data, where the second noise data is used to indicate a prediction result of the first noise data; Performing data denoising processing on the image denoising result based on the second noise data to obtain a data denoising feature, wherein the data denoising feature is used to represent an image feature representation corresponding to the image after the image denoising result is freed from the second noise data; Perform image decoding processing on the data denoising feature to obtain the second image.

8. The method according to claim 7, characterized in that Before performing feature decoding processing on at least one semantic feature encoding data among the plurality of semantic feature encoding data to obtain at least one semantic feature decoded data, the method further includes: Performing semantic segmentation on target semantic feature encoding data among the plurality of semantic feature encoding data to obtain a first semantic segmentation feature, where the first semantic segmentation feature is used to characterize a classification result of pixels in the first image; The iterative feature decoding process is performed on the p-th denoised feature coded data based on the semantic feature decoded data and the semantic feature coded data to obtain second noise data, comprising: Performing semantic segmentation on the p-th denoising feature encoding data to obtain a second semantic segmentation feature, where the second semantic segmentation feature is used to characterize a pixel classification result in the image denoising result; Performing feature splicing on the first semantic segmentation feature and the second semantic segmentation feature to obtain a semantic splicing feature; The semantic concatenation feature is iteratively decoded based on the semantic feature decoding data and the pth denoising feature encoding data to obtain the second noise data.

9. The method according to any one of claims 1 to 8, characterized in that: The defect recognition result of the image object includes the position distribution and defect category of the defects of the image object. The determining the defect recognition result corresponding to the first image based on the feature similarity between the first image and the second image includes: Extracting a first image feature representation corresponding to the first image, and extracting a second image feature representation corresponding to the second image; Based on the cosine similarity between the first image feature representation and the second image feature representation, a defect score distribution image is obtained, wherein the pixel values ​​in the defect score distribution image are used to indicate the difference between the pixel values ​​in the first image and the pixel values ​​in the second image; Performing feature upsampling on the defect score distribution image to obtain a defect position distribution image, wherein the defect position distribution image is used to indicate the position distribution of defects in the image object; Performing average pooling processing on the defect position distribution image to obtain the defect category; The defect position distribution image and the defect category are used as the defect recognition result.

10. The method according to any one of claims 1 to 8, characterized in that: The performing semantic analysis on the first image to obtain semantic analysis data includes: Inputting the first image into a pre-trained semantic analysis model, and outputting the semantic analysis data; The performing image denoising processing on the image denoising result based on the semantic analysis data to obtain a second image includes: The semantic analysis data and the image denoising result are input into a pre-trained image denoising model to obtain the second image.

11. The method according to claim 10, characterized in that Before inputting the first image into a pre-trained semantic analysis model and outputting the semantic analysis data, the method further includes: Obtaining an image noise addition result corresponding to the first sample image, wherein the image noise addition result is a result obtained by adding first sample noise data to the first sample image; Inputting the first sample image into a sample analysis model, and outputting semantic analysis data; Inputting the semantic analysis data and the image noise addition result into a sample denoising model, and outputting first predicted noise data; Based on the difference between the first sample noise data and the first predicted noise data, the sample analysis model and the sample denoising model are trained to obtain the semantic analysis model and the image denoising model.

12. An image processing device, characterized in that: The device comprises: An acquisition module, configured to acquire an image noise addition result corresponding to a first image, wherein the image noise addition result is a result obtained by adding first noise data to the first image, wherein the first image includes an image object; An analysis module, configured to perform semantic analysis on the first image to obtain semantic analysis data, wherein the semantic analysis data is used to characterize information of the image object in the first image; A denoising module, configured to perform image denoising processing on the image denoising result based on the semantic analysis data to obtain a second image; A determination module is used to determine a defect recognition result corresponding to the first image based on feature similarity between the first image and the second image, wherein the defect recognition result includes a defect condition of the image object.

13. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the image object recognition method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the image object recognition method according to any one of claims 1 to 11.

15. A computer program product, characterized in that It comprises computer instructions, which, when executed by a processor, implement the method for recognizing an image object as claimed in any one of claims 1 to 11.

Images