Image processing method, electronic device, storage medium, and program product

Abstract

The application provides an image processing method, an electronic device, a computer readable storage medium and a computer program product. The image processing method extracts a feature vector of a pixel in a to-be-processed image, performs vector uniformity measurement based on a pre-trained uniform pixel feature vector set, and outputs a uniformity heat map that can accurately reflect the uniformity of the pixels in the image. This facilitates the identification and optimization of non-uniform regions by the user, and does not require manual pre-setting of visual features, so that the image processing method has a more extensive application scenario.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more particularly to an image processing method, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Computer vision is an important branch of artificial intelligence. It refers to the ability of computers to visually perceive specific targets in place of human eyes and brains. For example, in industry, computers can be used to identify the color uniformity, brightness uniformity, or texture uniformity of products in order to detect product quality or defects.

[0003] However, in current technologies, computers rely on pre-designed visual features for comparison when performing visual recognition and judgment. This results in limited applicability across different application scenarios. Furthermore, after recognition and comparison, computers can only output a single judgment result, such as whether a product has defects, without being able to display the specific location of the defects. This makes it difficult for users to optimize the product and leads to a poor user experience. Summary of the Invention

[0004] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0005] This invention provides an image processing method, an electronic device, a computer-readable storage medium, and a computer program product, which can identify and label target pixels in an image, thereby improving image processing capabilities.

[0006] In a first aspect, embodiments of the present invention provide an image processing method, comprising: acquiring an image to be processed; acquiring feature vectors of pixels in the image to be processed; acquiring a pre-trained set of uniform pixel feature vectors; obtaining the vector uniformity of the pixels based on the set of uniform pixel feature vectors and the feature vectors of the pixels; and obtaining a uniformity heatmap of the image to be processed based on the vector uniformity of the pixels.

[0007] In a second aspect, embodiments of the present invention also provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the image processing method described above.

[0008] Thirdly, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions for performing the image processing method described above.

[0009] Fourthly, embodiments of the present invention also provide a computer program product, including a computer program or computer instructions, characterized in that the computer program or computer instructions are stored in a computer-readable storage medium, a processor of a computer device reads the computer program or computer instructions from the computer-readable storage medium, and the processor executes the computer program or computer instructions, causing the computer device to perform the image processing method as described above.

[0010] This invention includes: acquiring a to-be-processed image and feature vectors of pixels in the image; acquiring a pre-trained set of uniform pixel feature vectors; then, based on the set of uniform pixel feature vectors and the pixel feature vectors, obtaining the vector uniformity of the pixels; and finally, based on the pixel vector uniformity, obtaining a uniformity heatmap of the image to be processed. According to the scheme of this invention, by extracting feature vectors of pixels in the image to be processed and measuring vector uniformity based on a pre-trained set of uniform pixel feature vectors, a uniformity heatmap that accurately reflects the pixel uniformity in the image is output. This facilitates the identification and optimization of non-uniform regions by users, and eliminates the need for manual pre-setting of visual features, thus expanding the application scenarios of the image processing method.

[0011] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the description, claims, and drawings. Attached Figure Description

[0012] The accompanying drawings are provided to further understand the technical solutions of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the technical solutions of the present invention, and do not constitute a limitation on the technical solutions of the present invention.

[0013] Figure 1 This is a flowchart of an image processing method provided in one embodiment of the present invention;

[0014] Figure 2 yes Figure 1 A flowchart of a specific method for step S400;

[0015] Figure 3 This is a flowchart of an image processing method provided in one embodiment of the present invention;

[0016] Figure 4 This is a flowchart of the image feature extraction model training process provided in one embodiment of the present invention;

[0017] Figure 5This is a schematic diagram of the feature extraction network structure of an image feature extraction model provided in an embodiment of the present invention;

[0018] Figure 6 This is a schematic diagram of the target area provided in one embodiment of the present invention;

[0019] Figure 7 This is a flowchart of an image processing method provided in one embodiment of the present invention;

[0020] Figure 8 This is a schematic diagram illustrating the effect of an image processing method provided in one embodiment of the present invention;

[0021] Figure 9 This is a schematic diagram of an electronic device provided in one embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0023] It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0024] Image recognition and processing have significant practical implications in industrial machine vision. For example, in the tobacco industry, tobacco leaves are graded based on their color uniformity, i.e., color uniformity recognition; in the quality inspection of automobiles and electronic components, product defects are detected based on the brightness uniformity of images, i.e., brightness uniformity recognition; and in the textile industry, potential fabric defect areas are identified based on the texture uniformity of images, i.e., texture uniformity recognition.

[0025] However, in related technologies, the identification of specific regions in an image relies on specific visual features, such as color, brightness, or texture. Most of these visual features need to be pre-designed manually. For example, color uniformity is determined based on pixel statistical characteristics of color models like Lab and HSV; texture features are extracted using gray-level co-occurrence matrices and LBP texture descriptors, and then texture uniformity is determined based on feature statistical characteristics. The aforementioned methods for image recognition of non-uniform regions through pre-designed visual features have low universality when detecting product quality or defects, and cannot display non-uniformities in the image at the pixel level. This makes some minor defects difficult to observe with the naked eye, causing inconvenience for users.

[0026] This invention provides an image processing method, electronic device, computer-readable storage medium, and computer program product. The method involves acquiring an image to be processed and feature vectors of pixels within that image, obtaining a pre-trained set of uniform pixel feature vectors, then calculating the vector uniformity of the pixels based on the set of uniform pixel feature vectors and the pixel feature vectors. Finally, a uniformity heatmap of the image to be processed is obtained based on the pixel vector uniformity. According to the embodiments of this invention, by extracting feature vectors of pixels in the image to be processed and measuring vector uniformity based on a pre-trained set of uniform pixel feature vectors, a uniformity heatmap that accurately reflects the pixel uniformity in the image is output. This facilitates the identification and optimization of non-uniform regions for users and eliminates the need for manual pre-setting of visual features, thus broadening the application scenarios of the image processing method.

[0027] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0028] like Figure 1 As shown, Figure 1 This is a flowchart of an image processing method provided in one embodiment of the present invention. This image processing method can be applied to terminals, such as mobile phones, computers, cameras, and other electronic devices. Figure 1 In the example, the image processing method may include, but is not limited to, steps S100 to S500.

[0029] Step S100: Obtain the image to be processed.

[0030] It should be noted that there is no limitation on the spatial resolution for any image to be processed, that is, whether the image to be processed is a high-resolution high-definition image or a low-resolution blurry image, it has no impact on the implementation of the image processing method in the embodiments of this application.

[0031] Step S200: Obtain the feature vectors of pixels in the image to be processed.

[0032] It should be noted that, in order to obtain the feature vectors of pixels in the image to be processed, the image can be input into an image feature extraction model, and the output of the model is the feature vector of the pixels in the image to be processed. Depending on different scenarios or needs, feature extraction can be performed on the entire image to be processed, or it can be performed on a local part of the image. Local image feature extraction can reduce the feature extraction time, thereby improving the efficiency of feature extraction.

[0033] It should also be noted that in some application scenarios, various typical non-uniform images and their corresponding feature vectors can be obtained in advance through machine learning models or manual processing. Therefore, when obtaining the image to be processed, it can be classified or judged manually or by a model to obtain its corresponding feature vector. This method of obtaining feature vectors is simpler and faster, especially suitable for application scenarios that do not support running image feature extraction models at any time.

[0034] Step S300: Obtain the set of uniform pixel feature vectors obtained through pre-training.

[0035] It should be noted that a uniform pixel feature vector set is the set of pixel feature vectors output by an image feature extraction model after training in machine learning, and it can be obtained through pre-training. Training samples used to train the model can be complete images or partial images, and the resolution of the training samples can be the same or different.

[0036] It should be noted that image uniformity can be reflected in many aspects, such as color uniformity, brightness uniformity, and texture uniformity. The image processing method proposed in this embodiment can realize image uniformity recognition under various conditions and types. Three examples are used below for further explanation.

[0037] Example 1:

[0038] To obtain an image feature extraction model capable of extracting color features from images, we first use multiple complete images or partial image patches of complete images with uniform colors as training samples. We then train a ResNet50 convolutional neural network model pre-trained on the ImageNet large-scale dataset. Here, ResNet50 is an image feature extraction model. This model can extract features from pixels with uniform colors and represent them using a set of uniform pixel feature vectors.

[0039] Example 2:

[0040] To obtain an image feature extraction model capable of extracting texture features from images, multiple complete images or local image patches of complete images with uniform texture are used as training samples to train a pre-trained DenseNet convolutional neural network model. Here, DenseNet is an image feature extraction model that can extract features from pixels with uniform texture and then represent them through a set of uniform pixel feature vectors.

[0041] Example 3:

[0042] To obtain an image feature extraction model capable of extracting brightness features from images, multiple complete images or local image patches of complete images with uniform brightness are used as training samples. Deep learning is performed on other pre-trained convolutional neural network models. This model can extract pixel features with brightness texture and then represent them through a set of uniform pixel feature vectors.

[0043] Those skilled in the art will understand that the two embodiments described above use a deep convolutional neural network model as the image feature extraction model because it performs well in image recognition. However, other machine learning models, such as deep convolutional neural networks, can also be applied to the image processing methods provided in the embodiments of this application.

[0044] It should be noted that the uniform pixel feature vector set can be a set of image pixel feature vectors directly extracted after model training, or it can be a smaller set of image pixel features obtained by clustering the image pixel features extracted after model training.

[0045] In one embodiment, in order to improve the representational ability of uniform pixel feature vectors, feature aggregation is performed on the output features of the output layer of the image feature extraction model.

[0046] Step S400: Obtain the vector uniformity of the pixel based on the set of uniform pixel feature vectors and the feature vector of the pixel.

[0047] It's important to note that there are various methods for measuring the similarity between features, primarily depending on the computational relationship between feature vectors. For feature similarity measurement, the smaller the similarity, the greater the difference between the two feature vectors, and consequently, the greater the difference between image pixels. Conversely, the greater the similarity, the smaller the difference between the two feature vectors, and consequently, the smaller the difference between image pixels. For example, when identifying and processing whether color, brightness, and texture are uniform in an image, the similarity is compared between the feature vectors of pixels in the image to be processed and the feature vectors in the set of uniform pixel feature vectors. When the similarity is high, it means that the pixels in the image to be processed are similar in features to uniform pixels, and therefore may be identified as pixels with uniform color, brightness, and texture.

[0048] In one embodiment, the image processing method provides a metric for obtaining pixel uniformity by calculating cosine uniformity, such as... Figure 2 As shown, the specific steps include:

[0049] Reference Figure 2 As shown, in one embodiment, step S400 will be further described, and step S400 may include, but is not limited to, the following steps:

[0050] Step S411: Calculate the vector uniformity set of the pixel based on the pixel's feature vector and each uniform pixel feature vector in the set of uniform pixel feature vectors. The vector uniformity set includes vector uniformity elements.

[0051] Step S412: Determine the vector uniformity element with the largest value in the set of vector uniformity of pixels as the vector uniformity of pixels.

[0052] In one embodiment, the vector uniformity of a pixel can be cosine uniformity, assuming that the feature vector of pixel p in the image to be processed is v. p The set of uniform pixel feature vectors obtained from training is {v0, v1, ..., v...} N Then, the cosine uniformity of pixel p can be calculated using the following formula:

[0053]

[0054] In the above formula (1), s p V represents the cosine uniformity. q represents the uniform pixel feature vector in the set of uniform pixel feature vectors, and N represents the number of vectors in the set of uniform pixel feature vectors.

[0055] It is worth noting that the above calculation process only performs cosine uniformity calculation on a single pixel in the image to be processed. In practical applications of image processing methods, this calculation may be performed on multiple pixels or even all pixels in the image to be processed. Specifically, calculating the cosine value between pixel p and each feature vector in the set of uniform pixel feature vectors, and selecting the largest cosine similarity as the cosine similarity of pixel p, yields the feature vector that best represents pixel p. It is understandable that cosine uniformity is only one representation of the distance between two vectors. In practical applications, further calculations of the cosine uniformity may be needed to obtain the actual uniformity value of the pixels in the uniformity heatmap.

[0056] Step S500: Obtain the uniformity heatmap of the image to be processed based on the vector uniformity of the pixels.

[0057] It's important to note that a uniformity heatmap is a pixel-based graph that reflects the uniformity of pixels in an image. The color intensity of each pixel reflects the uniformity of the corresponding pixels in the image being processed. Using cosine uniformity to calculate pixel uniformity allows for manual adjustment of the heatmap display based on specific needs after obtaining the pixel uniformity values. For example, if dark colors represent non-uniform areas and light colors represent uniform areas, then for the cosine uniformity measurement method, pixels with smaller cosine uniformity values ​​should be represented by darker colors, and pixels with larger cosine uniformity values ​​by lighter colors. The pixel intensity is set according to user requirements, and the difference between different uniformity methods lies in the uniformity values ​​obtained after measurement.

[0058] It should be noted that the pixel uniformity heatmap can be a global pixel heatmap of the image to be processed, or a local pixel uniformity heatmap. Outputting a pixel uniformity heatmap helps users better observe the distribution of target pixels.

[0059] It should also be noted that pixel uniformity heatmap is a general term for heatmaps that represent the uniformity of an image. In practical applications, non-uniformity heatmaps can also be used to represent the uniformity of an image, and their generation method and principle are the same as those of uniformity heatmaps.

[0060] In the embodiments described later, the uniformity heatmap is further marked to make non-uniform pixels stand out more. For example... Figure 3 As shown, it includes at least the following steps:

[0061] Step S100: Obtain the image to be processed.

[0062] Step S200: Obtain the feature vectors of pixels in the image to be processed.

[0063] Step S300: Obtain the set of uniform pixel feature vectors obtained through pre-training.

[0064] Step S410: Obtain the cosine uniformity of the pixel based on the set of uniform pixel feature vectors and the feature vector of the pixel.

[0065] In this embodiment, cosine uniformity is used to measure the uniformity of pixels.

[0066] Step S510: Obtain the uniformity heatmap of the image to be processed based on the cosine uniformity of the pixels.

[0067] Step S600: Based on the uniformity heatmap and preset uniformity judgment conditions, identify non-uniform pixels among the pixels. In this embodiment, when using cosine similarity for uniformity measurement, the non-uniformity judgment condition is whether the cosine uniformity corresponding to the pixel is less than a first preset pixel uniformity threshold. Specifically, the calculated cosine uniformity of multiple pixels is compared with the first preset pixel uniformity threshold. For each pixel, if the cosine uniformity is greater than the first preset pixel uniformity threshold, then this pixel is a uniform pixel; if the cosine uniformity is less than the first preset pixel uniformity threshold, then this pixel is a non-uniform pixel.

[0068] It should be noted that in this embodiment, abnormal pixels are identified by determining whether the uniformity of each pixel is less than a preset pixel uniformity threshold, i.e., non-uniform pixels are identified.

[0069] Step S700: Mark non-uniform pixels. It should be noted that there are multiple methods for marking non-uniform pixels; any method that can distinguish between uniform and non-uniform pixels is considered to have marked non-uniform pixels.

[0070] Understandably, non-uniform pixels are identified by highlighting them in the image to be processed, for example, by circling or highlighting their outlines. It should be noted that the pixel uniformity heatmap can be a global pixel heatmap or a local pixel heatmap of the image to be processed. Outputting a pixel heatmap helps users better observe the distribution of target pixels.

[0071] The image processing method provided in this application extracts the image features of the image to be processed and compares them with the uniform pixel feature vector obtained through pre-training. This enables the identification of non-uniform pixels in the image that do not meet the feature requirements, eliminating the need for manual pre-setting of visual features and making the image processing method applicable to a wider range of scenarios. At the same time, it can also mark the identified non-uniform pixels, making it convenient for users to identify and optimize these non-uniform pixels, thus comprehensively improving the image processing capabilities.

[0072] like Figure 4 As shown, Figure 4 This is a flowchart of the image feature extraction model training process provided in one embodiment of the present invention, which includes at least the following steps:

[0073] Step S310: Obtain training samples, which are uniform images.

[0074] It should be noted that training samples can be complete images or local patches of images, and the spatial resolution of different training sample images can be inconsistent. During model training, the same batch of samples can be repeatedly input, or the same batch of samples can be divided into batches, with different batches of samples input in each iteration.

[0075] Step S320: Input the training samples into the image feature extraction model to obtain a set of uniform pixel feature vectors of the samples; wherein, the image feature extraction model is a convolutional neural network model.

[0076] In one embodiment, a convolutional neural network (CNN) is used as the image feature extraction model, which has a stronger feature extraction capability compared to other machine learning models. A CNN model pre-trained on large-scale datasets such as ImageNet and COCO is used to extract features of the image's pixel distribution. Furthermore, the unsupervised learning feature extraction method avoids manual labeling of the data, significantly reducing the workload of manual annotation and allowing pre-trained models on any large-scale dataset to be used in the image uniformity recognition scheme provided in this embodiment of the invention.

[0077] It is worth noting that in the image feature extraction process, the feature dimensionality is very high. To improve the representation ability of deep features, it is necessary to increase the feature receptive field and aggregate the output features of different convolutional layers. The topology of the convolutional neural network using feature aggregation is shown in the figure below. Figure 5 As shown.

[0078] In this embodiment, the image to be processed is input to the input layer 101. The output layers 102 to 105 are convolutional neural networks and are used for feature extraction. The three middle output layers of the convolutional neural network are used for feature aggregation. Assuming that output layer 3 104 is the j-th output layer, then output layer 2 103 is the j-1 layer, and output layer 4 105 is the j+1 layer.

[0079] Sample x i The feature map output by the j-th layer after inference from the pre-trained convolutional neural network model is represented as follows:

[0080]

[0081] In the above formula (2), C j H j W j These represent the number of channels, height, and width of the feature map, respectively, φ. ij This represents the output of sample i at layer j.

[0082] In this embodiment, the output of the middle three layers of the pre-trained model is used for feature aggregation, and the aggregated features are represented as follows:

[0083] F(x i )=f agg ({φ ij-1 ,φ ij ,φ ij+1}),j∈{2} (3)

[0084] In the above formula (3), f agg (·) is the aggregation operator, which first performs aggregation on the output features φ of the output layer 3 104. ij Perform 2x upsampling on the output feature φ of layer 4 (105). ij+1 Perform 4x upsampling, then compare the interpolated feature map with the feature map φ of the output layer 2103. ij-1 Channel merging is performed to aggregate features from three different layers; finally, the aggregated features are upsampled to ensure that the output spatial resolution is consistent with the input image resolution.

[0085] Assuming training sample x i The spatial resolution is H i ×W i The deep features extracted by the pre-trained convolutional neural network are represented as follows:

[0086] F i =f(x) i (4)

[0087] In the above formula (4), F i Let f(·) represent the deep feature representation extracted from the training samples through a pre-trained convolutional neural network, and let f(·) represent the feature extraction operator. The number of output channels of the feature extraction operator f(·) is C. f The spatial dimension is H i ×W i The feature space dimension is preserved consistent with the input image space resolution, meaning each pixel can be represented by a Ci. f The feature vector of dimension is used to represent the sample x. After feature extraction, the sample x i H can be obtained i ×W i C of uniform pixels f The feature vector of a dimension depth, where the feature vector of each pixel is represented as:

[0088]

[0089] In the above formula (5), v k Represents the depth feature vector of a pixel. The dimension of the depth feature vector of a pixel.

[0090] Through the above feature extraction, feature vectors of a large number of pixels can be obtained. Assuming that the number of pixels is M, that is, the preset target clustering feature vector dimension is M, the sample uniform pixel feature vector set can be expressed as:

[0091] Ω={v0,v1,…,v M} (6)

[0092] In the above formula (6), Ω represents the sample uniform pixel feature vector set, v M represents the sample uniform pixel feature vector.

[0093] Step S330: Perform clustering processing on the elements in the sample uniform pixel feature vector set to obtain a uniform pixel feature vector set.

[0094] In general, due to the large number of training samples and the high resolution of training sample images, the number of involved pixels is huge. To facilitate subsequent pixel-level uniformity measurement, it is necessary to cluster the feature set to reduce the magnitude of the number of feature vector pixels.

[0095] In this embodiment, the kmeans++ clustering algorithm is used to cluster the feature set. Assuming that the final number of clustered uniform pixels is N(N<<M), the finally obtained pixel feature vector set can be expressed as:

[0096] Ω′={v0,v1,…,v N} (7)

[0097] In the above formula (7), Ω' represents the uniform pixel feature vector set after clustering.

[0098] As Figure 6 shown, a schematic diagram of the target area is provided, where each square grid represents a pixel. In this embodiment, uniform texture pixels are identified and marked. It can be seen that pixels A, B, C, D, E, and F in the figure are all texture non-uniform pixels, among which the pixels, B, C, D, E, and F are adjacent to each other, forming a pixel area. In some application scenarios, users may not be very concerned about some non-uniform pixel areas with small areas. Therefore, the area of the pixel areas can be set in advance, and only the pixel areas with an area exceeding the preset area are marked, and these pixel areas that meet the marking conditions are defined as target areas. In this embodiment, contour lines are used to mark the target areas to make them stand out for the user's convenient observation.

[0099] In another embodiment, the number of target areas and the number of pixels contained in each target area can also be output and displayed, which enables the user to understand the image situation more accurately and facilitates the user's subsequent optimization of the image and even the product.

[0100] Since the image processing method provided in this application can be implemented using a pre-constructed image processing model, the specific process of implementing the image processing method provided in this application using a cosine-constructed image processing model is described below in one embodiment. Figure 7 As shown, the specific steps include:

[0101] Step S810: Load the image processing model, which includes a trained image feature extraction model and a set of feature vectors for uniform pixels.

[0102] Understandably, the feature vector set of uniform pixels is typically stored on disk to form a model file, which is loaded when needed. Image feature extraction is trained by pre-inputting multiple sample images with uniform color, texture, or brightness. The trained image feature extraction model can then extract features from the input image to be processed. The image feature extraction model can exist as a functional module within an image processing model.

[0103] In this embodiment, four uniform images are selected as training samples with an image resolution of 1080x1920. It is worth noting that the uniform sample in this embodiment can also be a local image patch of a uniform image, and the sizes of different training images can be different.

[0104] Step S820: Input the video frame into the image processing model.

[0105] In this embodiment, offline video is loaded from the disk and video frames are read as the input source, i.e., the image to be processed.

[0106] However, it is worth noting that network video streams can also be read as input sources via RTMP or RTSP.

[0107] Step S830: The image feature extraction model extracts feature vectors from the input video frames to obtain the feature vectors of pixels in the image to be processed.

[0108] In this embodiment, any pre-trained model on a large-scale dataset can be used for uniform pixel feature extraction in this invention. In this embodiment, a ResNet50 convolutional neural network model pre-trained on the ImageNet dataset is selected to extract features of uniformly distributed pixels and obtain the feature vector of the pixel.

[0109] In this embodiment, sample x i The spatial resolution is 1080x1920, and the deep features extracted by the pre-trained convolutional neural network are represented as follows:

[0110] F i =f(x) i )

[0111] The deep feature extraction operator f(·) outputs 3584 channels with a spatial dimension of 1080x1920. The feature space dimension is preserved consistent with the input image spatial resolution, meaning each pixel can be represented by a 3584-dimensional feature vector. Four 1080x1920 uniform sample images can yield M = 8,294,400 feature vectors of uniform pixels, each with a dimension of 3584. The set of sample uniform pixel feature vectors can be represented as:

[0112] Ω={v0,v1,…,v M}

[0113] In this embodiment, to reduce the computational cost of subsequent feature uniformity, the feature set is clustered, reducing the number of pixels in the feature vectors by an order of magnitude. The k-means++ clustering algorithm is used to cluster the feature vector set, resulting in a final cluster with N = 100 uniform pixels. The final clustered set of uniform pixel feature vectors can be represented as:

[0114] Ω′={v0,v1,…,v N}

[0115] Similar to the uniform sample learning process, this embodiment assumes that the spatial resolution of video frame y is 1080x1920, and the depth features F extracted by the pre-trained model are... y Represented as:

[0116] F y (i,j)=f(y),(1≤i≤1080,1≤j≤1920)

[0117] Among them, F y The dimensions are 3584×1080×1920.

[0118] Step S840: Calculate the uniformity of the feature vector of each pixel in the video frame for the image processing model.

[0119] Based on the set of uniform pixel feature vectors Ω′ that have been trained to each uniform pixel, the depth feature map F of the image to be processed is... y We perform a feature uniformity measurement, that is, a uniformity measurement, to obtain a uniformity heatmap for each pixel location:

[0120] S y (i,j),(1≤i≤1080,1≤j≤1920)

[0121] S y The value range is [0,1], where 0 indicates the worst uniformity and 1 indicates the best uniformity.

[0122] In this embodiment, the feature uniformity measurement adopts the vector cosine uniformity measurement method, which measures the pixel p to be measured and its depth feature vector v. p Based on the training obtained sample uniform pixel feature vector set {v0,v1,L,v... N The uniformity value of pixel p can be expressed as:

[0123] In this embodiment, N = 100.

[0124] Step S850: Display the pixel uniformity heatmap corresponding to the video frame.

[0125] Step S860: Extract non-uniform pixels that do not meet the uniformity standard and outline the non-uniform pixels.

[0126] like Figure 8 As shown, a schematic diagram of the effect of image processing according to the image processing method of the above embodiment is provided. It can be seen that the non-uniformity heatmap corresponding to the image to be processed is output. In the non-uniformity heatmap, 1 indicates the worst uniformity and 0 indicates the best uniformity. That is, the darker the color, the more non-uniform it is, and the lighter the color, the more uniform it is. Users can easily see the non-uniform pixels or regions in the image, which is intuitive and effective.

[0127] In other embodiments, for clearer identification, the outlines of non-uniform regions can be overlaid on the uniformity heatmap, and the number of non-uniform regions can be displayed.

[0128] The image processing method provided in this application extracts the image features of the image to be processed and compares them with the uniform pixel feature vector obtained through pre-training. This enables the identification of non-uniform pixels in the image that do not meet the feature requirements, eliminating the need for manual pre-setting of visual features and making the image processing method applicable to a wider range of scenarios. At the same time, it can also mark the identified non-uniform pixels, making it convenient for users to identify and optimize these non-uniform pixels, thus comprehensively improving the image processing capabilities.

[0129] In addition, one embodiment of the present invention provides an electronic device 110, which includes a processor 111, a memory 112, and a computer program stored in the memory and executable on the processor. The processor is capable of executing the computer program to implement the image processing method described above.

[0130] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0131] The non-transient software program and instructions required to implement the image processing method of the above embodiments are stored in memory. When executed by a processor, the image processing method of the above embodiments is executed, for example, the method described above. Figure 1 Method steps S100 to S500 Figure 2 Method steps S411 to S412 in the text Figure 3 Method steps S100 to S700 in the text Figure 4 Method steps S310 to S330 in the text Figure 7 Method steps S810 to S860.

[0132] The network element embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0133] Furthermore, one embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions that are executed by a processor or controller, for example, by a processor in the above-described network element embodiment, causing the processor to perform the image processing method described above, for example, performing the above-described... Figure 1 Method steps S100 to S500 Figure 2 Method steps S411 to S412 in the text Figure 3 Method steps S100 to S700 in the text Figure 4 Method steps S310 to S330 in the text Figure 7 Method steps S810 to S860.

[0134] Furthermore, embodiments of the present invention also provide a computer program product, including a computer program or computer instructions, characterized in that the computer program or computer instructions are stored in a computer-readable storage medium, a processor of a computer device reads the computer program or computer instructions from the computer-readable storage medium, and the processor executes the computer program or computer instructions, causing the computer device to perform the image processing method described above, for example, performing the above-described... Figure 1 Method steps S100 to S500 Figure 2 Method steps S411 to S412 in the text Figure 3 Method steps S100 to S700 in the text Figure 4 Method steps S310 to S330 in the text Figure 7 Method steps S810 to S860.

[0135] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0136] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. An image processing method, comprising: Obtain the image to be processed; The image to be processed is input into the image feature extraction model to obtain the feature vectors of the pixels in the image to be processed. Obtain the set of uniform pixel feature vectors obtained through pre-training; The vector uniformity of the pixel is obtained based on the set of uniform pixel feature vectors and the feature vector of the pixel. Based on the vector uniformity of the pixels, a uniformity heatmap of the image to be processed is obtained. The step of obtaining the vector uniformity of the pixel based on the set of uniform pixel feature vectors and the feature vector of the pixel includes: Based on the feature vector of the pixel and each uniform pixel feature vector in the set of uniform pixel feature vectors, the vector uniformity set of the pixel is calculated, and the vector uniformity set includes vector uniformity elements. The element with the largest value in the set of vector uniformity of the pixel is determined as the vector uniformity of the pixel. The uniform pixel feature vector set is obtained by the following steps: Obtain training samples, wherein the training samples are uniform image samples; The training samples are input into the image feature extraction model to obtain a set of uniform pixel feature vectors for the samples. Clustering is performed on the elements in the set of uniform pixel feature vectors of the samples to obtain the set of uniform pixel feature vectors. The image feature extraction model includes a first output layer, a second output layer, and a third output layer; The step of inputting the training samples into the image feature extraction model to obtain a set of uniform pixel feature vectors includes: The training samples are input into the image feature extraction model to obtain a first feature map output by the first output layer, a second feature map output by the second output layer, and a third feature map output by the third output layer. The second feature map is upsampled to obtain a second upsampled feature map; The third feature map is upsampled to obtain a third upsampled feature map. The first feature map is combined with the second upsampled feature map and the third upsampled feature map through channel merging to obtain the aggregated feature map; Based on the image resolution of the training samples, the aggregated feature map is upsampled to obtain the aggregated feature vector set; The aggregated feature vector set is determined as the sample uniform pixel feature vector set.

2. The method according to claim 1, characterized in that, The method further includes: Based on the uniformity heatmap and the preset non-uniformity judgment conditions, non-uniform pixels are identified among the pixels; the non-uniform pixels are then marked.

3. The method according to claim 2, characterized in that, The non-uniformity judgment condition includes: the vector uniformity corresponding to the non-uniform pixel is less than the preset pixel uniformity threshold. The step of determining non-uniform pixels among the pixels based on the uniformity heatmap and preset non-uniformity judgment conditions includes: The vector uniformity of the pixel is compared with the preset pixel uniformity threshold. Vector uniformity values ​​that are less than the preset pixel uniformity threshold are defined as target vector uniformity. The pixels corresponding to the uniformity of the target vector are defined as non-uniform pixels.

4. The method according to claim 2, characterized in that, The step of marking the non-uniform pixels includes: Determine whether there is a non-uniform region, wherein the non-uniform region includes a number of non-uniform pixels that are greater than a preset pixel number threshold; When it is determined that the non-uniform region exists, the non-uniform region is marked.

5. The method according to claim 4, characterized in that, The process of marking the non-uniform region includes: The non-uniform pixels are merged to obtain a non-uniform region; The outline of the non-uniform region is highlighted on the image to be processed.

6. The method according to any one of claims 2 to 3, characterized in that, The step of marking the non-uniform pixels includes: The non-uniform pixels are highlighted on the image to be processed.

7. The method according to claim 1, characterized in that, The step of clustering the elements in the set of uniform pixel feature vectors of the samples to obtain the set of uniform pixel feature vectors includes: Based on the preset clustering feature vector dimension, the elements in the sample uniform pixel feature vector set are clustered to obtain the clustered feature vector set. The clustered feature vector set is determined as the sample uniform pixel feature vector set.

8. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements the image processing method as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing computer-executable instructions for performing the image processing method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program or computer instructions, characterized in that, The computer program or the computer instructions are stored in a computer-readable storage medium, and the processor of the computer device reads the computer program or the computer instructions from the computer-readable storage medium. The processor executes the computer program or the computer instructions, causing the computer device to perform the image processing method as described in any one of claims 1 to 7.

Images