Image processing methods, apparatus, electronic devices and storage media

Abstract

This application provides an image processing method, apparatus, electronic device, and computer-readable storage medium based on artificial intelligence; it relates to computer vision technology in the field of artificial intelligence; the method includes: performing feature extraction processing on an image to obtain a feature map of the image; performing activation processing on the feature map at multiple levels, and downsampling the activation results at each level to obtain multiple feature vectors; fusing the multiple feature vectors to obtain a fused feature vector; performing multiple different types of image quality prediction processing based on the fused feature vector to obtain corresponding multiple different types of image quality prediction results; and fusing the multiple different types of image quality prediction results to obtain the quality of the image. This application can adapt to the quality recognition of any image in actual business scenarios and improve the accuracy of image quality recognition.

Description

Technical Field

[0001] This application relates to artificial intelligence technology and Internet technology, and in particular to an image processing method, apparatus, electronic device and computer-readable storage medium based on artificial intelligence. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. With the research and advancement of AI technology, it has been studied and applied in multiple fields.

[0003] Taking image quality recognition as an example, with the explosive growth of image data, it is crucial to quickly and accurately determine the quality of images (e.g., whether there is deformation or displacement). Related technologies typically use gradient features to predict image quality, but this method has low accuracy in image quality recognition, and there is currently no effective solution to this problem. Summary of the Invention

[0004] This application provides an image processing method, apparatus, electronic device, and computer-readable storage medium based on artificial intelligence, which can adapt to the quality recognition of any image in actual business scenarios and improve the accuracy of image quality recognition.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides an image processing method based on artificial intelligence, including:

[0007] The image is subjected to feature extraction processing to obtain the feature map of the image;

[0008] The feature map is subjected to activation processing at multiple levels, and the activation results at each level are downsampled to obtain multiple feature vectors.

[0009] The multiple feature vectors are fused to obtain a fused feature vector;

[0010] Based on the fused feature vector, multiple different types of image quality prediction processing are performed to obtain corresponding multiple different types of image quality prediction results.

[0011] The image quality is obtained by fusing the prediction results of the multiple different types of images.

[0012] This application provides an image processing device based on artificial intelligence, including:

[0013] The extraction module is used to perform feature extraction processing on the image to obtain the feature map of the image;

[0014] The activation module is used to perform activation processing on the feature map at multiple levels, and to downsample the activation results at each level to obtain multiple feature vectors.

[0015] The fusion module is used to fuse the multiple feature vectors to obtain a fused feature vector;

[0016] The prediction module is used to perform multiple different types of image quality prediction processing based on the fused feature vector, and obtain corresponding multiple different types of image quality prediction results respectively;

[0017] The quality module is used to fuse the prediction results of the multiple different types of image quality to obtain the quality of the image.

[0018] In the above scheme, the activation module is further configured to activate the input of the nth activation network through the nth activation network in the N cascaded activation networks to obtain the nth activation result; and transmit the nth activation result to the (n+1)th activation network for further activation processing; wherein, N is an integer greater than or equal to 2; n is an integer starting from 1 and increasing in value, and the value range of n satisfies 1≤n≤N-1; when n is 1, the input of the nth activation network is the feature map, and when n is 2≤n≤N-1, the input of the nth activation network is the activation result of the (n-1)th activation network.

[0019] In the above scheme, the activation module is further configured to perform channel-based average pooling on the input of the nth activation network to obtain the global content features of each channel corresponding to the input of the nth activation network; perform fully connected processing on the global content features of each channel corresponding to the input of the nth activation network to obtain the activation value of each channel corresponding to the input of the nth activation network; perform dot product processing on the activation value of each channel corresponding to the input of the nth activation network and the original content features of each channel in the input of the nth activation network; and perform downsampling processing on the result of the dot product processing to obtain the nth activation result.

[0020] In the above scheme, the activation module is further configured to perform the following processing for the activation results at each level: determine the sampling kernel and step size that are compatible with the activation results; sample the activation results sequentially according to the sampling kernel and the step size to obtain multiple sampling results; and combine the multiple sampling results to obtain the feature vector.

[0021] In the above scheme, the prediction module is further configured to perform at least two of the following image quality prediction processes: performing deformation prediction processing based on the fused feature vector to obtain the deformation of the image; performing sharpness prediction processing based on the fused feature vector to obtain the sharpness of the image; and performing aesthetics prediction processing based on the fused feature vector to obtain the aesthetics of the image.

[0022] In the above scheme, the prediction module is further configured to perform dimensionality reduction processing on the fused feature vector to obtain a first dimensionality-reduced feature vector; perform denoising processing on the first dimensionality-reduced feature vector to obtain a first denoised feature vector; map the first denoised feature vector to confidence scores belonging to different deformation degree labels, and determine the deformation degree label corresponding to the maximum confidence score as the deformation degree of the image.

[0023] In the above scheme, the prediction module is further configured to perform dimensionality reduction processing on the fused feature vector to obtain a second dimensionality-reduced feature vector; perform denoising processing on the second dimensionality-reduced feature vector to obtain a second denoised feature vector; map the second denoised feature vector to confidence scores belonging to different sharpness labels, and determine the sharpness label corresponding to the maximum confidence score as the sharpness of the image.

[0024] In the above scheme, the prediction module is further configured to perform dimensionality reduction processing on the fused feature vector to obtain a third dimensionality-reduced feature vector; perform denoising processing on the third dimensionality-reduced feature vector to obtain a third denoised feature vector; map the third denoised feature vector to confidence scores belonging to different aesthetic labels, and determine the aesthetic label corresponding to the maximum confidence score as the aesthetic score.

[0025] In the above scheme, the extraction module is further configured to normalize the image to obtain normalized image data; perform convolution processing on the normalized image data to obtain a convolutional feature map; and perform pooling processing on the convolutional feature map to obtain a feature map of the image.

[0026] In the above scheme, the fusion module is further configured to determine the weight corresponding to each feature vector; and to perform a weighted summation of the multiple feature vectors according to the weight corresponding to each feature vector to obtain the fused feature vector.

[0027] In the above scheme, the extraction module is further configured to identify a target region containing a target object from the image; normalize the target region to obtain normalized region data; perform convolution processing on the normalized region data to obtain a target convolutional feature map; perform pooling processing on the target convolutional feature map to obtain a feature map of the target region, and use the feature map of the target region as the feature map of the image.

[0028] In the above scheme, the quality module is further configured to identify a target region containing a target object from the image, and extract target features from the target region; map the target features to confidence levels belonging to different deformation labels, and determine the deformation label corresponding to the maximum confidence level as the deformation degree of the target object; and perform fusion processing on the image quality prediction results corresponding to the multiple different types and the deformation degree of the target object to obtain the quality of the image.

[0029] This application provides an electronic device, including:

[0030] Memory is used to store executable instructions for a computer;

[0031] The processor, when executing computer-executable instructions stored in the memory, implements the artificial intelligence-based image processing method provided in the embodiments of this application.

[0032] This application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the artificial intelligence-based image processing method provided in this application.

[0033] The embodiments of this application have the following beneficial effects:

[0034] Through multiple levels of activation processing, downsampling processing, and fusion processing, feature vectors can be extracted from images of different sizes, thus adapting to the quality recognition of any image in different business scenarios. By fusing multiple image quality prediction results corresponding to different types to determine the image quality, the accuracy of image quality recognition can be improved, thereby saving resources for quality recognition. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the architecture of an artificial intelligence-based image processing system 100 provided in an embodiment of this application;

[0036] Figure 2 This is a schematic diagram of the structure of the server 200 provided in the embodiments of this application;

[0037] Figure 3 This is a schematic diagram of the architecture of the machine learning model provided in the embodiments of this application;

[0038] Figure 4A and Figure 4B This is a schematic diagram of the architecture of the machine learning model provided in the embodiments of this application;

[0039] Figure 5 This is a flowchart illustrating the image processing method based on artificial intelligence provided in an embodiment of this application;

[0040] Figure 6A and Figure 6B This is a flowchart illustrating the image processing method based on artificial intelligence provided in an embodiment of this application;

[0041] Figure 7 This is a schematic diagram of the image quality recognition model provided in the embodiments of this application. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0044] In the following description, the terms "first / second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first / second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0046] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0047] 1) Computer Vision (CV): This is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing, tracking, and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0048] 2) Image quality (or subjective quality): It is generally determined by the image's sharpness, distortion, and aesthetics.

[0049] 3) Image resolution: refers to the amount of information stored in an image, specifically the number of pixels per inch. The unit of resolution is pixels per inch (PPI).

[0050] 4) Convolutional Neural Networks (CNN): These are feedforward neural networks that include convolutional computations and have a deep structure. They are generally composed of convolutional layers, pooling layers, and fully connected layers, and are mainly used in image classification, image detection, and image segmentation.

[0051] With the explosive growth of image data, the ability of backend systems to quickly and effectively input image information is crucial. In real-world business scenarios, such as the backend of a short video client, after a user uploads a cover image for a video, the system needs to review and store the cover image information. The review process includes checking whether the cover image is clear, aesthetically pleasing, and free from distortion. If the cover image is unclear, unattractive, or distorted, the user is prompted to re-upload it.

[0052] In some embodiments, the following issues are mainly considered when predicting image quality in actual business scenarios: (1) whether the image itself is clear, (2) the degree of image distortion, and (3) the aesthetics of the image. For the task of determining whether the image itself is clear, due to the special nature of this task, the size of the image will affect the image clarity prediction. For example, an image with average visual quality but a large resolution will be considered clear after scaling. Or some images may be visually fine, but after stretching, compression, or other processing, image distortion or reduced aesthetics may occur. In related technologies, images of fixed size are usually accepted as model input, and quality prediction cannot be performed for images of all sizes, thus the applicability is low. In view of the above technical problems, this application provides an image processing method based on artificial intelligence, which can adapt to the quality recognition of any image in actual business scenarios and improve the accuracy of image quality recognition. The following describes an exemplary application of the image processing method based on artificial intelligence provided in this application. The image processing method based on artificial intelligence provided in this application can be implemented by various electronic devices. For example, it can be implemented by a terminal or server alone, or by a terminal and server working together.

[0053] Next, we will describe an embodiment of this application using an example of implementation by a server and a terminal working together. See [link to relevant documentation]. Figure 1 , Figure 1 This is a schematic diagram of the architecture of an artificial intelligence-based image processing system 100 provided in an embodiment of this application.

[0054] The image processing system 100 based on artificial intelligence includes a server 200, a network 300, and a terminal 400, which will be described separately.

[0055] Server 200 is the backend server for client 410. It is used to receive user-uploaded images (such as cover images for short videos, shared images on social networks, etc.) sent by client 410. It is also used to determine the quality of user-uploaded images. When the quality of the user-uploaded image meets the qualification criteria, it stores the user-uploaded image in the database and sends a prompt message to client 410 to indicate that the image upload was successful. When the quality of the user-uploaded image does not meet the qualification criteria, it sends a prompt message to client 410 to prompt the user to re-upload the image.

[0056] Network 300, serving as the communication medium between server 200 and terminal 400, can be a wide area network (WAN), a local area network (LAN), or a combination of both.

[0057] Terminal 400 runs client 410, which is a client with image upload functionality, such as an instant messaging client, video client, microblogging client, short video client, etc. Client 410 is used to respond to image upload operations by sending the user-uploaded image to server 200; it is also used to receive prompt information sent by server 200 and display the prompt information on the human-computer interaction interface.

[0058] As an example, this can also be applied to image recommendation scenarios. Server 200 acquires multiple candidate images, determines the quality of each candidate image, selects those whose quality meets the qualification criteria, and sends them to client 410. Client 410 then displays the recommended images in the human-computer interaction interface. This improves the accuracy and efficiency of image recommendation, enhancing the user's image viewing experience.

[0059] The embodiments of this application can be implemented with the help of cloud technology, which refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to realize the computation, storage, processing, and sharing of data.

[0060] Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form resource pools, allowing for on-demand use with flexibility and convenience. Cloud computing technology will become a crucial support. The backend services of cloud computing systems require substantial computing and storage resources.

[0061] As an example, server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminal 400 can be various types of user terminals such as smartphones, tablets, in-vehicle terminals, smart wearable devices, laptops, desktop computers, and smart TVs. Terminal 400 and server 200 can be directly or indirectly connected via wired or wireless communication, and this embodiment does not impose any restrictions.

[0062] The following will explain... Figure 1 The structure of server 200 in [the document / reference]. See [the document / reference]. Figure 2 , Figure 2 This is a schematic diagram of the structure of the server 200 provided in the embodiments of this application. Figure 2The server 200 shown includes at least one processor 210, memory 240, and at least one network interface 220. The various components of server 200 are coupled together via a bus system 230. It is understood that the bus system 230 is used to implement communication between these components. In addition to a data bus, the bus system 230 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 230.

[0063] Processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0064] Memory 240 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 240 described in this application embodiment is intended to include any suitable type of memory. Memory 440 may optionally include one or more storage devices physically located away from processor 210.

[0065] In some embodiments, memory 240 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0066] Operating system 241 includes system programs for handling various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; network communication module 242 is used to reach other computer devices via one or more (wired or wireless) network interfaces 220, exemplary network interfaces 220 include: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0067] In some embodiments, the artificial intelligence-based image processing apparatus provided in this application can be implemented in software. Figure 2An AI-based image processing device 243, stored in memory 240, is shown. This device can be software in the form of programs and plugins, and includes the following software modules: extraction module 2431, activation module 2432, fusion module 2433, prediction module 2434, and quality module 2435. These modules can be logical function modules, and therefore can be arbitrarily combined or further divided according to the functions they implement. The functions of each module will be described below.

[0068] The following exemplarily describes a machine learning model for implementing the artificial intelligence-based image processing method of the embodiments of this application.

[0069] In some embodiments, machine learning models can be used to determine image quality. See also Figure 3 , Figure 3 This is a schematic diagram of the architecture of the machine learning model provided in the embodiments of this application. The machine learning model includes a feature extraction network, an activation network, a downsampling network, a first fusion network, a classification network, and a second fusion network.

[0070] As an example, a feature extraction network can be used to extract features from an image to obtain a feature map. Multiple (i.e., at least two) cascaded activation networks are then used to perform multi-level activation processing on the feature map, and a downsampling network connected in series with each activation network is used to downsample the activation results at each level, resulting in multiple feature vectors. A first fusion network is used to fuse these multiple feature vectors to obtain a fused feature vector. A classification network is then used to perform multiple different types of image quality prediction processing based on the fused feature vector, resulting in multiple corresponding image quality prediction results. Finally, a second fusion network is used to fuse these multiple different types of image quality prediction results to obtain the final image quality.

[0071] For example, feature extraction networks, activation networks, downsampling networks, first fusion networks, classification networks, and second fusion networks can include various types, such as convolutional neural networks, recurrent neural networks (RNNs), and multilayer feedforward neural networks.

[0072] Below, by Figure 1 The following description uses the server 200 in the example of executing the artificial intelligence-based image processing method provided in the embodiments of this application. See also... Figure 5 , Figure 5 This is a flowchart illustrating the artificial intelligence-based image processing method provided in the embodiments of this application, which will be combined with... Figure 5 The steps shown are explained.

[0073] In step S101, feature extraction processing is performed on the image to obtain the feature map of the image.

[0074] In some embodiments, the image is normalized to obtain normalized image data; the normalized image data is convolved to obtain a convolutional feature map; and the convolutional feature map is pooled to obtain a feature map of the image.

[0075] As an example, see Figure 4A , Figure 4A This is a schematic diagram of the architecture of the machine learning model provided in the embodiments of this application. Figure 3 The feature extraction network shown can include normalization layers, convolutional layers, and pooling layers. Thus, the normalization layer in the feature extraction network can normalize the image to obtain normalized image data; the convolutional layer in the feature extraction network can convolve the normalized image data to obtain a convolutional feature map; and the pooling layer in the feature extraction network can pool the convolutional feature map to obtain the image's feature map.

[0076] In some embodiments, the image can be a picture or a video frame from a video. Taking the image as a video frame from a video as an example, step S101 may specifically include: extracting video frames from the video and performing feature extraction processing on the video frames to obtain feature maps of the video frames. Thus, the quality of the video frames can be determined based on the feature maps, and the quality of the video can be determined based on the quality of the video frames.

[0077] As an example, multiple video frames are extracted from a video, and the quality of each video frame is determined. The average quality of the extracted video frames is determined as the quality of the video. Extracting multiple video frames from a video can be done by extracting all video frames from the video, randomly extracting multiple video frames from the video, or extracting one video frame at fixed intervals from the video, thus obtaining multiple video frames.

[0078] In some embodiments, a target region containing a target object is identified from an image; the target region is normalized to obtain normalized region data; the normalized region data is convolved to obtain a target convolutional feature map; the target convolutional feature map is pooled to obtain a feature map of the target region, and the feature map of the target region is used as the feature map of the image.

[0079] As an example, the type of the target object can be a person (e.g., a face, body, limbs, etc.), an animal, a plant, etc.

[0080] This application embodiment extracts feature maps from target regions of an image. Compared to extracting feature maps from the entire image, the feature maps extracted in this application embodiment can characterize the local features of the image, thereby ensuring the accuracy of the predicted image quality while saving computational resources.

[0081] In step S102, the feature map is activated at multiple levels, and the activation results at each level are downsampled to obtain multiple feature vectors.

[0082] In some embodiments, the process of performing multi-level activation processing on the feature map may include: activating the input of the nth activation network through the nth activation network in N cascaded activation networks to obtain the nth activation result; and transferring the nth activation result to the (n+1)th activation network for further activation processing; wherein N is an integer greater than or equal to 2; n is an integer starting from 1 and increasing in value, and the value range of n satisfies 1≤n≤N-1; when n is 1, the input of the nth activation network is the feature map, and when n is 2≤n≤N-1, the input of the nth activation network is the activation result of the (n-1)th activation network.

[0083] As a successor Figure 3 For example, the first activation network activates the feature map of the image output by the feature extraction network to obtain the first activation result; the second activation network activates the first activation result output by the first activation network to obtain the second activation result; the third activation network activates the second activation result output by the second activation network to obtain the third activation result; and so on, the (n+1)th activation network activates the nth activation result output by the nth activation network to obtain the (n+1)th activation result.

[0084] As an example, the activation process for the input of the nth activation network to obtain the nth activation result may include: performing channel-based average pooling on the input of the nth activation network to obtain the global content features corresponding to each channel of the input of the nth activation network; performing fully connected processing on the global content features corresponding to each channel of the input of the nth activation network to obtain the activation value corresponding to each channel of the input of the nth activation network; performing a dot product between the activation value corresponding to each channel of the input of the nth activation network and the original content features of each channel in the input of the nth activation network; and performing downsampling processing on the result of the dot product to obtain the nth activation result.

[0085] For example, the activation network mainly includes a squeezing layer (used to determine the global content features of each channel corresponding to the input of each activation network), an activation layer (used to determine the activation value of each channel corresponding to the input of each activation network), and a downsampling layer (used for downsampling processing). During convolution, the feature relationships of the channels are mixed with the spatial relationships learned by the convolution kernel. The activation network can extract this mixture, thereby directly learning the feature relationships of the channels. Since convolution only operates in a local space, it is difficult to obtain enough information to extract the relationships between channels. The squeezing layer encodes the entire spatial features of a channel into a global content feature (by performing channel-based average pooling). In principle, more complex aggregation strategies can also be used. After the squeezing layer obtains the global content features, another operation is needed to extract the channel features. The relationships between channels are learned through activation layers, meaning the non-linear relationships between each channel are learned, and these learned relationships are not mutually exclusive. Based on this, a threshold mechanism in the form of a sigmoid activation function is adopted. In addition, to reduce complexity and improve generalization ability, a structure containing two fully connected layers is adopted. The first fully connected layer plays a role in dimensionality reduction, and then the ReLU activation function is used for activation processing. The last fully connected layer is used to restore the original dimension. Finally, the learned activation values ​​of each channel (values ​​from 0 to 1) are multiplied by the original content features input to the squeezing layer. The above process actually learns the weight coefficients of each channel, thereby making the model more discriminative of the features of each channel. Finally, a downsampling layer is used to downsample the result of the multiplication, which can reduce dimensionality to reduce computational difficulty and improve the robustness of the model.

[0086] In some embodiments, downsampling the activation results at each level to obtain multiple feature vectors may include: performing the following processing on the activation results at each level: determining a sampling kernel and step size that are suitable for the activation results; sampling the activation results sequentially according to the sampling kernel and step size to obtain multiple sampling results; and combining the multiple sampling results to obtain a feature vector.

[0087] As a successor Figure 3 For example, the first activation result output by the first activation network is downsampled using the first downsampling network to obtain the first feature vector; the second activation result output by the second activation network is downsampled using the second downsampling network to obtain the second feature vector; the third activation result output by the third activation network is downsampled using the third downsampling network to obtain the third feature vector; and so on, the (n+1)th activation result output by the (n+1)th activation network is downsampled using the (n+1)th downsampling network to obtain the (n+1)th feature vector.

[0088] This application embodiment can transform feature maps of any size with the same number of channels into feature vectors of the same dimension, so that images of any size can be used as input to the model. This solves the technical problem that related technologies usually only accept images of fixed size as input to the model, and cannot perform quality prediction for images of all sizes, resulting in low adaptability to images of different sizes in actual business scenarios. This can improve the accuracy of image quality prediction.

[0089] In step S103, multiple feature vectors are fused to obtain a fused feature vector.

[0090] In some embodiments, the weight corresponding to each feature vector is determined; based on the weight corresponding to each feature vector, multiple feature vectors are weighted and summed to obtain a fused feature vector.

[0091] As a successor Figure 3 For example, through the first fusion network, the weights corresponding to the first feature vector output by the first downsampling network, ..., and the (n+1)th feature vector output by the (n+1)th downsampling network are determined respectively; according to the weights corresponding to each feature vector, the first feature vector, ..., and the (n+1)th feature vector are weighted and summed to obtain the fused feature vector. In this way, redundant information in the feature vector before fusion can be reduced, making the obtained fused feature vector more discriminative.

[0092] In step S104, multiple different types of image quality prediction processing are performed based on the fused feature vector to obtain corresponding multiple different types of image quality prediction results.

[0093] In some embodiments, see Figure 6A , Figure 6A This is a flowchart illustrating the image processing method based on artificial intelligence provided in an embodiment of this application. Figure 5 Step S104 may include steps S1041 to S1043, wherein at least two of steps S1041 to S1043 may be executed.

[0094] In step S1041, deformation degree prediction processing is performed based on the fused feature vector to obtain the deformation degree of the image.

[0095] In some embodiments, the fused feature vector is subjected to dimensionality reduction processing to obtain a first dimensionality-reduced feature vector; the first dimensionality-reduced feature vector is subjected to denoising processing to obtain a first denoised feature vector; the first denoised feature vector is mapped to confidence scores belonging to different deformation degree labels, and the deformation degree label corresponding to the maximum confidence score is determined as the deformation degree of the image.

[0096] As an example, see Figure 4B , Figure 4B This is a schematic diagram of the architecture of the machine learning model provided in the embodiments of this application. Figure 3 The classification network shown may include a deformability classification network, which includes a deformability dimensionality reduction fully connected layer, a deformability denoising (Dropout) layer, and a deformability mapping fully connected layer. Thus, the deformability dimensionality reduction fully connected layer reduces the dimensionality of the fused feature vector to obtain a first dimensionality-reduced feature vector; the deformability denoising layer denoises the first dimensionality-reduced feature vector to obtain a first denoised feature vector; and the deformability mapping fully connected layer maps the first denoised feature vector to confidence scores belonging to different deformability labels, with the deformability label corresponding to the highest confidence score being determined as the image deformability.

[0097] For example, the types of deformation labels include: severe deformation, slight deformation, and normal. "Severe deformation" means the deformed area occupies a proportion of the entire image exceeding a first deformation threshold; "slight deformation" means the deformed area occupies a proportion of the entire image that does not exceed the first deformation threshold and is not lower than a second deformation threshold; "normal" means the deformed area occupies a proportion of the entire image that is lower than the second deformation threshold. The first deformation threshold is greater than the second deformation threshold. The first and second deformation thresholds can be parameters obtained during the training of the machine learning model, or values ​​set by the user, client, or server.

[0098] In step S1042, sharpness prediction processing is performed based on the fused feature vector to obtain the image sharpness.

[0099] In some embodiments, the fused feature vector is dimensionality reduced to obtain a second dimensionality-reduced feature vector; the second dimensionality-reduced feature vector is denoised to obtain a second denoised feature vector; the second denoised feature vector is mapped to confidence scores belonging to different sharpness labels, and the sharpness label corresponding to the highest confidence score is determined as the sharpness of the image.

[0100] As a successor Figure 4B Example, Figure 3 The classification network shown may include a sharpness classification network, which includes a sharpness dimensionality reduction fully connected layer, a sharpness denoising layer, and a sharpness mapping fully connected layer. Thus, the sharpness dimensionality reduction fully connected layer reduces the fused feature vector to obtain a second dimensionality-reduced feature vector; the sharpness denoising layer denoises the second dimensionality-reduced feature vector to obtain a second denoised feature vector; and the sharpness mapping fully connected layer maps the second denoised feature vector to confidence scores belonging to different sharpness labels, with the sharpness label corresponding to the highest confidence score being determined as the image sharpness.

[0101] For example, the types of sharpness labels include: sharp, moderate, and blurry. "Sharp" means the number of pixels per unit area in the image exceeds a first sharpness threshold; "moderate" means the number of pixels per unit area in the image does not exceed the first sharpness threshold and is not lower than a second sharpness threshold; "blurry" means the number of pixels per unit area in the image is lower than the second sharpness threshold. The first sharpness threshold is greater than the second sharpness threshold. The first and second sharpness thresholds can be parameters obtained during the training of the machine learning model, or values ​​set by the user, client, or server.

[0102] In step S1043, aesthetic prediction processing is performed based on the fused feature vector to obtain the aesthetics of the image.

[0103] In some embodiments, the fused feature vector is dimensionality reduced to obtain a third dimensionality-reduced feature vector; the third dimensionality-reduced feature vector is denoised to obtain a third denoised feature vector; the third denoised feature vector is mapped to confidence scores belonging to different aesthetic labels, and the aesthetic label corresponding to the highest confidence score is determined as the aesthetic score.

[0104] As a successor Figure 4B Example, Figure 3 The classification network shown can include an aesthetics classification network, which includes an aesthetics dimensionality reduction fully connected layer, an aesthetics denoising layer, and an aesthetics mapping fully connected layer. Thus, the aesthetics dimensionality reduction fully connected layer reduces the dimensionality of the fused feature vector to obtain a third reduced-dimensional feature vector; the aesthetics denoising layer denoises the third reduced-dimensional feature vector to obtain a third denoised feature vector; and the aesthetics mapping fully connected layer maps the third denoised feature vector to confidence scores belonging to different aesthetics labels, with the aesthetics label corresponding to the highest confidence score being determined as the aesthetics score of the image.

[0105] For example, the types of aesthetic rating labels include: low aesthetic rating, generally low aesthetic rating, generally high aesthetic rating, and high aesthetic rating. "High aesthetic rating" can mean that the composition satisfaction score exceeds the first aesthetic rating threshold; "generally high aesthetic rating" can mean that the composition satisfaction score does not exceed the first aesthetic rating threshold and is not lower than the second aesthetic rating threshold; "generally low aesthetic rating" can mean that the composition satisfaction score is lower than the second aesthetic rating threshold and not lower than the third aesthetic rating threshold; and "low aesthetic rating" can mean that the composition satisfaction score is lower than the third aesthetic rating threshold. The first aesthetic rating threshold > the second aesthetic rating threshold > the third aesthetic rating threshold. These thresholds can be parameters obtained during the training of the machine learning model, or values ​​set by the user, client, or server.

[0106] In step S105, the image quality prediction results of multiple different types are fused to obtain the image quality.

[0107] As a successor Figure 3 For example, the image quality prediction results corresponding to multiple different types are fused through a second fusion network to obtain the image quality.

[0108] In some embodiments, a target region containing a target object is identified from an image, and target features are extracted from the target region; the target features are mapped to confidence levels belonging to different deformation labels, and the deformation label corresponding to the highest confidence level is determined as the deformation degree of the target object; the image quality prediction results corresponding to multiple different types and the deformation degree of the target object are fused to obtain the image quality.

[0109] As an example, the type of the target object can be a person (e.g., a face, body, limbs, etc.), an animal, a plant, etc.

[0110] Taking a person as an example, since people are a very important part of image quality, but because people may occupy a small proportion of the image, the model may not be able to capture the information of people well. Therefore, the accuracy of analyzing the overall image deformation is relatively low. Thus, the deformation of people in the image can be determined, and the image quality can be determined together with the image quality prediction results of multiple different types determined in step S104, thereby further improving the accuracy of predicting image quality.

[0111] In some embodiments, multiple image quality prediction results are quantized to obtain a score for each image quality prediction result; the weight corresponding to each image quality prediction result is determined; and the scores of multiple image quality prediction results are weighted and summed according to the weight corresponding to each image quality prediction result to obtain the image quality score.

[0112] Taking the image quality prediction result as the image distortion degree as an example, the distortion degree label types include: severe distortion, slight distortion, and normal. Thus, the distortion degree score corresponding to the distortion degree label "severe distortion" can be determined as a (e.g., 20), the distortion degree score corresponding to the distortion degree label "slight distortion" can be determined as b (e.g., 40), and the distortion degree score corresponding to the distortion degree label "normal" can be determined as c (e.g., 60), where a, b, and c are positive numbers, and a < b < c.

[0113] Taking the image quality prediction result as the image sharpness as an example, the sharpness label types include: sharp, fair, and blurry. Thus, the sharpness score corresponding to the sharpness label "blurry" can be determined as d (e.g., 20), the sharpness score corresponding to the sharpness label "fair" can be determined as e (e.g., 40), and the sharpness score corresponding to the sharpness label "sharp" can be determined as f (e.g., 60). Here, d, e, and f are positive numbers, and d < e < f.

[0114] Taking the image quality prediction result as the aesthetics of the image as an example, the types of aesthetics labels include: low aesthetics, generally low aesthetics, generally high aesthetics, and high aesthetics. Thus, the aesthetics score corresponding to the aesthetics label "low aesthetics" can be determined as g (e.g., 20), the aesthetics score corresponding to the aesthetics label "generally low aesthetics" can be determined as h (e.g., 40), the aesthetics score corresponding to the aesthetics label "generally high aesthetics" can be determined as i (e.g., 60), and the aesthetics score corresponding to the aesthetics label "high aesthetics" can be determined as j (e.g., 80). Here, g, h, i, and j are positive numbers, and g < h < i < j.

[0115] For example, when the image's distortion label is "slight distortion", the sharpness label is "sharp", and the aesthetics label is "generally high aesthetics", and the weights of the distortion, sharpness, and aesthetics labels are 1:1:1 respectively, the distortion score is 40, the sharpness score is 60, and the aesthetics score is 60. Thus, the image's quality score is 40 + 60 + 60 = 160.

[0116] Below, by Figure 1 The following description uses the example of a terminal 400 and a server 200 collaboratively implementing the artificial intelligence-based image processing method provided in this application embodiment. See also... Figure 6B , Figure 6B This is a schematic flowchart of the image processing method provided in the embodiments of this application, which will be combined with... Figure 6B The steps shown are explained.

[0117] In step S601, the terminal responds to the image upload operation by sending the image uploaded by the user to the server.

[0118] In step S602, the server performs feature extraction processing on the image to obtain the feature map of the image.

[0119] In step S603, the server performs activation processing on the feature map at multiple levels and downsamples the activation results at each level to obtain multiple feature vectors.

[0120] In step S604, the server performs a fusion process on multiple feature vectors to obtain a fused feature vector.

[0121] In step S605, the server performs multiple different types of image quality prediction processing based on the fused feature vector, and obtains corresponding multiple different types of image quality prediction results.

[0122] In step S606, the server fuses multiple different types of image quality prediction results to obtain the image quality.

[0123] In step S607, the server determines whether the image uploaded by the terminal is qualified based on the image quality and sends a corresponding prompt message to the terminal.

[0124] In some embodiments, when the quality of an image uploaded by a user meets the qualification criteria, the uploaded image is stored in the database, and a notification message indicating successful image upload is sent to the terminal; when the quality of an image uploaded by a user does not meet the qualification criteria, a notification message prompting the user to re-upload the image is sent to the terminal. The qualification criteria may be that the image's quality score is not lower than a quality score threshold. The quality score threshold may be a parameter obtained during the training of the machine learning model, or a value set by the user, client, or server.

[0125] It should be noted that the specific implementation methods in steps S602 to S606 are similar to those in steps S101 to S105 above, and will not be repeated here.

[0126] In this embodiment, the server has strong computing power and fast processing speed. The server detects the quality of the image uploaded by the user through the terminal and determines whether the image can be stored based on the image quality. This reduces the consumption of terminal computing resources while ensuring the storage of high-quality images.

[0127] The beneficial effects achieved by the embodiments of this application will be further illustrated below by comparing them with related technologies.

[0128] Related technology (1): Determining image quality by extracting gradient features. Specifically, since pixels in an image are discrete, differences can be used instead of derivatives in the image. That is, the gradient of an image can represent the difference between adjacent pixels. Thus, the gradient features of an image can be used to predict the quality of the image. The process of predicting image quality based on the gradient features of an image can include: extracting the gradient features of the image (e.g., Canny operator, Sober operator, Laplacian operator, etc.), directly weighting and summing these gradient features, comparing the weighted sum with a preset threshold to obtain the image quality, or fusing these gradient features and using them as input to commonly used machine learning models (e.g., support vector machine, random forest, etc.), and determining the output of the model as the image quality.

[0129] Related technology (2): Combining convolutional neural networks to predict image quality. Specifically, by building a convolutional neural network model, the labeled sample image data is put into the model for training, and finally the trained model is used to give the image quality label.

[0130] During implementation, the applicant found that the relevant technologies had low adaptability to images of different sizes in actual business scenarios. Specifically, the relevant technologies (1) had low adaptability to actual business scenarios. In actual business scenarios, it is usually necessary to consider the semantic information in the image and to label some special images. However, in the process of extracting the gradient features of the image, the gradient features do not consider the semantic information in the image. Therefore, the solution is difficult to combine with the actual business scenario to give a reasonable quality label. The relevant technologies (2) usually require images of fixed size as input to the model. There are usually two methods for selecting images of fixed size: the first is to crop the image to a fixed size based on the center of the image, and the second is to directly enlarge or reduce the image. The first method will lose the boundary information of the image, affecting the model's judgment of the overall quality of the image. The second method does not take into account the special nature of the task, that is, when the image is enlarged or reduced, it will affect the visual perception of the image quality. For example, some images have low resolution. After being reduced, the image quality will become clearer, and after being enlarged, the image quality will become blurry. Furthermore, reduction may also cause distortion. The composition of the enlarged or reduced image is different from that of the original image. Therefore, the solution will have a large deviation from the actual result.

[0131] To address the aforementioned technical issues, this application's embodiments employ multiple levels of activation processing, downsampling processing, and fusion processing to extract feature vectors representing semantic information from arbitrary images of different sizes, adapting to quality recognition of arbitrary images in various business scenarios. By fusing multiple image quality prediction results corresponding to different types to determine image quality, the accuracy of image quality recognition can be improved, thereby saving resources for quality recognition.

[0132] The following uses an image as an example to illustrate the artificial intelligence-based image processing method provided in the embodiments of this application.

[0133] With the increasing number of images on the internet in recent years, the ability of backend systems to quickly and effectively input image information is crucial. Taking the cover image of a user-uploaded short video as an example, after a user uploads a cover image, the backend server of the short video client needs to review whether the cover image is clear, aesthetically pleasing, and free from distortion. If the cover image is unclear, unattractive, or distorted, the user is prompted to re-upload the cover image. If the cover image is clear, aesthetically pleasing, and undistorted, the uploaded cover image is stored. Of course, image recommendations can also be based on image quality. For example, clear, aesthetically pleasing, and undistorted images are usually considered high-quality images and can be prioritized for recommendation to users.

[0134] This application utilizes deep learning methods to efficiently evaluate image quality without human intervention, facilitating image input for the backend system and saving labor costs. Compared to related technologies, this application allows the original image to be directly used as model input without adjusting the image size, and the model can simulate visual perception to predict image quality.

[0135] This application embodiment uses a multi-task model to determine the subjective quality of images. Considering that the three quality standards of distortion, sharpness, and aesthetics are closely related—for example, generally, blurry or distorted images do not have high aesthetics, and distorted images are generally blurry—this application embodiment trains the three tasks into a single model, which improves the accuracy of each task and saves backend resources. The specific implementation of the AI-based image processing method provided in this application embodiment will be described below.

[0136] (I) Labeling training samples

[0137] In some embodiments, training samples are labeled based on the nature of the images themselves and the needs of the business. For example, (1) Clarity: Quantitative standards are set for the three categories of clear, average, and blurry, and the training samples are labeled accordingly. (2) Distortion: Quantitative standards are set for the three categories of severe distortion, slight distortion, and normal, and the training samples are labeled accordingly. (3) Aesthetics: Quantitative standards are set for the four categories of low aesthetics (blurry, unreasonable composition), slightly lower aesthetics (blurry, acceptable composition), slightly higher aesthetics (clear or average clarity, relatively reasonable composition), and high aesthetics (clear image, reasonable composition), and the training samples are labeled accordingly.

[0138] (II) Image Quality Recognition Model (hereinafter referred to as the Model) Structure

[0139] In some embodiments, see Figure 7 , Figure 7 This is a schematic diagram of the image quality recognition model provided in the embodiments of this application. Figure 7 In the input model, an image 701 of size (w, h, c) is normalized and then input into the model. The backbone network of the model mainly consists of the SE network module (Senet Block, SqueezeExcitation Net Block). Figure 7 The diagram shows SE network modules 702-1, 702-2, 702-3, and 702-4, which correspond to the squeezing layer and activation layer in the activation network mentioned above, and the downsampling layer. Figure 7 The diagram shows downsampling layers 703-1, 703-2, 703-3, and 703-4, which are equivalent to the downsampling layers in the activation network mentioned above, and an adaptive downsampling layer (Adaptive Pool). Figure 7 The diagram shows adaptive downsampling layers 704-1, 704-2, 704-3, and 704-4 (equivalent to the downsampling network described above), and a fully connected layer (FC). Figure 7 The diagram shows a fully connected layer 705, a fully connected layer 708-1, a fully connected layer 708-2, and a fully connected layer 708-3, which are equivalent to the aforementioned dimensionality reduction fully connected layer and mapping fully connected layer.

[0140] This application embodiment uses a shared-bottom method to determine image quality. For example, the shared-bottom may include... Figure 7 The diagram shows a convolutional layer 71, a pooling layer 72, SE network modules 702-1, 702-2, 702-3, and 702-4, a downsampling layer 703-1, 703-2, 703-3, and 703-4, an adaptive downsampling layer 704-1, 704-2, 704-3, and 704-4, a vector fusion module 706, a fully connected layer 705, and a Dropout layer 707.

[0141] As an example, the SE network module includes convolutional layers, where kernel sizes of 5x5, 3x3, and 1x1 can be selected. The convolution process includes squeeze excitation (SE) and shortcut operations. Downsampling layers can use convolutional layers or pooling layers with a stride of 2 to downsample the image. Adaptive downsampling layers can transform feature maps of any size with the same number of channels into feature vectors of the same dimension, allowing convolutional neural network models to use images of any scale as input. Fully connected layers can be used for image quality classification.

[0142] As an example, Figure 7 In the process, the feature map after the downsampling layer is adaptively downsampled. After the adaptive downsampling, a feature vector of dimension (k, 1) is obtained (where k is related to the number of channels). Then, the vector fusion module 706 (equivalent to the first fusion network mentioned above) fuses these feature vectors together. Finally, the fully connected layers 708-1 (equivalent to the sharpness mapping fully connected layer mentioned above), 708-2 (equivalent to the deformation mapping fully connected layer mentioned above), and 708-3 (equivalent to the aesthetics mapping fully connected layer mentioned above) are used for classification processing. The output includes sharpness labels (including three categories: sharp, normal, and blurry) and the confidence scores of the corresponding labels, deformation labels (including three categories: severe deformation, slight deformation, and normal) and the confidence scores of the corresponding labels, and aesthetics labels (including four categories: low, slightly low, slightly high, and high) and the confidence scores of the corresponding labels.

[0143] For example, after an image enters the model, it first passes through the first convolutional layer 71 and pooling layer 72. This process is mainly for dimensionality reduction to reduce the computational cost of subsequent network layers. Then, it passes through the first SE network module 702-1 and downsampling layer 703-1, resulting in a feature map with dimensions (w_1, h_1, c_1), where c_1 is the fixed number of channels set in the convolutional layer of the SE network module 702-1. The main purpose of the downsampling layer 703-1 is to reduce dimensionality and improve the robustness of the model, by downsampling the features... Figure 1On the one hand, the feature map serves as the input to the next SE network module 702-2. On the other hand, adaptive downsampling is performed on the feature map. In this process, although w_1 and h_1 are different, the number of channels in the convolutional layer and downsampling layer 703-1 in the first SE network module 702-1 is fixed at c_1. After adaptive pooling, a feature vector with a fixed dimension of (c_1,1) is obtained. Assuming that the number of channels in the convolutional layers of the second SE network module 702-2, the third SE network module 702-3, and the fourth SE network module 702-4 are set to c_2, c_3, and c_4 respectively, then after each convolutional and downsampling process, feature maps of (w_2, h_2, c_2), (w_3, h_3, c_3), and (w_4, h_4, c_4) are obtained respectively. Since the size of the input image 701 is not fixed, w_2, h_2, w_3, h_3, w_4, and h_4 are also not fixed. However, the number of channels set in the convolutional layer is fixed, so c_2, c_3, and c_4 are fixed. After these feature maps are further processed by adaptive downsampling, feature vectors with fixed dimensions of (c_2, 1), (c_3, 1), and (c_4, 1) are obtained. Next, the four feature vectors are merged according to the second dimension through the vector fusion module 706 to obtain a feature vector with dimension (c_1+c_2+c_3+c_4, 1). At this time, the dimension of the feature vector is fixed. Then, it passes through the fully connected layer 705 (equivalent to the deformability reduction fully connected layer, sharpness reduction fully connected layer, and aesthetics reduction fully connected layer mentioned above) and the Dropout layer 707 (equivalent to the deformability denoising layer, sharpness denoising layer, and aesthetics denoising layer mentioned above). Among them, the Dropout layer 707 can effectively prevent the model from overfitting. Finally, it passes through the fully connected layer 708-1 for sharpness classification, through the fully connected layer 708-2 for deformability classification, and through the fully connected layer 708-3 for aesthetics classification. The loss function (Loss) selected for each fully connected layer in the model is cross-entropy. The total loss of the model is loss_1 (loss function of fully connected layer 708-1) + loss_2 (loss function of fully connected layer 708-2) + loss_3 (loss function of fully connected layer 708-3). Backpropagation is performed using the total loss.

[0144] In this embodiment, feature maps of different sizes are obtained after processing by different SE network modules and downsampling layers. Based on the characteristics of convolutional neural networks, namely the local invariance of the feature maps learned in each layer, the model ultimately learns stable image feature maps and meaningful semantic feature maps related to image quality. These feature maps are then converted into feature vectors and combined, and finally, after passing through a fully connected layer, an image quality classification and confidence score are provided.

[0145] In some embodiments, the accuracy of analyzing image distortion is relatively low in practical applications. This is mainly because people are an important part of image quality, but because people occupy a limited proportion in the image, the model cannot capture their information well. Therefore, a human detection model can be added to the image quality recognition model mentioned above. People detected by the human detection model are cropped into bounding boxes (i.e., the target area mentioned above). A CNN model (e.g., ResNet18) is used to perform three classifications on the bounding boxes to determine whether the person is deformed and the degree of deformation (including: severe deformation, slight deformation, and normal).

[0146] In summary, the embodiments of this application not only provide image quality judgments more suitable for business scenarios by combining the needs of business scenarios and the semantic information of images, but also allow images to be processed without cropping or resizing before entering the model, ensuring the maximum preservation of information in the original image. Based on this, image quality is determined, thus guaranteeing accuracy. In practical applications, the embodiments of this application can directly evaluate the quality of images and video cover images received from the backend, saving significant manpower costs. The results can also be applied to recommendation systems, where higher-quality images are recommended to users.

[0147] The following is combined with Figure 2 The implementation of the AI-based image processing apparatus provided in this application is an exemplary structure of a software module.

[0148] In some embodiments, such as Figure 2 As shown, the software modules stored in the AI-based image processing device 243 in the memory 240 may include:

[0149] The extraction module 2431 is used to perform feature extraction processing on the image to obtain the feature map of the image; the activation module 2432 is used to perform activation processing on the feature map at multiple levels and downsample the activation results at each level to obtain multiple feature vectors; the fusion module 2433 is used to fuse the multiple feature vectors to obtain a fused feature vector; the prediction module 2434 is used to perform multiple different types of image quality prediction processing based on the fused feature vector to obtain the corresponding multiple different types of image quality prediction results; the quality module 2435 is used to fuse the multiple different types of image quality prediction results to obtain the image quality.

[0150] In the above scheme, the activation module 2432 is also used to activate the input of the nth activation network through the nth activation network in the N cascaded activation networks to obtain the nth activation result; and to transmit the nth activation result to the (n+1)th activation network for further activation processing; where N is an integer greater than or equal to 2; n is an integer that increments from 1, and the value of n satisfies 1≤n≤N-1; when n is 1, the input of the nth activation network is the feature map, and when n is 2≤n≤N-1, the input of the nth activation network is the activation result of the (n-1)th activation network.

[0151] In the above scheme, the activation module 2432 is further configured to perform channel-based average pooling on the input of the nth activation network to obtain the global content features of each channel corresponding to the input of the nth activation network; perform fully connected processing on the global content features of each channel corresponding to the input of the nth activation network to obtain the activation value of each channel corresponding to the input of the nth activation network; perform dot product processing on the activation value of each channel corresponding to the input of the nth activation network and the original content features of each channel in the input of the nth activation network; and perform downsampling processing on the result of the dot product processing to obtain the nth activation result.

[0152] In the above scheme, the activation module 2432 is also used to perform the following processing for the activation results of each level: determine the sampling kernel and step size that are suitable for the activation results; sample the activation results sequentially according to the sampling kernel and step size to obtain multiple sampling results; and combine the multiple sampling results to obtain a feature vector.

[0153] In the above scheme, the prediction module 2434 is also used to perform at least two of the following image quality prediction processes: performing deformation prediction processing based on the fused feature vector to obtain the deformation of the image; performing sharpness prediction processing based on the fused feature vector to obtain the sharpness of the image; and performing aesthetics prediction processing based on the fused feature vector to obtain the aesthetics of the image.

[0154] In the above scheme, the prediction module 2434 is also used to perform dimensionality reduction processing on the fused feature vector to obtain a first dimensionality reduction feature vector; perform denoising processing on the first dimensionality reduction feature vector to obtain a first denoised feature vector; map the first denoised feature vector to confidence scores belonging to different deformation degree labels, and determine the deformation degree label corresponding to the maximum confidence score as the deformation degree of the image.

[0155] In the above scheme, the prediction module 2434 is also used to perform dimensionality reduction processing on the fused feature vector to obtain a second dimensionality reduction feature vector; to perform denoising processing on the second dimensionality reduction feature vector to obtain a second denoised feature vector; to map the second denoised feature vector to confidence scores belonging to different sharpness labels, and to determine the sharpness label corresponding to the maximum confidence score as the sharpness of the image.

[0156] In the above scheme, the prediction module 2434 is also used to perform dimensionality reduction processing on the fused feature vector to obtain a third dimensionality reduction feature vector; to perform denoising processing on the third dimensionality reduction feature vector to obtain a third denoised feature vector; to map the third denoised feature vector to confidence scores belonging to different aesthetic labels, and to determine the aesthetic label corresponding to the maximum confidence score as the aesthetic score.

[0157] In the above scheme, the extraction module 2431 is also used to normalize the image to obtain normalized image data; to perform convolution processing on the normalized image data to obtain a convolutional feature map; and to perform pooling processing on the convolutional feature map to obtain a feature map of the image.

[0158] In the above scheme, the fusion module 2433 is also used to determine the weight corresponding to each feature vector; based on the weight corresponding to each feature vector, the multiple feature vectors are weighted and summed to obtain the fused feature vector.

[0159] In the above scheme, the extraction module 2431 is also used to identify the target region containing the target object from the image; normalize the target region to obtain normalized region data; perform convolution processing on the normalized region data to obtain the target convolution feature map; perform pooling processing on the target convolution feature map to obtain the feature map of the target region, and use the feature map of the target region as the feature map of the image.

[0160] In the above scheme, the quality module 2435 is also used to identify the target region containing the target object from the image and extract the target features from the target region; map the target features to confidence scores belonging to different deformation degree labels, and determine the deformation degree label corresponding to the maximum confidence score as the deformation degree of the target object; and perform fusion processing on the image quality prediction results corresponding to multiple different types and the deformation degree of the target object to obtain the image quality.

[0161] In some embodiments, the machine learning model used to implement the artificial intelligence-based image processing method provided in the embodiments of this application can be stored in a blockchain network, so that... Figure 1 When the server 200 or terminal 400 needs to determine the quality of an image, it can directly obtain the corresponding machine learning model through the blockchain network and predict the quality of the image using the obtained machine learning model. This allows the server or terminal to predict the quality of the image without training the machine learning model, thereby saving the consumption of training resources for the server or terminal.

[0162] In some embodiments, the logic of the AI-based image processing method provided in this application can also be implemented in a smart contract. Different nodes determine the image quality by calling their respective smart contracts, and the final image quality is determined by taking the intersection of their data. This application embodiment can further improve the accuracy of image quality recognition through collaborative processing among multiple nodes.

[0163] Taking a computer device as an example, this application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the artificial intelligence-based image processing method described above in this application.

[0164] This application provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are executed by a processor, they cause the processor to execute the artificial intelligence-based image processing method provided in this application. For example... Figure 5 , Figure 6A and Figure 6B The image processing method based on artificial intelligence is shown.

[0165] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0166] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0167] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hypertext Markup Language document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0168] As an example, computer-executable instructions can be deployed to execute on a single computer device, or on multiple computer devices located in one location, or on multiple computer devices distributed across multiple locations and interconnected via a communication network.

[0169] In summary, the embodiments of this application, through multiple levels of activation processing, downsampling processing, and fusion processing, can extract feature vectors representing semantic information from arbitrary images of different sizes, adapting to the quality recognition of arbitrary images in different business scenarios; by fusing multiple image quality prediction results corresponding to different types to determine the image quality, the accuracy of image quality recognition can be improved, thereby saving resources for quality recognition.

[0170] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An image processing method based on artificial intelligence, characterized in that, The method includes: The image is subjected to feature extraction processing to obtain the feature map of the image; The feature map is subjected to activation processing at multiple levels, and the activation results at each level are downsampled to obtain multiple feature vectors. The multiple feature vectors are fused to obtain a fused feature vector; Based on the fused feature vector, multiple different types of image quality prediction processing are performed to obtain corresponding multiple different types of image quality prediction results. Identify the target region containing the target object from the image, and extract target features from the target region; The target features are mapped to confidence levels belonging to different deformation degree labels, and the deformation degree label corresponding to the highest confidence level is determined as the deformation degree of the target object. The image quality is obtained by fusing the image quality prediction results corresponding to the multiple different types and the deformation degree of the target object.

2. The method according to claim 1, characterized in that, The activation process performed on the feature map at multiple levels includes: The input of the nth activation network is activated through the nth activation network in a series of N cascaded activation networks to obtain the nth activation result. The nth activation result is transmitted to the (n+1)th activation network to continue the activation process; Wherein, N is an integer greater than or equal to 2; n is an integer whose value starts from 1 and increments, and the value of n satisfies 1≤n≤N-1; when n is 1, the input of the nth activation network is the feature map, and when n is 2≤n≤N-1, the input of the nth activation network is the activation result of the (n-1)th activation network.

3. The method according to claim 2, characterized in that, The activation process performed on the input of the nth activation network to obtain the nth activation result includes: The input of the nth activation network is subjected to channel-based average pooling to obtain the global content features of each channel corresponding to the input of the nth activation network. The global content features of each channel corresponding to the input of the nth activation network are processed by a fully connected method to obtain the activation value of each channel corresponding to the input of the nth activation network. The activation value of each channel corresponding to the input of the nth activation network is multiplied by the original content feature of each channel in the input of the nth activation network. The result of the dot product process is downsampled to obtain the nth activation result.

4. The method according to claim 1, characterized in that, The activation results at each level are downsampled to obtain multiple feature vectors, including: Perform the following processing on the activation results for each level: Determine the sampling kernel and step size that are compatible with the activation result; The activation result is sampled sequentially according to the sampling kernel and the step size to obtain multiple sampling results; The feature vector is obtained by combining the multiple sampling results.

5. The method according to claim 1, characterized in that, The process of performing multiple different types of image quality prediction based on the fused feature vector yields corresponding image quality prediction results of multiple different types, including: Perform at least two of the following image quality prediction processes: Based on the fused feature vector, deformation degree prediction processing is performed to obtain the deformation degree of the image; The image sharpness is obtained by performing sharpness prediction processing based on the fused feature vector. The aesthetics of the image are obtained by performing aesthetic prediction processing based on the fused feature vector.

6. The method according to claim 5, characterized in that, The deformation prediction process based on the fused feature vector to obtain the deformation of the image includes: The fused feature vector is then subjected to dimensionality reduction processing to obtain a first dimensionality-reduced feature vector; The first dimensionality-reduced feature vector is denoised to obtain the first denoised feature vector. The first denoised feature vector is mapped to confidence levels belonging to different deformation labels, and the deformation label corresponding to the highest confidence level is determined as the deformation of the image.

7. The method according to claim 5, characterized in that, The sharpness prediction process based on the fused feature vector to obtain the sharpness of the image includes: The fused feature vector is then subjected to dimensionality reduction processing to obtain a second dimensionality-reduced feature vector; The second dimensionality-reduced feature vector is denoised to obtain the second denoised feature vector. The second denoising feature vector is mapped to confidence levels belonging to different sharpness labels, and the sharpness label corresponding to the highest confidence level is determined as the sharpness of the image.

8. The method according to claim 5, characterized in that, The aesthetic prediction process based on the fused feature vector to obtain the aesthetic quality of the image includes: The fused feature vector is then subjected to dimensionality reduction processing to obtain a third dimensionality-reduced feature vector; The third dimensionality-reduced feature vector is denoised to obtain the third denoised feature vector; The third denoising feature vector is mapped to confidence scores belonging to different aesthetic labels, and the aesthetic label corresponding to the highest confidence score is determined as the aesthetic score.

9. The method according to claim 1, characterized in that, The process of extracting features from the image to obtain a feature map of the image includes: The image is normalized to obtain normalized image data; The normalized image data is convolved to obtain a convolutional feature map; The convolutional feature map is pooled to obtain the feature map of the image.

10. The method according to claim 1, characterized in that, The process of fusing the multiple feature vectors to obtain a fused feature vector includes: Determine the weight corresponding to each of the feature vectors; Based on the weights corresponding to each feature vector, the multiple feature vectors are weighted and summed to obtain the fused feature vector.

11. The method according to claim 1, characterized in that, The process of extracting features from the image to obtain a feature map of the image includes: Identify the target region containing the target object from the image; The target region is normalized to obtain normalized region data; The normalized region data is subjected to convolution processing to obtain the target convolutional feature map; The target convolutional feature map is pooled to obtain the feature map of the target region, and the feature map of the target region is used as the feature map of the image.

12. An image processing device based on artificial intelligence, characterized in that, The device includes: The extraction module is used to perform feature extraction processing on the image to obtain the feature map of the image; The activation module is used to perform activation processing on the feature map at multiple levels, and to downsample the activation results at each level to obtain multiple feature vectors. The fusion module is used to fuse the multiple feature vectors to obtain a fused feature vector; The prediction module is used to perform multiple different types of image quality prediction processing based on the fused feature vector, and obtain corresponding multiple different types of image quality prediction results respectively; The quality module is used to identify target regions containing target objects from the image and extract target features from the target regions; map the target features to confidence levels belonging to different deformation labels, and determine the deformation label corresponding to the highest confidence level as the deformation degree of the target object; and perform fusion processing on the image quality prediction results corresponding to the multiple different types and the deformation degree of the target object to obtain the image quality.

13. An electronic device, characterized in that, include: Memory is used to store executable instructions for a computer; A processor, when executing computer-executable instructions stored in the memory, implements the artificial intelligence-based image processing method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The device stores computer-executable instructions, which, when executed, are used to implement the artificial intelligence-based image processing method according to any one of claims 1 to 11.

15. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the artificial intelligence-based image processing method according to any one of claims 1 to 11.

Images