Image classification method, device, electronic device and storage medium

Abstract

The present application relates to the field of artificial intelligence, and more specifically, to an image classification method, device, electronic device, and storage medium, which can be applied to cover selection of advertising videos. The image classification method provided in the embodiment of the present application includes: acquiring an image to be classified; performing feature extraction on the image to be classified to obtain an initial feature map, the initial feature map including a plurality of initial feature pixels; performing enhancement processing on the initial feature map according to the correlation between the plurality of initial feature pixels to obtain a target feature map; and obtaining a clarity category of the image to be classified based on the target feature map. The image classification method, device, electronic device, and storage medium provided in the embodiment of the present application can improve the accuracy of the clarity classification results for the image to be classified.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and specifically to an image classification method, device, electronic device and storage medium. Background Art

[0002] Image clarity is one of the important indicators to measure image quality. In some actual image services, there will be a large number of images (for example, tens of millions of images), and these images need to be classified by clarity, and according to the clarity classification results, suitable images are selected for subsequent processing. At present, the classification of image clarity is usually implemented based on edge detection technology, for example, the Sobel edge detection algorithm, the Canny edge detection algorithm, etc. The edge detection technology is based on the variance of each pixel in the image, and calculates the edge and extreme points of the image. The smaller the variance, the fewer the edges and extreme points that represent the image, that is, the less texture the image contains, and the greater the probability that the image is blurred.

[0003] However, in the method of realizing image clarity classification based on edge detection technology, since the main evaluation factor is the variance of each pixel in the image, the accuracy of the clarity classification result is low. For example, if the entire image is a solid color image, the variance is 0, the image does not contain texture, but is clear and does not have blur. Summary of the invention

[0004] The embodiments of the present application provide an image classification method, device, electronic device and storage medium, which can improve the accuracy of the clarity classification results of the images to be classified.

[0005] The present application provides an image classification method, including:

[0006] Get the image to be classified;

[0007] Perform feature extraction on the image to be classified to obtain an initial feature map, where the initial feature map includes a plurality of initial feature pixels;

[0008] According to the correlation between multiple initial feature pixels, the initial feature map is enhanced to obtain a target feature map;

[0009] Based on the target feature map, the clarity category of the image to be classified is obtained.

[0010] Accordingly, an embodiment of the present application further provides an image classification device, comprising:

[0011] An image acquisition unit, used for acquiring an image to be classified;

[0012] A feature map acquisition unit is used to extract features of the image to be classified and obtain an initial feature map, where the initial feature map includes a plurality of initial feature pixels;

[0013] An enhancement processing unit, used to perform enhancement processing on the initial feature map according to the correlation between the multiple initial feature pixels to obtain a target feature map;

[0014] The classification unit is used to obtain the clarity category of the image to be classified based on the target feature map.

[0015] In some embodiments, the enhancement processing unit may include:

[0016] A first acquisition subunit is used to respectively acquire the correlation coefficient between the feature pixel to be processed and each initial feature pixel, and the feature pixel to be processed is any initial feature pixel;

[0017] The enhancement processing subunit is used to perform enhancement processing on the feature pixels to be processed according to the correlation coefficient to obtain the target feature pixels;

[0018] The second acquisition subunit is used to obtain a target feature map, where the target feature map includes all target feature pixels.

[0019] In some embodiments, the first acquisition subunit may include:

[0020] A third acquisition subunit is used to acquire a first pixel representation value of the feature pixel to be processed according to the position information of the feature pixel to be processed in the initial feature map;

[0021] A fourth acquisition subunit is used to acquire a second pixel representation value of each initial feature pixel according to the position information of each initial pixel feature in the initial feature map;

[0022] The fifth acquisition subunit is used to obtain a correlation coefficient corresponding to each initial pixel feature based on the first pixel representation value and all the second pixel representation values.

[0023] In some embodiments, the fifth acquisition subunit may include:

[0024] A first calculation subunit, configured to obtain a first calculation result based on the first pixel representation value and a target pixel representation value, wherein the target pixel representation value is any second pixel representation value;

[0025] A second calculation subunit, configured to obtain a second calculation result based on the first pixel representation value and all second pixel representation values;

[0026] The third calculation subunit is used to obtain the ratio of the first calculation result to the second calculation result as a correlation coefficient corresponding to the target pixel representation value.

[0027] In some embodiments, the first computing subunit is specifically configured to:

[0028] Obtaining a transposed result of a first pixel representation value;

[0029] Performing activation processing on the transposed result through a first preset activation function to obtain a first activation result;

[0030] Activate the target pixel representation value through a second preset activation function to obtain a second activation result;

[0031] Calculate a first product result of the first activation result and the second activation result;

[0032] The exponential function operation result with the target value as the base and the first product result as the independent variable is obtained as the first calculation result.

[0033] In some embodiments, the second computing subunit is specifically configured to:

[0034] Obtaining a transposed result of a first pixel representation value;

[0035] Performing activation processing on the transposed result through a first preset activation function to obtain a first activation result;

[0036] Performing activation processing on each second pixel representation value through a second preset activation function to obtain a third activation result;

[0037] respectively calculating second product results of the first activation result and each third activation result;

[0038] Obtaining respectively the exponential function operation results with the target value as the base and each second product result as the independent variable as the intermediate calculation result;

[0039] All intermediate calculation results are summed to obtain a second calculation result.

[0040] In some embodiments, the enhancement processing subunit may include:

[0041] A sixth acquisition subunit, configured to acquire a second pixel representation value of each initial feature pixel according to position information of each initial pixel feature in the initial feature map;

[0042] A fourth calculation subunit, used to calculate a product result of a correlation coefficient corresponding to the second pixel representation value and the second pixel representation value;

[0043] The fifth calculation subunit is used to sum all the product results to obtain the sum result as the target feature pixel.

[0044] In some embodiments, the classification unit may include:

[0045] A seventh acquisition subunit is used to obtain prediction probability values ​​corresponding to a plurality of pending categories by using the target feature map through the classification output layer, where the plurality of pending categories include clear, locally fuzzy, and globally fuzzy;

[0046] A probability value determination subunit is used to determine a target probability value with the largest value from a plurality of predicted probability values;

[0047] The classification subunit is used to take the undetermined category corresponding to the target probability value as the clarity category.

[0048] In some embodiments, the feature map acquisition unit may include:

[0049] An image input subunit, used for inputting the image to be classified into a lightweight feature extraction network;

[0050] The feature map output subunit is used to extract features from the image to be classified through a lightweight feature extraction network and output an initial feature map.

[0051] The embodiment of the present application also provides an electronic device, including a memory storing a plurality of instructions;

[0052] The processor loads instructions from the memory to execute the steps in any one of the image classification methods provided in the embodiments of the present application.

[0053] The embodiment of the present application also provides a computer-readable storage medium, which stores a plurality of instructions, and the instructions are suitable for a processor to load to execute the steps in any one of the image classification methods provided in the embodiment of the present application.

[0054] The embodiment of the present application can obtain an image to be classified; perform feature extraction on the image to be classified to obtain an initial feature map, wherein the initial feature map includes multiple initial feature pixels; perform enhancement processing on the initial feature map according to the correlation between the multiple initial feature pixels to obtain a target feature map; and obtain a clarity category of the image to be classified based on the target feature map.

[0055] In the present application, by performing feature extraction on the image to be classified, an initial feature map including multiple initial feature pixels is obtained, and then the initial feature map is enhanced according to the correlation between the multiple initial feature pixels to obtain a target feature map. That is, by paying attention to the correlation between the multiple initial feature pixels, the relationship between the high-dimensional features of the image to be classified is introduced to enhance the expressive power of the target feature map. Then, when there is a blurred area in the image to be classified, attention can be paid to the blurred area, thereby improving the accuracy of the clarity classification result of the image to be classified. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0057] Figure 1a is a schematic diagram of a scene of an image classification method provided in an embodiment of the present application;

[0058] Figure 1b It is a flowchart of the image classification method provided in the embodiment of the present application;

[0059] Figure 1c It is an auxiliary illustration of the image classification method provided in the embodiment of the present application;

[0060] Figure 1d It is an auxiliary illustration of the image classification method provided in the embodiment of the present application;

[0061] Figure 1e It is an auxiliary illustration of the image classification method provided in the embodiment of the present application;

[0062] Figure 1f It is an auxiliary illustration of the image classification method provided in the embodiment of the present application;

[0063] Figure 1g It is an auxiliary illustration of the image classification method provided in the embodiment of the present application;

[0064] Figure 1h It is an auxiliary illustration of the image classification method provided in the embodiment of the present application;

[0065] Figure 1i It is an auxiliary illustration of the image classification method provided in the embodiment of the present application;

[0066] Figure 2 is another flowchart of the image classification method provided in the embodiment of the present application;

[0067] Figure 3 is a schematic diagram of the structure of an image classification device provided in an embodiment of the present application;

[0068] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0070] The following is a brief introduction to the technologies that may be used in the embodiments of the present application.

[0071] Artificial Intelligence (AI) is a technology that uses digital computers to simulate human perception of the environment, acquire knowledge, and use knowledge. This technology can enable machines to have functions similar to human perception, reasoning, and decision-making. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation, and other major directions.

[0072] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0073] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0074] Autonomous driving technology usually includes high-precision maps, environmental perception, behavioral decision-making, path planning, motion control and other technologies. Autonomous driving technology has broad application prospects.

[0075] Embodiments of the present application provide an image classification method, device, electronic device, and storage medium.

[0076] The image classification device can be integrated into an electronic device, which can be a terminal, a server, etc. The terminal includes, but is not limited to, a mobile phone, a computer, an intelligent voice interaction device, an intelligent home appliance, a vehicle terminal, an aircraft, etc., and can be equipped with an Android or iOS system. The server can be a single server or a server cluster composed of multiple servers.

[0077] In some embodiments, the image classification device may also be integrated into multiple electronic devices. For example, the image classification device may be integrated into multiple servers, and the image classification method of the present application may be implemented by multiple servers.

[0078] It should be noted that the image classification method provided in the embodiment of the present application can be executed independently on the terminal, can be executed independently on the server, or can be executed jointly by the terminal and the server. The embodiment of the present application does not impose any specific restrictions on this.

[0079] Please refer to Figure 1a , taking the server 11 executing the image classification method as an example.

[0080] like Figure 1a As shown, the server 11 and the terminal 12 can be directly or indirectly connected by wired or wireless communication, and this application does not impose specific restrictions on this. The server 11 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. The server 11 is used to provide background services for applications running on the terminal 12.

[0081] The terminal 12 may be a mobile phone, a computer, an intelligent voice interaction device, a smart home appliance, a vehicle terminal, an aircraft, etc., but is not limited thereto, and may be equipped with an Android or Ios system. The server 11 may install and run an application that supports data upload. The application may be an application related to advertising playback (e.g., a video playback program, a music playback program, etc.), an album application, a social application, a shopping application, etc. Schematically, the server 11 may be a terminal used by a user, and a user account is logged into the application running in the server 11.

[0082] The server 11 may undertake the main image classification work, and the terminal 12 may undertake the secondary image classification work, or the server 11 may undertake the secondary image classification work, and the terminal 12 may undertake the main image classification work, or the server 11 or the terminal 12 may each undertake the image classification work independently.

[0083] Terminal 12 may generally refer to one of a plurality of terminals, and this embodiment is only illustrated by terminal 12. Those skilled in the art may know that the number of the above terminals may be more or less. For example, the above terminal may be only one, or the above terminal may be dozens or hundreds, or a greater number, in which case the implementation environment of the above image classification method also includes other terminals. The embodiment of the present application does not limit the number of terminals and device types.

[0084] Optionally, the above-mentioned wireless network or wired network uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, the data exchanged through the network is represented by technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.

[0085] like Figure 1a As shown, the server 11 can undertake the image classification work alone, and it can be used for:

[0086] Get the image to be classified;

[0087] Perform feature extraction on the image to be classified to obtain an initial feature map, where the initial feature map includes a plurality of initial feature pixels;

[0088] According to the correlation between multiple initial feature pixels, the initial feature map is enhanced to obtain a target feature map;

[0089] Based on the target feature map, the clarity category of the image to be classified is obtained.

[0090] Taking the example that the server undertakes the main image classification work and the terminal 12 undertakes the secondary image classification work, the terminal 12 can obtain the image to be classified to the server 11, and the server 11 receives the image to be classified; extracts features from the image to be classified to obtain an initial feature map, which includes multiple initial feature pixels; enhances the initial feature map according to the correlation between the multiple initial feature pixels to obtain a target feature map; obtains the clarity category of the image to be classified based on the target feature map, and then sends the clarity category to the terminal 12, as shown in the following figure. Figure 1a shown.

[0091] In an optional implementation, the image classification method provided in the embodiment of the present application can be applied to the cover selection of advertising videos. The application scenario is introduced below.

[0092] Firstly, a plurality of advertising images are captured from an advertising video, and then an image to be classified is obtained, where the image to be classified is any advertising image. Thereafter, feature extraction is performed on the image to be classified to obtain an initial feature map, where the initial feature map includes a plurality of initial feature pixels, and according to the correlation between the plurality of initial feature pixels, the initial feature map is enhanced to obtain a target feature map. Next, based on the target feature map, the clarity category of the image to be classified is obtained.

[0093] Finally, one advertising image is selected from all advertising images whose clarity category is clear as the cover of the advertising video.

[0094] In an optional implementation, the image classification method provided in the embodiment of the present application can be applied to the quality scoring of images taken by devices such as mobile phones with cameras, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, etc. The application scenarios are introduced below.

[0095] First, an image to be classified taken by a camera of the device is obtained. Thereafter, feature extraction is performed on the image to be classified to obtain an initial feature map, which includes multiple initial feature pixels. The initial feature map is enhanced according to the correlation between the multiple initial feature pixels to obtain a target feature map. Next, based on the target feature map, a clarity category of the image to be classified is obtained.

[0096] Finally, if the clarity category of the image to be classified is clear, the image to be classified is saved;

[0097] If the clarity category of the image to be classified is partially blurred, a query message is generated and displayed; if first instruction information for instructing to save the image to be classified is received, the image to be classified is saved; if second instruction information for deleting the image to be classified is received, the image to be classified is deleted;

[0098] If the clarity category of the image to be classified is global blur, the image to be classified is deleted.

[0099] The following will be combined Figure 1b As shown, the specific process of an image classification method involving artificial intelligence provided by the embodiment of the present application is described. It should be noted that the serial numbers of the following embodiments are not used as a limitation on the preferred order of the embodiments of the present application.

[0100] 110. Obtain an image to be classified.

[0101] The image to be classified may be any image that needs to obtain a definition category.

[0102] Taking the application of the image classification method to the cover selection of an advertising video as an example, the image to be classified can be captured from the advertising video. For example, after the advertising video is acquired, one advertising image can be captured from the advertising video at intervals of a target number of advertising images as the image to be classified. The target number can be set according to actual application requirements, and the embodiments of the present application do not impose specific restrictions on this.

[0103] Taking the application of image classification methods to quality scoring of images taken by devices such as mobile phones with cameras, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, and aircraft as an example, the image taken by the device through the camera is the image to be classified. After detecting that the device has taken the image to be classified through the camera, the image to be classified can be obtained.

[0104] 120. Perform feature extraction on the image to be classified to obtain an initial feature map, where the initial feature map includes a plurality of initial feature pixels.

[0105] Among them, the initial feature map (Feature Map) can be the processing result generated after the image to be classified is processed by the artificial neural network through convolution and other processes, representing an image feature in the neural network space. The initial feature map can be represented by a one-dimensional vector, or a three-dimensional feature map formed by superimposing multiple two-dimensional feature maps of non-1*1 size. If the initial feature map is represented by a one-dimensional vector, then each vector element in the one-dimensional vector can represent an initial feature pixel. If the initial feature map is a three-dimensional feature map, then multiple pixel points corresponding to the position in the three-dimensional feature map can be collectively used as an initial feature pixel.

[0106] For example, Figure 1cIn the example, the initial feature map is a one-dimensional vector with a size of 1*48. Then, in the one-dimensional vector, each vector element can represent an initial feature pixel.

[0107] For example, Figure 1d In , the initial feature map is a three-dimensional feature map with a size of 4*4*3. Then, multiple pixels corresponding to the positions in the three-dimensional feature map can be used as an initial feature pixel. For example, for the convenience of description, Figure 1d The virtual coordinate system O-XYZ is created in Figure 1d Three pixels with the same X-axis coordinate and Y-axis coordinate are collectively regarded as an initial feature pixel.

[0108] In some embodiments, the artificial neural network can be a lightweight feature extraction network to improve image classification efficiency, for example, MobileNetV1, MobileNetV2, MobileNetV3, Xception, ShuffleNetV1, ShuffleNetV2 and other feature extraction networks.

[0109] Based on the above description, in the embodiment of the present application, step 120 may include:

[0110] Input the image to be classified into a lightweight feature extraction network;

[0111] Through the lightweight feature extraction network, features are extracted from the image to be classified and the initial feature map is output.

[0112] In actual implementation, in order to further improve the efficiency of image classification, in the embodiment of the present application, the network structure of the lightweight feature extraction network can be further simplified.

[0113] Taking the lightweight feature extraction network MobileNetV2 as an example, its network structure can be further simplified as shown in Table 1.

[0114] Table 1

[0115]

[0116] That is, the simplified network structure may include a series of convolutional layers 2d-1, a set of bottleneck layers, a convolutional layer 2d-2, an average pooling layer, and a convolutional layer 2d-3, wherein the bottleneck layer set may include a plurality of series-connected bottleneck layers, for example, six series-connected bottleneck layers. After the image to be classified is input into the above network structure, it is processed by the convolutional layer 2d-1, the set of bottleneck layers, the convolutional layer 2d-2, the average pooling layer, and the convolutional layer 2d-3 in sequence, and features are extracted, and an initial feature map is output. As shown in Table 1, the size of the initial feature map is 1*1*320, that is, the size is 1*1, the number of channels is 320, and it can be represented as a one-dimensional vector of 1*320. Of course, as mentioned above, the output initial feature map may also be a three-dimensional feature map, for example, a three-dimensional feature map of size 8*8*5, that is, the size is 8*8, and the number of channels is 5.

[0117] In actual implementation, in order to improve the network effect, each convolution layer can be connected to a first activation layer and a batch normalization (BN) layer in sequence. For example, a first activation layer and a BN layer are connected after convolution layer 2d-1, and then the first bottleneck layer is connected. For another example, a first activation layer and a BN layer are connected after convolution layer 2d-2, and then the average pooling layer is connected. For another example, a first activation layer and a BN layer are connected after convolution layer 2d-3. Among them, the first activation layer can be implemented by a ReLU activation function, which is used to adjust the processing object to a fixed range [0, +∞].

[0118] In addition, please combine Figure 1e The bottleneck layer may include an expansion layer (Expansion Layer), a depthwise convolution layer (Depthwise Convolution) and a point-by-point convolution layer (Projection Layer) in series, wherein the expansion layer is used to perform point-by-point convolution on the input features, that is, in the depth direction, the input features are weightedly combined to generate a new feature image. In the depthwise convolution layer, one convolution kernel is responsible for one channel, and the input features of one channel are only processed by one convolution kernel in the depthwise convolution layer. The point-by-point convolution layer is also used to perform point-by-point convolution on the input features, that is, in the depth direction, the input features are weightedly combined to generate a new feature image.

[0119] The convolution kernel parameters of the extension layer can be 1*1*N1*N2, the convolution kernel parameters of the depth convolution layer can be 3*3*N3, and the convolution kernel parameters of the point-by-point convolution layer can be 1*1*N4*N5, where N1 is equal to the number of input channels of the extension layer, N2 is used to represent the expected number of output channels of the extension layer, N3 is equal to the number of input channels of the depth convolution layer, N4 is equal to the number of input channels of the point-by-point convolution layer, and N5 is used to represent the expected number of output channels of the point-by-point convolution layer.

[0120] For example, in a bottleneck layer, the number of input channels of the expansion layer is 3, then N1=3, the number of output channels expected by the expansion layer is 5, then N2=5, the number of input channels of the depthwise convolution layer is 5, then N3=5, the number of input channels of the pointwise convolution layer is 5, then N4=5, the number of output channels expected by the pointwise convolution layer is 4, then N5=4. Combined with this example, the processing flow of the input features by the above bottleneck layer is as follows: Figure 1f In addition, it can be understood that in the application embodiment itself, the specific values ​​of N1, N2, N3, N4 and N5 can be set according to actual application requirements, and the application embodiment does not make specific restrictions on this.

[0121] In addition, please combine Figure 1g In the embodiment of the present application, in order to further improve the network effect, in the bottleneck layer, a BN layer and a second activation layer can be connected in sequence after the expansion layer, and then the depth convolution layer can be connected. Similarly, a BN layer and a second activation layer can be connected in sequence after the depth convolution layer, and then the point-by-point convolution layer can be connected. A BN layer can be connected after the point-by-point convolution layer. Among them, the second activation layer can be implemented by the ReLU6 activation function.

[0122] In addition, it should be noted that in the bottleneck layer set, there may be a situation where the input features and output features of some bottleneck layers have the same size. For example, the network structure shown in Table 1 can be further refined, as shown in Table 2 after refinement.

[0123] Table 2

[0124]

[0125]

[0126] Among them, the input features and output features of the bottleneck layer 5 have the same size, that is, the same size and number of channels. Then, the input features of the expansion layer in the bottleneck layer 5 and the output features of the point-by-point convolution layer can be matrix added, and the operation results can be used as the output features of the bottleneck layer 5. That is, the network structure of the bottleneck layer 5 can be as follows Figure 1h The network structure of other bottleneck layers (bottleneck layer 1, bottleneck layer 2, bottleneck layer 3, bottleneck layer 4 and bottleneck layer 6) can still be as follows Figure 1e or Figure 1f shown.

[0127] In addition, it should be noted that in Table 1 and Table 2, the average pooling layer can be replaced by a maximum pooling layer (Max Pool), and the embodiment of the present application does not impose any specific limitation on this.

[0128] 130. According to the correlation between the multiple initial feature pixels, the initial feature map is enhanced to obtain a target feature map.

[0129] For image processing, each pixel in any image cannot be isolated, and it must be associated with other pixels in the image. The mutual association of a large number of pixels produces various objects in the image. In the embodiment of the present application, the correlation between multiple initial feature pixels is used to characterize the mutual association of each pixel in the image to be classified.

[0130] In actual implementation, the correlation between multiple initial feature pixels may include the correlation between any initial feature pixel and all initial feature pixels. After obtaining the correlation between multiple initial feature pixels, the initial feature map can be enhanced according to the correlation between the multiple initial feature pixels to obtain the target feature map. Among them, the enhancement of the initial feature map can be understood as: enhancing each initial feature pixel in the initial feature map, that is, for each initial feature pixel, on the basis of its own existing pixel features, superimposing the correlation features between it and all initial feature pixels.

[0131] Based on the above description, in an embodiment of the present application, an autocorrelation feature layer will be added after the artificial neural network used to "extract features from the image to be classified and obtain an initial feature map", which is used to enhance the initial feature map according to the correlation between multiple initial feature pixels to obtain a target feature map.

[0132] Taking the artificial neural network as an example, the network structure obtained by further simplifying the network structure of MobileNetV2 (as shown in Table 1) is added to obtain the network structure shown in Table 3.

[0133] Table 3

[0134]

[0135] Further, in the embodiment of the present application, step S130 may specifically include:

[0136] Respectively obtain the correlation coefficient between the feature pixel to be processed and each initial feature pixel, the feature pixel to be processed is any initial feature pixel;

[0137] According to the correlation coefficient, the feature pixels to be processed are enhanced to obtain the target feature pixels;

[0138] A target feature map is obtained, and the target feature map includes all target feature pixels.

[0139] Among them, “respectively obtaining the correlation coefficient between the feature pixel to be processed and each initial feature pixel” can be understood as: for any initial feature pixel among multiple initial feature pixels, it should be used as the feature pixel to be processed, and then the correlation coefficient between the feature pixel to be processed and each initial feature pixel is obtained respectively. Further, in the embodiment of the present application, “respectively obtaining the correlation coefficient between the feature pixel to be processed and each initial feature pixel” can specifically include:

[0140] Obtaining a first pixel representation value of the feature pixel to be processed according to position information of the feature pixel to be processed in the initial feature map;

[0141] Obtaining a second pixel representation value of each initial feature pixel according to the position information of each initial pixel feature in the initial feature map;

[0142] Based on the first pixel characterization value and all the second pixel characterization values, a correlation coefficient corresponding to each initial pixel feature is obtained.

[0143] For example, the size of the initial feature map is 1*1*48, that is, the size is 1*1, the number of channels is 48, and it can be represented as a one-dimensional vector of 1*48, as shown in the following example: Figure 1c As shown. Then, if the position information of the feature pixel to be processed in the initial feature map is 01, it can be mapped to the first vector element in the initial feature map according to the position information as the feature pixel to be processed, and then its pixel value is used as the first pixel representation value, that is, at this time, the first pixel representation value only includes a vector element value. Similarly, the second pixel representation value of each initial feature pixel can be obtained according to the position information of each initial pixel feature in the initial feature map.

[0144] For another example, the size of the initial feature map is 4*4*3, that is, the size is 4*4 and the number of channels is 3. Figure 1d As shown. Then, if the position information of the feature pixel to be processed in the initial feature map is X=1, Y=1, then the three pixel points whose X-axis coordinates and Y-axis coordinates are both 1 can be mapped to the initial feature map according to the position information, and they are collectively used as an initial feature pixel, and then the pixel values ​​corresponding to these three pixel points are collectively used as the first pixel representation value, that is, at this time, the first pixel representation value includes three pixel values. Similarly, the second pixel representation value of each initial feature pixel can be obtained according to the position information of each initial pixel feature in the initial feature map.

[0145] Further, in the embodiment of the present application, “obtaining a correlation coefficient corresponding to each initial pixel feature based on the first pixel representation value and all second pixel representation values” may specifically include:

[0146] Obtaining a first calculation result based on the first pixel representation value and the target pixel representation value, wherein the target pixel representation value is any second pixel representation value;

[0147] Obtaining a second calculation result based on the first pixel representation value and all second pixel representation values;

[0148] A ratio of the first calculation result to the second calculation result is obtained as a correlation coefficient corresponding to the target pixel representation value.

[0149] Further, in the embodiment of the present application, “obtaining a first calculation result based on the first pixel representation value and the target pixel representation value” may specifically include:

[0150] Obtaining a transposed result of a first pixel representation value;

[0151] Performing activation processing on the transposed result through a first preset activation function to obtain a first activation result;

[0152] The target pixel representation value is activated by a second preset activation function to obtain a second activation result.

[0153] Calculate a first product result of the first activation result and the second activation result;

[0154] The exponential function operation result with the target value as the base and the first product result as the independent variable is obtained as the first calculation result.

[0155] The first preset activation function may be any one of activation functions such as ReLU, ReLU6 and Sigmoid, and similarly, the second preset activation function may be any one of activation functions such as ReLU, ReLU6 and Sigmoid. The ReLU activation function is used to adjust the processing object to a fixed range [0, +∞], the ReLU6 activation function is used to adjust the processing object to a fixed range [0, 6], and the Sigmoid activation function is used to adjust the processing object to a fixed range [0, 1].

[0156] The target value can be e, that is, 2.71828...

[0157] Further, in the embodiment of the present application, “obtaining a second calculation result based on the first pixel representation value and all second pixel representation values” may specifically include:

[0158] Obtaining a transposed result of a first pixel representation value;

[0159] Performing activation processing on the transposed result through a first preset activation function to obtain a first activation result;

[0160] Performing activation processing on each second pixel representation value through a second preset activation function to obtain a third activation result;

[0161] respectively calculating second product results of the first activation result and each third activation result;

[0162] Obtaining respectively the exponential function operation results with the target value as the base and each second product result as the independent variable as the intermediate calculation result;

[0163] All intermediate calculation results are summed to obtain a second calculation result.

[0164] As mentioned above, the first preset activation function can be any one of the activation functions such as ReLU, ReLU6 and Sigmoid. Similarly, the second preset activation function can be any one of the activation functions such as ReLU, ReLU6 and Sigmoid.

[0165] The target value can be e, that is, 2.71828...

[0166] Combined with the above description, the operation process of "obtaining the ratio of the first calculation result to the second calculation result as the correlation coefficient corresponding to the target pixel representation value" can be expressed by formula (1):

[0167]

[0168] Among them, the numerator is the first calculation result, and the denominator is the second calculation result. Specifically, α represents the first preset activation function, p represents the position information of the feature pixel to be processed in the initial feature map, fp represents the first pixel representation value of the feature pixel to be processed, β represents the second preset activation function, i represents the position information of a certain initial pixel feature in the initial feature map, and the value range of i is: the position information of the first initial pixel feature in the initial feature map to the position information of the last initial pixel feature in the initial feature map, with Figure 1c Taking the initial feature map shown in the figure as an example, the value of i ranges from 01 to 48, and fi represents the second pixel representation value of the initial feature pixel when the position information is i.

[0169] It should be noted that in the embodiment of the present application, if the initial feature map is a one-dimensional vector, for example, the size of the initial feature map is 1*1*48, that is, the size is 1*1, the number of channels is 48, and it can be represented as a one-dimensional vector of 1*48, as shown in the following example: Figure 1c As shown, the first pixel representation value only includes one vector element value, and its transposed result is itself.

[0170] If the initial feature map is a three-dimensional feature map, for example, the size of the initial feature map is 4*4*3, that is, the size is 4*4 and the number of channels is 3, as shown in the following example: Figure 1d As shown. Continuing with the example that the position information of the feature pixel to be processed in the initial feature map is X=1, Y=1, the pixel values ​​corresponding to the three pixel points whose X-axis coordinates and Y-axis coordinates are both 1 can be represented as row vectors (A1, A2, A3), and collectively used as the first pixel representation value, then the transposed result is the column vector corresponding to the row vector (A1, A2, A3).

[0171] Furthermore, in the embodiment of the present application, “performing enhancement processing on the feature pixels to be processed according to the correlation coefficient to obtain the target feature pixels” may specifically include:

[0172] Obtaining a second pixel representation value of each initial feature pixel according to the position information of each initial pixel feature in the initial feature map;

[0173] Calculate the product of the correlation coefficient corresponding to the second pixel representation value and the second pixel representation value;

[0174] All product results are summed up to obtain the summed result as the target feature pixel.

[0175] In an embodiment of the present application, when executing the step of "respectively obtaining the correlation coefficient between the feature pixel to be processed and each initial feature pixel", the second pixel representation value of each initial feature pixel has been obtained based on the position information of each initial pixel feature in the initial feature map. Therefore, when executing "enhancing the feature pixel to be processed according to the correlation coefficient to obtain the target feature pixel", the second pixel representation value that has been obtained can be reused to improve image classification efficiency.

[0176] Furthermore, the process of acquiring target feature pixels can be expressed by formula (2):

[0177]

[0178] Where i represents the position information of an initial pixel feature in the initial feature map, and the value range of i is: the position information of the first initial pixel feature in the initial feature map to the position information of the last initial pixel feature in the initial feature map, Figure 1c Taking the initial feature map shown as an example, the value of i is 01~48, fi represents the second pixel representation value of the initial feature pixel when the position information is i, fi represents the second pixel representation value of the initial feature pixel when the position information is i, Wpi represents the correlation coefficient corresponding to the second pixel representation value fi, for example, Wp01 represents the correlation coefficient corresponding to the second pixel representation value f01.

[0179] 140. Based on the target feature map, obtain the clarity category of the image to be classified.

[0180] In the embodiment of the present application, the clarity category of the image to be classified may be any one of clear, locally blurred and globally blurred.

[0181] Among them, clarity means that there is no blur in the image to be classified, local blur is used to characterize that there is blur in the image to be classified, and the blurred area does not completely cover the image to be classified, that is, only part of the image to be classified has blur, and global blur is used to characterize that there is blur in the image to be classified, and the blurred area completely covers the image to be classified, that is, the entire image to be classified is in a blurred state.

[0182] In actual implementation, the target feature map can be input into the classification output layer, and then based on the output result of the classification output layer, the definition category of the image to be classified can be obtained. Based on this, in the embodiment of the present application, "obtaining the definition category of the image to be classified based on the target feature map" can specifically include:

[0183] Through the classification output layer, using the target feature map, the predicted probability values ​​corresponding to multiple pending categories are obtained, and the multiple pending categories include clear, locally fuzzy and globally fuzzy;

[0184] From multiple predicted probability values, determine the target probability value with the largest value;

[0185] The undetermined category corresponding to the target probability value is used as the clarity category.

[0186] The classification output layer can be a fully connected network, and the last layer of the fully connected network is a fully connected layer with a size of 1*1*3. During the training process of the image classification model (including the lightweight feature extraction network, the autocorrelation feature layer, and the classification output layer), the fully connected network can be initialized using a Gaussian distribution with a variance of 0.01 and a mean of 0.

[0187] In addition, in the embodiment of the present application, the sum of the predicted probability values ​​corresponding to the multiple pending categories is 1. For example, the predicted probability value corresponding to clarity is 0.8, the predicted probability value corresponding to local blur is 0.2, and the predicted probability value corresponding to global blur is 0, then the target probability value is 0.8, and the pending category corresponding to the target probability value of 0.8 can be used as the clarity category, that is, clarity is used as the clarity category of the image to be classified.

[0188] Since all target feature pixels included in the target feature map have been enhanced, the difference between the blurred area and the clear area can be better expressed, especially the boundary between the blurred area and the clear area. This allows the image classification model to focus on the correlation between multiple initial feature pixels and introduce the relationship between the high-dimensional features of the image to be classified, so as to further focus on the blurred area, enhance the expressive power of the target feature map, and thus improve attention to the blurred area.

[0189] Finally, it should be noted that in the embodiments of the present application, the image classification model can be pre-trained. Figure 1i Taking the application of the image classification method provided in the embodiment of the present application to the cover selection of an advertising video as an example, the clarity category of the image to be classified can be output through the image classification model, which can be any one of clear, locally blurred and globally blurred.

[0190] Continuing to take the application of the image classification method to the cover selection of advertising videos as an example, during the training process of the image classification model, the training data set used can come from the advertising scene. For example, 30 advertising images are uniformly extracted from each advertising video, and then the clarity of each advertising image is marked to see whether there is a blurred area. The clarity annotation content can be: clear, globally blurred, and locally blurred. Among them, there can be 2000 clear advertising images, 1890 locally blurred advertising images, and 600 globally blurred advertising images. The embodiment of the present application does not impose specific restrictions on this.

[0191] Furthermore, during the training of the image classification model, the cross entropy loss function can be used to optimize the network parameters.

[0192] The mathematical expression of the cross entropy loss function is:

[0193] L=-[ylogy′+(1-y)log(1-y′)]

[0194] Among them, y is the clarity label, y' is the probability value corresponding to a certain undetermined category predicted during the training process of the image classification model, and the cross entropy loss function can maximize the probability value of the image classification model predicting that an image to be classified belongs to the correct undetermined category, thereby improving the training efficiency.

[0195] In addition, in an embodiment of the present application, the template parameter w and bias parameter b of each convolution kernel in the image classification model (for example, the aforementioned convolution layer 2d-1, convolution layer 2d-2, convolution layer 2d-3, and the convolution kernel included in the expansion layer, depth convolution layer and point-by-point convolution layer in the bottleneck layer) can be solved based on stochastic gradient descent (SGD). In each iteration, the prediction result error is calculated and back-propagated to the image classification model, the gradient is calculated, and the template parameter w and bias parameter b of each convolution kernel in the image classification model are updated.

[0196] In an embodiment of the present application, features are extracted from the image to be classified to obtain an initial feature map including multiple initial feature pixels, and then the initial feature map is enhanced based on the correlation between the multiple initial feature pixels to obtain a target feature map. That is, by paying attention to the correlation between the multiple initial feature pixels, the relationship between the high-dimensional features of the image to be classified is introduced to enhance the expressive power of the target feature map. Then, when there is a blurred area in the image to be classified, attention can be paid to the blurred area, thereby improving the accuracy of the clarity classification result of the image to be classified.

[0197] The image classification method described in the above embodiment will be further described in detail below.

[0198] In the embodiment of the present application, the image classification device is integrated into Figure 1a Taking the server 11 shown as an example, the image classification method provided in the embodiment of the present application is described in detail.

[0199] like Figure 2 As shown, the specific process of an image classification method is as follows:

[0200] 210. Obtain an image to be classified.

[0201] The image to be classified may be any image that needs to obtain a definition category.

[0202] 220. Through a lightweight feature extraction network, specifically MobileNetV2, feature extraction is performed on the image to be classified to obtain an initial feature map, which includes multiple initial feature pixels.

[0203] You can obtain MobileNetV2 pre-trained on the ImageNet dataset as a lightweight feature extraction network, perform feature extraction on the classified image, and obtain the initial feature map.

[0204] 230. According to the position information of the feature pixel to be processed in the initial feature map, obtain the first pixel representation value of the feature pixel to be processed, according to the position information of each initial pixel feature in the initial feature map, obtain the second pixel representation value of each initial feature pixel, and based on the first pixel representation value and the target pixel representation value, obtain a first calculation result, the target pixel representation value is any second pixel representation value, and based on the first pixel representation value and all second pixel representation values, obtain a second calculation result, and then obtain the ratio of the first calculation result to the second calculation result as a correlation coefficient corresponding to the target pixel representation value.

[0205] Further, in the embodiment of the present application, “obtaining a first calculation result based on the first pixel representation value and the target pixel representation value” may specifically include:

[0206] Obtaining a transposed result of a first pixel representation value;

[0207] Performing activation processing on the transposed result through a first preset activation function to obtain a first activation result;

[0208] The target pixel representation value is activated by a second preset activation function to obtain a second activation result.

[0209] Calculate a first product result of the first activation result and the second activation result;

[0210] The exponential function operation result with the target value as the base and the first product result as the independent variable is obtained as the first calculation result.

[0211] The first preset activation function may be any one of activation functions such as ReLU, ReLU6 and Sigmoid, and similarly, the second preset activation function may be any one of activation functions such as ReLU, ReLU6 and Sigmoid. The ReLU activation function is used to adjust the processing object to a fixed range [0, +∞], the ReLU6 activation function is used to adjust the processing object to a fixed range [0, 6], and the Sigmoid activation function is used to adjust the processing object to a fixed range [0, 1].

[0212] The target value can be e, that is, 2.71828...

[0213] Further, in the embodiment of the present application, “obtaining a second calculation result based on the first pixel representation value and all second pixel representation values” may specifically include:

[0214] Obtaining a transposed result of a first pixel representation value;

[0215] Performing activation processing on the transposed result through a first preset activation function to obtain a first activation result;

[0216] Performing activation processing on each third pixel representation value through a second preset activation function to obtain a third activation result;

[0217] respectively calculating second product results of the first activation result and each second activation result;

[0218] Obtaining respectively the exponential function operation results with the target value as the base and each second product result as the independent variable as the intermediate calculation result;

[0219] All intermediate calculation results are summed to obtain a second calculation result.

[0220] As mentioned above, the first preset activation function can be any one of the activation functions such as ReLU, ReLU6 and Sigmoid. Similarly, the second preset activation function can be any one of the activation functions such as ReLU, ReLU6 and Sigmoid.

[0221] The target value can be e, that is, 2.71828...

[0222] 240. According to the position information of each initial pixel feature in the initial feature map, the second pixel representation value of each initial feature pixel is obtained, and the product result of the correlation coefficient corresponding to the second pixel representation value and the second pixel representation value is calculated, and then all the product results are summed to obtain the summed result as the target feature pixel.

[0223] 250. Through the classification output layer, using the target feature map, the predicted probability values ​​corresponding to multiple pending categories are obtained, and the multiple pending categories include clear, locally blurred and globally blurred. From the multiple predicted probability values, the target probability value with the largest value is determined, and the pending category corresponding to the target probability value is used as the clarity category.

[0224] The classification output layer can be a fully connected network, and the last layer of the fully connected network is a fully connected layer with a size of 1*1*3. During the training process of the image classification model (including the lightweight feature extraction network, the autocorrelation feature layer, and the classification output layer), the fully connected network can be initialized using a Gaussian distribution with a variance of 0.01 and a mean of 0.

[0225] In addition, in the embodiment of the present application, the sum of the predicted probability values ​​corresponding to the multiple pending categories is 1. For example, the predicted probability value corresponding to clarity is 0.8, the predicted probability value corresponding to local blur is 0.2, and the predicted probability value corresponding to global blur is 0, then the target probability value is 0.8, and the pending category corresponding to the target probability value of 0.8 can be used as the clarity category, that is, clarity is used as the clarity category of the image to be classified.

[0226] Since all target feature pixels included in the target feature map have been enhanced, the difference between the blurred area and the clear area can be better expressed, especially the boundary between the blurred area and the clear area. This allows the image classification model to focus on the correlation between multiple initial feature pixels and introduce the relationship between the high-dimensional features of the image to be classified, so as to further focus on the blurred area, enhance the expressive power of the target feature map, and thus improve attention to the blurred area.

[0227] In an embodiment of the present application, features are extracted from the image to be classified to obtain an initial feature map including multiple initial feature pixels, and then the initial feature map is enhanced based on the correlation between the multiple initial feature pixels to obtain a target feature map. That is, by paying attention to the correlation between the multiple initial feature pixels, the relationship between the high-dimensional features of the image to be classified is introduced to enhance the expressive power of the target feature map. Then, when there is a blurred area in the image to be classified, attention can be paid to the blurred area, thereby improving the accuracy of the clarity classification result of the image to be classified.

[0228] In order to better implement the above image classification method, the embodiment of the present application also provides an image classification device, which can be integrated in an electronic device, and the electronic device can be a terminal, a server, etc. Among them, the terminal can be a mobile phone, a computer, an intelligent voice interaction device, a smart home appliance, a vehicle terminal, an aircraft, etc., and it can be equipped with an Android or Ios system, and the server can be a single server or a server cluster composed of multiple servers.

[0229] For example, in the embodiment of the present application, the method of the embodiment of the present application is described in detail by taking the image classification device being specifically integrated in the server as an example.

[0230] For example, Figure 3 As shown, the image classification device 300 may include:

[0231] An image acquisition unit 310 is used to acquire an image to be classified;

[0232] A feature map acquisition unit 320 is used to extract features from the image to be classified to obtain an initial feature map, where the initial feature map includes a plurality of initial feature pixels;

[0233] An enhancement processing unit 330 is used to perform enhancement processing on the initial feature map according to the correlation between the multiple initial feature pixels to obtain a target feature map;

[0234] The classification unit is used to obtain the clarity category of the image to be classified based on the target feature map.

[0235] In some embodiments, the enhancement processing unit 330 may include:

[0236] A first acquisition subunit is used to respectively acquire the correlation coefficient between the feature pixel to be processed and each initial feature pixel, and the feature pixel to be processed is any initial feature pixel;

[0237] The enhancement processing subunit is used to perform enhancement processing on the feature pixels to be processed according to the correlation coefficient to obtain the target feature pixels;

[0238] The second acquisition subunit is used to obtain a target feature map, where the target feature map includes all target feature pixels.

[0239] In some embodiments, the first acquisition subunit may include:

[0240] A third acquisition subunit is used to acquire a first pixel representation value of the feature pixel to be processed according to the position information of the feature pixel to be processed in the initial feature map;

[0241] A fourth acquisition subunit is used to acquire a second pixel representation value of each initial feature pixel according to the position information of each initial pixel feature in the initial feature map;

[0242] The fifth acquisition subunit is used to obtain a correlation coefficient corresponding to each initial pixel feature based on the first pixel representation value and all the second pixel representation values.

[0243] In some embodiments, the fifth acquisition subunit may include:

[0244] A first calculation subunit, configured to obtain a first calculation result based on the first pixel representation value and a target pixel representation value, wherein the target pixel representation value is any second pixel representation value;

[0245] A second calculation subunit, configured to obtain a second calculation result based on the first pixel representation value and all second pixel representation values;

[0246] The third calculation subunit is used to obtain the ratio of the first calculation result to the second calculation result as a correlation coefficient corresponding to the target pixel representation value.

[0247] In some embodiments, the first computing subunit is specifically configured to:

[0248] Obtaining a transposed result of a first pixel representation value;

[0249] Performing activation processing on the transposed result through a first preset activation function to obtain a first activation result;

[0250] Activate the target pixel representation value through a second preset activation function to obtain a second activation result;

[0251] Calculate a first product result of the first activation result and the second activation result;

[0252] The exponential function operation result with the target value as the base and the first product result as the independent variable is obtained as the first calculation result.

[0253] In some embodiments, the second computing subunit is specifically configured to:

[0254] Obtaining a transposed result of a first pixel representation value;

[0255] Performing activation processing on the transposed result through a first preset activation function to obtain a first activation result;

[0256] Performing activation processing on each second pixel representation value through a second preset activation function to obtain a third activation result;

[0257] respectively calculating second product results of the first activation result and each second activation result;

[0258] Obtaining respectively the exponential function operation results with the target value as the base and each second product result as the independent variable as the intermediate calculation result;

[0259] All intermediate calculation results are summed to obtain a second calculation result.

[0260] In some embodiments, the enhancement processing subunit may include:

[0261] A sixth acquisition subunit, configured to acquire a second pixel representation value of each initial feature pixel according to position information of each initial pixel feature in the initial feature map;

[0262] A fourth calculation subunit, used to calculate a product result of a correlation coefficient corresponding to the second pixel representation value and the second pixel representation value;

[0263] The fifth calculation subunit is used to sum all the product results to obtain the sum result as the target feature pixel.

[0264] In some embodiments, the classification unit may include:

[0265] A seventh acquisition subunit is used to obtain prediction probability values ​​corresponding to a plurality of pending categories by using the target feature map through the classification output layer, where the plurality of pending categories include clear, locally fuzzy, and globally fuzzy;

[0266] A probability value determination subunit is used to determine a target probability value with the largest value from a plurality of predicted probability values;

[0267] The classification subunit is used to take the undetermined category corresponding to the target probability value as the clarity category.

[0268] In some embodiments, the feature map acquisition unit 320 may include:

[0269] An image input subunit, used for inputting the image to be classified into a lightweight feature extraction network;

[0270] The feature map output subunit is used to extract features from the image to be classified through a lightweight feature extraction network and output an initial feature map.

[0271] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

[0272] As can be seen from the above, the image classification device 300 of the embodiment of the present application can acquire the image to be classified by the image acquisition unit 310; the feature map acquisition unit 320 performs feature extraction on the image to be classified to obtain an initial feature map, and the initial feature map includes multiple initial feature pixels; the enhancement processing unit 330 performs enhancement processing on the initial feature map according to the correlation between the multiple initial feature pixels to obtain a target feature map; the classification unit obtains the clarity category of the image to be classified based on the target feature map.

[0273] In an embodiment of the present application, features are extracted from the image to be classified to obtain an initial feature map including multiple initial feature pixels, and then the initial feature map is enhanced based on the correlation between the multiple initial feature pixels to obtain a target feature map. That is, by paying attention to the correlation between the multiple initial feature pixels, the relationship between the high-dimensional features of the image to be classified is introduced to enhance the expressive power of the target feature map. Then, when there is a blurred area in the image to be classified, attention can be paid to the blurred area, thereby improving the accuracy of the clarity classification result of the image to be classified.

[0274] The embodiment of the present application also provides an electronic device, which can be a terminal, a server, etc. Among them, the terminal can be a mobile phone, a computer, an intelligent voice interaction device, a smart home appliance, a vehicle terminal, an aircraft, etc., and it can be equipped with an Android or iOS system, and the server can be a single server or a server cluster composed of multiple servers, etc.

[0275] In some embodiments, the image classification device may also be integrated into multiple electronic devices. For example, the image classification device may be integrated into multiple servers, and the image classification method of the present application may be implemented by multiple servers.

[0276] In the embodiments of the present application, the electronic device of the embodiments of the present application will be taken as an example for detailed description. Figure 4As shown, it shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:

[0277] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, an input module 404, and a communication module 405. Those skilled in the art will appreciate that Figure 4 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.

[0278] The processor 401 is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402. In some embodiments, the processor 401 may include one or more processing cores; in some embodiments, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0279] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (e.g., a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0280] The electronic device also includes a power supply 403 for supplying power to various components. In some embodiments, the power supply 403 can be logically connected to the processor 401 through a power management system, so that the power management system can manage charging, discharging, power consumption, etc. The power supply 403 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0281] The electronic device may further include an input module 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0282] The electronic device may further include a communication module 405. In some embodiments, the communication module 405 may include a wireless module. The electronic device may perform short-range wireless transmission through the wireless module of the communication module 405, thereby providing the user with wireless broadband Internet access. For example, the communication module 405 may be used to help the user send and receive emails, browse web pages, and access streaming media.

[0283] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically, in the embodiment of the present application, the processor 401 in the electronic device will load the executable file corresponding to the process of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application program stored in the memory 402, thereby realizing various functions, as follows:

[0284] Get the image to be classified;

[0285] Perform feature extraction on the image to be classified to obtain an initial feature map, where the initial feature map includes a plurality of initial feature pixels;

[0286] According to the correlation between multiple initial feature pixels, the initial feature map is enhanced to obtain a target feature map;

[0287] Based on the target feature map, the clarity category of the image to be classified is obtained.

[0288] In some embodiments, a computer program product is also proposed, including a computer program or instructions, which implement the steps in any of the above-mentioned image classification methods when executed by a processor.

[0289] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0290] In an embodiment of the present application, features are extracted from the image to be classified to obtain an initial feature map including multiple initial feature pixels, and then the initial feature map is enhanced based on the correlation between the multiple initial feature pixels to obtain a target feature map. That is, by paying attention to the correlation between the multiple initial feature pixels, the relationship between the high-dimensional features of the image to be classified is introduced to enhance the expressive power of the target feature map. Then, when there is a blurred area in the image to be classified, attention can be paid to the blurred area, thereby improving the accuracy of the clarity classification result of the image to be classified.

[0291] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0292] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any one of the image classification methods provided in the embodiments of the present application. For example, the instructions can execute the following steps:

[0293] Get the image to be classified;

[0294] Perform feature extraction on the image to be classified to obtain an initial feature map, where the initial feature map includes a plurality of initial feature pixels;

[0295] According to the correlation between multiple initial feature pixels, the initial feature map is enhanced to obtain a target feature map;

[0296] Based on the target feature map, the clarity category of the image to be classified is obtained.

[0297] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0298] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs any one of the image classification methods provided in the above embodiments.

[0299] Since the instructions stored in the storage medium can execute the steps in any image classification method provided in the embodiments of the present application, the beneficial effects that can be achieved by any image classification method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0300] The above is a detailed introduction to an image classification method, device, electronic device and computer-readable storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An image classification method, characterized in that: include: Get the image to be classified; Extracting features from the image to be classified to obtain an initial feature map, wherein the initial feature map includes a plurality of initial feature pixels; According to the correlation between the multiple initial feature pixels, the initial feature map is enhanced to obtain a target feature map; Based on the target feature map, obtaining the clarity category of the image to be classified; The method of performing enhancement processing on the initial feature map according to the correlation between the multiple initial feature pixels to obtain the target feature map comprises: respectively obtaining the correlation coefficient between the feature pixel to be processed and each of the initial feature pixels, wherein the feature pixel to be processed is any of the initial feature pixels; performing enhancement processing on the feature pixel to be processed according to the correlation coefficient to obtain the target feature pixel; and obtaining the target feature map, wherein the target feature map includes all the target feature pixels; Wherein, respectively obtaining the correlation coefficient between the feature pixel to be processed and each of the initial feature pixels comprises: obtaining a first pixel representation value of the feature pixel to be processed according to the position information of the feature pixel to be processed in the initial feature map; obtaining a second pixel representation value of each of the initial feature pixels according to the position information of each of the initial feature pixels in the initial feature map; and obtaining a correlation coefficient corresponding to each of the initial feature pixels based on the first pixel representation value and all of the second pixel representation values; Among them, the correlation coefficient corresponding to each of the initial feature pixels is obtained based on the first pixel representation value and all the second pixel representation values, including: obtaining a first calculation result based on the first pixel representation value and the target pixel representation value, the target pixel representation value is any of the second pixel representation values; obtaining a second calculation result based on the first pixel representation value and all the second pixel representation values; obtaining the ratio of the first calculation result to the second calculation result as the correlation coefficient corresponding to the target pixel representation value.

2. The image classification method according to claim 1, characterized in that: The obtaining a first calculation result based on the first pixel representation value and the target pixel representation value includes: Obtaining a transposed result of the first pixel representation value; Performing activation processing on the transposed result through a first preset activation function to obtain a first activation result; Performing activation processing on the target pixel representation value through a second preset activation function to obtain a second activation result; Calculating a first product result of the first activation result and the second activation result; An exponential function operation result with the target value as the base and the first product result as the independent variable is obtained as the first calculation result.

3. The image classification method according to claim 1, characterized in that: The obtaining a second calculation result based on the first pixel representation value and all the second pixel representation values ​​includes: Obtaining a transposed result of the first pixel representation value; Performing activation processing on the transposed result through a first preset activation function to obtain a first activation result; Performing activation processing on each of the second pixel representation values ​​through a second preset activation function to obtain a third activation result; respectively calculating a second product result of the first activation result and each of the third activation results; respectively obtaining exponential function operation results with the target value as the base and each of the second multiplication results as the independent variable as the intermediate calculation results; All the intermediate calculation results are summed to obtain a second calculation result.

4. The image classification method according to claim 1, characterized in that: The step of performing enhancement processing on the feature pixel to be processed according to the correlation coefficient to obtain the target feature pixel comprises: Acquire a second pixel representation value of each of the initial feature pixels according to the position information of each of the initial feature pixels in the initial feature map; Calculating a product result of a correlation coefficient corresponding to the second pixel representation value and the second pixel representation value; All the product results are summed up to obtain a summed result as the target feature pixel.

5. The image classification method according to claim 1, characterized in that: The obtaining, based on the target feature map, the clarity category of the image to be classified comprises: Obtain prediction probability values ​​corresponding to a plurality of pending categories by using the target feature map through the classification output layer, wherein the plurality of pending categories include clear, locally fuzzy, and globally fuzzy; Determining a target probability value with the largest value from among the plurality of predicted probability values; The undetermined category corresponding to the target probability value is used as the clarity category.

6. The image classification method according to claim 1, characterized in that: The step of extracting features from the image to be classified to obtain an initial feature map includes: Inputting the image to be classified into a lightweight feature extraction network; The lightweight feature extraction network is used to extract features from the image to be classified, and the initial feature map is output.

7. An image classification device, characterized in that: include: An image acquisition unit, used for acquiring an image to be classified; A feature map acquisition unit, used to extract features from the image to be classified to obtain an initial feature map, wherein the initial feature map includes a plurality of initial feature pixels; An enhancement processing unit, configured to perform enhancement processing on the initial feature map according to the correlation between the plurality of initial feature pixels to obtain a target feature map; A classification unit, used for obtaining a definition category of the image to be classified based on the target feature map; Wherein, the enhancement processing unit comprises: A first acquisition subunit is used to respectively acquire the correlation coefficient between the feature pixel to be processed and each initial feature pixel, and the feature pixel to be processed is any initial feature pixel; The enhancement processing subunit is used to perform enhancement processing on the feature pixels to be processed according to the correlation coefficient to obtain the target feature pixels; A second acquisition subunit is used to obtain a target feature map, where the target feature map includes all target feature pixels; Wherein, the first acquisition subunit includes: A third acquisition subunit is used to acquire a first pixel representation value of the feature pixel to be processed according to the position information of the feature pixel to be processed in the initial feature map; A fourth acquisition subunit, configured to acquire a second pixel representation value of each initial feature pixel according to position information of each initial feature pixel in the initial feature map; a fifth acquisition subunit, configured to obtain a correlation coefficient corresponding to each initial feature pixel based on the first pixel representation value and all second pixel representation values; Wherein, the fifth acquisition subunit includes: A first calculation subunit is used to obtain a first calculation result based on the first pixel representation value and the target pixel representation value, where the target pixel representation value is any second pixel representation value; A second calculation subunit, configured to obtain a second calculation result based on the first pixel representation value and all second pixel representation values; The third calculation subunit is used to obtain the ratio of the first calculation result to the second calculation result as a correlation coefficient corresponding to the target pixel representation value.

8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a plurality of instructions; The processor loads instructions from the memory to execute the steps in the image classification method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the image classification method described in any one of claims 1 to 6.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps in the image classification method described in any one of claims 1 to 6 are implemented.

Images