Method, device, terminal device and storage medium for determining image attributes

Abstract

The present application provides a method for determining image attributes, a terminal device and a storage medium, the method comprising obtaining an image to be identified; inputting the image to be identified into a pre-trained image attribute classification model, and outputting an image angle probability value and a text type probability value of the image to be identified respectively; the image attribute classification model is obtained by training an improved Efficient network using image samples marked with image angle information and text type information, the improved Efficient network comprising a plurality of Attention‑MBConv modules, the Attention‑MBConv module being formed by adding an attention module after the last feature output layer of the MBConv module; determining the image angle and the text type respectively according to the image angle probability value and the text type probability value; the method can simultaneously identify the image angle and the text type, reducing the amount of operation and calculation.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method for determining image attributes, a terminal device, and a computer-readable storage medium. Background Art

[0002] With the development of science and technology, the demand for image processing and recognition is increasing, such as ID card recognition, bill text recognition, automatic test paper grading, photo search, etc. However, in some practical application scenarios, such as photo search, in order to improve the accuracy of the search, it is often necessary to judge multiple attributes of the image before image recognition (such as text recognition), such as the direction of the image text (i.e., whether it is tilted), the type of the image text (i.e., whether the text is handwritten or printed), etc. The existing image attribute determination schemes basically classify and judge the single attribute of the image separately, that is, each model scheme can only recognize a single attribute of the image, so for images with multiple attributes, it is necessary to design multiple independent model schemes, and the model scheme design is complex and time-consuming.

[0003] Application Contents

[0004] In view of this, an embodiment of the present application provides a method for determining image attributes, a terminal device and a computer-readable storage medium to overcome the problem that each model scheme in the prior art can only identify a single attribute of an image, and for images with multiple attributes, multiple independent model schemes need to be designed, and the model scheme design is complex and time-consuming.

[0005] In a first aspect, an embodiment of the present application provides a method for determining image attributes, the method comprising: obtaining an image to be identified;

[0006] Input the image to be identified into a pre-trained image attribute classification model, and output the image angle probability value and the text type probability value of the image to be identified respectively; wherein the image attribute classification model is obtained by training an improved Efficient network using image samples marked with image angle information and text type information, and the improved Efficient network includes a plurality of Attention-MBConv modules, each of which is connected in series, and the Attention-MBConv module is formed by adding an attention module after the last feature output layer of the MBConv module;

[0007] The image angle and the text type are determined according to the image angle probability value and the text type probability value respectively.

[0008] In a second aspect, an embodiment of the present application provides a device for determining image attributes, the device comprising:

[0009] An image acquisition module to be identified, used to acquire an image to be identified;

[0010] A probability value output module, used for inputting the image to be identified into a pre-trained image attribute classification model, and outputting the image angle probability value and the text type probability value of the image to be identified respectively; wherein the image attribute classification model is obtained by training an improved Efficient network using image samples marked with image angle information and text type information, the improved Efficient network includes a plurality of Attention-MBConv modules, each of the Attention-MBConv modules is connected in series, and the Attention-MBConv module is formed by adding an attention module after the last feature output layer of the MBConv module;

[0011] The angle type determination module is used to determine the image angle and the text type according to the image angle probability value and the text type probability value respectively.

[0012] In a third aspect, an embodiment of the present application provides a terminal device, comprising: a memory; one or more processors coupled to the memory; one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the method for determining image attributes provided in the first aspect above.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the method for determining image attributes provided in the first aspect.

[0014] The embodiment of the present application provides a method for determining image attributes, a terminal device, and a computer-readable storage medium, which obtain an image to be identified; input the image to be identified into a pre-trained image attribute classification model, and output an image angle probability value and a text type probability value of the image to be identified respectively; wherein the image attribute classification model is obtained by training an improved Efficient network using image samples marked with image angle information and text type information, and the improved Efficient network includes multiple Attention-MBConv modules, each Attention-MBConv module is connected in series, and the Attention-MBConv module is formed by adding an attention module after the last feature output layer of the MBConv module; the image angle and the text type are determined according to the image angle probability value and the text type probability value; since the method uses image samples marked with image angle information and text type information (i.e., known image angle and text type) to train the improved Efficient network to obtain a pre-trained image attribute classification model, the image angle and text type of the image to be identified can be identified respectively based on the image attribute classification model, and it can be seen that the method only needs to design and train one model to simultaneously identify the image angle and text type, thereby reducing the operation and calculation amount. In addition, the improved Efficient network can simultaneously learn and train multiple image attributes, which can promote each other's convolutional neural network to learn better deep features and further improve the accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0016] Figure 1 A schematic diagram of an application scenario of the method for determining image attributes provided in an embodiment of the present application;

[0017] Figure 2 A flowchart of a method for determining image attributes is provided for one embodiment of the present application;

[0018] Figure 3 A schematic diagram of the structure of an improved Efficient-B0 network provided in one embodiment of the present application;

[0019] Figure 4 A schematic diagram of a process for training an image attribute classification model provided in one embodiment of the present application;

[0020] Figure 5 A schematic diagram of the structure of an Attention-MBConv module provided in one embodiment of the present application;

[0021] Figure 6 A schematic diagram of the structure of an attention module (Attention Block) provided for one embodiment of the present application;

[0022] Figure 7 A schematic flow chart of a method for determining image attributes provided by another embodiment of the present application;

[0023] Figure 8 A schematic diagram of the structure of an apparatus for determining image attributes provided by one embodiment of the present application;

[0024] Fig. 9 A schematic diagram of the structure of a terminal device provided in one embodiment of the present application;

[0025] Fig.10 A schematic diagram of the structure of a computer-readable storage medium provided in one embodiment of the present application. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0027] In order to explain the present application in more detail, a method, apparatus, terminal device and computer storage medium for determining image attributes provided by the present application are specifically described below in conjunction with the accompanying drawings.

[0028] Please refer to Figure 1 , Figure 1A schematic diagram of an application scenario of the method for determining image attributes provided in an embodiment of the present application is shown, and the application scenario includes a terminal device 102 and a server 104 provided in an embodiment of the present application, wherein a network is provided between the server 104 and the terminal device 102. The network is used to provide a medium for a communication link between the terminal device 102 and the server 104. Among them, the terminal device 102 interacts with the server 104 through the network to receive or send messages, etc., and the server 104 can be a server that provides various services. Among them, the server 104 can be used to perform the method for determining image attributes provided in an embodiment of the present application, and the server 104 can obtain the image to be identified from the terminal device 102, and then input the image to be identified into a pre-trained image attribute classification model to output the image angle probability value and the text type probability value of the image to be identified respectively, but determine the image angle and the text type according to the image angle probability value and the text type probability value. In addition, the server 104 can also return the determined result to the terminal device 102, etc. In addition, the server 104 can store a pre-trained image attribute classification model; the terminal device 102 can be used to take a photo of an object to be identified (such as a test paper, etc.) to generate an image to be identified.

[0029] Optionally, the terminal device 102 may be any electronic device with a display screen, including but not limited to a smart phone and a computer device, wherein the computer device may be at least one of a desktop computer, a portable computer, a laptop computer, a tablet computer, etc. The terminal device 102 may generally refer to one of a plurality of terminal devices. In addition, the terminal device 102 may also be used to execute a method for determining an image attribute provided in an embodiment of the present application.

[0030] It should be understood that the number of terminal devices, networks and servers is only illustrative. According to implementation requirements, there can be any number of terminal devices, networks and servers. For example, the server can be a server cluster composed of multiple servers.

[0031] Based on this, an embodiment of the present application provides a method for determining image attributes. Figure 2 , Figure 2 A schematic diagram of a method for determining image attributes provided by an embodiment of the present application is shown. Figure 1 The server in the example is used to illustrate the following steps:

[0032] Step S110, obtaining an image to be recognized;

[0033] Before performing text recognition on an image, it is necessary to determine the attributes of the image. Therefore, in this embodiment, the image to be recognized refers to any image that needs to be recognized. Among them, the image to be recognized can be a picture generated by an image acquisition device (such as a smart terminal, a camera device, etc.) taking a document, certificate, test paper, book, etc.

[0034] In addition, the angle (i.e., whether the image is tilted), color, size, resolution, etc. of the image to be recognized are not limited, as long as they meet the minimum requirements for image recognition.

[0035] Step S120, inputting the image to be identified into a pre-trained image attribute classification model, and outputting the image angle probability value and the text type probability value of the image to be identified respectively; wherein the image attribute classification model is obtained by training an improved Efficient network using image samples marked with image angle information and text type information, and the improved Efficient network includes a plurality of Attention-MBConv modules, each Attention-MBConv module is connected in series, and the Attention-MBConv module is formed by adding an attention module after the last feature output layer of the MBConv module;

[0036] Specifically, model training is to give an input vector and a target output value, then input the input vector into one or more network structures or functions to obtain the actual output value, calculate the bias based on the target output value and the actual output value, and determine whether the bias is within the allowable range; if it is within the allowable range, the training ends and the relevant parameters are fixed; if it is not within the allowable range, some parameters in the network structure or function are continuously adjusted until the bias is within the allowable range or a certain end condition is reached, the training ends and the relevant parameters are fixed, and finally the trained model can be obtained based on the fixed relevant parameters.

[0037] The training of the image attribute classification model in this embodiment is actually as follows: inputting the image sample marked with the image angle information and the text type information as the input vector into the improved Efficient network, and taking the image angle and the text type of the image sample as the target output value; finding the hidden layer, outputting the output of each layer unit, finding the deviation between the target classification result and the actual classification result, when the deviation is within the unacceptable range, calculating the error of the neurons in the network layer, finding the error gradient, and updating the weight, finding the hidden layer again, outputting the output of each layer unit, finding the deviation between the target classification result and the actual classification result, until the deviation is within the acceptable range, the training ends, and the weight and threshold are fixed, thereby obtaining the pre-trained image attribute classification model. In addition, in this embodiment, an improved Efficient network is adopted, wherein the improved Efficient network is a multi-task neural network, which can analyze and judge multiple tasks (in this embodiment, multiple attributes of the image) at the same time, greatly simplifying the process of image attribute detection, and also improving the effect and accuracy of image attribute detection. In addition, the improved Efficient network includes multiple Attention-MBConv modules, each of which is connected in series. The Attention-MBConv module is formed by adding an attention module after the last feature output layer of the MBConv module. Adding an attention module means introducing an attention mechanism into the MBConv module, which can further improve the accuracy of text detection. Among them, the MBConv module is also called the moving flip bottleneck convolution.

[0038] The image angle information is used to determine the image angle of the image sample; the text type information is used to determine the text type of the image sample. Among them, the image angle can usually refer to any angle in the range of 0 to 360°; but in practical applications, it is not necessary to determine the exact angle; in addition, for the convenience of model training, the image angle can be divided into 4 categories in this embodiment, namely 0°, 90°, 270° and 360°. The text type mainly includes two categories: handwriting and print (or typeface). It is worth noting that the text type in this embodiment does not refer to the font type (such as Songti, Kaiti, etc.); handwriting or print can include different font types.

[0039] The image angle probability value refers to the probability that the image to be recognized belongs to a certain angle category; the text type probability value refers to the probability that the image to be recognized belongs to a certain text category.

[0040] In addition, the MBConv module is usually composed of multiple layers (usually including convolutional layers and ReLU layers), and each layer outputs a feature map. The last feature output layer usually refers to the last layer of the MBConv module.

[0041] Step S130, determining the image angle and the text type according to the image angle probability value and the text type probability value respectively.

[0042] Specifically, the image to be identified is processed by the image attribute classification model, and the image angle probability value and the text type probability value can be outputted, and then the angle category corresponding to the largest image angle probability value is selected as the image angle, and the text type corresponding to the largest text type probability value is selected as the text type. For example, the image A to be identified is input into the image attribute classification model for processing, and the output result is that the probability that the image angle of A is 0° is one-fifth, the probability that the image angle of A is 270° is four-fifths, the probability that the text type of A is printed is three-quarters, and the probability that the text type of A is printed is one-quarter, so the final image angle of A is 270°, and the text type of A is printed.

[0043] In order to facilitate understanding of the present solution, a detailed embodiment is given. For example, when a user needs to search for the correct solution to a certain question from a question bank, the user can use a smart terminal such as a mobile phone to shoot the question to form an image to be identified; since the image may be tilted during the shooting process, and the question may include multiple text types, for example, a printed math question A, the user writes some text with a pen next to A, etc., then the text type at this time includes both printed and handwritten text, and it is necessary to distinguish between printed and handwritten text when identifying the question. After obtaining the image to be identified, the image to be identified is uploaded to a question bank-related server, and the question bank-related server can use the pre-trained image attribute classification model in the embodiment of the present invention to judge the image angle and text category of the image to be identified, and determine the image angle (i.e., the question angle) and the text category corresponding to the question according to the judgment result (i.e., determine that the question is printed, thereby excluding some texts handwritten by the user next to A), and then identify the question information based on optical character recognition, and then find the corresponding question and related solution from the question bank according to the question information.

[0044] The method for determining image attributes in the embodiment of the present application obtains an image to be identified; inputs the image to be identified into a pre-trained image attribute classification model, and outputs an image angle probability value and a text type probability value of the image to be identified respectively; wherein the image attribute classification model is obtained by training an improved Efficient network using image samples marked with image angle information and text type information, and the improved Efficient network includes multiple Attention-MBConv modules, each Attention-MBConv module is connected in series, and the Attention-MBConv module is formed by adding an attention module after the last feature output layer of the MBConv module; the image angle and the text type are determined according to the image angle probability value and the text type probability value; since the method uses image samples marked with image angle information and text type information (i.e., known image angle and text type) to train the improved Efficient network to obtain a pre-trained image attribute classification model, the image angle and text type of the image to be identified can be identified respectively based on the image attribute classification model. It can be seen that the method only needs to design and train one model to simultaneously identify the image angle and text type, thereby reducing the operation and calculation amount. In addition, the improved Efficient network can simultaneously learn and train multiple image attributes, which can promote each other's convolutional neural network to learn better deep features and further improve the accuracy.

[0045] In one embodiment, the improved Efficient network includes a deep feature extraction network, two first fully connected layers, two second fully connected layers and two softmax classification layers; wherein the deep feature extraction network includes multiple Attention-MBConv modules and two convolution modules, and the Attention-MBConv module is arranged between the two convolution modules.

[0046] In an optional implementation, the improved Efficient network may be an improved Efficient-B0 network. The improved Efficient-B0 network generally includes two convolution modules, multiple Attention-MBConv modules, a global average pooling layer, and an FC classification layer (see Figure 3 As shown, Figure 3 Only part of the content is shown in the figure). Optionally, the number of Attention-MBConv modules can be 16, wherein the 16 Attention-MBConv modules are connected in series to form a multi-level structure, through which multiple or multi-stage flip bottleneck convolution operations can be performed on the image. In addition, the multi-level structure is set between two convolution modules.

[0047] The global average pooling layer is mainly used to perform global average pooling on the deep feature maps output by the convolution module and the Attention-MBConv module. The FC classification layer is mainly used to process the deep feature maps after global average pooling and output the classification results. In this embodiment, the FC classification layer has two first fully connected layers, two second fully connected layers and two softmax classification layers, wherein the two first fully connected layers can be fully connected layers of 512 neurons; the two second fully connected layers can be a fully connected layer of 4 neurons and a fully connected layer of 2 neurons, wherein the fully connected layer of 4 neurons is usually used to classify image angles, and the fully connected layer of 2 neurons is usually used to classify text types.

[0048] Please refer to Figure 4 , the training method of the image attribute classification model includes:

[0049] Step S210, obtaining an image sample, and marking the image sample with the image angle information and the text type information to obtain an image sample marked with the image angle information and the text type information;

[0050] In this embodiment, a relatively large number (for example, several thousand, tens of thousands, etc.) of image samples must first be prepared. The image samples can be collected by photographing equipment, or they can be crawled from a picture database or a test question database on the Internet. Generally, the more image samples there are, the more accurate the trained model will be; but too many image samples will reduce the speed of model training. Therefore, in practical applications, it is sufficient to select an appropriate number of image samples, but when preparing image samples, the samples should be diversified as much as possible. When preparing image samples, a data training set can be established, and the image samples can be stored in the data training set.

[0051] After the image samples are acquired, the image samples need to be labeled with image angle information and text type information. Optionally, during the labeling process, the image angle labels can be used to label the image samples with image angle information, and the text type labels can be used to label the image samples with text type information.

[0052] Step S220, inputting the image sample marked with the image angle information and the text type information into the deep feature extraction network for feature extraction to obtain a deep feature map;

[0053] Specifically, the image samples marked with image angle information and text type information are first input into the first convolution module for convolution processing to obtain a feature map; then the feature map is input into a multi-level structure formed by multiple Attention-MBConv modules, and multiple flipped bottleneck convolution operations are performed, and then the deep feature map is output through the second convolution module.

[0054] Step S230, inputting the deep feature map into two first fully connected layers, two second fully connected layers and two softmax classification layers in sequence, and outputting the actual classification result;

[0055] Specifically, before the deep feature map is sequentially input into the first fully connected layer, the second fully connected layer and the softmax classification layer, it is usually further included to input the deep feature map into the global average pooling layer for global average pooling, and after global averaging, the deep feature image is input into the subsequent classification layer for classification processing to obtain the actual classification result. Among them, the deep feature map usually outputs the image angle probability value and the text type probability value after passing through the two first fully connected layers, the two second fully connected layers and the two softmax classification layers, and then the actual classification result is determined according to the image angle probability value and the text type probability value.

[0056] Step S240, adjusting the weights of the improved Efficient network until the deviation between the actual classification result and the target classification result is within an allowable range, the training is completed, and an image attribute classification model is obtained.

[0057] Specifically, the target image angle and text type (i.e., the target classification result) can be determined based on the image angle information and text type information, and then the actual classification result (i.e., the result obtained by classifying the image sample using the improved Efficient network) is checked to see if its deviation is within the allowable range. If not, the weight of the improved Efficient network is adjusted and the actual classification result is re-output until the deviation between the actual classification result and the target classification result is within the allowable range. The training is completed and the image attribute classification model is obtained. The allowable range can be a preset value, indicating the maximum acceptable deviation between the actual classification result and the target classification result.

[0058] This model training method only needs to be trained once to train the image angle and text type classification models at the same time, which greatly reduces the operation steps and improves efficiency.

[0059] For ease of understanding, a detailed embodiment is given. Take the improved Efficient-B0 network as an example to illustrate the process of image attribute classification model training: input image samples marked with image angle information and text type information to the first stage to the ninth stage for processing, and finally output the image angle probability value and text type probability value of the image to be identified, wherein the first stage is convolution module -> batch normalization -> Swish activation function; the second stage: 1 Attention-MBConv module, the third stage: 2 Attention-MBConv modules, the fourth stage: 2 Attention-MBConv modules, the fifth stage: 3 Attention-MBConv modules, the sixth stage: 3 Attention-MBConv modules, the seventh stage: 4 Attention-MBConv modules, the eighth stage: 1 Attention-MBConv module, the ninth stage: convolution module -> batch normalization -> Swish activation function -> global average pooling -> random inactivation -> fully connected layer -> Softmax layer, and finally output.

[0060] In one embodiment, the Attention-MBConv module includes a first convolutional layer, a ReLU layer, a second convolutional layer and an attention module; the step of inputting the image sample into the deep feature extraction network for feature extraction to obtain the deep feature map includes:

[0061] Input the first feature map into the first convolution layer for convolution processing to obtain a second feature map; wherein the first feature map is a feature map output by the convolution module located before the current Attention-MBConv module and the Attention-MBConv module processing the image sample marked with image angle information and text type information;

[0062] The second feature map is input into the ReLU layer for ReLU and deep convolution processing to obtain a third feature map; the third feature map is input into the second convolution layer for ReLU and convolution processing to obtain a fourth feature map; the fourth feature map is input into the attention module for weighted calculation to obtain a weighted feature map; the weighted feature map is input into the Attention-MBConv module or the convolution module located after the current Attention-MBConv module for output to obtain a deep feature map.

[0063] Specifically, Figure 5As shown in the figure, the Attention-MBConv module includes the first convolutional layer (i.e., the first block in the figure), the ReLU layer (i.e., the second block in the figure, 3×3, ReLU6, Dwise), the second convolutional layer (i.e., the third block in the figure, ReLU6, 1×1) and the attention module (i.e., the Attention Block in the figure); please refer to Figure 6 The attention module includes the Scale layer and the Global Pooling layer, 1×1Conv layer, BN / ReLU layer, 1×1Conv layer, BN layer and Sigmoid function layer connected from top to bottom, where the Sigmoid function layer is connected to the Scale layer. The BN and ReLU layers represent nonlinear activation units, which are mainly used to avoid the problem that the convolutional neural network can only fit linear data but not nonlinear data, which has never caused the subsequent model training to be inaccurate or even lead to training failure. The attention module is mainly used to perform weighted processing on the fourth feature map, so that the feature map output after weighting is more accurate, thereby making the image attribute classification model obtained by subsequent training more accurate.

[0064] In this embodiment, the Attention-MBConv module may refer to any Attention-MBConv module in the improved Efficient network. The feature map input to the first convolutional layer is called the first feature map, wherein the first feature map is usually a feature map obtained by inputting an image sample marked with image angle information and text type information into the first stage of the improved Efficient network through a convolution module, batch normalization, and Swish activation function, or a feature map output by other Attention-MBConv modules. Among them, other Attention-MBConv modules refer to Attention-MBConv modules other than the current Attention-MBConv module and before the current Attention-MBConv module.

[0065] The first feature map is convolved, batch standardized, and normalized by the first convolution layer to obtain a second feature map, and then the second feature map is input into the ReLU layer for ReLU, deep convolution, batch standardization, and normalization to obtain a third feature map. The third feature map is input into the second convolution layer for ReLU, convolution, batch standardization, and normalization to obtain a fourth feature map; the fourth feature map is input into the attention module for weighted calculation to obtain a weighted feature map; the weighted feature map is input into the Attention-MBConv module or the convolution module after the current Attention-MBConv module for output to obtain a deep feature map.

[0066] In one embodiment, the step of adjusting the weights of the improved Efficient network includes:

[0067] The gradient descent method is used to adjust the weights of the improved Efficient network, and the loss function adopts the weighted loss function.

[0068] Specifically, the gradient descent method is an iterative method that can be used to solve the model parameters of the machine learning algorithm. When solving the minimum value of the loss function, the gradient descent method can be used to iterate step by step to obtain the minimized loss function and model parameter value.

[0069] The loss function used when the gradient descent method is used to adjust the weight of the improved Efficient network is a weighted loss function. Optionally, the weights of the two attributes of image angle and font type in the weighted loss function can be 1:2.

[0070] In one embodiment, before the step of inputting the image sample marked with the image angle information and the text type information into the deep feature extraction network for feature extraction, the step includes:

[0071] The image samples marked with image angle information and text type information are scaled according to a preset scaling range.

[0072] Specifically, before using image samples to train the improved Efficient network, the image samples need to be scaled. Each image sample is scaled to a uniform size range. The preset scaling range is a preset value, such as 300×300. It should be understood that the value of the preset scaling range is not unique and other values ​​may be taken.

[0073] In some embodiments, before the step of inputting the image sample marked with the image angle information and the text type information into the deep feature extraction network for feature extraction, the step further includes:

[0074] One or more enhancement processing methods are selected according to a preset selection probability to perform enhancement processing on the image samples after scaling processing; when multiple enhancement processing methods are selected, the running order of each enhancement processing method is random, and the enhancement processing method includes random transformation order processing, brightness transformation processing, compression processing, motion blur processing and noise addition processing on the RGB three channels of the image sample.

[0075] Specifically, in order to enhance the generalization ability of the pre-trained text formula localization model and prevent the model from overfitting, the image samples need to be enhanced before the image samples are used to train the improved Efficient network. The specific enhancement method includes random transformation order processing, brightness transformation processing, compression processing, motion blur processing and noise addition processing of the RGB three channels of the image samples, where the added noise can be Gaussian noise and salt and pepper noise.

[0076] Brightness conversion processing usually refers to multiplying an image sample by a coefficient to change the brightness of the image sample. The coefficient can be a random number in the range of 0.7-1.3.

[0077] Motion blur is the moving effect in scene images, which is usually produced during the photo shooting process. In order to ensure the diversity and authenticity of image samples, motion blur is performed on image samples.

[0078] In addition, for an image sample, image enhancement processing is optional (i.e., it may be performed or not), and the image enhancement processing method may be selected from one or more of them. Each method may be selected according to its preset selection probability, and the probability of each method being selected may be equal or different.

[0079] In an optional implementation, the probability of each enhancement processing method being selected may be set to 0.5.

[0080] In one embodiment, the compression process includes JPEG compression; the step of selecting one or more enhancement processing methods according to a preset selection probability to perform enhancement processing on the image sample after the scaling process includes:

[0081] The scaled image samples are compressed using JPEG compression according to a preset compression strength.

[0082] Among them, JPEG compression is an image compression method based on DCT transformation, Zig-Zag scanning and Huffman coding. In order to prevent over-compression during the image compression process, the compression strength should be kept within a preset compression range. Optionally, the preset compression range can be 0 to 40.

[0083] It should be understood that the algorithm used in image compression is not limited to the JPEG compression algorithm mentioned in this embodiment. According to the technical inspiration of the present invention, those skilled in the art may also use other compression algorithms to perform image compression.

[0084] For ease of understanding, a detailed implementation example is given. The method for determining image attributes mainly includes two parts: model training stage and model inference stage (ie, application stage). Figure 7The model training phase mainly includes: inputting image samples with known image angles and font type labels, scaling the image samples to 300×300, and enhancing the scaled image samples; then inputting the enhanced image samples into the deep feature extraction network in the improved Efficient-B0 network for deep feature extraction, and then inputting the deep features into two fully connected layers with 512 neurons, and then passing through fully connected layers with 4 and 2 neurons respectively, and outputting the softmax layer to the probability values ​​of the two attributes of angle and text type, adjusting the weights of the improved Efficient-B0 network, and completing the model training. The model inference stage is mainly as follows: the image to be recognized (the image can also be scaled to 300×300 first) is input into the deep feature extraction network in the trained improved Efficient-B0 network for deep feature extraction, and then the deep features are input into two fully connected layers with 512 neurons, and then passed through fully connected layers with 4 and 2 neurons respectively, and the softmax layer outputs the image angle probability value and text type probability value. The categories with the highest probability are image angle and text type.

[0085] It should be understood that although Figure 2-3 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Figure 2-3 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0086] The above-mentioned embodiment disclosed in the present application describes in detail a method for determining image attributes. The above-mentioned method disclosed in the present application can be implemented using various forms of equipment. Therefore, the present application also discloses an apparatus for determining image attributes corresponding to the above-mentioned method. A specific embodiment is given below for detailed description.

[0087] See also Figure 8 , is a device for determining image attributes disclosed in an embodiment of the present application, mainly comprising:

[0088] The to-be-recognized image acquisition module 802 is used to acquire the to-be-recognized image;

[0089] A probability value output module 804 is used to input the image to be identified into a pre-trained image attribute classification model, and output the image angle probability value and the text type probability value of the image to be identified respectively; wherein the image attribute classification model is obtained by training an improved Efficient network using image samples marked with image angle information and text type information, and the improved Efficient network includes a plurality of Attention-MBConv modules, each of which is connected in series, and the Attention-MBConv module is formed by adding an attention module after the last feature output layer of the MBConv module;

[0090] The angle type determination module 806 is used to determine the image angle and the text type according to the image angle probability value and the text type probability value respectively.

[0091] In one embodiment, the improved Efficient network includes a deep feature extraction network, a global average pooling layer, two first fully connected layers, two second fully connected layers and two softmax classification layers; wherein the deep feature extraction network includes multiple Attention-MBConv modules and two convolution modules, and the Attention-MBConv module is arranged between the two convolution modules; including:

[0092] A sample acquisition module, used for acquiring image samples;

[0093] A sample marking module is used to mark image samples using image angle information and text type information to obtain image samples marked with image angle information and text type information;

[0094] A feature extraction module, used for inputting image samples marked with image angle information and text type information into a deep feature extraction network for feature extraction to obtain a deep feature map;

[0095] The result output module is used to input the deep feature map into two first fully connected layers, two second fully connected layers and two softmax classification layers in sequence, and output the actual classification results;

[0096] The weight adjustment module is used to adjust the weight of the improved Efficient network until the deviation between the actual classification result and the target classification result is within the allowable range, and the training is completed to obtain the image attribute classification model.

[0097] In one embodiment, the Attention-MBConv module includes a first convolution layer, a ReLU layer, a second convolution layer and an attention module; the feature extraction module includes: a second feature map obtaining module, a third feature map obtaining module, a fourth feature map obtaining module and a weighted calculation module;

[0098] A second feature map obtaining module is used to input the first feature map into the first convolution layer for convolution processing to obtain a second feature map; wherein the first feature map is a feature map output by the convolution module located before the current Attention-MBConv module and the Attention-MBConv module processing the image sample marked with image angle information and text type information;

[0099] A third feature map obtaining module, used for inputting the second feature map into the ReLU layer for ReLU and depth convolution processing to obtain a third feature map;

[0100] A fourth feature map obtaining module, used for inputting the third feature map into the second convolutional layer to perform ReLU, convolution, batch normalization and normalization processing to obtain a fourth feature map;

[0101] A weighted calculation module, used for inputting the fourth feature map into the attention module for weighted calculation to obtain a weighted feature map;

[0102] The feature extraction module is also used to input the weighted feature map into the Attention-MBConv module or the convolution module located after the current Attention-MBConv module for output to obtain a deep feature map.

[0103] In one embodiment, the weight adjustment module is also used to adjust the weights of the Efficien network improved by the gradient descent method; wherein the loss function adopts a weighted loss function.

[0104] In one embodiment, it further includes:

[0105] The zoom processing module is used to perform zoom processing on the image samples marked with image angle information and text type information according to a preset zoom range.

[0106] In one embodiment, it further includes:

[0107] The enhancement processing module is used to select one or more enhancement processing methods according to a preset selection probability to enhance the image samples after scaling processing; wherein, when multiple enhancement processing methods are selected, the running order of each enhancement processing method is random, and the enhancement processing method includes random transformation order processing, brightness transformation processing, compression processing, motion blur processing and noise addition processing of the RGB three channels of the image sample.

[0108] In one embodiment, the compression process includes JPEG compression; the enhancement processing module includes a compression processing module:

[0109] The compression processing module is used to compress the image samples after the scaling processing by using JPEG compression and according to a preset compression strength.

[0110] The specific definition of the device for determining image attributes can be found in the definition of the method above, which will not be repeated here. Each module in the above device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above modules can be embedded in or independent of the processor in the terminal device in the form of hardware, or can be stored in the memory of the terminal device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0111] Please refer to Fig. 9 , Fig. 9 It shows a structural block diagram of a terminal device provided by an embodiment of the present application. The terminal device 90 may be a terminal device. The terminal device 90 in the present application may include one or more of the following components: a processor 92, a memory 94, and one or more applications, wherein the one or more applications may be stored in the memory 94 and configured to be executed by one or more processors 92, and the one or more applications are configured to execute the method described in the above method embodiment applied to the terminal device, and may also be configured to execute the method described in the above method embodiment applied to the method for determining image attributes.

[0112] The processor 92 may include one or more processing cores. The processor 92 uses various interfaces and lines to connect the various parts of the entire terminal device 90, and executes various functions and processes data of the terminal device 90 by running or executing instructions, programs, code sets or instruction sets stored in the memory 94, and calling data stored in the memory 92. Optionally, the processor 94 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 92 can integrate one or more combinations of a central processing unit (CPU), a graphics processing unit (GPU) reporting verifier of buried data, and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 92, but may be implemented separately through a communication chip.

[0113] The memory 94 may include a random access memory (RAM) or a read-only memory (ROM). The memory 94 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 94 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data created by the terminal device 90 during use, etc.

[0114] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the terminal device to which the scheme of the present application is applied. The specific terminal device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0115] In summary, the terminal device provided in the embodiment of the present application is used to implement the corresponding method for determining image attributes in the aforementioned method embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0116] See also Fig.10, which shows a block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable storage medium 100 stores program code, which can be called by a processor to execute the method described in the above-mentioned method for determining image attributes, or can be called by a processor to execute the method described in the above-mentioned method for determining image attributes.

[0117] The computer readable storage medium 100 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 100 includes a non-transitory computer-readable storage medium. The computer readable storage medium 100 has storage space for program code 1002 that executes any method step of the above method. These program codes can be read from or written to one or more computer program products. The program code 1002 can be compressed, for example, in an appropriate form.

[0118] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0119] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for determining image attributes, It is characterized in that The method comprises: Obtain an image to be recognized; Input the image to be identified into a pre-trained image attribute classification model, and output the image angle probability value and the text type probability value of the image to be identified respectively; wherein the image attribute classification model is obtained by training an improved Efficient network using image samples marked with image angle information and text type information, and the improved Efficient network includes a plurality of Attention-MBConv modules, each of which is connected in series, and the Attention-MBConv module is formed by adding an attention module after the last feature output layer of the MBConv module; Determine the image angle and the text type according to the image angle probability value and the text type probability value respectively; The improved Efficient network includes a deep feature extraction network, a global average pooling layer, two first fully connected layers, two second fully connected layers and two softmax classification layers; wherein the deep feature extraction network includes a plurality of Attention-MBConv modules and two convolution modules, and the Attention-MBConv module is arranged between the two convolution modules; the training method of the image attribute classification model includes: Acquire an image sample, and mark the image sample with the image angle information and the text type information to obtain an image sample marked with the image angle information and the text type information; Inputting the image sample marked with image angle information and text type information into the deep feature extraction network to extract features and obtain a deep feature map; Input the deep feature map into two of the first fully connected layers, two of the second fully connected layers and two of the softmax classification layers in sequence, and output an actual classification result; The weights of the improved Efficient network are adjusted until the deviation between the actual classification result and the target classification result is within an allowable range, and the training is completed to obtain the image attribute classification model; The Attention-MBConv module includes a first convolutional layer, a ReLU layer, a second convolutional layer and an attention module; The step of inputting the image sample into the deep feature extraction network for feature extraction to obtain a deep feature map includes: Inputting the first feature map into the first convolution layer for convolution processing to obtain a second feature map; wherein the first feature map is a feature map output by the convolution module located before the current Attention-MBConv module and the Attention-MBConv module processing the image sample marked with image angle information and text type information; Inputting the second feature map into the ReLU layer to perform ReLU and depth convolution processing to obtain a third feature map; Inputting the third feature map into the second convolutional layer for convolution processing to obtain a fourth feature map; Inputting the fourth feature map into the attention module for weighted calculation to obtain a weighted feature map; The weighted feature map is input into the Attention-MBConv module located after the current Attention-MBConv module or the convolution module for output to obtain the depth feature map.

2. The method according to claim 1, It is characterized in that The step of adjusting the weights of the improved Efficient network includes: The weights of the improved Efficient network using the gradient descent method are adjusted; wherein the loss function uses a weighted loss function.

3. The method according to any one of claims 1 to 2, It is characterized in that Before the step of inputting the image sample marked with the image angle information and the text type information into the deep feature extraction network for feature extraction, the method includes: The image sample marked with the image angle information and the text type information is scaled according to a preset scaling range.

4. The method according to claim 3, It is characterized in that Before the step of inputting the image sample marked with the image angle information and the text type information into the deep feature extraction network for feature extraction, the method further includes: Select one or more enhancement processing methods according to a preset selection probability to perform enhancement processing on the image sample after the scaling processing; When multiple enhancement processing methods are selected, the running order of each enhancement processing method is random, and the enhancement processing method includes performing random transformation order processing, brightness transformation processing, compression processing, motion blur processing and noise addition processing on the RGB three channels of the image sample.

5. The method according to claim 4, It is characterized in that The compression process includes JPEG compression; the step of selecting one or more enhancement processing methods according to a preset selection probability to perform enhancement processing on the image sample after the scaling process includes: The image samples after scaling are compressed using the JPEG compression according to a preset compression strength.

6. A device for determining image attributes, It is characterized in that The device comprises: An image acquisition module to be identified, used to acquire an image to be identified; A probability value output module, used for inputting the image to be identified into a pre-trained image attribute classification model, and outputting the image angle probability value and the text type probability value of the image to be identified respectively; wherein the image attribute classification model is obtained by training an improved Efficient network using image samples marked with image angle information and text type information, the improved Efficient network includes a plurality of Attention-MBConv modules, each of the Attention-MBConv modules is connected in series, and the Attention-MBConv module is formed by adding an attention module after the last feature output layer of the MBConv module; An angle type determination module, used to determine the image angle and the text type according to the image angle probability value and the text type probability value respectively; The improved Efficient network includes a deep feature extraction network, a global average pooling layer, two first fully connected layers, two second fully connected layers and two softmax classification layers; wherein the deep feature extraction network includes a plurality of Attention-MBConv modules and two convolution modules, and the Attention-MBConv module is arranged between the two convolution modules; and includes: A sample acquisition module, used for acquiring image samples; A sample marking module is used to mark image samples using image angle information and text type information to obtain image samples marked with image angle information and text type information; A feature extraction module, used for inputting image samples marked with image angle information and text type information into a deep feature extraction network for feature extraction to obtain a deep feature map; The result output module is used to input the deep feature map into two first fully connected layers, two second fully connected layers and two softmax classification layers in sequence, and output the actual classification results; The weight adjustment module is used to adjust the weight of the improved Efficient network until the deviation between the actual classification result and the target classification result is within the allowable range, and the training is completed to obtain the image attribute classification model; The Attention-MBConv module includes the first convolutional layer, the ReLU layer, the second convolutional layer, and the attention module; The feature extraction module includes: a second feature map obtaining module, a third feature map obtaining module, a fourth feature map obtaining module and a weighted calculation module; A second feature map obtaining module is used to input the first feature map into the first convolution layer for convolution processing to obtain a second feature map; wherein the first feature map is a feature map output by the convolution module located before the current Attention-MBConv module and the Attention-MBConv module processing the image sample marked with image angle information and text type information; A third feature map obtaining module, used for inputting the second feature map into the ReLU layer for ReLU and depth convolution processing to obtain a third feature map; A fourth feature map obtaining module, used for inputting the third feature map into the second convolutional layer to perform ReLU, convolution, batch normalization and normalization processing to obtain a fourth feature map; A weighted calculation module, used for inputting the fourth feature map into the attention module for weighted calculation to obtain a weighted feature map; The feature extraction module is also used to input the weighted feature map into the Attention-MBConv module or the convolution module located after the current Attention-MBConv module for output to obtain a deep feature map.

7. A terminal device, It is characterized in that include: Memory; One or more processors coupled to the memory; One or more applications, wherein the one or more applications are stored in a memory and configured to be executed by one or more processors, and the one or more applications are configured to execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 5.

Images