Image classification method and device, storage medium and electronic device

Abstract

The invention discloses an image classification method and device, a storage medium and an electronic device, and relates to the field of computers.The image classification method comprises the steps that a to-be-detected image is screened out from an image database according to a quality filtering threshold value, and the image quality score of the to-be-detected image is lower than the quality filtering threshold value; the to-be-detected image is input into a feature recognition model to be processed, a feature recognition result is obtained, the feature recognition model comprises a plurality of feature recognition module strings, the feature recognition module strings are connected in series, each feature recognition module string comprises a plurality of feature recognition modules, and the feature recognition modules are connected in series; classifying the feature recognition result through a classification model to obtain a classification result; by adopting the above scheme, the problem that in the prior art, multiple models need to be trained to complete classification for images with different resolutions, and the calculation cost is too high is solved.

Description

Technical Field

[0001] The present application relates to the field of computers, and more specifically, to an image classification method and device, a storage medium, and an electronic device. Background Art

[0002] In the process of intelligent construction and transformation, adding computer vision algorithms based on machine learning has become a common operation. However, although there is a certain hardware budget, due to objective reasons, such as ensuring the utilization of existing equipment and avoiding repeated installation of equipment with high installation difficulty and poor power supply conditions, in actual applications, it is usually necessary to use the images provided by the existing camera equipment in the site to a certain extent. However, the input of such applications is usually a camera. In applications such as tower and blade detection in wind farms, it is especially necessary to use the images provided by a large number of outdoor cameras, but different models of cameras often face the problem of different resolutions. In the actual application of computer vision algorithm models, when the model encounters such natural changes as weather changes, noise types and compression artifacts, and the resolution of the input image is different from that of the training set, the recognition accuracy will be significantly reduced. In order to ensure the recognition accuracy, traditional solutions usually train multiple models for different resolutions or input changes (such as designing and training three models for the three resolutions of 256*256, 1080*960, and 1080*720 that may be available in this scene), but the computational cost of this method is too high.

[0003] Regarding the related technology, the existing technology needs to train multiple models to complete the classification of images with different resolutions, and the computational cost is too high. No effective solution has been proposed yet.

[0004] Therefore, it is necessary to improve the related technology to overcome the above-mentioned defects in the related technology. Summary of the invention

[0005] The embodiments of the present application provide an image classification method and device, a storage medium, and an electronic device to at least solve the problem in the related art that the prior art requires training multiple models to complete classification for images of different resolutions, resulting in excessively high computational costs.

[0006] According to one aspect of an embodiment of the present application, there is provided an image classification method, comprising: screening out an image to be detected from an image database according to a quality filtering threshold, wherein the image quality score of the image to be detected is lower than the quality filtering threshold; inputting the image to be detected into a feature recognition model for processing to obtain a feature recognition result, wherein the feature recognition model comprises a plurality of feature recognition module strings, the plurality of feature recognition module strings are connected in parallel, each feature recognition module string comprises a plurality of feature recognition modules, the plurality of feature recognition modules are connected in series; and classifying the feature recognition result by a classification model to obtain a classification result.

[0007] In an exemplary embodiment, the image to be detected is input into a feature recognition model for processing to obtain a feature recognition result, including: inputting the image to be detected into the feature recognition model; processing the image to be detected by the plurality of feature recognition module strings respectively to obtain a plurality of sub-feature recognition results; and determining the feature recognition result according to the plurality of sub-feature recognition results.

[0008] In an exemplary embodiment, the image to be detected is processed respectively by the multiple feature recognition module strings to obtain multiple sub-feature recognition results, including: performing Fourier space change processing on the image to be detected by the first feature recognition module to obtain a first processing result, and performing convolution layer processing on the image to be detected by the first feature recognition module to obtain a second processing result, wherein the feature recognition module string includes the first feature recognition module; performing nonlinear mapping on the sum of the first processing result and the second processing result by an activation function to obtain a third processing result; inputting the third processing result into the second feature recognition module for processing, and inputting the obtained fourth processing result into the next feature recognition module in the feature recognition module string for processing, until the multiple feature recognition modules are traversed to obtain a first sub-feature recognition result, wherein the multiple sub-feature recognition results include the first sub-feature recognition result.

[0009] In an exemplary embodiment, the image to be detected is subjected to Fourier space change processing by a first feature recognition module to obtain a first processing result, including: performing Fourier transform on the image to be detected to obtain a first sub-processing result; processing the first sub-processing result by an all-pass filter to obtain a second sub-processing result; performing a zero-filling operation on the second sub-processing result, and performing an inverse Fourier transform on the obtained third sub-processing result to obtain the first processing result.

[0010] In an exemplary embodiment, the first feature recognition module performs convolution layer processing on the image to be detected to obtain a second processing result, including: processing the image to be detected through a target convolution layer to obtain a fourth sub-processing result; and performing an interpolation operation on the fourth sub-processing result to obtain the second processing result.

[0011] In an exemplary embodiment, performing nonlinear mapping on the sum of the first processing result and the second processing result through an activation function to obtain a third processing result includes: calculating the sum of the first processing result and the second processing result to obtain a fifth processing result; and calculating the sum of the first processing result and the second processing result through the formula Perform nonlinear mapping on the fifth processing result to obtain the third processing result, wherein GELU(x) is the activation function and x is the fifth processing result.

[0012] According to another aspect of an embodiment of the present application, an image classification device is also provided, including: a screening module, used to screen out images to be detected from an image database according to a quality filtering threshold, wherein the image quality score of the images to be detected is lower than the quality filtering threshold; a processing module, used to input the images to be detected into a feature recognition model for processing to obtain feature recognition results, wherein the feature recognition model includes a plurality of feature recognition module strings, the plurality of feature recognition module strings are connected in parallel, each feature recognition module string includes a plurality of feature recognition modules, the plurality of feature recognition modules are connected in series; a classification module, used to classify the feature recognition results through a classification model to obtain classification results.

[0013] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned image classification method when running.

[0014] According to another aspect of an embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the image classification method through the computer program.

[0015] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps of the method described in each embodiment of the present application are implemented.

[0016] Through the present application, low-quality images to be detected whose image quality scores are lower than the quality filtering threshold are screened out from an image database, and then the images to be detected are input into a feature recognition model for processing to obtain feature recognition results, wherein the feature recognition module includes multiple feature recognition module strings, the multiple feature recognition module strings are connected in parallel, and the multiple feature recognition modules in one feature recognition module string are connected in series; finally, the feature recognition results are classified by a classification model to obtain classification results; adopting the above scheme, a new feature recognition model is proposed, through which image feature recognition of multiple clarity can be completed, the image classification accuracy is improved, and the cost is reduced; thereby solving the problem in the related art that the prior art needs to train multiple models to complete classification for images of different resolutions, and the computational cost is too high. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 It is a hardware structure block diagram of a computer terminal of an image classification method according to an embodiment of the present application;

[0020] Figure 2 is a flow chart of an image classification method according to an embodiment of the present application;

[0021] Figure 3 It is a schematic diagram of the use process of an improved FNO model according to an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of a processing flow of a feature recognition model according to an embodiment of the present application;

[0023] Figure 5 It is a structural block diagram of an image classification device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] The method embodiments provided in the embodiments of the present application can be executed in a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 is a hardware structure block diagram of a computer terminal of an image classification method according to an embodiment of the present application. Figure 1 As shown, the computer terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor (Central Processing Unit, MCU) or a programmable logic device (Field Programmable Gate Array, FPGA)) and a memory 104 for storing data, wherein the above-mentioned computer terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0027] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the image classification method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0028] A wireless network provided by a communication provider of a computer terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0029] The following is an explanation of the professional terms that appear in this application:

[0030] Operator Learning: Operator Learning is a machine learning method that focuses on learning and using linear or nonlinear operators, rather than just learning vectors or matrices. Operator learning focuses on learning the mapping from input space to output space, where the mapping itself is an operator, not just a vector or matrix. This method is applicable to various fields, such as signal processing, quantum computing, etc., and can handle more complex data structures and problems. The goal of operator learning is to learn the optimal operator so that the input data can be mapped to the output space in the best way to meet the needs of a specific task.

[0031] FNO: Fourier neural operator, is based on replacing the kernel integral operator with a convolution operator defined in Fourier space. The input function of this operator is defined on a well-defined, equally spaced grid of points, and outputs the domain of interest at the same grid points. The network parameters are defined and learned in Fourier space, rather than in physical space, i.e. the coefficients of the Fourier series of the output function are learned from data. It is currently mainly used in theoretical fields such as solving partial differential equations.

[0032] ResNet: Residual Network, by introducing residual connections to solve the gradient vanishing and gradient exploding problems in deep neural networks, makes it possible to train very deep networks. Its core idea is to introduce residual connections in each residual block, allowing information to jump between network layers and better preserve the original input information. ResNet is widely used in tasks such as image classification, object detection, and semantic segmentation.

[0033] In this embodiment, an image classification method is provided. Figure 2 is a flow chart of an image classification method according to an embodiment of the present application. Figure 2 As shown, the process includes the following steps S202-S206:

[0034] Step S202: Screening out an image to be detected from an image database according to a quality filtering threshold, wherein the image quality score of the image to be detected is lower than the quality filtering threshold;

[0035] Step S204: inputting the image to be detected into a feature recognition model for processing to obtain a feature recognition result, wherein the feature recognition model includes a plurality of feature recognition module strings, the plurality of feature recognition module strings are connected in parallel, each feature recognition module string includes a plurality of feature recognition modules, the plurality of feature recognition modules are connected in series;

[0036] Step S206: classify the feature recognition result through a classification model to obtain a classification result.

[0037] Through the above steps, low-quality images to be detected whose image quality scores are lower than the quality filtering threshold are screened out from the image database, and then the images to be detected are input into the feature recognition model for processing to obtain feature recognition results, wherein the feature recognition module includes multiple feature recognition module strings, multiple feature recognition module strings are connected in parallel, and multiple feature recognition modules in one feature recognition module string are connected in series; finally, the feature recognition results are classified by the classification model to obtain classification results; adopting the above scheme, a new feature recognition model is proposed, through which image feature recognition of multiple clarity can be completed, the image classification accuracy is improved, and the cost is reduced; thereby solving the problem in the related technology that the prior art needs to train multiple models to complete classification for images of different resolutions, and the calculation cost is too high.

[0038] In an exemplary embodiment, the image to be detected is input into a feature recognition model for processing to obtain a feature recognition result, including: inputting the image to be detected into the feature recognition model; processing the image to be detected by the plurality of feature recognition module strings respectively to obtain a plurality of sub-feature recognition results; and determining the feature recognition result according to the plurality of sub-feature recognition results.

[0039] Optionally, the feature recognition module is an improved FNO model, which is composed of multiple improved FNO operation modules (i.e., feature recognition modules). In the embodiment of the present application, there are n*m ​​operation modules, where n is the number of FNO modules in a module string (i.e., feature recognition module string), and m is the number of FNO module strings; the image to be detected is processed separately by multiple parallel feature recognition module strings to obtain m sub-feature recognition results, and finally the feature recognition result is determined based on these multiple sub-feature recognition results.

[0040] Optionally, after obtaining multiple sub-feature recognition results, these sub-feature recognition results are added and summed, and then the sum result of 3*m channels is converted into 3 channels through the linear layer of the model; finally, the feature recognition result is obtained through batch normalization layer processing.

[0041] Further, the image to be detected is processed respectively by the multiple feature recognition module strings to obtain multiple sub-feature recognition results, including: performing Fourier space change processing on the image to be detected by the first feature recognition module to obtain a first processing result, and performing convolution layer processing on the image to be detected by the first feature recognition module to obtain a second processing result, wherein the feature recognition module string includes the first feature recognition module; performing nonlinear mapping on the sum of the first processing result and the second processing result by an activation function to obtain a third processing result; inputting the third processing result into the second feature recognition module for processing, and inputting the obtained fourth processing result into the next feature recognition module in the feature recognition module string for processing, until the multiple feature recognition modules are traversed to obtain a first sub-feature recognition result, wherein the multiple sub-feature recognition results include the first sub-feature recognition result.

[0042] Specifically, the process of processing the image to be detected by each feature recognition module string includes: first, copying the image to be detected into two copies, one copy is processed by Fourier space transformation to obtain a first processing result, and the other copy is processed by convolution layer to obtain a second processing result; after both copies are calculated, they are added and summed, and then the activation function σ is used for nonlinear mapping to obtain a third processing result. At this time, the processing process of one feature recognition module is completed; then the processing result of the previous feature recognition module is input into the next feature recognition module for the same processing flow, until all the multiple feature recognition modules have completed the processing, and the first sub-feature recognition result, that is, the processing result of a feature recognition module string is obtained.

[0043] Optionally, the image to be detected is subjected to Fourier space transform processing by a first feature recognition module to obtain a first processing result, including: performing Fourier transform on the image to be detected to obtain a first sub-processing result; processing the first sub-processing result by an all-pass filter to obtain a second sub-processing result; performing a zero-filling operation on the second sub-processing result, and performing an inverse Fourier transform on the obtained third sub-processing result to obtain the first processing result.

[0044] To process the Fourier space change of the detected image, the formula To complete, is the Fourier transform, h l-1 Transformed into Fourier space, and then with Perform product operation, h l-1 is the output of the previous feature recognition module (i.e. the hidden vector obtained by the previous module). It is an all-pass filter that retains the three channels of the image and all wave frequency pixels. rand W r represents the height and width of the input image resolution r. X is the formula for the zero-filling operation, which is used to perform secondary processing on data with insufficient resolution to maintain the consistency of the overall resolution of the data set. is the inverse Fourier transform, which converts the data back to the source space.

[0045] Optionally, the image to be detected is processed by the first feature recognition module through a convolution layer to obtain a second processing result, including: processing the image to be detected through a target convolution layer to obtain a fourth sub-processing result; and interpolating the fourth sub-processing result to obtain the second processing result.

[0046] The image to be detected is processed by convolution layer, which can be obtained by formula F W =ψ(h l-1 W) is completed, W is a convolution layer, and a 1*1 convolution layer is used in the embodiment of the present application; ψ is the formula embodiment of the interpolation operation.

[0047] Optionally, performing nonlinear mapping on the sum of the first processing result and the second processing result through an activation function to obtain a third processing result includes: calculating the sum of the first processing result and the second processing result to obtain a fifth processing result; and obtaining a third processing result through the formula Perform nonlinear mapping on the fifth processing result to obtain the third processing result, wherein GELU(x) is the activation function and x is the fifth processing result.

[0048] First, the first processing result and the second processing result are added to obtain the fifth processing result, and then the fifth processing result is nonlinearly mapped using an activation function. The activation function used in this embodiment is GELU, and the specific calculation formula is: x is the fifth processing result of the input.

[0049] In an optional embodiment, in order to solve the problem of poor performance of computer vision models in actual applications when input images are inconsistent with the training set size or have affected clarity, the FNO model is applied and improved, and this model is fused with the existing image recognition model during the training phase. The overall input and output of the fused new model are consistent with the original model, and no additional image data conversion process is required when it is deployed again. The usage process of this model is as follows: Figure 3 As shown, including:

[0050] 3.1. Put the source image to be identified from the image database into the software equipped with algorithms and programs;

[0051] 3.2. Process the data according to the original data processing method of the software, adjust the filter settings, and relax the low-quality image data filtering threshold (i.e. the above-mentioned quality filtering threshold);

[0052] 3.3. Put the image to be detected as input into the image recognition algorithm (including the above-mentioned feature recognition model and classification model);

[0053] 3.4. The software obtains and outputs the judgment results, such as whether the person in the image is suspected of illegal operation;

[0054] 3.5. End this usage process.

[0055] Specifically, the processing flow of the above feature recognition model is as follows: Figure 4 As shown, the following steps are included:

[0056] 1. Start by inputting image data (i.e. the image to be detected mentioned above);

[0057] 2. Improved FNO model (i.e. the above-mentioned feature recognition model). The improved FNO model consists of multiple improved FNO operation modules (feature recognition modules). Figure 4 middle Represents the first i ,l j ) hidden vectors (i.e., the third processing result), in this embodiment, the number of operation modules is n*m, n is the number of FNO modules in a single module string, m is the number of FNO module strings, l i ∈{0,…,m},l j ∈{0,…,n}.

[0058] The logic of multiple serial feature recognition modules is shown in Formula 1:

[0059] h l =Block(h l-1 ) Formula 1;

[0060] Among them, h l-1 is the output of the previous module, that is, the hidden vector obtained by the previous module, h l The output of this module, Block is used to represent the operation logic of a single improved FNO. Specifically, the operation logic is as follows:

[0061] As shown in formula 2:

[0062]

[0063] h l-1 is copied into two copies, one of which is processed by Fourier space transformation. The calculation process is shown in Formula 3; one part is processed by the convolution layer, F WThe calculation process of is shown in Formula 4. After both parts are calculated, they are added and summed, and then the activation function σ is used for nonlinear mapping. The activation function used in this embodiment is GELU, and the specific calculation formula is shown in Formula 5.

[0064]

[0065] in, is the Fourier transform, h l-1 Transformed into Fourier space, and then with Perform a product operation. It is an all-pass filter that retains the three channels of the image and all wave frequency pixels. r and W r represents the height and width of the input image resolution r. χ is the formula for the zero-filling operation, which is used to perform secondary processing on data with insufficient resolution to maintain the consistency of the overall resolution of the data set. is the inverse Fourier transform, which converts the data back to the source space.

[0066] F W =ψ(h l-1 W) Formula 4;

[0067] W is a convolution layer, and a 1x 1 convolution layer is used in this embodiment; ψ is a formula embodiment of the interpolation operation.

[0068]

[0069] After obtaining the processing results of multiple module strings, the module strings are connected in parallel, and the processing results of m module strings are added together; then they are input into the linear layer, and the 3*m channels are converted into 3 channels through this linear layer; finally, they are processed by the batch normalization layer to obtain the feature recognition results.

[0070] 3. Obtain the hidden layer result of the output of the axis FNO model in step 2 (i.e. the above-mentioned feature recognition result);

[0071] 4. The output hidden layer of the FNO model is put into the pre-trained backbone model as input. In this embodiment, the backbone model selected in actual application is the ResNet model (i.e., the above-mentioned classification model). In theory, various other image recognition models are also suitable;

[0072] 5. Output the judgment and classification results of the backbone model;

[0073] 6. End.

[0074] Through the above method, the FNO-based operator learning method is improved, so that the model has better feature extraction capabilities when facing pictures with low clarity (low pixels or clarity affected by weather and other issues), and has good multi-size image training capabilities; at the same time, the original visual recognition model has a lower recognition threshold than before, and can recognize some low-quality images that originally could not pass the data quality screening, obtain more comprehensive data, and enable the algorithm to play a greater role. At the same time, more comprehensive information is fed back to the companies that apply this algorithm. For example, when the image clarity returned by the camera in foggy weather is low, it will not only improve the recognition accuracy of the target in the image that could originally pass the data filtering, but also can perform image recognition on more images, improve the overall statistical accuracy of the system, and improve the database.

[0075] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0076] In this embodiment, an image classification device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0077] Figure 5 : is a structural block diagram of an image classification device according to an embodiment of the present application, the device comprising:

[0078] A screening module 52, configured to screen out an image to be detected from an image database according to a quality filtering threshold, wherein the image quality score of the image to be detected is lower than the quality filtering threshold;

[0079] A processing module 54 is used to input the image to be detected into a feature recognition model for processing to obtain a feature recognition result, wherein the feature recognition model includes a plurality of feature recognition module strings, the plurality of feature recognition module strings are connected in parallel, each feature recognition module string includes a plurality of feature recognition modules, the plurality of feature recognition modules are connected in series;

[0080] The classification module 56 is used to classify the feature recognition result through a classification model to obtain a classification result.

[0081] Through the above-mentioned device, low-quality images to be detected whose image quality scores are lower than the quality filtering threshold are screened out from the image database, and then the images to be detected are input into the feature recognition model for processing to obtain feature recognition results, the feature recognition module includes multiple feature recognition module strings, the multiple feature recognition module strings are connected in parallel, and the multiple feature recognition modules in a feature recognition module string are connected in series; finally, the feature recognition results are classified by the classification model to obtain classification results; using the above-mentioned scheme, a new feature recognition model is proposed, through which image feature recognition of multiple clarity can be completed, the image classification accuracy is improved, and the cost is reduced; thereby solving the problem in the related technology that the existing technology needs to train multiple models to complete classification for images of different resolutions, and the calculation cost is too high.

[0082] Optionally, the above-mentioned processing module 54 is also used to input the image to be detected into the feature recognition model; process the image to be detected respectively through the multiple feature recognition module strings to obtain multiple sub-feature recognition results; and determine the feature recognition result based on the multiple sub-feature recognition results.

[0083] Optionally, the feature recognition module is an improved FNO model, which is composed of multiple improved FNO operation modules (i.e., feature recognition modules). In the embodiment of the present application, there are n*m ​​operation modules, where n is the number of FNO modules in a module string (i.e., feature recognition module string), and m is the number of FNO module strings; the image to be detected is processed separately by multiple parallel feature recognition module strings to obtain m sub-feature recognition results, and finally the feature recognition result is determined based on these multiple sub-feature recognition results.

[0084] Optionally, after obtaining multiple sub-feature recognition results, these sub-feature recognition results are added and summed, and then the sum result of 3*m channels is converted into 3 channels through the linear layer of the model; finally, the feature recognition result is obtained through batch normalization layer processing.

[0085] Optionally, the above-mentioned processing module 54 is also used to perform Fourier space transform processing on the image to be detected through the first feature recognition module to obtain a first processing result, and to perform convolution layer processing on the image to be detected through the first feature recognition module to obtain a second processing result, wherein the feature recognition module string includes the first feature recognition module; non-linearly map the sum of the first processing result and the second processing result through an activation function to obtain a third processing result; input the third processing result into the second feature recognition module for processing, and input the obtained fourth processing result into the next feature recognition module in the feature recognition module string for processing, until the multiple feature recognition modules are traversed, and a first sub-feature recognition result is obtained, wherein the multiple sub-feature recognition results include the first sub-feature recognition result.

[0086] Specifically, the process of processing the image to be detected by each feature recognition module string includes: first, copying the image to be detected into two copies, one copy is processed by Fourier space transformation to obtain a first processing result, and the other copy is processed by convolution layer to obtain a second processing result; after both copies are calculated, they are added and summed, and then the activation function σ is used for nonlinear mapping to obtain a third processing result. At this time, the processing process of one feature recognition module is completed; then the processing result of the previous feature recognition module is input into the next feature recognition module for the same processing flow, until all the multiple feature recognition modules have completed the processing, and the first sub-feature recognition result, that is, the processing result of a feature recognition module string is obtained.

[0087] Optionally, the above-mentioned processing module 54 is also used to perform Fourier transform on the image to be detected to obtain a first sub-processing result; process the first sub-processing result through an all-pass filter to obtain a second sub-processing result; perform zero-filling operation on the second sub-processing result, and perform inverse Fourier transform on the obtained third sub-processing result to obtain the first processing result.

[0088] To process the Fourier space change of the detected image, the formula To complete, is the Fourier transform, h l-1 Transformed into Fourier space, and then with Perform product operation, h l-1 is the output of the previous feature recognition module (i.e. the hidden vector obtained by the previous module). It is an all-pass filter that retains the three channels of the image and all wave frequency pixels. r and W rrepresents the height and width of the input image resolution r. χ is the formula for the zero-filling operation, which is used to perform secondary processing on data with insufficient resolution to maintain the consistency of the overall resolution of the data set. is the inverse Fourier transform, which converts the data back to the source space.

[0089] Optionally, the processing module 54 is further configured to process the image to be detected through a target convolution layer to obtain a fourth sub-processing result; and perform an interpolation operation on the fourth sub-processing result to obtain the second processing result.

[0090] The image to be detected is processed by convolution layer, which can be obtained by formula F W =ψ(h l-1 W) is completed, W is a convolution layer, and a 1*1 convolution layer is used in the embodiment of the present application; ψ is the formula embodiment of the interpolation operation.

[0091] Optionally, the processing module 54 is further configured to calculate the sum of the first processing result and the second processing result to obtain a fifth processing result; by formula Perform nonlinear mapping on the fifth processing result to obtain the third processing result, wherein GELU(x) is the activation function and x is the fifth processing result.

[0092] First, the first processing result and the second processing result are added to obtain the fifth processing result, and then the fifth processing result is nonlinearly mapped using an activation function. The activation function used in this embodiment is GELU, and the specific calculation formula is: x is the fifth processing result of the input.

[0093] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0094] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0095] S1, filtering out an image to be detected from an image database according to a quality filtering threshold, wherein the image quality score of the image to be detected is lower than the quality filtering threshold;

[0096] S2, inputting the image to be detected into a feature recognition model for processing to obtain a feature recognition result, wherein the feature recognition model includes a plurality of feature recognition module strings, the plurality of feature recognition module strings are connected in parallel, each feature recognition module string includes a plurality of feature recognition modules, the plurality of feature recognition modules are connected in series;

[0097] S3, classifying the feature recognition result through a classification model to obtain a classification result.

[0098] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0099] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0100] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0101] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0102] S1, filtering out an image to be detected from an image database according to a quality filtering threshold, wherein the image quality score of the image to be detected is lower than the quality filtering threshold;

[0103] S2, inputting the image to be detected into a feature recognition model for processing to obtain a feature recognition result, wherein the feature recognition model includes a plurality of feature recognition module strings, the plurality of feature recognition module strings are connected in parallel, each feature recognition module string includes a plurality of feature recognition modules, the plurality of feature recognition modules are connected in series;

[0104] S3, classifying the feature recognition result through a classification model to obtain a classification result.

[0105] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0106] An embodiment of the present application further provides a computer program product, comprising a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program product, and when the computer program is executed by a processor, the steps of the method described in each embodiment of the present application are implemented.

[0107] Optionally, in this embodiment, the above computer program may be configured to implement the following steps when executed by a processor:

[0108] S1, filtering out an image to be detected from an image database according to a quality filtering threshold, wherein the image quality score of the image to be detected is lower than the quality filtering threshold;

[0109] S2, inputting the image to be detected into a feature recognition model for processing to obtain a feature recognition result, wherein the feature recognition model includes a plurality of feature recognition module strings, the plurality of feature recognition module strings are connected in parallel, each feature recognition module string includes a plurality of feature recognition modules, the plurality of feature recognition modules are connected in series;

[0110] S3, classifying the feature recognition result through a classification model to obtain a classification result.

[0111] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0112] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0113] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image classification method, It is characterized in that include: Filtering an image to be detected from an image database according to a quality filtering threshold, wherein the image quality score of the image to be detected is lower than the quality filtering threshold; Inputting the image to be detected into a feature recognition model for processing to obtain a feature recognition result, wherein the feature recognition model includes a plurality of feature recognition module strings, the plurality of feature recognition module strings are connected in parallel, each feature recognition module string includes a plurality of feature recognition modules, the plurality of feature recognition modules are connected in series; The feature recognition result is classified by a classification model to obtain a classification result.

2. The method according to claim 1, It is characterized in that The image to be detected is input into the feature recognition model for processing to obtain a feature recognition result, including: Inputting the image to be detected into the feature recognition model; The image to be detected is processed respectively by the plurality of feature recognition module strings to obtain a plurality of sub-feature recognition results; The feature recognition result is determined according to the multiple sub-feature recognition results.

3. The method according to claim 2, It is characterized in that The image to be detected is processed respectively by the plurality of feature recognition module strings to obtain a plurality of sub-feature recognition results, including: Performing Fourier space variation processing on the image to be detected by a first feature recognition module to obtain a first processing result, and performing convolution layer processing on the image to be detected by the first feature recognition module to obtain a second processing result, wherein the feature recognition module string includes the first feature recognition module; performing nonlinear mapping on the sum of the first processing result and the second processing result by an activation function to obtain a third processing result; The third processing result is input into the second feature recognition module for processing, and the fourth processing result is input into the next feature recognition module in the feature recognition module string for processing, until the multiple feature recognition modules are traversed and a first sub-feature recognition result is obtained, wherein the multiple sub-feature recognition results include the first sub-feature recognition result.

4. The method according to claim 3, It is characterized in that The first feature recognition module performs Fourier space change processing on the image to be detected to obtain a first processing result, including: Performing Fourier transform on the image to be detected to obtain a first sub-processing result; The first sub-processing result is processed by an all-pass filter to obtain a second sub-processing result; the second sub-processing result is zero-filled, and the obtained third sub-processing result is inversely Fourier transformed to obtain the first processing result.

5. The method according to claim 3, It is characterized in that The first feature recognition module performs convolution layer processing on the image to be detected to obtain a second processing result, including: Processing the image to be detected through a target convolution layer to obtain a fourth sub-processing result; An interpolation operation is performed on the fourth sub-processing result to obtain the second processing result.

6. The method according to claim 3, It is characterized in that Performing nonlinear mapping on the sum of the first processing result and the second processing result through an activation function to obtain a third processing result, including: Calculate the sum of the first processing result and the second processing result to obtain a fifth processing result; Perform nonlinear mapping on the fifth processing result to obtain the third processing result, wherein GELU(x) is the activation function and x is the fifth processing result.

7. An image classification device, It is characterized in that include: A screening module, used for screening out the image to be detected from the image database according to the quality filtering threshold, wherein the image quality score of the image to be detected is lower than the quality filtering threshold; A processing module, used for inputting the image to be detected into a feature recognition model for processing to obtain a feature recognition result, wherein the feature recognition model includes a plurality of feature recognition module strings, the plurality of feature recognition module strings are connected in parallel, each feature recognition module string includes a plurality of feature recognition modules, the plurality of feature recognition modules are connected in series; The classification module is used to classify the feature recognition results through a classification model to obtain classification results.

8. A computer-readable storage medium, It is characterized in that The computer-readable storage medium includes a stored program, wherein the program executes the method according to any one of claims 1 to 6 when executed.

9. An electronic device comprising a memory and a processor, It is characterized in that A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 6 through the computer program.

10. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images