A fine-grained image classification method and device
By extracting multi-scale features in fine-grained image classification and combining global features and component features, the problems of large calculation amount and high time delay in the prior art are solved, and efficient fine-grained image classification is achieved.
Patent Information
- Application Number
- CN202410983007.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-07-22
AI Technical Summary
The existing fine-grained image classification methods have large calculation volume and high time delay, making it difficult to meet the needs of edge-end applications.
By acquiring the first feature maps of multiple different scales of the target image, feature enhancement, predict the score map, extract the most distinguished component features, and determine the categories based on global features, reducing the number of feature extractions and reducing the amount of calculation.
It achieves low computational volume and time delay, improves the accuracy of fine-grained image classification, and meets the needs of edge-end applications.
Smart Images

Figure CN118521835B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of image classification, and in particular to a fine-grained image classification method and device. Background Art
[0002] Fine-grained image classification refers to identifying subcategories of objects that are very similar in appearance and semantic information, such as identifying different types of "birds" (such as "crows", "sparrows", "seagulls", etc.), or identifying different types of cars (such as "Mercedes-Benz", "BMW", "Audi", etc.). Fine-grained image recognition has broad application prospects.
[0003] Compared with traditional image classification, the difference and difficulty of fine-grained image recognition are: first, the differences between different subclasses are very subtle, usually reflected in local details, and second, the intra-class differences are relatively large, that is, different objects of the same subclass also have large differences in shape, posture, color, background, etc. In other words, compared with traditional image classification, fine-grained image classification is more difficult.
[0004] The current fine-grained image classification method is to first extract features from the target image, then generate an attention map based on the extracted features, then locate the target area from the target image based on the attention map, then extract features from the target area, and finally determine the category of the target image based on the features extracted from the target area. The current fine-grained image classification method requires secondary feature extraction, and the computational complexity of secondary feature extraction is large. That is, the current fine-grained image classification method has a large computational complexity and high latency, making it difficult to meet the needs of edge applications. Summary of the invention
[0005] In view of this, the present application provides a fine-grained image classification method and device to solve the problem that the current fine-grained image classification method has a large amount of calculation, high latency, and is difficult to meet the needs of edge applications. The technical solution is as follows:
[0006] The first aspect of the present application provides a fine-grained image classification method, comprising:
[0007] Get the target image;
[0008] Obtaining N first feature maps of different scales for the target image, wherein the first feature map of the smallest scale among the N first feature maps is used as a global feature, and N is an integer greater than 1;
[0009] Perform feature enhancement on the N first feature maps to obtain N second feature maps of different scales;
[0010] Predict a score map according to each second feature map, and obtain score maps corresponding to N second feature maps, wherein the score map has the same width and height as the corresponding second feature map, and the score at each position in the score map can represent the discriminability of the feature at the corresponding position in the corresponding second feature map;
[0011] Extracting the most discriminative component features from the N second feature maps according to the score maps respectively corresponding to the N second feature maps;
[0012] The category of the target image is determined according to the global features and the component features.
[0013] In a possible implementation, extracting the most discriminative component feature from the N second feature maps according to the score maps respectively corresponding to the N second feature maps includes:
[0014] For each of the N second feature maps:
[0015] Determine K maximum scores from the score map corresponding to the second feature map, where K is an integer greater than 0;
[0016] The most discriminative component features are extracted from the second feature map according to the positions of the K maximum scores.
[0017] In a possible implementation, acquiring N first feature maps of different scales for the target image; performing feature enhancement on the N first feature maps to obtain N second feature maps of different scales; predicting a score map according to each second feature map to obtain score maps corresponding to the N second feature maps; extracting the most discriminative component features from the N second feature maps according to the score maps corresponding to the N second feature maps; and determining the category of the target image according to the global features and the component features, includes:
[0018] Using a pre-trained fine-grained image classification model, obtain N first feature maps of different scales for the target image;
[0019] Using the fine-grained image classification model, feature enhancement is performed on the N first feature maps to obtain N second feature maps of different scales;
[0020] Using the fine-grained image classification model, based on each second feature map, a score map is predicted to obtain score maps corresponding to the N second feature maps respectively;
[0021] Using the fine-grained image classification model, based on the score maps corresponding to the N second feature maps, the most discriminative component features are extracted from the N second feature maps;
[0022] Determining the category of the target image by using the fine-grained image classification model based on the global features and the component features;
[0023] The fine-grained image classification model is trained using training images that are only labeled with categories.
[0024] In a possible implementation, the training process of the fine-grained image classification model includes:
[0025] Using a fine-grained image classification model, obtaining the global features of the training image, N second feature maps of different scales, score maps and classification maps corresponding to the N second feature maps, and the most discriminative component features of the training image, wherein the value at each position in the classification map is the probability corresponding to a plurality of set categories predicted according to the feature at the corresponding position in the corresponding second feature map;
[0026] Using a fine-grained image classification model, based on the global features and the most discriminative component features of the training image, predict the probability distribution of the training image on the plurality of set categories;
[0027] Determine the prediction loss for the training image according to the score map and the classification map respectively corresponding to the N second feature maps of the training image, the probability distribution of the training image on the multiple set categories, and the category annotated by the training image;
[0028] Parameters of the fine-grained image classification model are updated according to the prediction loss for the training image.
[0029] In a possible implementation, determining the prediction loss for the training image according to the score map and the classification map respectively corresponding to the N second feature maps of the training image, the probability distribution of the training image on the multiple set categories, and the category annotated by the training image includes:
[0030] Determine a first category prediction loss for the training image according to the probability distribution of the training image on the multiple set categories and the category annotated by the training image;
[0031] Determine a second category prediction loss for the training image according to classification maps respectively corresponding to the N second feature maps of the training image and the category annotated by the training image;
[0032] The feature score prediction loss for the training image is determined according to the score map and the classification map respectively corresponding to the N second feature maps of the training image and the category annotated by the training image.
[0033] In a possible implementation, the training process of the fine-grained image classification model further includes:
[0034] Using a fine-grained image classification model, generating attention maps corresponding to the N second feature maps of the training image respectively;
[0035] Processing the attention maps corresponding to the N second feature maps of the training image into binary mask maps, to obtain binary mask maps corresponding to the N second feature maps of the training image;
[0036] The step of determining the feature score prediction loss for the training image according to the score map and the classification map respectively corresponding to the N second feature maps of the training image and the category annotated by the training image comprises:
[0037] Multiplying the N second feature maps of the training image by the corresponding binary mask maps respectively to obtain the latest classification maps corresponding to the N second feature maps of the training image respectively;
[0038] Determine the feature score prediction loss for the training image according to the category annotated by the training image and the score map and the latest classification map respectively corresponding to the N second feature maps of the training image.
[0039] In a possible implementation, determining the feature score prediction loss for the training image according to the category annotated by the training image and the score map and the latest classification map respectively corresponding to the N second feature maps of the training image includes:
[0040] Obtain the probability corresponding to the category marked by the training image from multiple probabilities at each position in the latest classification map corresponding to each second feature map of the training image, so as to obtain the target classification maps corresponding to the N second feature maps of the training image respectively;
[0041] Determine the feature score prediction loss according to the target classification map and the score map respectively corresponding to the N second feature maps of the training image.
[0042] In a possible implementation, the training process of the fine-grained image classification model further includes:
[0043] Performing enhancement processing on the training image to obtain a processed training image;
[0044] Using a fine-grained image classification model, obtaining global features and most discriminative component features of the processed training image;
[0045] Using a fine-grained image classification model, based on the global features and the most discriminative component features of the processed training images, predict the probability distribution of the processed training images on the multiple set categories;
[0046] Determining a probability distribution consistency loss according to the probability distribution of the training image on the multiple set categories and the probability distribution of the processed training image on the multiple set categories;
[0047] Determining a category prediction loss for the processed training image according to the probability distribution of the processed training image on the multiple set categories and the category annotated by the training image;
[0048] The updating of parameters of the fine-grained image classification model according to the prediction loss of the training image comprises:
[0049] Parameters of a fine-grained image classification model are updated according to the prediction loss for the training image, the probability distribution consistency loss, and the category prediction loss for the processed training image.
[0050] In a possible implementation manner, the performing enhancement processing on the training image includes:
[0051] Based on a predetermined homography matrix, a homography transformation is performed on the training image.
[0052] A second aspect of the present application provides a fine-grained image recognition device, comprising: an image acquisition module, a multi-scale feature acquisition module, a feature processing module, an information prediction module, a component feature acquisition module, and an image category determination module;
[0053] The image acquisition module is used to acquire a target image;
[0054] The multi-scale feature acquisition module is used to acquire N first feature maps of different scales for the target image, wherein the first feature map of the smallest scale among the N first feature maps is used as the global feature, and N is an integer greater than 1;
[0055] The feature processing module is used to perform feature enhancement on the N first feature maps to obtain N second feature maps of different scales;
[0056] The information prediction module is used to predict a score map according to each second feature map to obtain score maps corresponding to the N second feature maps, wherein the score map has the same width and height as the corresponding second feature map, and the score of each position in the score map can characterize the discriminability of the feature at the corresponding position in the corresponding second feature map;
[0057] The component feature acquisition module is used to extract the most discriminative component feature from the N second feature maps according to the score maps respectively corresponding to the N second feature maps;
[0058] The image category determination module is used to determine the category of the target image according to the global feature and the component feature.
[0059] By means of the above technical scheme, the fine-grained image classification method provided by the present application, after obtaining the target image, first obtains multiple first feature maps of different scales for the target image, wherein the feature map of the smallest scale is used as the global feature of the target image, and then performs feature enhancement on the multiple first feature maps to obtain N second feature maps of different scales, and then predicts the score map according to each second feature map to obtain the score maps corresponding to the N second feature maps, and then extracts the most discriminative component features from the multiple second feature maps according to the score maps corresponding to the multiple second feature maps, and finally determines the category of the target image according to the global features and the most discriminative component features. Since the fine-grained image classification method provided by the present application only needs to perform feature extraction once, the amount of calculation is small and the delay is short. Since the fine-grained image classification method provided by the present application determines the category of the target image based on the global features combined with the most discriminative component features, a relatively accurate fine-grained category can be obtained. In summary, the fine-grained image classification method provided by the present application has a low amount of calculation, a short delay and a high fine-grained classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0061] Figure 1 A schematic diagram of a hardware structure of a terminal provided in an embodiment of the present application;
[0062] Figure 2 A schematic diagram of a hardware structure of a server provided in an embodiment of the present application;
[0063] Figure 3 A schematic diagram of the process of a fine-grained image classification method provided in an embodiment of the present application;
[0064] Figure 4 A schematic diagram of a first training method for a fine-grained image classification model provided in an embodiment of the present application;
[0065] Figure 5 A schematic diagram of the structure of a fine-grained image classification model provided in an embodiment of the present application;
[0066] Figure 6A schematic diagram of obtaining identification component features provided by an embodiment of the present application;
[0067] Figure 7 A schematic diagram of a second training method for a fine-grained image classification model provided in an embodiment of the present application;
[0068] Figure 8 A schematic diagram of the training process of the fine-grained image classification model provided in an embodiment of the present application;
[0069] Fig. 9 A schematic diagram of the structure of a fine-grained image classification device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0070] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.
[0071] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0072] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0073] The fine-grained image classification method provided in this application can be applied to terminals as well as servers.
[0074] Next, the product form of the above-mentioned terminal is described.
[0075] The above-mentioned terminal can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiments of the present application do not impose any restrictions on this.
[0076] Figure 1 An optional hardware structure diagram of a terminal is shown.
[0077] refer to Figure 1 As shown, the terminal may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 1 This is merely an example of a terminal and does not constitute a limitation on the terminal. The terminal may include more or fewer components than shown in the figure, or a combination of certain components, or different components.
[0078] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the terminal. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.
[0079] The display unit 140 may be used to display information input by a user or information provided to a user, various menus of the terminal, an interactive interface, file display, and / or playback of any multimedia file.
[0080] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.
[0081] The processor 170 is the control center of the terminal. It uses various interfaces and lines to connect various parts of the entire terminal. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.
[0082] Among them, the memory 120 can be used to store software codes related to the fine-grained image classification method, the processor 170 can call the software code stored in the memory 120, and can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to implement corresponding functions.
[0083] The radio frequency unit 110 (optional) can be used to receive and send information or receive and send signals during a call. For example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the radio frequency unit 110 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LowNoiseAmplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General PacketRadio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.
[0084] In the embodiment of the present application, the radio frequency unit 110 can send data to other devices, and can also receive data sent by other devices. It should be understood that the radio frequency unit 110 is optional, and can be replaced by other communication interfaces, such as a network port.
[0085] The terminal also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions.
[0086] The terminal also includes an external interface 180, which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal to communicate with other devices, or to connect a charger to charge the terminal.
[0087] Although not shown, the terminal may further include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be described in detail here.
[0088] Next, the product form of the above server is described.
[0089] Figure 2 A structural diagram of the above server is provided, such as Figure 2As shown, the server includes a bus 201, a processor 202, a communication interface 203 and a memory 204. The processor 202, the memory 204 and the communication interface 203 communicate with each other via the bus 201.
[0090] The bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 2 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0091] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).
[0092] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard drive (HDD), or a solid state drive (SSD).
[0093] The memory 204 may be used to store software codes related to the fine-grained image classification method, and the processor 202 may call the software codes stored in the memory 204 or schedule other units to implement corresponding functions.
[0094] It should be understood that the above-mentioned terminals and servers can be centralized or distributed devices, and the processors in the above-mentioned terminals and servers (such as processor 170 and processor 202) can be hardware circuits (such as application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), general-purpose processors, digital signal processors (DSP), microprocessors or microcontrollers, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.
[0095] Next, the fine-grained image classification method provided by this application is introduced through the following embodiments.
[0096] See also Figure 3 , shows a schematic flow chart of a fine-grained image classification method provided in an embodiment of the present application, which may include:
[0097] Step S301: Acquire a target image.
[0098] The target image is an image of a category to be determined.
[0099] Step S302: Obtain N first feature maps of different scales for the target image.
[0100] Among them, the first feature map with the smallest scale among the N first feature maps is used as the global feature of the target image, and N is an integer greater than 1.
[0101] Exemplarily, N=3, and three first feature maps of different scales can be obtained for the target image, and the first feature map of the smallest scale among the three first feature maps of different scales can be used as the global feature of the target image.
[0102] Step S303: performing feature enhancement on the N first feature maps of the target image to obtain N second feature maps of different scales of the target image.
[0103] The feature enhancement is used to integrate the feature information of the first feature maps of other scales into the first feature map of each scale, that is, each second feature map contains feature information of multiple scales.
[0104] Step S304: predicting a score map according to each second feature map of the target image, and obtaining score maps corresponding to the N second feature maps of the target image.
[0105] Among them, the score map has the same width and height as the corresponding second feature map, and the score at each position in the score map can represent the discriminability of the feature at the corresponding position in the corresponding second feature map.
[0106] Step S305: extracting the most discriminative component feature from the N second feature maps of the target image according to the score maps respectively corresponding to the N second feature maps of the target image.
[0107] In a possible implementation, according to the score maps respectively corresponding to the N second feature maps of the target image, the process of extracting the most discriminative component features from the N second feature maps of the target image may include: for each of the N second feature maps of the target image, determining K maximum scores from the score map corresponding to the second feature map, and extracting the most discriminative component features from the second feature map according to the positions of the K maximum scores (the features corresponding to the K maximum scores in the second feature map are the most discriminative component features). K is an integer greater than 0, and the specific value of K may be determined according to the specific application scenario.
[0108] Step S306: Determine the category of the target image based on the global features and the most discriminative component features of the target image.
[0109] The embodiment of the present application determines the category based on two aspects of features (i.e., global features and the most discriminative component features) at the same time. Combining the most discriminative component features on the basis of global features can better distinguish fine-grained categories.
[0110] The fine-grained image classification method provided in the embodiment of the present application, after obtaining the target image, first obtains multiple first feature maps of different scales for the target image, wherein the feature map of the smallest scale is used as the global feature of the target image, and then performs feature enhancement on the multiple first feature maps to obtain N second feature maps of different scales, and then predicts the score map according to each second feature map to obtain the score maps corresponding to the N second feature maps, and then extracts the most discriminative component features from the multiple second feature maps according to the score maps corresponding to the multiple second feature maps, and finally determines the category of the target image according to the global features and the most discriminative component features. Since the fine-grained image classification method provided in the embodiment of the present application only needs to perform feature extraction once, the amount of calculation is small and the delay is short. Since the present application determines the category of the target image based on two aspects of features (i.e., global features and the most discriminative local features), a relatively accurate classification result can be obtained. In summary, the fine-grained image classification method provided in the embodiment of the present application has a low amount of calculation, a short delay and a high accuracy of fine-grained classification.
[0111] In a possible implementation, steps S302 to S306 in the above embodiment can be implemented using a model, that is, a pre-trained fine-grained image classification model is used to obtain N first feature maps of different scales for the target image, and the fine-grained image classification model is used to perform feature enhancement on the N first feature maps of the target image to obtain N second feature maps of different scales of the target image, and the fine-grained image classification model is used to predict a score map based on each second feature map of the target image to obtain score maps corresponding to the N second feature maps of the target image, and the fine-grained image classification model is used to extract the most discriminative component features from the N second feature maps of the target image based on the score maps corresponding to the N second feature maps of the target image, and the fine-grained image classification model is used to determine the category of the target image based on the global features of the target image and the most discriminative component features. The fine-grained image classification model is trained using training images that are only labeled with categories.
[0112] Next, the training process of the fine-grained image classification model is introduced.
[0113] There are many ways to train a fine-grained image classification model. This embodiment provides two optional training methods.
[0114] See also Figure 4 , which shows a flow chart of the first training method of the fine-grained image classification model, which may include:
[0115] Step S401: using a fine-grained image classification model, obtaining global features of a training image, N second feature maps of different scales, score maps and classification maps corresponding to the N second feature maps, and the most discriminative component features of the training image.
[0116] Specifically, the specific implementation process of step S401 may include:
[0117] Step a1: Using a fine-grained image classification model, obtain N first feature maps of different scales for the training image.
[0118] Among the N first feature maps of different scales of the training image, the first feature map with the smallest scale is used as the global feature of the training image.
[0119] In one possible implementation, Figure 5 As shown, the fine-grained image classification model may include a feature extraction module 501, and the training image may be input into the feature extraction module 501 for feature extraction to obtain N first feature maps of different scales. Optionally, the feature extraction module 501 may use a ResNet-50 network. Of course, this embodiment is not limited thereto. For example, the feature extraction module 501 may also use a Swin-T network.
[0120] Exemplarily, the feature extraction module 501 uses a ResNet-50 network, which includes multiple stages. The first feature maps of N different scales of the above-mentioned training image can be feature maps C3, C4, and C5 output by the last three stages of the ResNet-50 network.
[0121] Step a2: Using a fine-grained image classification model, perform feature enhancement on N first feature maps of the training image to obtain N second feature maps of different scales of the training image.
[0122] like Figure 5 As shown, the fine-grained image classification model may include a discriminative component feature acquisition module 502. Further, the discriminative component feature acquisition module 502 may include a feature enhancement module 5021. The N first feature maps of the training image may be input into the feature enhancement module 5021 for feature enhancement processing to obtain N second feature maps of the training image. Optionally, the feature enhancement module 5021 may use a feature pyramid network. Figure 6 , Figure 6 The left half shows a schematic diagram of P3, P4, and P5 obtained by performing feature enhancement on the above C3, C4, and C5 by the feature pyramid network. P3, P4, and P5 are second feature maps of three different scales.
[0123] Step a3: Using the fine-grained image classification model, based on each second feature map of the training image, predict the score map and the classification map, and obtain the score map and the classification map corresponding to the N second feature maps of the training image.
[0124] Among them, the score map and the classification map have the same width and height as the corresponding second feature map. The score at each position in the score map can characterize the discriminability of the feature at the corresponding position in the corresponding second feature map. The value at each position in the classification map is the probability corresponding to multiple set categories predicted based on the feature at the corresponding position in the corresponding second feature map.
[0125] For the i-th second feature map of the training image, it can be processed by a 1×1 convolution layer to obtain an initial score map with a dimension of 1, a width and a height, and an initial classification map with a dimension of C (C is the number of set categories) and a width and a height. After obtaining the initial score map and the initial classification map, the values of the initial score map and the initial classification map are processed into values within the range of (0, 1) (for example, a sigmoid operation can be performed on the initial score map and the initial classification map), and the processed score map is used as the final score map, and the processed classification map is used as the final classification map.
[0126] like Figure 5 As shown, the discriminative component feature acquisition module 502 may also include a score map and classification map prediction module 5022, and the N second feature maps of the training image may be input into the score map and classification map prediction module 5022. The score map and classification map prediction module 5022 predicts a score map and a classification map based on each second feature map of the training image, and outputs a score map and a classification map corresponding to the N second feature maps of the training image.
[0127] Step a4: using the fine-grained image classification model, based on the score maps corresponding to the N second feature maps of the training image, the most discriminative component features are extracted from the N second feature maps of the training image.
[0128] Specifically, for each second feature map of the training image, K maximum scores are determined from the score map corresponding to the second feature map, and the most discriminative component features are extracted from the second feature map according to the positions of the K maximum scores. Figure 6 As shown in FIG. 1 , after obtaining the score map corresponding to P5, the K maximum scores (i.e., Top K scores) are determined from the score map corresponding to P5. According to the positions of the K maximum scores, features are extracted from the corresponding positions of P5. The extracted features are the most discriminative component features. The most discriminative component features are extracted from P3 and P4 in the same way.
[0129] like Figure 5As shown, the discriminative component feature acquisition module may further include a discriminative component feature extraction module 5023, which may input the score maps corresponding to the N second feature maps of the training image and the N second feature maps of the training image into the discriminative component feature extraction module 5023, and the discriminative component feature extraction module 5023 extracts the most discriminative component feature from the N second feature maps of the training image according to the score maps corresponding to the N second feature maps of the training image.
[0130] It should be noted that during the training process, the discriminative component feature acquisition module 502 can effectively learn the discriminative component features, so that the model can better distinguish fine-grained categories.
[0131] Step S402: using a fine-grained image classification model, based on the global features and the most discriminative component features of the training image, predict the probability distribution of the training image in multiple set categories.
[0132] like Figure 5 As shown, the fine-grained image classification model can also include a category prediction module 503, which can splice the global features of the training image with the most discriminative component features of the training image, and input the spliced features into the category prediction module 503. The category prediction module 503 predicts the probability distribution of the training image in multiple set categories based on the input features.
[0133] Step S403: Determine the prediction loss for the training image according to the score map and the classification map respectively corresponding to the N second feature maps of the training image, the probability distribution of the training image in multiple set categories, and the category annotated by the training image.
[0134] The specific implementation process of step S403 may include:
[0135] Step S403a: Determine the first category prediction loss for the training image according to the probability distribution of the training image in multiple set categories and the category annotated by the training image.
[0136] The first category prediction loss for the training image can adopt the cross entropy loss, which can be expressed as:
[0137] (1)
[0138] Among them, θ represents the parameters of the above feature extraction module (such as ResNet-50 network), represents the training image, represents the probability distribution of training images in multiple set categories (this probability distribution is an unnormalized probability distribution), and y represents the category annotated by the training image.
[0139] Step S403b: Determine the second category prediction loss for the training image according to the classification maps corresponding to the N second feature maps of the training image and the category annotated by the training image.
[0140] The Focal loss can be used for the second category prediction loss of the training image. Specifically, for each second feature map of the training image, the Focal loss is determined according to the second feature map and the category annotated by the training image. After obtaining the Focal losses corresponding to the N second feature maps of the training image, the Focal losses corresponding to the N second feature maps of the training image can be summed to obtain the second category prediction loss for the training image. .
[0141] Step S403c: Determine the feature score prediction loss for the training image according to the score map and the classification map respectively corresponding to the N second feature maps of the training image and the category annotated by the training image.
[0142] In a possible implementation, the feature score prediction loss for the training image can be determined directly based on the score map and the classification map respectively corresponding to the N second feature maps of the training image and the category annotated by the training image.
[0143] Considering that there is a large part of background area in the training image, these areas should not be supervised. In order to obtain better training effect, in another possible implementation, a fine-grained image classification model can be used to generate attention maps corresponding to the N second feature maps of the training image, and the attention maps corresponding to the N second feature maps of the training image are processed into binary mask maps to obtain binary mask maps corresponding to the N second feature maps of the training image. Then, when determining the feature score prediction loss for the training image according to the score map and classification map corresponding to the N second feature maps of the training image and the category annotated by the training image, the classification map corresponding to the N second feature maps of the training image can be multiplied by the corresponding binary mask map to obtain the latest classification map corresponding to the N second feature maps of the training image, and then the feature score prediction loss for the training image can be determined according to the category annotated by the training image and the score map and the latest classification map corresponding to the N second feature maps of the training image.
[0144] like Figure 5 As shown, the fine-grained image classification model can also include an attention module 504, and the N second feature maps of the training image can be input into the attention module 504. The attention module 504 generates an attention map according to each second feature map of the training image to obtain attention maps corresponding to the N second feature maps of the training image.
[0145] In one possible implementation, the attention module 504 may generate the i-th second feature map of the training image in the following manner: The corresponding attention map :First, the second feature map Perform the average and maximum calculations on the channel dimension, expressed as as well as , then, and They are cascaded together and then subjected to m×m convolution, batch normalization, RELU, m×m convolution, and Sigmoid operations, where m represents the convolution kernel size.
[0146] After obtaining the i-th second feature map of the training image The corresponding attention map After that, it can be processed into a binary mask image Specifically, we can first calculate The average , and then press the formula , to obtain a binary mask image :
[0147] (2)
[0148] in, ,visible, The value of each position is either 0 or 1.
[0149] Obtained in the above manner Afterwards, you can The second feature map of the i-th training image The corresponding classification diagram Multiply, and the multiplication result is used as the i-th second feature map Corresponding latest classification map ( ),Will and Multiplying together, we only have Only positions with a median value of 1 will be supervised.
[0150] like Figure 6 As shown in the figure, the attention module generates an attention map based on P5. After obtaining the attention map, the attention map is processed into a binary mask map, and then the binary mask map is multiplied by the classification map corresponding to P5 to obtain the latest classification map corresponding to P5. The same method can be used to obtain the latest classification maps corresponding to P3 and P4 respectively.
[0151] After obtaining the latest classification maps corresponding to the N second feature maps of the training image in the above manner, the feature score prediction loss for the training image can be determined according to the category annotated by the training image and the score map and the latest classification map corresponding to the N second feature maps of the training image. The specific determination process of the feature score prediction loss is as follows:
[0152] Step b1, obtaining the probability corresponding to the category annotated in the training image from multiple probabilities at each position in the latest classification map corresponding to each second feature map of the training image, so as to obtain the target classification maps corresponding to the N second feature maps of the training image respectively.
[0153] Step b2: Determine the feature score prediction loss according to the target classification map and the score map corresponding to the N second feature maps of the training image.
[0154] Specifically, the feature score prediction loss corresponding to the i-th second feature map of the training image can be determined as follows:
[0155] (3)
[0156] Among them, H i , W i , C represent the height, width, and dimension of the i-th second feature map of the training image respectively, Represents the score map corresponding to the i-th second feature map of the training image, Represents the target classification map corresponding to the i-th second feature map of the training image.
[0157] The feature score prediction loss corresponding to the N second feature maps of the training image can be summed to obtain the final feature score prediction loss .
[0158] Step S404: Update the parameters of the fine-grained image classification model according to the prediction loss for the training image.
[0159] The first category prediction loss for the training image (the above ), the second category prediction loss for the training images (the above ) and the feature score prediction loss for the training images (the above ) fusion, and update the parameters of the fine-grained image classification model according to the fusion loss. It can be expressed as:
[0160] (4)
[0161] Among them, α is The corresponding weight, β, is The corresponding weight, γ, is The specific values of the corresponding weights, α, β, and γ, can be set according to the application scenario. For example, α=β=γ=1 can be set.
[0162] It should be noted that the training goal of the fine-grained image classification model in this embodiment is to make the categories predicted based on the global features and the most discriminative component features of the training image consistent with the categories annotated with the training image, to make the categories predicted based on the N second features of the training image consistent with the categories annotated with the training image, and to make the scores in the score map corresponding to the N second features of the training image approach the corresponding probability values in the corresponding target classification map.
[0163] The fine-grained image classification model is trained for multiple rounds in the above manner until the training end condition is met (for example, the model converges, or a preset number of training rounds is reached). After the training is completed, a fine-grained image classification model with high fine-grained classification accuracy can be obtained.
[0164] See also Figure 7 , showing a flow chart of a second training method for a fine-grained image classification model, which may include:
[0165] Step S701: performing enhancement processing on the training image to obtain a processed training image.
[0166] In a possible implementation, a homography transformation may be performed on the training image based on a predetermined homography matrix.
[0167] The homography matrix can be determined as follows: first initialize four pairs of points, namely pt1: [[0, 0], [0,1], [1, 1], [1, 0]], pt2: [[0, 0], [0, 1], [1, 1], [1, 0]], then use a truncated normal distribution to sample within a predefined range of translation, scaling, in-plane rotation, and symmetric perspective distortion, and then apply these operations to pt2 to obtain the transformed points , assuming that the homography matrix to be solved is H, we can get the following equation:
[0168] (5)
[0169] The polynomial solver function in NumPy can be used to solve equation (5) to obtain the value of H. Assuming that I represents the original training image, the homography transformation of I can be performed by the following equation to obtain the transformed image I′:
[0170] (6)
[0171] It should be noted that applying random homography transformation to training images can make the fine-grained image classification model more robust to some difficult samples caused by pose changes.
[0172] like Figure 8 As shown, after the training image is enhanced, the original training image and the processed training image are input into the fine-grained image classification model for processing.
[0173] Step S702a: Using the fine-grained image classification model, obtain the global features of the training image, N second feature maps of different scales, the score maps corresponding to the N second feature maps, the classification map, and the most discriminative component features of the training image.
[0174] The specific implementation process and related description of step S702a can refer to the specific implementation process and related description of step S401, which will not be described in detail in this embodiment.
[0175] Step S702b: Using the fine-grained image classification model, obtain the global features and the most discriminative component features of the processed training image.
[0176] The implementation process of obtaining the global features and the most discriminative component features of the processed training image using the fine-grained image classification model is similar to the implementation process of obtaining the global features and the most discriminative component features of the training image using the fine-grained image classification model, and this embodiment will not be repeated here.
[0177] Step S703a: Using the fine-grained image classification model, based on the global features and the most discriminative component features of the training image, predict the probability distribution of the training image in multiple set categories.
[0178] The specific implementation process and related description of step S703a can refer to the specific implementation process and related description of step S402, which will not be described in detail in this embodiment.
[0179] Step S703b: using the fine-grained image classification model, based on the global features and the most discriminative component features of the processed training image, predict the probability distribution of the processed training image in multiple set categories.
[0180] The global features of the training image can be spliced with the most discriminative component features, and the fine-grained image classification model can be used to predict the probability distribution of the processed training image on multiple set categories based on the spliced features.
[0181] Step S704a: Determine the prediction loss for the training image according to the score map and the classification map respectively corresponding to the N second feature maps of the training image, the probability distribution of the training image in multiple set categories, and the category annotated by the training image.
[0182] The specific implementation process and related description of step S704a can refer to the specific implementation process and related description of step 403, which will not be described in detail in this embodiment.
[0183] Step S704b: Determine the category prediction loss for the processed training image according to the probability distribution of the processed training image in multiple set categories and the category annotated by the training image.
[0184] The cross entropy loss can be used for the category prediction loss of the processed training image, which can be expressed as:
[0185] (7)
[0186] in, represents the processed training image, Represents the processed training image The probability distribution over multiple set categories (this probability distribution is an unnormalized probability distribution, such as Figure 8 logits_aug in ), y represents the category of the original training image annotation, and Figure 8 Logits_raw in is the training image The probability distribution over multiple set categories (this probability distribution is an unnormalized probability distribution).
[0187] Step S704c: Determine the probability distribution consistency loss according to the probability distribution of the training image on the multiple set categories and the probability distribution of the processed training image on the multiple set categories.
[0188] The probability distribution consistency loss can be determined as follows:
[0189] (8)
[0190] in, represents the probability distribution of the original training image on multiple set categories (normalized probability distribution), Represents the processed training image The probability distribution over multiple set categories (normalized probability distribution), represents the softmax function, represents the Kullback-Leibler (KL) divergence, and T represents the distillation temperature.
[0191] It should be noted that introducing probability distribution consistency loss based on homography transformation of training images can prompt the fine-grained image classification model to mine richer information, so as to improve the classification accuracy and robustness of the fine-grained image classification model to changes in target posture.
[0192] Step S705: Update the parameters of the fine-grained image classification model according to the prediction loss for the training image, the category prediction loss for the processed training image, and the probability distribution consistency loss.
[0193] The prediction loss for the training images (including the above , , ), the category prediction loss for the processed training images (the above ) and probability distribution consistency loss (the above ) fusion, and update the parameters of the fine-grained image classification model according to the fusion loss.
[0194] Among them, the loss after fusion It can be expressed as:
[0195] (9)
[0196] Among them, α is The corresponding weight, β, is The corresponding weight, γ, is The corresponding weight μ is The corresponding weight, δ is The specific values of the corresponding weights, α, β, γ, μ, and δ, can be set according to the application scenario. For example, α=β=γ=μ=δ=1 can be set.
[0197] It should be noted that the training goal of the fine-grained image classification model in this embodiment is to make the categories predicted based on the global features and the most discriminative component features of the training image consistent with the categories annotated with the training image, to make the categories predicted based on the N second features of the training image consistent with the categories annotated with the training image, to make the scores in the score map corresponding to the N second features of the training image approach the corresponding probability values in the corresponding target classification map, to make the categories predicted based on the global features and the most discriminative component features of the processed training image consistent with the categories annotated with the training image, and to make the probability distribution of the training image on multiple set categories consistent with the probability distribution of the processed training image on multiple set categories.
[0198] The fine-grained image classification model is trained for multiple rounds in the above manner until the training end condition is met (for example, the model converges, or a preset number of training rounds is reached). After the training is completed, a fine-grained image classification model with higher fine-grained classification accuracy and more robust to target posture changes can be obtained. In addition, it should be noted that since the training images used to train the fine-grained image classification model in this application only need to be labeled with categories, and do not need to be labeled with discriminative components, the implementation cost is relatively low.
[0199] After testing, the fine-grained image classification model trained in the above manner can achieve high classification accuracy on the three datasets of CUB-2002-2011, StanfordCars and FGVC Aircraft. The following table shows the classification accuracy on the three datasets when the feature extraction model in the fine-grained image classification model adopts the ResNet-50 network, and the classification accuracy on the three training datasets when the feature extraction model in the fine-grained image classification model adopts the Swin-T network.
[0200] Table 1 Classification accuracy of fine-grained image classification models on three datasets
[0201]
[0202] In addition, the performance of the discriminative component feature acquisition module in the fine-grained image classification model was tested using the above three data sets. After testing, in the bird recognition task, the components corresponding to the features acquired by the discriminative component feature acquisition module mainly include the head, torso, wings, etc. In vehicle model recognition, the components corresponding to the features acquired by the discriminative component feature acquisition module mainly include tires, lights and body, etc. In the aircraft recognition task, the components corresponding to the features acquired by the discriminative component feature acquisition module include wings, nose and tail, etc. It can be seen that the discriminative component feature acquisition module of the fine-grained image classification model in the present application can effectively acquire the features of discriminative components.
[0203] After the fine-grained image classification model is obtained by training in any of the above methods, the feature extraction module in the fine-grained image classification model can be used to obtain N first feature maps of different scales for the target image, wherein the first feature map of the smallest scale is used as the global feature. Then, the feature enhancement module in the fine-grained image classification model is used to enhance the features of the N first feature maps of the target image to obtain N second feature maps of different scales of the target image. Then, the score map and classification map prediction module in the fine-grained image classification model are used to predict the score map based on each second feature map of the target image to obtain the score maps corresponding to the N second feature maps of the target image. Then, the discriminative component feature extraction module of the fine-grained image classification model is used to extract the most discriminative component features from the N second feature maps of the target image based on the score maps corresponding to the N second feature maps of the target image. Finally, the category prediction module of the fine-grained image classification model is used to determine the category of the target image based on the global features and the most discriminative component features of the target image.
[0204] The above introduces the fine-grained image classification method provided by the embodiment of the present application. The following will introduce the device corresponding to the above-mentioned fine-grained image classification method.
[0205] See also Fig. 9 , Fig. 9 A schematic diagram of the structure of a fine-grained image classification device provided in an embodiment of the present application, wherein the fine-grained image classification device includes: an image acquisition module 901, a multi-scale feature acquisition module 902, a feature processing module 903, an information prediction module 904, a component feature acquisition module 905 and an image category determination module 906.
[0206] The image acquisition module 901 is used to acquire a target image.
[0207] The multi-scale feature acquisition module 902 is used to acquire N first feature maps of different scales for the target image, wherein the first feature map of the smallest scale among the N first feature maps is used as the global feature, and N is an integer greater than 1.
[0208] The feature processing module 903 is used to perform feature enhancement on the N first feature maps to obtain N second feature maps of different scales.
[0209] The information prediction module 904 is used to predict a score map based on each second feature map to obtain score maps corresponding to N second feature maps, wherein the score map has the same width and height as the corresponding second feature map, and the score of each position in the score map can characterize the discriminability of the feature at the corresponding position in the corresponding second feature map.
[0210] The component feature acquisition module 905 is used to extract the most discriminative component feature from the N second feature maps according to the score maps respectively corresponding to the N second feature maps.
[0211] The image category determination module 906 is used to determine the category of the target image according to the global features and the component features.
[0212] In a possible implementation, when the component feature acquisition module 905 extracts the most discriminative component feature from the N second feature maps according to the score maps respectively corresponding to the N second feature maps, it is specifically used to:
[0213] For each second feature map in the N second feature maps, K maximum scores are determined from the score map corresponding to the second feature map, where K is an integer greater than 0; and the most discriminative component feature is extracted from the second feature map according to the positions of the K maximum scores.
[0214] In one possible implementation, the multi-scale feature acquisition module 902, the feature processing module 903, the information prediction module 904, the component feature acquisition module 905 and the image category determination module 906 can be implemented by a pre-trained fine-grained image classification model, which is trained using training images that are only labeled with categories.
[0215] The fine-grained image classification device provided in the embodiment of the present application may also include a model training module. The model training module is used to:
[0216] Using a fine-grained image classification model, obtaining global features of a training image, N second feature maps of different scales, score maps and classification maps corresponding to the N second feature maps, and the most discriminative component features of the training image, wherein the value at each position in the classification map is the probability corresponding to a plurality of set categories predicted according to the feature at the corresponding position in the corresponding second feature map;
[0217] Using a fine-grained image classification model, based on the global features and the most discriminative component features of the training image, predict the probability distribution of the training image on the plurality of set categories;
[0218] Determine the prediction loss for the training image according to the score map and the classification map respectively corresponding to the N second feature maps of the training image, the probability distribution of the training image in multiple set categories, and the category annotated by the training image;
[0219] Parameters of a fine-grained image classification model are updated based on the prediction loss for training images.
[0220] In a possible implementation, when the model training module determines the prediction loss for the training image according to the score map and the classification map respectively corresponding to the N second feature maps of the training image, the probability distribution of the training image in multiple set categories, and the category annotated by the training image, it is specifically used to:
[0221] Determine a first category prediction loss for the training image according to a probability distribution of the training image on a plurality of set categories and a category annotated by the training image;
[0222] Determine a second category prediction loss for the training image according to the classification maps corresponding to the N second feature maps of the training image and the category annotated by the training image;
[0223] The feature score prediction loss for the training image is determined according to the score map and the classification map respectively corresponding to the N second feature maps of the training image and the category annotated by the training image.
[0224] In a possible implementation, the model training module is also used to:
[0225] A fine-grained image classification model is used to generate attention maps corresponding to N second feature maps of the training image. The attention maps corresponding to the N second feature maps of the training image are processed into binary mask maps to obtain binary mask maps corresponding to the N second feature maps of the training image.
[0226] The model training module is specifically used to determine the feature score prediction loss for the training image according to the score map and the classification map respectively corresponding to the N second feature maps of the training image and the category annotated by the training image:
[0227] Multiply the N second feature maps of the training image with the corresponding binary mask maps to obtain the latest classification maps corresponding to the N second feature maps of the training image; determine the feature score prediction loss for the training image according to the categories annotated on the training image and the score maps and the latest classification maps corresponding to the N second feature maps of the training image.
[0228] In a possible implementation, when the model training module determines the feature score prediction loss for the training image according to the category annotated by the training image and the score map and the latest classification map respectively corresponding to the N second feature maps of the training image, it is specifically used to:
[0229] The probability corresponding to the category annotated in the training image is obtained from multiple probabilities at each position in the latest classification map corresponding to each second feature map of the training image to obtain the target classification maps corresponding to the N second feature maps of the training image respectively; the feature score prediction loss is determined according to the target classification maps and score maps corresponding to the N second feature maps of the training image respectively.
[0230] In a possible implementation, the model training module is also used to:
[0231] Performing enhancement processing on the training image to obtain a processed training image;
[0232] Using fine-grained image classification models, we obtain the global features and the most discriminative component features of the processed training images;
[0233] Using a fine-grained image classification model, the probability distribution of the processed training image on multiple set categories is predicted based on the global features and the most discriminative component features of the processed training image;
[0234] Determining a probability distribution consistency loss according to the probability distribution of the training image on the multiple set categories and the probability distribution of the processed training image on the multiple set categories;
[0235] According to the probability distribution of the processed training image on multiple set categories and the category annotated by the training image, the category prediction loss for the processed training image is determined.
[0236] When the model training module updates the parameters of the fine-grained image classification model according to the prediction loss for the training image, it is specifically used to update the parameters of the fine-grained image classification model according to the prediction loss for the training image, the probability distribution consistency loss, and the category prediction loss for the processed training image.
[0237] In a possible implementation, when the model training module performs enhancement processing on the training image, it is specifically used to perform homography transformation on the training image based on a predetermined homography matrix.
[0238] When the fine-grained image classification device provided in the embodiment of the present application performs fine-grained image classification, since only one feature extraction is required, the amount of calculation is small and the delay is short. Since the category of the target image is determined based on two aspects of features (i.e., global features and the most discriminative local features), a relatively accurate fine-grained classification result can be obtained. In summary, the fine-grained image classification device provided in the embodiment of the present application has a low amount of calculation, a short delay, and a high fine-grained classification accuracy.
[0239] An embodiment of the present application also provides an electronic device, which may include: at least one processor, at least one communication interface, at least one memory and at least one communication bus.
[0240] In the embodiment of the present application, the number of the processor, the communication interface, the memory, and the communication bus is at least one, and the processor, the communication interface, and the memory communicate with each other through the communication bus;
[0241] The processor may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application;
[0242] The memory may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;
[0243] The memory stores a program, and the processor can call the program stored in the memory, where the program is used to implement the steps of the fine-grained image classification method provided in the above embodiment.
[0244] An embodiment of the present application also provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the fine-grained image classification method provided in the above embodiment.
[0245] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the steps of the fine-grained image classification method provided in the above embodiment.
[0246] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.
[0247] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0248] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.
[0249] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.
Claims
1. A fine-grained image classification method, characterized in that: include: Get the target image; Obtaining N first feature maps of different scales for the target image, wherein the first feature map of the smallest scale among the N first feature maps is used as a global feature, and N is an integer greater than 1; Perform feature enhancement on the N first feature maps to obtain N second feature maps of different scales; Predict a score map according to each second feature map, and obtain score maps corresponding to N second feature maps, wherein the score map has the same width and height as the corresponding second feature map, and the score at each position in the score map can represent the discriminability of the feature at the corresponding position in the corresponding second feature map; Extracting the most discriminative component features from the N second feature maps according to the score maps respectively corresponding to the N second feature maps; Determining the category of the target image according to the global features and the component features; The extracting the most discriminative component feature from the N second feature maps according to the score maps respectively corresponding to the N second feature maps comprises: For each of the N second feature maps: Determine K maximum scores from the score map corresponding to the second feature map, where K is an integer greater than 0; The most discriminative component features are extracted from the second feature map according to the positions of the K maximum scores.
2. The fine-grained image classification method according to claim 1, characterized in that: The method comprises: obtaining N first feature maps of different scales for the target image; performing feature enhancement on the N first feature maps to obtain N second feature maps of different scales; predicting a score map according to each second feature map to obtain score maps corresponding to the N second feature maps; and extracting the most discriminative component features from the N second feature maps according to the score maps corresponding to the N second feature maps; Determining the category of the target image according to the global feature and the component feature includes: Using a pre-trained fine-grained image classification model, obtain N first feature maps of different scales for the target image; Using the fine-grained image classification model, feature enhancement is performed on the N first feature maps to obtain N second feature maps of different scales; Using the fine-grained image classification model, based on each second feature map, a score map is predicted to obtain score maps corresponding to the N second feature maps respectively; Using the fine-grained image classification model, based on the score maps corresponding to the N second feature maps, the most discriminative component features are extracted from the N second feature maps; Determining the category of the target image by using the fine-grained image classification model based on the global features and the component features; The fine-grained image classification model is trained using training images that are only labeled with categories.
3. The fine-grained image classification method according to claim 2, characterized in that: The training process of the fine-grained image classification model includes: Using a fine-grained image classification model, obtaining the global features of the training image, N second feature maps of different scales, score maps and classification maps corresponding to the N second feature maps, and the most discriminative component features of the training image, wherein the value at each position in the classification map is the probability corresponding to a plurality of set categories predicted according to the feature at the corresponding position in the corresponding second feature map; Using a fine-grained image classification model, based on the global features and the most discriminative component features of the training image, predict the probability distribution of the training image on the plurality of set categories; Determine the prediction loss for the training image according to the score map and the classification map respectively corresponding to the N second feature maps of the training image, the probability distribution of the training image on the multiple set categories, and the category annotated by the training image; Parameters of the fine-grained image classification model are updated according to the prediction loss for the training image.
4. The fine-grained image classification method according to claim 3, characterized in that: The step of determining the prediction loss for the training image according to the score map and the classification map respectively corresponding to the N second feature maps of the training image, the probability distribution of the training image on the multiple set categories, and the category annotated by the training image comprises: Determine a first category prediction loss for the training image according to the probability distribution of the training image on the multiple set categories and the category annotated by the training image; Determine a second category prediction loss for the training image according to classification maps respectively corresponding to the N second feature maps of the training image and the category annotated by the training image; The feature score prediction loss for the training image is determined according to the score map and the classification map respectively corresponding to the N second feature maps of the training image and the category annotated by the training image.
5. The fine-grained image classification method according to claim 4, characterized in that: The training process of the fine-grained image classification model also includes: Using a fine-grained image classification model, generating attention maps corresponding to the N second feature maps of the training image respectively; Processing the attention maps corresponding to the N second feature maps of the training image into binary mask maps, to obtain binary mask maps corresponding to the N second feature maps of the training image; The step of determining the feature score prediction loss for the training image according to the score map and the classification map respectively corresponding to the N second feature maps of the training image and the category annotated by the training image comprises: Multiplying the N second feature maps of the training image by the corresponding binary mask maps respectively to obtain the latest classification maps corresponding to the N second feature maps of the training image respectively; Determine the feature score prediction loss for the training image according to the category annotated by the training image and the score map and the latest classification map respectively corresponding to the N second feature maps of the training image.
6. The fine-grained image classification method according to claim 5, characterized in that: The step of determining the feature score prediction loss for the training image according to the category annotated by the training image and the score map and the latest classification map respectively corresponding to the N second feature maps of the training image includes: Obtain the probability corresponding to the category marked in the training image from multiple probabilities at each position in the latest classification map corresponding to each second feature map of the training image, so as to obtain the target classification maps corresponding to the N second feature maps of the training image respectively; Determine the feature score prediction loss according to the target classification map and the score map respectively corresponding to the N second feature maps of the training image.
7. The fine-grained image classification method according to claim 3, characterized in that: The training process of the fine-grained image classification model also includes: Performing enhancement processing on the training image to obtain a processed training image; Using a fine-grained image classification model, obtaining global features and most discriminative component features of the processed training image; Using a fine-grained image classification model, based on the global features and the most discriminative component features of the processed training images, predict the probability distribution of the processed training images on the multiple set categories; Determining a probability distribution consistency loss according to the probability distribution of the training image on the multiple set categories and the probability distribution of the processed training image on the multiple set categories; Determining a category prediction loss for the processed training image according to the probability distribution of the processed training image on the multiple set categories and the category annotated by the training image; The updating of parameters of the fine-grained image classification model according to the prediction loss of the training image comprises: Parameters of a fine-grained image classification model are updated according to the prediction loss for the training image, the probability distribution consistency loss, and the category prediction loss for the processed training image.
8. The fine-grained image classification method according to claim 7, characterized in that: The step of performing enhancement processing on the training image comprises: Based on a predetermined homography matrix, a homography transformation is performed on the training image.
9. A fine-grained image recognition device, characterized in that: include: Image acquisition module, multi-scale feature acquisition module, feature processing module, information prediction module, component feature acquisition module and image category determination module; The image acquisition module is used to acquire a target image; The multi-scale feature acquisition module is used to acquire N first feature maps of different scales for the target image, wherein the first feature map of the smallest scale among the N first feature maps is used as the global feature, and N is an integer greater than 1; The feature processing module is used to perform feature enhancement on the N first feature maps to obtain N second feature maps of different scales; The information prediction module is used to predict a score map according to each second feature map to obtain score maps corresponding to the N second feature maps, wherein the score map has the same width and height as the corresponding second feature map, and the score of each position in the score map can characterize the discriminability of the feature at the corresponding position in the corresponding second feature map; The component feature acquisition module is used to extract the most discriminative component feature from the N second feature maps according to the score maps respectively corresponding to the N second feature maps; The image category determination module is used to determine the category of the target image according to the global feature and the component feature; The component feature acquisition module extracts the most discriminative component feature from the N second feature maps according to the score maps respectively corresponding to the N second feature maps, including: For each of the N second feature maps: Determine K maximum scores from the score map corresponding to the second feature map, where K is an integer greater than 0; The most discriminative component features are extracted from the second feature map according to the positions of the K maximum scores.
