Image classification method, device, storage medium and electronic device

Abstract

An image classification method, device, storage medium and electronic device, wherein a target image to be classified is determined (101), a pre-trained fine-grained classification model is called to perform fine-grained classification on the target image (102), and a fine-grained category of the target image output by the fine-grained classification model is obtained (103).

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image recognition technology, and in particular to an image classification method, device, storage medium and electronic device. Background Art

[0002] Image classification mainly includes coarse-grained image classification and fine-grained image classification. Fine-grained image classification is also called subcategory image classification. Its purpose is to divide coarse-grained large categories into more detailed subcategories, such as distinguishing bird species, car models, dog breeds, etc. Summary of the invention

[0003] The present application provides an image classification method, device, storage medium and electronic device, which can realize fine-grained classification of images.

[0004] In a first aspect, the present application provides an image classification method, comprising:

[0005] Determine the target image that needs to be classified;

[0006] Calling a pre-trained fine-grained classification model to perform fine-grained classification on the target image, wherein the fine-grained classification model includes a feature extraction module, a feature optimization module and a fine-grained classification module, the feature extraction module is used to extract image features of the target image, the feature optimization module is used to optimize the image features to obtain optimized image features, and the fine-grained classification module performs fine-grained classification on the optimized image features to obtain fine-grained categories of the target image;

[0007] Obtain a fine-grained category of the target image output by the fine-grained classification model.

[0008] In a second aspect, the present application also provides an image classification device, comprising:

[0009] An image determination component, used to determine a target image that needs to be classified;

[0010] A model calling component, used to call a pre-trained fine-grained classification model to perform fine-grained classification on the target image, wherein the fine-grained classification model includes a feature extraction module, a feature optimization module and a fine-grained classification module, the feature extraction module is used to extract image features of the target image, the feature optimization module is used to optimize the image features to obtain optimized image features, and the fine-grained classification module performs fine-grained classification on the optimized image features to obtain fine-grained categories of the target image;

[0011] The category acquisition component is used to obtain the fine-grained category of the target image output by the fine-grained classification model.

[0012] In a third aspect, the present application further provides a storage medium having a computer program stored thereon, wherein when the computer program is executed on a computer, the computer is caused to execute:

[0013] Determine the target image that needs to be classified;

[0014] Calling a pre-trained fine-grained classification model to perform fine-grained classification on the target image, wherein the fine-grained classification model includes a feature extraction module, a feature optimization module and a fine-grained classification module, the feature extraction module is used to extract image features of the target image, the feature optimization module is used to optimize the image features to obtain optimized image features, and the fine-grained classification module performs fine-grained classification on the optimized image features to obtain fine-grained categories of the target image;

[0015] Obtain a fine-grained category of the target image output by the fine-grained classification model.

[0016] In a fourth aspect, the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor calls the computer program stored in the memory to execute:

[0017] Determine the target image that needs to be classified;

[0018] Calling a pre-trained fine-grained classification model to perform fine-grained classification on the target image, wherein the fine-grained classification model includes a feature extraction module, a feature optimization module and a fine-grained classification module, the feature extraction module is used to extract image features of the target image, the feature optimization module is used to optimize the image features to obtain optimized image features, and the fine-grained classification module performs fine-grained classification on the optimized image features to obtain fine-grained categories of the target image;

[0019] Obtain a fine-grained category of the target image output by the fine-grained classification model. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The technical solution and beneficial effects of the present application will be made apparent by describing in detail the specific implementation modes of the present application in conjunction with the accompanying drawings.

[0021] Figure 1 It is a flow chart of the image classification method provided in the embodiment of the present application.

[0022] Figure 2 This is an example diagram of triggering image classification in an embodiment of the present application.

[0023] Figure 3This is an example diagram of the image classification interface provided in the embodiments of the present application.

[0024] Figure 4 This is an example diagram of the selection sub-interface provided in the embodiments of the present application.

[0025] Figure 5 It is a schematic diagram of the architecture of the fine-grained classification model provided in an embodiment of the present application.

[0026] Figure 6 It is an example diagram of the target image provided in the embodiment of the present application.

[0027] Figure 7 This is another architectural diagram of the fine-grained classification model provided in an embodiment of the present application.

[0028] Figure 8 It is a schematic diagram of the architecture of a machine learning network provided in an embodiment of the present application.

[0029] Fig. 9 It is another flowchart of the image classification method provided in an embodiment of the present application.

[0030] Fig.10 It is a structural schematic diagram of the image classification device provided in an embodiment of the present application.

[0031] Fig.11 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] Please refer to the drawings, where the same component symbols represent the same components. The principle of the present application is illustrated by implementing it in an appropriate computing environment. The following description is based on the illustrated specific embodiments of the present application, which should not be considered as limiting other specific embodiments of the present application that are not described in detail herein.

[0033] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0034] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0035] Among them, machine learning (ML) is a multi-disciplinary interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning and other technologies.

[0036] The technical solution provided in the embodiments of the present application relates to the machine learning technology of artificial intelligence, which is specifically described by the following embodiments:

[0037] The embodiment of the present application provides an image classification method, an image classification device, a storage medium, and an electronic device, wherein the execution subject of the image classification method can be the image classification device provided in the embodiment of the present application, or an electronic device integrating the image classification device, wherein the image classification device can be implemented in hardware or software. The electronic device can be a smart phone, a tablet computer, a PDA, a laptop computer, or a desktop computer, etc., which is equipped with a processor (including but not limited to a general-purpose processor, a customized processor, etc.) and has processing capabilities.

[0038] Please refer to Figure 1 , Figure 1 : is a flow chart of an image classification method provided in an embodiment of the present application. The flow of the image classification method may include:

[0039] In 101, a target image that needs to be classified is determined.

[0040] In an embodiment of the present application, the electronic device can determine the target image that needs to be classified based on a preset image classification cycle and according to preset image selection rules, or when receiving an image classification instruction input by the user, determine the target image that needs to be classified according to the image classification operation input by the user, and so on.

[0041] It should be noted that the embodiments of the present application do not specifically limit the settings of the image classification cycle, image selection rules, and image classification instructions. They can be set by the electronic device according to user input, or by the manufacturer of the electronic device as default settings, etc.

[0042] For example, assuming that the image classification cycle is pre-configured as a natural week starting on Monday, and the image selection rule is configured as "selecting captured images for image classification", the electronic device can automatically trigger image classification every Monday and determine the captured image as the target image that needs to be classified.

[0043] For example, please refer to Figure 2 , an electronic device provides a "classification" control for triggering image classification in an image browsing interface. In which, the rectangles in the figure represent different images, and the circular boxes in the rectangles represent the "selection" control for selecting the corresponding images. The user can click the selection control corresponding to an image to select the image, and can click the selection control corresponding to the image again to cancel the selection of the image. Figure 2 As shown, after the user selects an image to be classified, the user inputs an image classification instruction to the electronic device by clicking the classification control, wherein the image classification instruction carries indication information indicating the image selected by the user. Accordingly, the electronic device determines the image selected by the user as a target image to be classified according to the indication information in the image classification instruction input by the user.

[0044] For another example, the electronic device may receive an input image classification instruction through an image classification interface including a request input interface, such as Figure 3 As shown, the request input interface can be in the form of an input box, in which the user can type the identification information of the image to be classified, and enter confirmation information (such as directly pressing the Enter key on the keyboard) to enter the image classification instruction, which carries the identification information of the image to be classified. Accordingly, the electronic device can determine the target image to be classified according to the identification information in the received image classification instruction.

[0045] For example, in Figure 3 The image classification interface also includes an "open" control. On the one hand, when the electronic device detects that the open control is triggered, it will overlay the image classification interface with a selection sub-interface (such as Figure 4As shown), the selection sub-interface provides the user with thumbnails of images that can be classified, such as thumbnails of image A, image B, image C, image D, image E, image F, etc., for the user to search and select the thumbnails of the images that need to be classified; on the other hand, after selecting the thumbnail of the image that needs to be classified, the user can trigger the confirmation control provided by the selection sub-interface to input an image classification instruction to the electronic device, and the image classification instruction is associated with the thumbnail of the image selected by the user, instructing the electronic device to use the image selected by the user as the target image that needs to be classified.

[0046] In addition, a person skilled in the art may also set other specific implementation methods of the input image classification instructions according to actual needs, and the present invention does not impose any specific limitation on this.

[0047] In 102, a pre-trained fine-grained classification model is called to perform fine-grained classification on the target image, wherein the fine-grained classification model includes a feature extraction module, a feature optimization module and a fine-grained classification module, the feature extraction module is used to extract image features of the target image, the feature optimization module is used to optimize the image features to obtain optimized image features, and the fine-grained classification module performs fine-grained classification on the optimized image features to obtain fine-grained categories of the target image.

[0048] It should be noted that in the present application, a fine-grained classification model is pre-trained using a machine learning method, and the fine-grained classification model is configured to perform fine-grained classification on images.

[0049] Please refer to Figure 5 The fine-grained classification model provided in this application consists of two parts, namely a feature extraction module for extracting features, a feature optimization module for optimizing features, and a fine-grained classification module for performing fine-grained classification according to features.

[0050] The electronic device first inputs the target image into the feature extraction module, and extracts features of the target image based on the feature extraction module, thereby obtaining image features of the target image.

[0051] Exemplarily, the feature extraction module includes N layers of convolutional layers. When performing feature extraction on a target image based on the feature extraction module, the electronic device first performs convolution calculation on the target image based on the first convolutional layer to obtain a feature map, recorded as the first layer feature map; then, the electronic device performs convolution calculation on the first layer feature map based on the second convolutional layer to obtain a new feature map, recorded as the second layer feature map; and so on, until an Nth layer feature map is obtained based on the Nth convolutional layer calculation, and the Nth layer feature map is used as the image feature obtained by performing feature extraction on the target image.

[0052] It should be noted that the value of N in this application, that is, the number of convolutional layers constituting the feature extraction module, is not specifically limited and can be set by ordinary technicians in this field according to actual needs. For example, in the embodiment of this application, the value of N is 9.

[0053] After the electronic device extracts the image features of the target image based on the feature extraction module, it further optimizes the image features according to a preset optimization strategy based on the feature optimization module to obtain optimized image features, which are recorded as optimized image features.

[0054] After the electronic device optimizes the extracted image features based on the feature optimization module to obtain optimized image features, it further performs fine-grained classification on the optimized image features based on the fine-grained classification module to obtain fine-grained categories of the target image. The fine-grained classification module may be a fully connected layer.

[0055] For example, please refer to Figure 6 ,when Figure 6 When the image on the left is determined as the target image, its coarse-grained category is "dog". Based on the fine-grained classification module, the optimized image features corresponding to the target image are fine-grained classified, and its fine-grained category is obtained as "corgi", which means that the breed of the dog in the target image is corgi; Figure 6 When the image on the right is determined as the target image, it can be seen that its coarse-grained category is also "dog". Based on the fine-grained classification module, the optimized image features corresponding to the target image are fine-grained classified, and its fine-grained category is obtained as "Husky", which means that the breed of the dog in the target image is Husky.

[0056] In 103, the fine-grained category of the target image output by the fine-grained classification model is obtained.

[0057] In an embodiment of the present application, after the fine-grained classification model completes the fine-grained classification of the target image, the electronic device can obtain the classified fine-grained category from the fine-grained classification module of the fine-grained classification model.

[0058] As can be seen from the above, the present application determines the target image that needs to be classified, and calls the pre-trained fine-grained classification model to perform fine-grained classification on the target image, wherein the fine-grained classification model includes a feature extraction module, a feature optimization module and a fine-grained classification module, the feature extraction module is used to extract the image features of the target image, the feature optimization module is used to optimize the image features to obtain the optimized image features, the fine-grained classification module performs fine-grained classification on the optimized image features to obtain the fine-grained category of the target image, and then obtains the fine-grained category of the target image output by the fine-grained classification model. Therefore, the present application does not require manual fine-grained classification, and can efficiently achieve fine-grained classification of images.

[0059] In one embodiment, determining a target image that needs to be classified includes:

[0060] When the image classification cycle is reached, the newly added image in the image classification cycle is determined as the target image.

[0061] In an embodiment of the present application, when the electronic device reaches an image classification cycle, it triggers the determination of target images that need to be classified. Among them, the electronic device can directly use the newly added images in the image classification cycle as target images. For example, in an image classification cycle, the electronic device adds 20 images, and the electronic device uses these 20 images as target images that need to be classified.

[0062] In one embodiment, “determining a target image that needs to be classified” includes:

[0063] (1) determining the image in the preset storage path as the target image; or,

[0064] (2) determining an image in a preset image format as a target image; or,

[0065] (3) An image in a preset image format under a preset storage path is determined as a target image.

[0066] The embodiments of the present application do not specifically limit the setting of the preset storage path and the preset image format, which can be set by the electronic device according to the user input, or by the manufacturer of the electronic device. It should be noted that the preset storage path can be configured as one or more, and correspondingly, the preset image format can be configured as one or more.

[0067] For example, assuming that the user requires the electronic device to classify the captured images, the preset storage path can be configured as the storage path of the images captured by the electronic device. For example, if the electronic device is based on the Android system, the preset storage path is configured as " / storage / 0 / DCIM". In this way, the electronic device will identify all images in the file directory "DCIM" corresponding to / storage / 0 / DCIM as target images that need to be classified.

[0068] For another example, assuming that the user requires the electronic device to classify images in a certain image format, the preset image format can be configured as the image format specified by the user. For example, if the user requires the electronic device to classify images in the "JPG" format, the preset image format can be configured as the "JPG" format. In this way, the electronic device will identify all local "JPG" format images as target images that need to be classified.

[0069] For another example, assuming that the user requires the electronic device to classify images in a certain image format, the preset storage path can be configured as the storage path of the images taken by the electronic device, and the preset image format can be configured as the image format specified by the user. For example, if the electronic device is based on the Android system, the preset storage path is configured as " / storage / 0 / DCIM". In addition, if the user requires the electronic device to classify images in the "JPG" format, the preset image format is configured as the "JPG" format. In this way, the electronic device will determine all images in the "JPG" format in the file directory "DCIM" corresponding to / storage / 0 / DCIM as target images that need to be classified.

[0070] In one embodiment, the fine-grained classification model further includes a dimension reduction module, which is used to perform dimension reduction on the optimized image features to obtain the optimized image features after dimension reduction;

[0071] The fine-grained classification module is also used to perform classification prediction on the optimized image features after dimensionality reduction to obtain the fine-grained categories of the target image.

[0072] Please refer to Figure 7 The fine-grained classification model provided in this application also includes a dimensionality reduction module, which can be a pooling layer. Figure 7 As shown, one end of the dimensionality reduction module is connected to the feature optimization module, and the other end is connected to the fine-grained classification module.

[0073] In the embodiment of the present application, after the electronic device optimizes the image extracted by the feature extraction module based on the feature optimization module, it does not directly perform fine-grained classification based on the optimized image features obtained by optimization, but inputs the optimized image features into the dimension reduction module, thereby performing feature dimension reduction processing on the optimized image features based on the dimension reduction module to obtain the optimized image features after dimension reduction. Then, the electronic device inputs the optimized image features after dimension reduction into the fine-grained classification module for fine-grained classification, and accordingly obtains the fine-grained category of the target image.

[0074] In one embodiment, after obtaining the fine-grained category of the target image output by the fine-grained classification model, the method further includes:

[0075] A storage path is allocated to the target image according to the fine-grained category, and the target image is stored in the storage path.

[0076] In the embodiment of the present application, in order to facilitate the user to browse images, the electronic device also classifies and stores the target images according to the fine-grained categories obtained by fine-grained classification of the target images.

[0077] The electronic device may allocate a storage path for each fine-grained category and store the corresponding target image in the allocated storage path. For example, if the target image is classified into nine categories, the electronic device will allocate nine different storage paths for storing the target images of the corresponding categories.

[0078] In one embodiment, after storing the target image in the allocated storage path, the method further includes:

[0079] For each target image in the storage path, obtain browsing behavior data of the user browsing each target image, and obtain the creation time of each target image;

[0080] Performing weighted summation on the browsing behavior data and creation time of each target image to obtain a weighted sum value of each target image;

[0081] The target images are sorted according to their weighted sum values.

[0082] In the embodiment of the present application, after the target images are classified, the target images of each category (ie, the target images under each storage path) are also sorted.

[0083] The browsing behavior data includes relevant data describing the user's browsing behavior. For example, the browsing behavior data includes the number of times the user browses the target image, the opening time and the closing time of the user's browsing of the target image each time, and so on.

[0084] In addition to obtaining the browsing behavior data of the user browsing each target image, the electronic device also obtains the creation time of each target image, wherein the creation time is the difference between the current time and the generation time of the target image.

[0085] It should be noted that the above-mentioned current moment does not specifically refer to a certain moment, but refers to the moment when the electronic device performs the operation of "obtaining the creation time of each target image". In addition, the embodiment of the present application does not specifically limit the method of generating the target image. For example, if a target image is generated by an electronic device through shooting, the generation moment of the target image is the shooting moment when the electronic device obtains the target image; for another example, if a target image is generated by an electronic device through downloading from the Internet, the generation moment of the target image is the download moment when the electronic device obtains the target image through the Internet, and so on.

[0086] In an embodiment of the present application, after obtaining the browsing behavior data and creation time of each target image, the electronic device performs weighted summation on the obtained browsing behavior data and creation time according to a preset weighted summation algorithm to obtain a weighted sum value corresponding to each target image.

[0087] Among them, the browsing behavior data can reflect the characteristics of the user's browsing behavior, and the creation time is the characteristic of the image itself. The purpose of the electronic device weighted summing the acquired browsing behavior data and creation time is to comprehensively evaluate the target image in combination with the target image's own characteristics and user characteristics outside the image. In this way, the weighted sum value obtained by the weighted summation is also the "score" obtained by comprehensively evaluating the target image. The high or low score reflects the probability that the target image may be browsed by the user.

[0088] In the embodiment of the present application, after obtaining the weighted sum value of each target image, the electronic device sorts the target images in descending order according to the weighted sum value.

[0089] In one embodiment, a weighted sum is performed on the browsing behavior data and the creation time of each target image to obtain a weighted sum value of each target image, including:

[0090] According to the browsing behavior data of each target image, the number of browsing times of each target image and the browsing duration of each browsing are obtained;

[0091] According to the number of views of each target image and the viewing time of each view, the average viewing time of each target image is obtained;

[0092] Normalize the number of views, average viewing time, and creation time of each target image;

[0093] The normalized browsing times, average browsing time, and creation time of each target image are weighted and summed to obtain the weighted sum value of each target image.

[0094] In an embodiment of the present application, when a target image is browsed by a user, the electronic device records the browsing behavior data of the user browsing the target image, wherein the browsing behavior data includes but is not limited to the number of times the user browses the target image, the opening time and the closing time of the user browsing the target image each time, and so on.

[0095] Therefore, when the electronic device performs weighted summation of the browsing behavior data and creation time of each target image, it can directly extract the number of views of each target image (i.e., the number of times the user browses the target image) from the browsing behavior data of each target image, and obtain the browsing time of each target image each time based on the "opening time and closing time of each target image browsed by the user" in the browsing behavior data of each target image.

[0096] After obtaining the number of views of each target image and the viewing time of each view, the electronic device further calculates the average viewing time of each target image according to the number of views of each target image and the viewing time of each view. It should be noted that those skilled in the art can understand that the average viewing time referred to here is the average viewing time of a single target image, not the average viewing time of multiple target images.

[0097] In addition, in the embodiment of the present application, corresponding weight values ​​are pre-assigned to the three data of the number of views, average browsing time and creation time, respectively, but the values ​​of the weight values ​​corresponding to the number of views, average browsing time and creation time are not specifically limited, and can be set by ordinary technicians in this field according to actual needs. For example, the weight value corresponding to the number of views can be set to 0.3, the weight value corresponding to the average browsing time can be set to 0.2, and the weight value corresponding to the creation time can be set to 0.5.

[0098] In order to improve the efficiency of weighted summation, when the electronic device performs weighted summation of the number of views, average viewing time and creation time of each target image, it first normalizes the number of views, average viewing time and creation time of each target image, and normalizes the number of views, average viewing time and creation time of each target image to the same numerical range.

[0099] Then, the electronic device performs weighted summation on the normalized browsing times, average browsing time and creation time of each target image according to a preset weighted summation algorithm to obtain a weighted sum value corresponding to each target image.

[0100] In one embodiment, the image features include a feature map, and the feature optimization module is used to transpose the feature map to obtain a transposed feature map, and perform matrix multiplication on the feature map and the transposed feature map, and use the result of the matrix multiplication as the optimized image feature.

[0101] Taking the image feature as a feature map as an example, the present application provides a method for optimizing the image feature.

[0102] The electronic device first performs a transposition process on the extracted image features, that is, the feature map of the target image, based on the feature optimization module to obtain a transposed feature map, which is recorded as a transposed feature map. Then, the electronic device further performs a matrix multiplication process on the original feature map and the transposed feature map based on the feature optimization module, and uses the result of the matrix multiplication as the optimized image feature for fine-grained classification.

[0103] In one embodiment, before determining the target image to be classified, the method further includes:

[0104] (1) obtaining a plurality of sample images, and obtaining fine-grained category labels and coarse-grained category labels of the sample images;

[0105] (2) constructing a machine learning network, wherein the machine learning network includes a first branch network, a second branch network and a binary classification module of the same structure, wherein the first branch network includes a feature extraction module, a feature optimization module and a fine-grained classification module, and the binary classification module is connected to the feature extraction module of the first branch network and the feature extraction module of the second branch network;

[0106] (3) selecting a first sample image from the plurality of sample images, extracting image features of the first sample image based on a feature extraction module of the first branch network, optimizing the first sample image through a feature optimization module of the first branch network, and inputting the image features into a fine-grained classification module of the first branch network for fine-grained classification to obtain a first predicted fine-grained category;

[0107] (4) selecting a second sample image from the plurality of sample images, extracting image features of the second sample image based on a feature extraction module of the second branch network, optimizing the image features through a feature optimization module of the second branch network, and inputting the image features into a fine-grained classification module of the second branch network for fine-grained classification to obtain a second predicted fine-grained category;

[0108] (5) fusing the image features of the first sample image and the image features of the second sample image to obtain a fused image feature, and predicting whether the coarse-grained categories of the first sample image and the second sample image are the same based on a binary classification module to obtain a prediction result;

[0109] (6) obtaining a first classification loss of the first branch network according to the first predicted fine-grained category and the fine-grained category label of the first sample image, obtaining a second classification loss of the second branch network according to the second predicted fine-grained category and the fine-grained category label of the second sample image, and obtaining a third classification loss according to the prediction result, the coarse-grained category of the first sample image and the second sample image;

[0110] (7) Obtaining the corresponding total loss according to the first classification loss, the second classification loss, and the third classification loss, and adjusting the parameters of the first branch network and the second branch network according to the total loss until the training is terminated when a preset training stop condition is met;

[0111] (8) Select a branch network from the first branch network and the second branch network as a fine-grained classification model.

[0112] This application also provides an optional method for training a fine-grained classification model.

[0113] The electronic device first obtains a plurality of sample images, and obtains a fine-grained category label and a coarse-grained category label of each sample image. The fine-grained category label is used to describe the fine-grained category of the sample image, and the coarse-grained label is used to describe the coarse-grained category of the sample image.

[0114] It should be noted that the present application does not impose any specific restrictions on the method and quantity of obtaining sample images, and can be configured by ordinary technicians in the field according to actual needs. For example, the electronic device can crawl images from the Internet as sample images, and receive fine-grained category labels and coarse-grained category labels of the manually annotated sample images.

[0115] In addition, electronic devices also build machine learning networks, please refer to Figure 8 The constructed machine learning network includes two branch networks with the same structure and a binary classification module. Each branch network includes a feature extraction module, a feature optimization module and a fine-grained classification module. For the convenience of distinction, one of the branch networks is recorded as the first branch network and the other branch network is recorded as the second branch network. In addition, the binary classification module is connected to the feature extraction module of the first branch network and the feature extraction module of the second branch network.

[0116] After the machine learning network is constructed, the electronic device uses the acquired sample images to train the machine learning network.

[0117] Among them, the electronic device selects two sample images from the multiple sample images obtained, records one of the sample images as the first sample image, and records the other sample image as the second sample image. For the first sample image, the electronic device extracts the image features of the first sample image based on the feature extraction module of the first branch network, optimizes it through the feature optimization module of the first branch network, and inputs it into the fine-grained classification module of the first branch network for fine-grained classification to obtain a first predicted fine-grained category. For the second sample image, the electronic device extracts the image features of the second sample image based on the feature extraction module of the second branch network, optimizes it through the feature optimization module of the second branch network, and inputs it into the fine-grained classification module of the second branch network for fine-grained classification to obtain a second predicted fine-grained category.

[0118] In addition, the electronic device also fuses the image features of the first sample image and the image features of the second sample image to obtain a fused image feature, and predicts whether the coarse-grained categories of the first sample image and the second sample image are the same based on the fused image feature based on the binary classification module to obtain a prediction result.

[0119] In addition, the electronic device also obtains the first classification loss of the first branch network according to the first predicted fine-grained category and the fine-grained category label of the first sample image, obtains the second classification loss of the second branch network according to the second predicted fine-grained category and the fine-grained category label of the second sample image, and obtains the third classification loss according to the prediction result, the coarse-grained category of the first sample image and the second sample image. Among them, the loss function for obtaining the aforementioned first classification loss, second classification loss and third classification loss can be configured by ordinary technicians in this field according to actual needs, and this application does not make specific restrictions on this.

[0120] Then, the electronic device obtains the corresponding total loss according to the first classification loss, the second classification loss and the third classification loss, which can be expressed as:

[0121] L total =Loss1+Loss2+Loss3;

[0122] Among them, L total Represents the total loss, Loss1 represents the first classification loss, Loss2 represents the second classification loss, and Loss3 represents the third classification loss.

[0123] After obtaining the total loss, the electronic device adjusts the parameters of the first branch network and the second branch network according to the total loss. It should be noted that the goal of model training is to minimize the total loss. Therefore, after determining the total loss each time, the parameters of the first branch network and the second branch network can be adjusted in the direction of minimizing the total loss.

[0124] As described above, the parameters of the first branch network and the second branch network are continuously adjusted until the preset training stop condition is met, and the training is terminated. The preset training stop condition can be set by ordinary technicians in this field according to actual needs, and the embodiment of the present application does not make specific restrictions on this.

[0125] For example, the preset training stop condition is configured as follows: stop training when the total loss reaches the minimum value;

[0126] For another example, the preset training stop condition is configured as: stopping training when the number of iterations of the parameter reaches a preset number.

[0127] When the preset training stop condition is met, the electronic device determines that the first branch network and the second branch in the machine learning network can accurately classify the image in fine-grained manner. At this time, a branch network is selected from the first branch network and the second branch network as a fine-grained classification model for fine-grained classification of the image.

[0128] It should be noted that the present application does not impose any specific restrictions on how to select a fine-grained classification model from the first branch network and the second branch network. For example, in an embodiment of the present application, the electronic device can randomly select a branch network from the first branch network and the second branch network as a fine-grained classification model.

[0129] In one embodiment, the third classification loss is obtained according to the following formula:

[0130] Loss3=-[η*y*log(p)+(1-y)*log(1-p)];

[0131] Among them, Loss3 represents the third classification loss; η represents the correction coefficient (take an empirical value, such as 0.3-0.5 in this application); y is used to characterize whether the coarse-grained category label of the first sample image is the same as the coarse-grained category label of the second sample image, and can be determined by ordinary technicians in this field according to actual needs; p represents the prediction result.

[0132] In the embodiment of the present application, the purpose of increasing the correction coefficient η is to reduce the contribution of y*log(p) to the total loss when the coarse-grained category labels of the first sample image and the second sample image are different, so that the first branch network and the second branch network understand that the first sample image and the second sample image are similar in a certain dimension.

[0133] In one embodiment, selecting a branch network from the first branch network and the second branch network as a fine-grained classification model includes:

[0134] Obtaining the classification accuracy of the first branch network, and obtaining the classification accuracy of the second branch network;

[0135] A branch network with higher classification accuracy is selected from the first branch network and the second branch network as a fine-grained classification model.

[0136] The present application provides a method for selecting a fine-grained classification model, wherein the electronic device obtains the classification accuracy of a first branch network and the classification accuracy of a second branch network respectively, and then selects a branch network with a higher classification accuracy from the first branch network and the second branch network as a fine-grained classification model.

[0137] In one embodiment, fusing the image feature of the first sample image and the image feature of the second sample image to obtain the fused image feature includes:

[0138] Channel merging is performed on the image features of the first sample image and the image features of the second sample image, and the result of the channel merging is used as the fused image feature.

[0139] For example, the electronic device may use the Concat method to perform channel merging of the image features of the first sample image and the image features of the second sample image, and use the channel merging result as the fused image feature.

[0140] Please refer to Fig. 9 , the present application also provides a model training method, the process of the model training method can be:

[0141] In 201, a plurality of sample images are obtained, and fine-grained category labels and coarse-grained category labels of the sample images are obtained.

[0142] The electronic device first obtains a plurality of sample images, and obtains a fine-grained category label and a coarse-grained category label of each sample image. The fine-grained category label is used to describe the fine-grained category of the sample image, and the coarse-grained label is used to describe the coarse-grained category of the sample image.

[0143] It should be noted that the present application does not impose any specific restrictions on the method and quantity of obtaining sample images, and can be configured by ordinary technicians in the field according to actual needs. Fig.10 , you can pre-acquire some sample images from the ImageNet dataset and store them in the electronic device, crawl some sample images from the Internet and store them in the electronic device, and manually collect some sample images from the Internet.

[0144] After acquiring the sample image, the electronic device further receives the manually annotated fine-grained category label and the coarse-grained category label of the sample image, wherein the fine-grained category label is used to describe the fine-grained category of the sample image, and the coarse-grained label is used to describe the coarse-grained category of the sample image.

[0145] In 202, a machine learning network is constructed, and the machine learning network includes a first branch network, a second branch network and a binary classification module with the same structure. The first branch network includes a feature extraction module, a feature optimization module and a fine-grained classification module. The binary classification module is connected to the feature extraction module of the first branch network and the feature extraction module of the second branch network.

[0146] Please refer to Figure 8 The constructed machine learning network includes two branch networks with the same structure and a binary classification module. Each branch network includes a feature extraction module, a feature optimization module and a fine-grained classification module. For the convenience of distinction, one of the branch networks is recorded as the first branch network and the other branch network is recorded as the second branch network. In addition, the binary classification module is connected to the feature extraction module of the first branch network and the feature extraction module of the second branch network.

[0147] In 203, a first sample image is selected from a plurality of sample images, and image features of the first sample image are extracted based on a feature extraction module of the first branch network. After being optimized by a feature optimization module of the first branch network, the image features are input into a fine-grained classification module of the first branch network for fine-grained classification to obtain a first predicted fine-grained category.

[0148] In 204, a second sample image is selected from multiple sample images, and image features of the second sample image are extracted based on the feature extraction module of the second branch network. After being optimized by the feature optimization module of the second branch network, the image features are input into the fine-grained classification module of the second branch network for fine-grained classification to obtain a second predicted fine-grained category.

[0149] Among them, the electronic device selects two sample images from the multiple sample images obtained, records one of the sample images as the first sample image, and records the other sample image as the second sample image. For the first sample image, the electronic device extracts the image features of the first sample image based on the feature extraction module of the first branch network, optimizes it through the feature optimization module of the first branch network, and inputs it into the fine-grained classification module of the first branch network for fine-grained classification to obtain a first predicted fine-grained category. For the second sample image, the electronic device extracts the image features of the second sample image based on the feature extraction module of the second branch network, optimizes it through the feature optimization module of the second branch network, and inputs it into the fine-grained classification module of the second branch network for fine-grained classification to obtain a second predicted fine-grained category.

[0150] In 205 , the image features of the first sample image and the image features of the second sample image are fused to obtain fused image features, and based on the binary classification module, a prediction is made based on the fused image features as to whether the coarse-grained categories of the first sample image and the second sample image are the same to obtain a prediction result.

[0151] In addition, the electronic device also fuses the image features of the first sample image and the image features of the second sample image to obtain a fused image feature, and predicts whether the coarse-grained categories of the first sample image and the second sample image are the same based on the fused image feature based on the binary classification module to obtain a prediction result.

[0152] For example, the electronic device may use the Concat method to perform channel merging of the image features of the first sample image and the image features of the second sample image, and use the channel merging result as the fused image feature.

[0153] In 206, a first classification loss of the first branch network is obtained according to the first predicted fine-grained category and the fine-grained category label of the first sample image, a second classification loss of the second branch network is obtained according to the second predicted fine-grained category and the fine-grained category label of the second sample image, and a third classification loss is obtained according to the prediction result, the coarse-grained category of the first sample image and the second sample image.

[0154] Among them, the loss function for obtaining the aforementioned first classification loss, second classification loss and third classification loss can be configured by ordinary technicians in this field according to actual needs, and this application does not make any specific restrictions on this.

[0155] For example, the third classification loss is obtained according to the following formula:

[0156] Loss3=-[η*y*log(p)+(1-y)*log(1-p)];

[0157] Among them, Loss3 represents the third classification loss; η represents the correction coefficient (take an empirical value, such as 0.3-0.5 in this application); y is used to characterize whether the coarse-grained category label of the first sample image is the same as the coarse-grained category label of the second sample image, and can be determined by ordinary technicians in this field according to actual needs; p represents the prediction result.

[0158] In the embodiment of the present application, the purpose of increasing the correction coefficient η is to reduce the contribution of y*log(p) to the total loss when the coarse-grained category labels of the first sample image and the second sample image are different, so that the first branch network and the second branch network understand that the first sample image and the second sample image are similar in a certain dimension.

[0159] In 207, the corresponding total loss is obtained according to the first classification loss, the second classification loss and the third classification loss, and the parameters of the first branch network and the second branch network are adjusted according to the total loss until the training is terminated when a preset training stop condition is met.

[0160] For example, the total loss is obtained according to the following formula:

[0161] L total =Loss1+Loss2+Loss3;

[0162] Among them, L total Represents the total loss, Loss1 represents the first classification loss, Loss2 represents the second classification loss, and Loss3 represents the third classification loss.

[0163] After obtaining the total loss, the electronic device adjusts the parameters of the first branch network and the second branch network according to the total loss. It should be noted that the goal of model training is to minimize the total loss. Therefore, after determining the total loss each time, the parameters of the first branch network and the second branch network can be adjusted in the direction of minimizing the total loss.

[0164] As described above, the parameters of the first branch network and the second branch network are continuously adjusted until the preset training stop condition is met, and the training is terminated. The preset training stop condition can be set by ordinary technicians in this field according to actual needs, and the embodiment of the present application does not make specific restrictions on this.

[0165] For example, the preset training stop condition is configured as follows: stop training when the total loss reaches the minimum value;

[0166] For another example, the preset training stop condition is configured as: stopping training when the number of iterations of the parameter reaches a preset number.

[0167] In 208 , a branch network is selected from the first branch network and the second branch network as a fine-grained classification model.

[0168] When the preset training stop condition is met, the electronic device determines that the first branch network and the second branch in the machine learning network can accurately classify the image in fine-grained manner. At this time, a branch network is selected from the first branch network and the second branch network as a fine-grained classification model for fine-grained classification of the image.

[0169] Please refer to Fig.10 , Fig.10 3 is a schematic diagram of the structure of the image classification device provided by the present application. The image classification device may include: an image determination component 301, a model calling component 302 and a category acquisition component 303.

[0170] An image determination component 301 is used to determine a target image that needs to be classified;

[0171] The model calling component 302 is used to call the pre-trained fine-grained classification model to perform fine-grained classification on the target image, wherein the fine-grained classification model includes a feature extraction module, a feature optimization module and a fine-grained classification module, the feature extraction module is used to extract image features of the target image, the feature optimization module is used to optimize the image features to obtain optimized image features, and the fine-grained classification module performs fine-grained classification on the optimized image features to obtain fine-grained categories of the target image;

[0172] The category acquisition component 303 is used to acquire the fine-grained category of the target image output by the fine-grained classification model.

[0173] In one embodiment, when determining a target image that needs to be classified, the image determination component 301 is used to:

[0174] When the image classification cycle is reached, the newly added image in the image classification cycle is determined as the target image.

[0175] In one embodiment, the fine-grained classification model further includes a dimension reduction module, which is used to perform dimension reduction on the optimized image features to obtain the optimized image features after dimension reduction;

[0176] The fine-grained classification module is also used to perform classification prediction on the optimized image features after dimensionality reduction to obtain the fine-grained categories of the target image.

[0177] In one embodiment, the image classification device provided by the present application also includes a classification storage component, which is used to allocate a storage path to the target image according to the fine-grained category after obtaining the fine-grained category of the target image output by the fine-grained classification model, and store the target image in the storage path.

[0178] In one embodiment, the image features include a feature map, and the feature optimization module is used to transpose the feature map to obtain a transposed feature map, and perform matrix multiplication on the feature map and the transposed feature map, and use the result of the matrix multiplication as the optimized image feature.

[0179] In one embodiment, the image classification device provided by the present application further includes a model training component, which is used to:

[0180] Acquire multiple sample images, and obtain fine-grained category labels and coarse-grained category labels of the sample images;

[0181] Constructing a machine learning network, the machine learning network includes a first branch network, a second branch network and a binary classification module with the same structure, the first branch network includes a feature extraction module, a feature optimization module and a fine-grained classification module, and the binary classification module is connected to the feature extraction module of the first branch network and the feature extraction module of the second branch network;

[0182] Selecting a first sample image from a plurality of sample images, extracting image features of the first sample image based on a feature extraction module of the first branch network, optimizing the first sample image through a feature optimization module of the first branch network, and inputting the image features into a fine-grained classification module of the first branch network for fine-grained classification to obtain a first predicted fine-grained category;

[0183] A second sample image is selected from the plurality of sample images, and image features of the second sample image are extracted based on a feature extraction module of the second branch network, and after being optimized by a feature optimization module of the second branch network, the image features are input into a fine-grained classification module of the second branch network for fine-grained classification to obtain a second predicted fine-grained category;

[0184] Fusing the image features of the first sample image and the image features of the second sample image to obtain fused image features, and predicting whether the coarse-grained categories of the first sample image and the second sample image are the same based on a binary classification module to obtain a prediction result;

[0185] Obtaining a first classification loss of a first branch network according to the first predicted fine-grained category and the fine-grained category label of the first sample image, obtaining a second classification loss of a second branch network according to the second predicted fine-grained category and the fine-grained category label of the second sample image, and obtaining a third classification loss according to the prediction result, the coarse-grained category of the first sample image and the second sample image;

[0186] Obtain the corresponding total loss according to the first classification loss, the second classification loss, and the third classification loss, and adjust the parameters of the first branch network and the second branch network according to the total loss until the training is terminated when the preset training stop condition is met;

[0187] A branch network is selected from the first branch network and the second branch network as a fine-grained classification model.

[0188] In one embodiment, the third classification loss is obtained according to the following formula:

[0189] Loss3=-[η*y*log(p)+(1-y)*log(1-p)];

[0190] Among them, Loss3 represents the third classification loss, η represents the correction coefficient, y is used to characterize whether the coarse-grained category label of the first sample image is the same as the coarse-grained category label of the second sample image, and p represents the prediction result.

[0191] In one embodiment, when a branch network is selected from the first branch network and the second branch network as a fine-grained classification model, the model training component is used to:

[0192] Obtaining the classification accuracy of the first branch network, and obtaining the classification accuracy of the second branch network;

[0193] A branch network with higher classification accuracy is selected from the first branch network and the second branch network as the fine-grained classification model.

[0194] In one embodiment, when fusing the image feature of the first sample image and the image feature of the second sample image to obtain the fused image feature, the model training component is used to:

[0195] Channel merging is performed on the image features of the first sample image and the image features of the second sample image, and the result of the channel merging is used as the fused image feature.

[0196] It should be noted that the image classification device provided in the embodiment of the present application belongs to the same concept as the image classification method in the above embodiment. Any method provided in the image classification method embodiment can be run on the image classification device. The specific implementation process is detailed in the above image classification method embodiment and will not be repeated here.

[0197] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program stored therein is executed on a computer, the computer executes the image classification method provided in the embodiment of the present application. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0198] The present application also provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is used to execute the image classification method provided in the present application by calling the computer program stored in the memory.

[0199] For example, the electronic device may be a mobile terminal such as a tablet computer or a smart phone. Fig.11 , Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0200] The processor 401 is electrically connected to the memory 402 .

[0201] The processor 401 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device, executes various functions of the electronic device and processes data by running or loading computer programs stored in the memory 402 and calling data stored in the memory 402.

[0202] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the computer programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, a computer program required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0203] In the embodiment of the present application, the processor 401 in the electronic device will load instructions corresponding to the processes of one or more computer programs into the memory 402 according to the following steps, and the processor 401 will run the computer program stored in the memory 402 to implement various functions, such as:

[0204] Determine the target image that needs to be classified;

[0205] Calling a pre-trained fine-grained classification model to perform fine-grained classification on a target image, wherein the fine-grained classification model includes a feature extraction module, a feature optimization module and a fine-grained classification module, wherein the feature extraction module is used to extract image features of the target image, the feature optimization module is used to optimize the image features to obtain optimized image features, and the fine-grained classification module performs fine-grained classification on the optimized image features to obtain fine-grained categories of the target image;

[0206] Get the fine-grained category of the target image output by the fine-grained classification model.

[0207] It should be noted that the electronic device provided in the embodiment of the present application belongs to the same concept as the image classification method in the above embodiment. Any method provided in the image classification method embodiment can be run on the electronic device. The specific implementation process is detailed in the feature extraction method embodiment and will not be repeated here.

[0208] It should be noted that, for the image classification method of the embodiment of the present application, ordinary testers in the field can understand that all or part of the process of implementing the image classification method of the embodiment of the present application can be completed by controlling the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, such as a memory of an electronic device, and executed by at least one processor in the electronic device, and the execution process may include the process of the embodiment of the image classification method. Among them, the storage medium can be a disk, an optical disk, a read-only memory, a random access memory, etc.

[0209] For the data screening device of the embodiment of the present application, each functional module can be integrated into a processing chip, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk.

Claims

1. A method for image classification, wherein: The image classification method comprises: Determine the target image that needs to be classified; Calling a pre-trained fine-grained classification model to perform fine-grained classification on the target image, wherein the fine-grained classification model includes a feature extraction module, a feature optimization module and a fine-grained classification module, the feature extraction module is used to extract image features of the target image, the feature optimization module is used to optimize the image features to obtain optimized image features, and the fine-grained classification module performs fine-grained classification on the optimized image features to obtain fine-grained categories of the target image; Obtaining the fine-grained category of the target image output by the fine-grained classification model; The fine-grained classification model is obtained by the following training method: Acquire multiple sample images, and acquire fine-grained category labels and coarse-grained category labels of the sample images; Constructing a machine learning network, the machine learning network comprising a first branch network, a second branch network and a binary classification module having the same structure, the first branch network comprising a feature extraction module, a feature optimization module and a fine-grained classification module, the binary classification module being connected to the feature extraction module of the first branch network and the feature extraction module of the second branch network; Selecting a first sample image from the multiple sample images, and extracting image features of the first sample image based on a feature extraction module of the first branch network, optimizing the first branch network through a feature optimization module, and inputting the image features into a fine-grained classification module of the first branch network for fine-grained classification to obtain a first predicted fine-grained category; Selecting a second sample image from the multiple sample images, extracting image features of the second sample image based on the feature extraction module of the second branch network, optimizing the image features through the feature optimization module of the second branch network, and inputting the image features into the fine-grained classification module of the second branch network for fine-grained classification to obtain a second predicted fine-grained category; fusing the image features of the first sample image and the image features of the second sample image to obtain a fused image feature, and predicting whether the coarse-grained categories of the first sample image and the second sample image are the same based on the fused image feature based on the binary classification module to obtain a prediction result; Obtaining a first classification loss of the first branch network according to the first predicted fine-grained category and the fine-grained category label of the first sample image, obtaining a second classification loss of the second branch network according to the second predicted fine-grained category and the fine-grained category label of the second sample image, and obtaining a third classification loss according to the prediction result, the coarse-grained category of the first sample image, and the second sample image; Obtaining a corresponding total loss according to the first classification loss, the second classification loss, and the third classification loss, and adjusting parameters of the first branch network and the second branch network according to the total loss until the training is terminated when a preset training stop condition is met; A branch network is selected from the first branch network and the second branch network as the fine-grained classification model.

2. The image classification method according to claim 1, wherein: The step of determining a target image that needs to be classified includes: When the image classification period is reached, the image newly added in the image classification period is determined as the target image.

3. The image classification method according to claim 1, wherein: The fine-grained classification model further includes a dimension reduction module, which is used to perform feature dimension reduction on the optimized image features to obtain optimized image features after dimension reduction; The fine-grained classification module is also used to perform fine-grained classification on the optimized image features after dimensionality reduction to obtain the fine-grained category of the target image.

4. The image classification method according to claim 1, wherein: After obtaining the fine-grained category of the target image output by the fine-grained classification model, the method further includes: A storage path is allocated to the target image according to the fine-grained category, and the target image is stored in the storage path.

5. The image classification method according to claim 1, wherein: The image feature includes a feature map, and the feature optimization module is used to perform a transposition process on the feature map to obtain a transposed feature map, and perform a matrix multiplication process on the feature map and the transposed feature map, and use the result of the matrix multiplication as the optimized image feature.

6. The image classification method according to claim 1, wherein: The third classification loss is obtained according to the following formula: Loss3=-[η*y*log(p)+(1-y)*log(1-p)]; Among them, Loss3 represents the third classification loss, η represents the correction coefficient, y is used to characterize whether the coarse-grained category label of the first sample image is the same as the coarse-grained category label of the second sample image, and p represents the prediction result.

7. The image classification method according to claim 5, wherein: The selecting a branch network from the first branch network and the second branch network as the fine-grained classification model includes: Obtaining the classification accuracy of the first branch network, and obtaining the classification accuracy of the second branch network; A branch network with higher classification accuracy is selected from the first branch network and the second branch network as the fine-grained classification model.

8. The image classification method according to claim 5, wherein: The fusing the image feature of the first sample image and the image feature of the second sample image to obtain a fused image feature includes: Channel merging is performed on the image features of the first sample image and the image features of the second sample image, and a result of the channel merging is used as the fused image feature.

9. An image classification device, wherein: include: A model training component, used to obtain a plurality of sample images, and obtain fine-grained category labels and coarse-grained category labels of the sample images; Construct a machine learning network, the machine learning network includes a first branch network and a second branch network with the same structure, and a binary classification module, the first branch network includes a feature extraction module, a feature optimization module and a fine-grained classification module, and the binary classification module is connected to the feature extraction module of the first branch network and the feature extraction module of the second branch network; select a first sample image from the multiple sample images, and extract the image features of the first sample image based on the feature extraction module of the first branch network, optimize it through the feature optimization module of the first branch network, and input it into the fine-grained classification module of the first branch network for fine-grained classification to obtain a first predicted fine-grained category; select a second sample image from the multiple sample images, and extract the image features of the second sample image based on the feature extraction module of the second branch network, and optimize it through the feature optimization module of the second branch network, and input it into the fine-grained classification module of the second branch network for fine-grained classification to obtain a second predicted fine-grained category; fuse the image features of the first sample image The method comprises the following steps: obtaining a first branch network and a second branch network based on the image feature of the first sample image and the image feature of the second sample image, and obtaining a fused image feature based on the image feature of the first sample image and the image feature of the second sample image, and predicting whether the coarse-grained categories of the first sample image and the second sample image are the same based on the fused image feature based on the binary classification module to obtain a prediction result; obtaining a first classification loss of the first branch network based on the first predicted fine-grained category and the fine-grained category label of the first sample image, obtaining a second classification loss of the second branch network based on the second predicted fine-grained category and the fine-grained category label of the second sample image, and obtaining a third classification loss based on the prediction result and the coarse-grained categories of the first sample image and the second sample image; obtaining a corresponding total loss based on the first classification loss, the second classification loss and the third classification loss, and adjusting the parameters of the first branch network and the second branch network based on the total loss until the training is terminated when a preset training stop condition is met; selecting a branch network from the first branch network and the second branch network as a fine-grained classification model; An image determination component, used to determine a target image that needs to be classified; A model calling component, used to call a pre-trained fine-grained classification model to perform fine-grained classification on the target image, wherein the fine-grained classification model includes a feature extraction module, a feature optimization module and a fine-grained classification module, the feature extraction module is used to extract image features of the target image, the feature optimization module is used to optimize the image features to obtain optimized image features, and the fine-grained classification module performs fine-grained classification on the optimized image features to obtain fine-grained categories of the target image; The category acquisition component is used to obtain the fine-grained category of the target image output by the fine-grained classification model.

10. A storage medium, wherein: The storage medium stores a computer program, which, when executed on a computer, causes the computer to execute: Determine the target image that needs to be classified; Calling a pre-trained fine-grained classification model to perform fine-grained classification on the target image, wherein the fine-grained classification model includes a feature extraction module, a feature optimization module and a fine-grained classification module, the feature extraction module is used to extract image features of the target image, the feature optimization module is used to optimize the image features to obtain optimized image features, and the fine-grained classification module performs fine-grained classification on the optimized image features to obtain fine-grained categories of the target image; Obtaining the fine-grained category of the target image output by the fine-grained classification model; The fine-grained classification model is obtained by the following training method: Acquire multiple sample images, and acquire fine-grained category labels and coarse-grained category labels of the sample images; Constructing a machine learning network, the machine learning network comprising a first branch network, a second branch network and a binary classification module having the same structure, the first branch network comprising a feature extraction module, a feature optimization module and a fine-grained classification module, the binary classification module being connected to the feature extraction module of the first branch network and the feature extraction module of the second branch network; Selecting a first sample image from the multiple sample images, and extracting image features of the first sample image based on a feature extraction module of the first branch network, optimizing the first branch network through a feature optimization module, and inputting the image features into a fine-grained classification module of the first branch network for fine-grained classification to obtain a first predicted fine-grained category; Selecting a second sample image from the multiple sample images, extracting image features of the second sample image based on the feature extraction module of the second branch network, optimizing the image features through the feature optimization module of the second branch network, and inputting the image features into the fine-grained classification module of the second branch network for fine-grained classification to obtain a second predicted fine-grained category; fusing the image features of the first sample image and the image features of the second sample image to obtain a fused image feature, and predicting whether the coarse-grained categories of the first sample image and the second sample image are the same based on the fused image feature based on the binary classification module to obtain a prediction result; Obtaining a first classification loss of the first branch network according to the first predicted fine-grained category and the fine-grained category label of the first sample image, obtaining a second classification loss of the second branch network according to the second predicted fine-grained category and the fine-grained category label of the second sample image, and obtaining a third classification loss according to the prediction result, the coarse-grained category of the first sample image, and the second sample image; Obtaining a corresponding total loss according to the first classification loss, the second classification loss, and the third classification loss, and adjusting parameters of the first branch network and the second branch network according to the total loss until the training is terminated when a preset training stop condition is met; A branch network is selected from the first branch network and the second branch network as the fine-grained classification model.

11. An electronic device, wherein: The electronic device includes a processor and a memory, wherein a computer program is stored in the memory, and the processor is configured to execute, by calling the computer program stored in the memory: Determine the target image that needs to be classified; Calling a pre-trained fine-grained classification model to perform fine-grained classification on the target image, wherein the fine-grained classification model includes a feature extraction module, a feature optimization module and a fine-grained classification module, the feature extraction module is used to extract image features of the target image, the feature optimization module is used to optimize the image features to obtain optimized image features, and the fine-grained classification module performs fine-grained classification on the optimized image features to obtain fine-grained categories of the target image; Obtaining the fine-grained category of the target image output by the fine-grained classification model; The fine-grained classification model is obtained by the following training method: Acquire multiple sample images, and acquire fine-grained category labels and coarse-grained category labels of the sample images; Constructing a machine learning network, the machine learning network comprising a first branch network, a second branch network and a binary classification module having the same structure, the first branch network comprising a feature extraction module, a feature optimization module and a fine-grained classification module, the binary classification module being connected to the feature extraction module of the first branch network and the feature extraction module of the second branch network; Selecting a first sample image from the multiple sample images, and extracting image features of the first sample image based on a feature extraction module of the first branch network, optimizing the first branch network through a feature optimization module, and inputting the image features into a fine-grained classification module of the first branch network for fine-grained classification to obtain a first predicted fine-grained category; Selecting a second sample image from the multiple sample images, extracting image features of the second sample image based on the feature extraction module of the second branch network, optimizing the image features through the feature optimization module of the second branch network, and inputting the image features into the fine-grained classification module of the second branch network for fine-grained classification to obtain a second predicted fine-grained category; fusing the image features of the first sample image and the image features of the second sample image to obtain a fused image feature, and predicting whether the coarse-grained categories of the first sample image and the second sample image are the same based on the fused image feature based on the binary classification module to obtain a prediction result; Obtaining a first classification loss of the first branch network according to the first predicted fine-grained category and the fine-grained category label of the first sample image, obtaining a second classification loss of the second branch network according to the second predicted fine-grained category and the fine-grained category label of the second sample image, and obtaining a third classification loss according to the prediction result, the coarse-grained category of the first sample image, and the second sample image; Obtaining a corresponding total loss according to the first classification loss, the second classification loss, and the third classification loss, and adjusting parameters of the first branch network and the second branch network according to the total loss until the training is terminated when a preset training stop condition is met; A branch network is selected from the first branch network and the second branch network as the fine-grained classification model.

12. The electronic device according to claim 11, wherein: When determining a target image that needs to be classified, the processor is used to execute: When the image classification period is reached, the image newly added in the image classification period is determined as the target image.

13. The electronic device according to claim 11, wherein: The fine-grained classification model further includes a dimension reduction module, which performs feature dimension reduction on the optimized image features to obtain optimized image features after dimension reduction; The fine-grained classification module is also used to perform fine-grained classification on the optimized image features after dimensionality reduction to obtain the fine-grained category of the target image.

14. The electronic device according to claim 11, after acquiring the fine-grained category of the target image output by the fine-grained classification model, the processor is further configured to execute: A storage path is allocated to the target image according to the fine-grained category, and the target image is stored in the storage path.

15. The electronic device according to claim 11, wherein: The image feature includes a feature map, and the feature optimization module is used to perform a transposition process on the feature map to obtain a transposed feature map, and perform a matrix multiplication process on the feature map and the transposed feature map, and use the result of the matrix multiplication as the optimized image feature.

16. The electronic device according to claim 11, wherein: The third classification loss is obtained according to the following formula: Loss3=-[η*y*log(p)+(1-y)*log(1-p)]; Among them, Loss3 represents the third classification loss, η represents the correction coefficient, y is used to characterize whether the coarse-grained category label of the first sample image is the same as the coarse-grained category label of the second sample image, and p represents the prediction result.

17. The electronic device according to claim 16, wherein: When selecting a branch network from the first branch network and the second branch network as the fine-grained classification model, the processor is configured to execute: Obtaining the classification accuracy of the first branch network, and obtaining the classification accuracy of the second branch network; A branch network with higher classification accuracy is selected from the first branch network and the second branch network as the fine-grained classification model.

18. The electronic device according to claim 16, wherein: When the image feature of the first sample image and the image feature of the second sample image are fused to obtain the fused image feature, the processor is configured to execute: Channel merging is performed on the image features of the first sample image and the image features of the second sample image, and a result of the channel merging is used as the fused image feature.

Images